The incident place at Fisheries Ghat, an area beside the Karnaphuli River, around 11am Friday, said fire service official Rabiul Alam.
The dead victim has been identified as Russel Dey, a 35-year-old banker.
"They were walking past the hut when it came down on them, leaving them injured. The three were rescued and taken to the Chittagong Medical College and Hospital," said Alam.
"The saw mill made from tin and wood were taken apart by strong winds," said Megnad Tangchanggya, assistant meteorologist at Patenga Met Office. "There is no possibility of a tornado or cyclone."
However, heavy showers will continue over Chittagong division, said a Met Office forecast.
There are also fears that the downpour would lead to more landslides in the hilly areas of Chittagong and Sylhet.