The deaths occurred when a small overcrowded boat carrying 112 people set out to cross one of the busiest shipping lanes in the world and panic took hold among the passengers not far from the shore
As of 7 pm local time, only six survivors had been pulled from the debris, Xinhua quoted the district government as saying.
It was unknown how many people were in the buildings when the accident happened.
Xinhua did not give a cause for the pre-dawn collapse in Wenzhou's Lucheng industrial district in Zhejiang province and said that investigations were still going on.
The collapsed buildings were built by villagers and five adjacent houses built in the 1970s remain standing, though rescuers are demolishing them to avoid secondary disasters, Xinhua said.