Parents have been advised to prevent children from being outdoors until sunset
The extent of loss and cause of fire were not immediately clear.
Factory Supervisor Md Khorshed Alam said the fire had originated at the warehouse where fabrics and yarn had been stored.
Five firefighting units doused the flames around 4pm, three hours after it had started.
Alam said the factory was opened a month ago.