The workers were constructing a toilet at a school next to the temple in Faridpur’s Madhukhali Upazila
Another person was injured in the accident at Raghabpur on Mymensingh-Kishoreganj road on Friday evening.
The victims were not immediately identified.
Mymensingh Kotwali Model Police Station's OC Abu Md Fazlul Karim said three of the victims had died on the spot.
Two others succumbed to their injuries at the Mymensingh Medical College and Hospital.