Hundreds of thousands of Rohingyas have fled violence in Myanmar's Rakhine since a military crackdown began on Aug 25.
The incident was reported Saturday morning from the makeshift Rohingya camp on the Balukhali hills in Cox's Bazar's Ukhiya, said local police.
The victims include a woman, estimated to be in her 30s, and three minors, Ukhiya police's Inspector Md Kay Kislu told bdnews24.com.
He said a wild elephant from the forest near the refugee camp barged in around 12:45pm, leaving the Rohingyas fleeing in panic.
"Two persons died on the spot leaving several others injured, who were taken to the medical camp run by MSF (Médecins Sans Frontières). Two others succumbed to their injuries around 1:30pm," said Inspector Kislu.
A similar incident left two refugees killed about a month ago.
International agencies say an estimated 536,000 people crossed the border into the southeastern Bangladesh district of Cox's Bazar in seven weeks.
According to the Inter Sector Coordination Group or ISCG of aid agencies, hosted by International Organisation for Migration or IOM, some 15,000 people entered between Oct 9 and 11.