"It is a tragic and disastrous event," General Tito Castellanos, director of the INPEC prison agency, told local Caracol Radio early on Tuesday. "There was a situation, apparently a riot, the prisoners lit some mattresses and a conflagration occurred."
Castellanos later confirmed a death toll of 51 people - 49 who died in the prison and two who died after being taken to hospital.
"Unfortunately the majority of the dead died because of smoke inhalation," he told Caracol. "We have two (injured people) who have been sent to Cali. Right now their diagnosis is with the doctors and we expect a report."
Twenty-four people are being treated in hospital and the fire was put out by local firefighters, Castellanos added.
The prison has a total of 1,267 inmates and the cell block where the fire occurred houses 180.
An official read a list of prisoners still alive and unhurt to anxious family members gathered outside the prison, some of whom shouted for joy when they heard their loved one's name.
In Colombia, as in many Latin American countries, prisons are highly overcrowded.
Colombia's jails have a capacity for 81,000 inmates but currently house about 97,000, according to official figures.
"I have given instructions to carry forward investigations that allow us to clarify this terrible situation," President Ivan Duque, who is on a visit to Portugal, said on Twitter.
Prison violence "obliges the complete re-imagining of prisons policy toward a humanisation of jail and dignity for the prisoner," Colombia's President-elect Gustavo Petro, who takes office in August, said on Twitter.
Colombia released some prisoners during the coronavirus pandemic after some two dozen inmates were killed during protests in 2020 against crowded conditions and lack of services in jail.
Hundreds have died in prisons in neighbouring Ecuador over the last year, in what the government there says is violence connected to drug gang competition and which it has failed to quell.