The 81-year-old breathed his last around 6:30am on Tuesday at Dhaka's Samorita Hospital.
Rahman was admitted there 10 days ago after he suffered a stroke, the hospital's General Manager Nurul Islam Tuhin told bdnews24.com.
Rahman served as the chief justice for 14 months from January 2000.
Then he was appointed chief adviser to the caretaker government because he was the final chief justice to retire during the Awami League’s 1996-2001 term.
The Eighth Parliamentary Elections were held under the caretaker government on Oct 1, 2001. The BNP-led Four-party Alliance came to power in that election.
On Oct 10, Justice Rahman handed over power to the new prime minister, Khaleda Zia.
Leaders of the ruling Awami League have criticised Rahman’s caretaker government on several occasions.
In April this year, its chief and Prime Minister Sheikh Hasina blamed an international conspiracy to gain control over Bangladesh's natural gas for her party's defeat in the 2001 polls.
"I wanted to use the country's resources for the people first...Wanted to keep a 50-year reserve, then to consider whether to sell or not," she told at a party event.
Hasina claimed that she and Khaleda attended a reception hosted by chief adviser Rahman for the visiting former US President Jimmy Carter in 2011, where the issue of gas export was raised and that she 'refused'.
The Awami League leader said she left the reception after that, but the BNP leaders remained.
The Supreme Court adjourned its Tuesday's sessions over his death.
High Court Division's Additional Registrar Md Sabbir Foyez told bdnews24.com that the namaz-e-janaza of the late judge will be held at the top court premises after the Zuhr prayers.
Born on Mar 1, 1936 in Jessore, Justice Rahman was the son of eminent Jessore lawyer Khan Bahadur Lutfur Rahman.
His maternal uncle Nurul Huda had also served as a High Court judge.
After completing his master's in English from Dhaka University in 1956, he enrolled for the LLB programme at the same university.
He taught at the Shahid Suhrawardy College before starting to practice law in 1960.
Rahman was an apprentice to Bangladesh's first attorney general MH Khandaker.
He was appointed as a High Court judge in 1979 and was appointed to the Appellate Division in 1991.
Rahman retired as the chief justice on Feb 28, 2001. He is survived by his wife and three daughters.