Balandin, swimming in lane eight after qualifying as the slowest finalist, touched the wall in two minutes, 7.46 seconds, just 0.07 seconds ahead of Josh Prenot of the United States.
Russia's Anton Chupkov took the bronze in lane seven, with 0.24 seconds separating the top three.
In swimming finals, the fastest qualifiers take the central lanes four and five, and a win in the outside lane is a rarity.
"There are pluses and minuses. On the one hand you don't see your opponents and so it's very hard," Balandin said.
"On the other hand the plus is that if you don't see them you can concentrate just on the result instead of whether to shadow someone."
The Russian-speaking Kazakh said he and Chupkov had joked about their lanes before the race.
On a video feed, the two could be seen chatting in the call room before the race.
"While everyone else was concentrating very hard and getting nervous, we were relaxed and talking like friends," the 21-year-old said.
"To be honest, we were saying we just needed to get through the next two minutes and it would all be over."
Balandin's mother watched in tears as he paraded with his gold medal around the poolside.
Japan's Yasuhiro Koseki had seized control of the race from the outset, leading until the 150-metre mark and well under world record pace before he tied up over the final lap.
Balandin learned of Rahimov's gold only after winning his own, saying he had been "a bit cut off from the world" for the previous hour.
"It's probably the biggest honor and the best thing I could do for my country," he said.
"I'm very proud of this. This is history, the first swimming medal for Kazakhstan, and I'm very glad I wrote it."