The results in the heartland rural states of Rajasthan, Chhattisgarh and Madhya Pradesh could force the federal government run by Modi's Hindu nationalist Bharatiya Janata Party (BJP) to raise spending in the countryside, where more than two-thirds of India's 1.3 billion people live.
Political analysts said the BJP's defeat would underscore rural dismay with the government and could help unite the opposition led by the Congress party. Modi is personally popular but has been criticised for failing to deliver jobs for young people and better conditions for farmers.
Reacting late on Tuesday to the results, Modi wrote on Twitter: "Victory and defeat are an integral part of life. Today's results will further our resolve to serve people and work even harder for the development of India."
Congress has ruled India for most of its post-independence era after 1947 but was decimated by Modi's BJP in national polls in 2014. Since then, it had struggled to make major inroads, even in state polls.
Gandhi, the fourth generation scion of the Nehru-Gandhi dynasty, has sought to build a coalition of regional groups.
On Tuesday, celebrations erupted outside the Congress party headquarters in New Delhi, with supporters dancing, setting off firecrackers and brandishing posters praising Gandhi.
"We are going to provide the states with a vision and a government they can be proud of," Gandhi told reporters.
"There is a feeling among people that the promises made by the prime minister ... have not been fulfilled."
CONGRESS CONFIDENT
In Madhya Pradesh, the most important of the five states that have held assembly elections in recent weeks, Congress was leading with 113 seats while the BJP had 110 out of 230.
Regional parties were leading in two smaller states that also voted, Telangana in the south and Mizoram in the northeast.
Congress said it was confident it could form governments in all three big states. The BJP previously ruled all three, for three terms in two of them.
The BJP said the state results would not necessarily dictate what happens in next year's general election.
Investors said the BJP had not fared as badly as feared, and nationally would still likely have an edge over an opposition coalition in the general election.
"That said, we maintain our view that the BJP will secure victory in the general election, which would allow PM Modi to get his reform agenda back on track after a lacklustre 2018."
Markets recovered from sharp early losses and ended marginally higher, though the central bank governor's sudden resignation on Monday kept investors nervous.
A lawmaker for the BJP said it had erred in focusing its campaign on partisan themes, such as the building of a Hindu temple at a site disputed by Muslims, instead of offering jobs and growth.
"We forgot the issue of development that Modi took up in 2014," said Sanjay Kakade.