The oath of office and secrecy was administered by President Pranab Mukherjee at the Rashtrapati Bhavan, the presidential palace, with the setting sun serving as a backdrop to the momentous event attended by leaders from eight countries including Pakistan Prime Minister Nawaz Sharif.
A council of ministers, numbering around 45, also took oath.
The Modi government succeeds the government of prime minister Manmohan Singh whose UPA coalition government was in power for 10 years.
Millions of Indians watched the inauguration live on television as the 63-year-old Hindu nationalist leader, once thought of as too divisive to lead the world's largest democracy, took his oath along with his cabinet members in the palace forecourt.
The low-caste son of a tea stall-owner, Modi won India's first parliamentary majority after 25 years of coalition governments, giving him ample room to advance economic reforms that started over two decades ago but stalled in recent years.
Modi invited leaders from across South Asia to his swearing-in ceremony, including Prime Minister Nawaz Sharif from rival power Pakistan, an unprecedented gesture.
Even before his inauguration, Modi had some recognition abroad, treated by many with suspicion - and by some as a pariah - for Hindu-Muslim violence that erupted 12 years ago in Gujarat, the western state he ruled.
The US administration denied Modi a visa in 2005, but President Barack Obama has now invited him to the White House.
He has spoken with the presidents of the United States and Russia and he has become one of only three people that Japanese Prime Minister Shinzo Abe follows on Twitter.
Looking on were some 4,000 guests, ranging from members of the defeated Nehru-Gandhi family to top industrialists, Bollywood actors, Hindu holy men and the leaders of several neighboring countries.
Supporters exploded fireworks in celebration a few blocks away at the headquarters of his Bharatiya Janata Party (BJP) after he was sworn in.
Modi kicked things off on Sunday with an announcement that he would streamline the cabinet and move to a more centralized system of governing aimed at breaking decision-making bottlenecks widely blamed for dragging down economic growth.
Modi said he would appoint 'super ministers' in charge of several departments but by late on Monday evening had not revealed the portfolios for the cabinet.
Arun Jaitley, 61, will be the finance minister as well as running the defense ministry, party sources said. One of the top corporate lawyers in the country and a close Modi aide, Jaitley served in a previous BJP administration as commerce minister.
Pakistan's Sharif said after arriving in Delhi that South Asia's bitterest rivals now had an opportunity to turn a page in their history of troubled relations.
He said the nuclear-armed neighbors, which were traumatically separated at the end of British rule in 1947 and have fought three wars since, should together rid their region of the instability that has plagued them for decades.
"We should remove fears, mistrust and misgivings about each other," he told the Indian news network NDTV.
The BJP has long advocated a tough stance on Pakistan, with which India has a major territorial dispute in Kashmir, and Modi has been seen as a hardliner on issues of national security.
In that respect, Modi's decision to invite Sharif for his inauguration and bilateral talks on Tuesday came as a surprise and raised hopes for a thaw in relations between the rivals, which have been particularly frosty since 2008 attacks on the city of Mumbai by Pakistan-based militants.
As a gesture of goodwill following invitations to the ceremony, Pakistan and Sri Lanka released dozens of Indian fishermen jailed for straying into their territorial waters.
China has built a port in Sri Lanka and is involved in upgrading another in Bangladesh, besides military and civil assistance to long-time ally Pakistan, heightening Delhi's anxieties of being boxed in.
"Modi has appreciated the much-neglected fact that foreign policy begins at the nation's borders," wrote foreign policy analyst C Raja Mohan in the Indian Express. "As a realist, however, Modi should be aware that major breakthroughs are unlikely amid the current flux within Pakistan."