The news is the latest in a series of embarrassments for Tokyo organisers that have sparked outrage at home and abroad, and comes just days after a well-known musician was forced to step down as composer for the ceremony after old reports of his bullying and abusive behaviour surfaced.
Abe, who famously dressed up as the titular plumber from video game Super Mario at the Rio Games to represent Japan, played an outsized role in attracting the Olympics to Tokyo.
At the time of the bid, Abe and his supporters hoped the Olympics would parallel the 1964 Tokyo Games heralding the nation's revival after decades of economic stagnation and also mark its recovery from a devastating nuclear and natural disaster in 2011.
Instead, the Games, delayed a year because of the global pandemic, has faced a series of setbacks and scandals.
Earlier this year the head of the Tokyo 2020 organising committee resigned after making sexist remarks and Tokyo Olympics creative head followed after he made derogatory comments about a popular Japanese female entertainer.
On Thursday, Tokyo Game organisers fired Kentaro Kobayashi over a joke he made about the Holocaust as part of his comedy act in the 1990s that recently resurfaced in domestic media.
"It has come to our notice that Kobayashi used comments that made fun of a tragic fact in history in his past performance," said Seiko Hashimoto, who heads the organising committee.
Earlier, the Simon Wiesenthal Center, an international Jewish human rights organisation, released a statement condemning Kobayashi's past behaviour.
"Any association of this person to the Tokyo Olympics would insult the memory of six million Jews and make a cruel mockery of the Paralympics," said Abraham Cooper, a rabbi and global social action director of the centre.
The opening ceremony on Friday is set to be a subdued affair, with Japanese media reporting that less than 950 people-including only around 15 global leaders - are set to attend.
First Lady Jill Biden lands in Tokyo on Thursday afternoon for the Games' opening ceremony, raising expectations she might also use her attendance to discuss vaccines with Prime Minister Yoshihide Suga.
Biden has been travelling across the United States urging more people to get inoculated.
Only a third of Japanese have had at least one dose of the vaccine, fuelling public concerns that the Olympics could become a super-spreader event. Already 87 Olympic-related personnel, including athletes, have tested positive for COVID-19, forcing athlete withdrawals and pushing teammates to isolate.
NHK said Abe decided against attending the ceremony after the Japanese government declared a state of emergency and virus restrictions over Tokyo. Abe's office could not immediately be reached on Thursday, a public holiday in Japan.
In a recent poll in the Asahi newspaper, 68 percent of respondents expressed doubt about the ability of Olympic organisers to control virus infections, with 55 percent saying they opposed the Games going ahead.
Olympics competition has already begun, with the Japanese women's softball team getting the hosts off to a winning start on Wednesday.
The second day of softball began early Thursday under cloudy skies with the United States defeating Canada by a run to go 2-0 in the standings.
Between matches in rural Fukushima, which was devastated by the 2011 nuclear disaster, players were on the lookout for a brown bear that had been spotted this week.
"I'm kind of disappointed I did not get to see it," US pitcher Monica Abbott said.