Organisers confirmed that a visitor from abroad who was working for the Olympics tested positive in a routine check on Friday. The person's nationality was not revealed due to privacy concerns.
The other cases included two members of the media, seven contractors and five Games personnel.
The case at the athletes' village, a 44-hectare site built on Tokyo's waterfront, is particularly worrying as a majority of the 11,000 competitors will be staying there.
Originally intended to showcase Japan's recovery from its 2011 earthquake and nuclear disaster, the Tokyo Olympics has become an exercise in damage limitation.
Postponed for a year due to the global pandemic, it is being held mostly without spectators and under tight quarantine rules. Most athletes are starting to arrive for the Games, which run from Jul 23 through Aug 8.
The Japanese public has been wary about hosting the Games amid a resurgence in new coronavirus infections and worries that an influx of visitors may create a super-spreader event, straining an already-stretched medical system.
Only around 20 percent of the population is fully vaccinated.
Although Japan has escaped the explosive outbreaks of other nations, it has more than 820,000 cases and about 15,000 deaths. The number of new cases in host city Tokyo, which is in its fourth state of emergency over the virus, has been over 1,000 for four straight days.
Tokyo 2020 President Seiko Hashimoto acknowledged the public's concerns.
"I understand that there are still many worrying factors. Organisers must try to make sure that people will understand that these games are safe and secure," she told a news conference on Saturday.
So far, more than 40 people involved in the Games, including Japanese and foreigners, have tested positive.
Toshiro Muto, chief of the Tokyo 2020 organising committee, said on Saturday that officials were working on the assumption that there would be positive COVID-19 cases.
A key part of the anti-contagion measures is daily saliva testing of the athletes who take part, as well as frequent testing of others involved in the event. Visitors' movements are also due to be monitored and restricted.
But in a sign that organisers were already finding rules difficult to enforce, Ugandan weightlifter Julius Ssekitoleko went missing from his team's training site in Osaka on Friday.
Authorities are still looking for him, according to Games organisers. Media reports said he left behind a note saying he wanted to stay and work in Japan, as life in Uganda was difficult.