Two athletes from the same country and competing in the same sport staying in the village in the Harumi waterfront district tested positive for the virus, organisers said without providing further details.
Organisers on Sunday reported 10 new cases connected to the Olympics, including a third athlete who was not staying in the village, down from 15 new cases a day earlier.
South Africa also reported three positive cases in its soccer squad - two players and an analyst. It was not immediately clear if those cases were identified as part of the same testing programme.
An International Olympic Committee member from South Korea tested positive for the coronavirus on landing in Tokyo. Ryu Seung-min, a former Olympic athlete, is vaccinated, reflecting the infection risk even from vaccinated attendees.
The new infections are testing the layered testing regime designed to ensure COVID cases are quickly caught and isolated. Proponents argue that the growing number of cases underscores the strength of the testing system.
Infection rates are climbing among Tokyo's general population, topping 1,000 new cases for five consecutive days. Polls show many Japanese oppose holding the Games with the influx of overseas visitors it entails.
Prime Minister Yoshihide Suga is betting his political fortunes on successfully pulling off the Games while suppressing rising COVID cases.
Public support for his cabinet has slid to 35.9 percent, a Kyodo poll showed on Sunday, the lowest since he took power last September. Just 29.4 percent think the fourth state of emergency, which began last Monday, is effective, according to the poll.
The rainy season ended in Tokyo on Friday, bringing blue skies and intense heat. The burden on participants has been increased by virus countermeasures like masking.
"While we have been dealing with COVID matters, we haven't taken our focus off the heat," said David Hughes, chief medical officer at the Australian Olympic Committee.
Officials point to heat countermeasures including the distribution of drinks and salt tablets and the use of misting towers and cooling vests.
The delayed Olympics was intended to showcase a modern, diverse Japan at a time of rising regional rivalries but the pandemic has left the country hosting a pared down event.
Athletes continue to question the compromises organisers have made, with Maya Yoshida, captain of Japan's soccer team, calling for the decision to hold the Games behind closed doors to be reconsidered.