There is an 80 percent likelihood that the number of people on the planet, currently 7.2 billion, will increase to between 9.6 billion and 12.3 billion by 2100, the researchers said.
They also saw an 80 percent probability that Africa's population will rise to between 3.5 billion and 5.1 billion by 2100 from about 1 billion today.
The study, led by UN demographer Patrick Gerland and University of Washington statistician and sociologist Adrian Raftery and published in the journal Science, foresees only a 30 percent chance that earth's population will stop rising this century.
"Previous forecasts did indeed forecast a levelling off of the world population around 2050, and in some cases a decline," Raftery said.
Raftery said the new projections arise from data that clearly establishes that birth rates in sub-Saharan Africa have not been decreasing as quickly as some experts had expected, a trend that was "not as clear when previous forecasts were made."
Raftery said the researchers used data on population, fertility, mortality and migration from every country and then predicted future rates using new statistical models.
Some of the figures, such as the median projection of the population hitting 10.9 billion by 2100, mirror a UN report published in 2013.
UN demographer Gerland said sub-Saharan Africa countries already with big populations and high fertility levels are expected to drive population growth, including Nigeria, Tanzania, Democratic Republic of the Congo, Niger, Uganda, Ethiopia, Kenya, Zambia, Mozambique and Mali.
The world's population reached 1 billion in the early 19th century, doubled to 2 billion in the 1920s and doubled again to 6 billion in the 1990s. It hit 7 billion in 2011.
The findings underscore worries expressed for decades by some experts about a planet growing more crowded and humankind exhausting natural resources, struggling to produce enough food or cope with poverty and infectious diseases.
Raftery said African nations could benefit by intensifying policies to lower fertility rates, with studies showing that greater access to contraceptives and more education for girls and women can be effective.
The researchers projected that Asia's population, now 4.4 billion, will peak at around 5 billion people in 2050, then begin to decline.
They forecast that the populations of North America, Europe and Latin America will stay below 1 billion each by 2100.
Among the experts who had predicted the global population rise would peter out was a 2010 report by Austrian demographer Wolfgang Lutz.
He forecast it likely would reach 8 billion to 10 billion by 2050 but "population stabilisation and the onset of a decline are likely" in the second half of the century.