Does Forecasting Have Room At The Top?

Superforecasting is the art/science/sport of predicting the future - for example, who will win elections, which countries will fight wars, when key technologies will be discovered. Over the past few years, it went from an obscure academic subfield to a multibillion dollar industry in the form of prediction markets. More recently, AI superforecasters have come close to the accuracy of top humans, and their performance is rising rapidly. In a year or two, we'll see one of the following patterns:

Either humans have already come close to some fundamental limit on the predictability of world events - in which case AIs will plateau at or slightly above the human level - or the trend line will continue until AIs are far beyond top humans. By analogy to superintelligence, the natural term for the second situation would be "superforecasting"; since that's already in use, we can cringely call it "ultraforecasting".

Daniel Reeves1 makes the case for scenario A here. He describes a study he coauthored in 2010, which found that, on a variety of questions related to sports games and movie box office receipts2, prediction markets only outperformed simple boring statistical models by 3-6%. Maybe those statistical models are close to the best that it's possible to do; the rest is what the mathematicians call aleatoric uncertainty - irreducible complexity downstream of chaotic systems that entirely resist modeling.

He could be right. This post isn't meant to be a decisive refutation, but rather a description of why I'm still about 70-30 expecting Scenario B.

https://www.astralcodexten.com/p/does-forecasting-have-room-at-the