An AI system has delivered a powerful rebuttal to the claim that large language models (LLMs) just “regurgitate” their training data. By placing in the top ten of a fiercely competitive international forecasting contest, ManticAI’s system has shown it can perform “genuine reasoning” about future events, according to its creators.
The British startup’s AI came in eighth in the Metaculus Cup, a competition where entrants predicted the outcomes of 60 complex real-world events over the summer. The questions, which ranged from election results to the likelihood of a public feud between Trump and Musk, could not be answered by simply searching a database of past information.
Toby Shevlane, ManticAI’s co-founder and a former Google DeepMind researcher, said, “You can’t predict the future like that.” He explained that their system uses a variety of AI agents from different developers to tackle problems from multiple angles. Agents are assigned roles like historical researcher, scenario analyst, and current events monitor, with their combined insights forming the basis for the final prediction.
This sophisticated process allowed the AI to develop unique perspectives. Shevlane noted that the system often disagreed with the consensus view of human forecasters, who tend to cluster around an average. This suggests AI can act as a check against common human biases like herd behavior, providing a more objective, data-driven viewpoint.
While acknowledging AI’s rapid progress, experts point out that it can still struggle with logical consistency checks on highly complex, multi-stage forecasts. However, the performance of ManticAI has convinced many that the future of prediction is collaborative. The goal will be to merge the tireless, unbiased analytical power of AI with the nuanced judgment and intuition that human experts still possess.
41