More Than 100 AI Agents Failed to Beat the Fantasy Soccer Average

By Marcus Bennett

Adding more than 100 AI agents did not materially improve Will Knight’s fantasy soccer selections, and his experimental Premier League squad did not outscore the average manager over the following weeks.

In his AI Lab newsletter, Knight connected the hobby test to a larger research and market contest: AI forecasting systems can beat an average person, yet they still trail specialized human experts on some of the hardest events to predict.

What did the AI squad actually do?

A few weeks before his assessment, Knight deployed a small horde of AI agents to construct and manage an experimental roster. He wanted to know whether a swarm of bots could assemble a winning Premier League fantasy team as AI agents spread through nearly every part of life.

Several years earlier, Knight had tried a tool from the UK’s Turing Institute that selected teams after modeling previous fantasy league data. The tool ignored real-world information, including player injuries and management changes, and produced remarkably bonkers suggestions.

On a flight to visit family, Knight asked Codex for help. He supplied a couple of relevant research papers, then requested a harness—instructions and code that guide agents through defined steps—to check recent news and run a prediction model.

Codex quickly produced a setup that appeared to collate and analyze more information than Knight could manage alone. The initial output remained poor: it left Erling Haaland on the bench.

Knight then divided the work among specialized agents. He said the change significantly improved the suggestions.

• Several agents read news and blog posts.
• Others analyzed recent results.
• More agents ran predictive models.
• A group critiqued proposed selections.

The division of labor drove the improvement in Knight’s account. Each agent contributed a specific kind of information or analysis, while another group evaluated the resulting team.

Knight then increased the system to more than 100 agents. The extra scale did not materially change the output and chiefly ran up his API bill, meaning the charges associated with model access and usage.

Even so, Knight said the gain over the earlier Turing Institute tool was striking. The multi-agent design improved his selections; adding more agents did not.

Can commercial models forecast better than people?

According to Haifeng Xu, a computer scientist at the University of Chicago who studies AI uses in market situations, newer models may forecast better because they can consume and analyze huge volumes of real-world information and orchestrate more complex systems.

Xu’s explanation was direct: “AI is just getting smarter in understanding the world—I feel like that's the fundamental reason they’re improving.”

Late last year, Xu noticed the growing popularity of prediction markets such as Kalshi and Polymarket. He and several colleagues began testing how the strongest commercial models would perform on those markets.

The researchers asked models to forecast events including the NBA Championship and who President Trump would meet with.

Xu recalled the moment the results emerged: “Very quickly we started to realize that models did really well.” The models sometimes performed better than the average person without any special instructions or guidance, Xu said.

In their testing, GPT-5 placed first, followed in order by Grok 4, Claude Sonnet, Gemini and Llama.

That result puts the fantasy experiment in a broader context. Knight’s team used current news, recent match results and multiple models, but Xu’s research team found that leading commercial systems could already exceed an average human forecaster on some events.

What do the new benchmarks show?

Xu and his colleagues went on to develop Prophet Arena, a benchmark that measures a model’s ability to make predictions. It belongs to a broader group of benchmarks designed to test AI’s future-gazing capabilities.

Another, Prediction Arena, came from researchers at Harvard University and a company called Arcada Labs. In their experiment, they gave frontier models $10,000 to wager on prediction markets over 57 days.

The models consistently lost money, the researchers reported. They nevertheless found evidence that the models improved consistently over time, combining poor near-term returns with visible progress.

Xu said AI makes predictions most effectively through a custom harness. He also identified specialized agents as the best approach, with distinct jobs such as reading news, collecting other data and analyzing the whole system.

Knight’s team followed the same basic pattern. Its gains came from assigning agents separate functions, not from increasing their number beyond 100.

Where do human forecasters still lead?

AI still often lags behind highly skilled experts known as superforecasters. Xu says people cannot process as much information as a model, but they understand the real world more deeply, including the dynamics behind a news story.

Yet Knight said there is growing evidence that the superforecaster advantage is eroding. The Metaculus Cup recently showed that AI can sometimes beat even the best human predictors.

A benchmark called ForecastBench could provide further evidence that AI can match human forecasting skills when the results of an ongoing contest conclude soon. Knight expects major laboratories to focus on future-gazing as a new way to measure progress in machine intelligence.

What changes for trading?

Knight argued that AI may soon upend industries predicated on trading, placing that prospect in the same broad pattern as the changes he associates with math, coding and law. His forecast for the labor market is equally direct: the strongest human traders may move from making every prediction to supervising and orchestrating agents that forecast better than they can.

Across the prediction-market experiments Knight described, the competitive unit was no longer a single forecast but a system that gathered news, ran models and critiqued its own outputs. That architecture explains why adding agents beyond 100 raised cost more than it improved Knight’s test.

For now, however, Knight’s AI team did not outscore the average over the past weeks. He called the suggestions unimpressive and said he would not use them in his own selections.

Knight still expects the models to improve, potentially enough to make fantasy football too easy to dominate. He hopes that outcome does not arrive, because it would remove the fun of earning bragging rights by winning on his own.

Knight expects major laboratories to turn to future-gazing as a measure of machine-intelligence progress. The practical threshold for traders is whether custom, multi-agent systems can close the remaining gap with superforecasters, rather than merely outperform the average person.