Valory's Open-Weight Olas Predict Model Nears GPT-5.6 on Forecasting

Valory says its new open-weight forecasting model, Olas-Predict-Qwen3.8-27B, comes within 1.2 percentage points of GPT-5.6 on a test of real-world prediction markets. The 27-billion-parameter specialist reached 67.3% balanced accuracy in the company's internal evaluation, against 68.5% for the GPT-5.6 comparator, when both systems received the same market evidence.

The result speaks to a practical question in AI development: how much of a specialist task's performance comes from a model's broad knowledge, and how much from relevant evidence and focused training? Valory's experiment suggests a smaller, adaptable model can compete on a narrowly defined forecasting task, while also showing that a stronger foundation model accounts for much of the improvement.

The stakes are real for developers building forecasting workflows. An open-weight forecaster (a model whose trained parameters anyone can download and run) gives organizations the option to host and adapt the model themselves, rather than depend entirely on a remote proprietary service. That is the deployment story behind the announcement, and the benchmark table is the research story.

What does the comparison measure?

Valory reports results across 6,824 researchable prediction markets. The systems used common prompts and evidence at temperature zero, and Valory applied a date filter tied to the GPT-5.6 comparator's stated February 16, 2026 knowledge cutoff. The comparison evaluates how models reason over supplied evidence, not how well each system gathers its own information.

That scope matters. These are company-reported results on selected markets, not an independent demonstration of general equivalence between the models. The final comparison is also separate from an earlier 10,916-sample experiment and from Valory's previously published 14-billion-parameter study.

Balanced accuracy gives equal weight to performance on the two outcome classes, preventing a common outcome from dominating the result. Valory also reports Brier score, which evaluates probability forecasts and penalizes confident errors. Lower Brier scores are better. A forecaster should be rewarded for well-calibrated uncertainty, not just for picking the eventual winner.

Valory's reported final comparison subset:

• Historic GPT-4.1 baseline — Brier score 0.3110, balanced accuracy 58.5%
• Olas-Predict-R1-14B — Brier score 0.2722, balanced accuracy 58.9%
• Qwen3.8-27B before domain training — Brier score 0.2235, balanced accuracy 66.4%
• Olas-Predict-Qwen3.8-27B — Brier score 0.2124, balanced accuracy 67.3%
• GPT-5.6 comparator — Brier score 0.2029, balanced accuracy 68.5%

The historical GPT-4.1 baseline should be read as that specific baseline, not as a new universal score for GPT-4.1.

Where does the gain actually come from?

The new model improves on Olas-Predict-R1-14B by approximately 22% on Brier error and 8.4 percentage points on balanced accuracy in this comparison. But the untrained Qwen3.8-27B starting point already reaches 66.4% balanced accuracy and a 0.2235 Brier score, accounting for much of the step forward.

Domain training supplies the additional improvement. Brier error falls by approximately 5%, while balanced accuracy rises by 0.9 percentage points. Expected calibration error (a summary of how well stated confidence matches observed outcomes) falls from 0.1630 to 0.1429, an improvement of approximately 12.3%.

This is a more useful finding than treating the entire generational gain as the effect of fine-tuning. A stronger base model and specialized training contribute separately. The latter appears to improve how the model expresses probabilities — an important capability when a downstream system must decide whether the evidence is strong enough to act.

How does the forecasting system work?

Valory describes a training process that uses reviewed forecasting examples and reinforcement learning to improve probability estimates. A separate tool gathers market intelligence and evidence, then supplies that material to the model.

This separation is central to the architecture. The language model is one component of a forecasting system: evidence retrieval determines which information is available, while the model interprets it and produces a probability. Strong reasoning over weak or outdated evidence can still produce a poor forecast.

The split also creates room to improve the components independently. Developers can refine retrieval, change how evidence is presented, or update the forecaster without rebuilding the entire agent. The reported scores evaluate a particular combination of evidence and model behavior, so they should not be transferred automatically to a different retrieval pipeline.

How does Olas turn agent activity into training data?

Valory is a core contributor to Olas, whose Predict ecosystem connects agents that create markets, trade, provide forecasts, and finalize outcomes. Traders can purchase forecasting services from specialist agents through the Mech Marketplace. Once events resolve, forecasts can be compared with outcomes, creating records for evaluation and future training.

That feedback loop gives the project a source of task-specific evidence. Instead of judging a forecast solely by whether its explanation sounds convincing, the system can eventually compare its probability estimate with a resolved event. Market resolution still defines the target, so question wording and resolution quality remain consequential.

Valory's earlier published work used more than 200,000 agent prediction records to specialize Olas-Predict-R1-14B. A separate later-market evaluation reported 75.8% accuracy across 2,628 markets. That figure is ordinary accuracy from a different evaluation; it is not directly comparable with the new announcement's 67.3% balanced accuracy.

What do open weights change for deployment?

For an organization building a forecasting workflow, an open-weight model can create more control over model updates, serving infrastructure, and the handling of supplied evidence.

A 27-billion-parameter model still requires suitable hardware, with memory, latency, and throughput depending on precision, context length, and concurrency. The announcement does not provide measured deployment costs or serving benchmarks for the new model, so the parameter count alone does not establish a particular operating-cost advantage.

Forecasting quality also differs from trading performance. Returns depend on market prices, transaction costs, execution, and strategy as well as predictions — a distinction reflected in the limitations of Valory's earlier model card.

The significance of Olas-Predict-Qwen3.8-27B is therefore focused but meaningful: Valory reports a specialist model approaching its frontier comparator on supplied-evidence forecasting, with measurable gains from domain training. Its next opportunity is to turn the continuing stream of resolved predictions into improvements that hold up across later events and real deployments.

By Marcus Bennett