Ask a frontier model to summarize a news article and it hallucinates a fact roughly 5-10% of the time. Ask the same model to compute a financial metric that requires combining three numbers from a 10-K, and on the hardest cases the error rate can be more than ten times higher. This isn't a vague impression — it's what two independently run 2026 benchmarks measure when you put them side by side. Financial data doesn't just contain more numbers than typical text; it demands multi-step arithmetic over context-dependent, easily-confused figures, and that combination is where today's models still break.
TLDR:
- FAITH, a 2026 benchmark built from 453 S&P 500 10-K filings, finds that even top models like Claude-Sonnet-4 and Gemini-2.5-Pro hold up on simple lookups but lose 10-20+ points of accuracy on multi-step financial calculations — and GPT-4.1's accuracy on the hardest category falls to 30%.
- The same top models hallucinate far less on general-domain text: 5.6-10.3% on a leading public benchmark, versus a 10-70% error rate on FAITH's hardest financial reasoning category. That gap is the "financial hallucination premium."
- Separately, FinanceBench shows the pipeline matters as much as the model: GPT-4-Turbo's failure rate drops from 81% to 21% purely by handing it the correct source documents instead of a generic retrieval pipeline.
Financial reasoning has a measurable hallucination premium
FAITH (A Framework for Assessing Intrinsic Tabular Hallucinations in Finance) tests models on a task designed to isolate hallucination cleanly: it masks a numeric value inside a real 10-K's Management Discussion & Analysis section and asks the model to recover it from the surrounding tables and text. Built from 453 S&P 500 companies' 2024 annual reports sourced via SEC EDGAR, the benchmark grades models across four reasoning tiers, from a straight lookup to a calculation requiring three or more linked figures.
| Model | Overall accuracy | Multivariate Calculation accuracy | Point drop |
|---|---|---|---|
| Claude-Sonnet-4 | 95.6% | 80.0% | 15.6 pts |
| Gemini-2.5-Pro | 91.9% | 90.0% | 1.9 pts |
| GPT-4.1 | 89.2% | 30.0% | 59.2 pts |
Source: FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance, arXiv:2508.05201, Main Split results (Table 3), accessed July 2026.
Figure 1: Accuracy degrades sharply once a financial question requires chaining multiple figures together, and the degradation isn't uniform across models. Source: FAITH, arXiv:2508.05201.
Gemini-2.5-Pro's resilience comes from a specific behavior the paper's authors highlight in a case study: on a question requiring the model to infer an equity investment value from a property's total debt and ownership percentage, only Gemini-2.5-Pro correctly chained the table's debt figure with the text's ownership percentage — GPT-4.1 and Claude-Sonnet-4 both defaulted to a simpler (wrong) calculation that ignored the table entirely. The paper also documents a common failure mode worth naming directly: "scale errors," where a model recovers the right digits but the wrong magnitude — reporting "$150" instead of "$150 million." Correcting for scale errors alone would lift one open-source model's accuracy by 20 points, which tells you how often the underlying number is almost right in a way that's easy to miss on manual review.
The premium shows up against general-text benchmarks too
FAITH's numbers only mean something in context. Vectara's Hallucination Leaderboard, a continuously updated public benchmark, measures how often models hallucinate while summarizing ~7,700 general-domain documents — news, science, legal, business, and more. It's a different task than FAITH's masked-span prediction, but it's the best available apples-to-apples read on how these same models behave outside finance.
| Model | General-text hallucination rate | Financial multi-step error rate | Ratio |
|---|---|---|---|
| GPT-4.1 | 5.6% | 70.0% | 12.5× |
| Claude-Sonnet-4 | 10.3% | 20.0% | 1.9× |
| Gemini-2.5-Pro | 7.0% | 10.0% | 1.4× |
Sources: Vectara Hallucination Leaderboard (HHEM-2.3), updated May 2026; FAITH, arXiv:2508.05201 (financial rate = 100% − Multivariate Calculation accuracy, Main Split).
The two benchmarks use different tasks and shouldn't be read as a single unified error rate — but the direction is consistent and worth taking seriously: for every model tested, the financial multi-step error rate is higher than the same model's general-text hallucination rate, and for at least one frontier model the gap is an order of magnitude. Multi-step financial arithmetic isn't "just another kind of text" for these models; it's a harder task than the one general hallucination leaderboards measure.
Even good models fail when the pipeline feeds them badly
A second, independent benchmark shows the premium compounds with a separate problem: what actually reaches the model. FinanceBench tests GPT-4-Turbo on open-ended financial questions over 10-Ks, 10-Qs, and earnings releases.
| Retrieval condition | GPT-4-Turbo failure rate |
|---|---|
| Standard retrieval pipeline (naive RAG) | 81% |
| Correct source documents given directly | 21% |
Source: FinanceBench: A New Benchmark for Financial Question Answering, arXiv:2311.11944, with Patronus AI.
Figure 2: The same model, the same questions — a 60-point swing driven entirely by what reached the model's context. Source: FinanceBench, arXiv:2311.11944.
That 60-point gap is a retrieval failure, not a reasoning failure. The model didn't get worse at math; it got worse documents. Combined with FAITH's finding that even perfectly-retrieved context still produces a 10-20+ point accuracy drop on hard reasoning, the picture for anyone building a financial agent is two separate problems stacked on top of each other: get the right document to the model, and structure it so the model doesn't have to chain too much arithmetic across too much unstructured text.
Where FocusAlpha fits
Both failure modes point the same direction: less depends on picking the "best" model and more on what data reaches it and in what shape. It's the argument for putting a trusted data layer under your agents. FocusAlpha addresses the retrieval half directly: its SEC filings API returns structured, cited filings, financials, and earnings-call transcripts instead of a raw-document retrieval pipeline an agent has to get right on its own, and it reduces the reasoning half by returning pre-normalized figures (revenue, margins, growth rates) instead of requiring an agent to chain values across tables itself. Neither eliminates hallucination; both narrow the gap this data shows.
FAQ
Why do AI models hallucinate more on financial data than general text?
Financial questions often require chaining multiple figures from tables and surrounding text — a calculation, not a lookup — and that reasoning complexity is where models degrade most. On the FAITH benchmark, models that score above 90% on simple lookups can drop to 30-80% on tasks requiring three or more linked values (FAITH, arXiv:2508.05201).
What is the FAITH benchmark?
FAITH (A Framework for Assessing Intrinsic Tabular Hallucinations in Finance) is a 2026 evaluation built from 453 S&P 500 companies' 10-K filings. It masks a numeric value in a sentence and asks a model to recover it from the surrounding tables and text, then grades accuracy across four reasoning complexity tiers (arXiv:2508.05201).
Which AI model hallucinates least on financial data?
Among the models FAITH tested, Claude-Sonnet-4 and Gemini-2.5-Pro led on the hardest "Multivariate Calculation" category, at 80.0% and 90.0% accuracy respectively on the Main Split — well ahead of GPT-4.1's 30.0% on the same category, even though GPT-4.1 was competitive on simpler lookups (FAITH, arXiv:2508.05201).
Does better retrieval fix financial hallucinations?
It fixes part of the problem. On FinanceBench, GPT-4-Turbo's failure rate on financial questions dropped from 81% to 21% simply by supplying the correct source documents instead of relying on a standard retrieval pipeline — but even with perfect retrieval, a 21% failure rate (and FAITH's 10-20+ point accuracy drop on complex reasoning) shows retrieval alone doesn't close the gap (FinanceBench, arXiv:2311.11944; FAITH, arXiv:2508.05201).
What is a "scale error" in financial AI hallucinations?
A scale error is when a model recovers the correct digits but the wrong order of magnitude — for example, reporting "$150" when the source says "$150 million." FAITH's authors found this single error type was common enough that correcting for it alone raised one model's value-accuracy by 20 points (FAITH, arXiv:2508.05201).
How can I reduce financial data hallucinations in my AI agent?
The evidence points to two levers: improve what reaches the model (structured, cited retrieval instead of generic document search) and reduce how much arithmetic the model has to chain itself (pre-normalized figures instead of raw tables it must combine on the fly). Both measurably narrow the gap in the benchmarks above.
What is FocusAlpha?
FocusAlpha is a SEC filings API and agent-ready financial data layer: it turns SEC filings (10-K, 10-Q, 8-K, 13F), earnings-call transcripts, and other trusted company communications into structured, normalized data where every value keeps its citation back to the source document. AI agents connect via API or MCP to research public companies from complete, trusted information.