--- title: "Why a \"Smarter Model\" Can't Fix \"Worse Data\"" slug: why-a-smarter-model-cant-fix-worse-data date: 2026-07-15 excerpt: "The most common question we get: won't better models make the data layer unnecessary? The benchmark evidence points the other way — the ceiling on answer quality is set by the data." category: Data Infrastructure tags: ["Data Infrastructure", "AI & Agents"] author: Jennifer Ma authorRole: "Co-founder & CEO" status: published --- The most common question we hear from investment teams evaluating agent infrastructure is some version of our own [FAQ](https://www.focusalpha.ai/#faq): *"Will better models make the data layer unnecessary?"* Our answer has always been no — better models improve reasoning, but they still depend on the information they can access. That used to be a thesis. Three years of benchmark evidence has made it a measurement: on financial tasks, the failure rate of a frontier model moves 4× with the quality of its context, while moving up a model generation barely moves it at all. **TLDR:** - The same GPT-4-Turbo fails [81% of FinanceBench financial questions with a standard retrieval setup, but 21% when handed the full documents](https://arxiv.org/abs/2311.11944) — a 4× swing from data alone. - Model progress doesn't remove the data dependency: [all 18 frontier models Chroma tested degrade as context grows noisier and longer](https://www.trychroma.com/research/context-rot), and [2026-frontier compressors still flip investment decisions in 21–53% of cases](https://arxiv.org/abs/2606.29251). - The industry is pricing this in: Gartner predicts organizations will [abandon 60% of AI projects unsupported by AI-ready data through 2026](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk). ## Same model, different data: a 4× gap The cleanest evidence comes from [FinanceBench](https://arxiv.org/abs/2311.11944), a 10,231-question benchmark on public-company financials built by Patronus AI with real filings as ground truth. The authors evaluated 16 model configurations on a 150-case sample with 2,400 manually reviewed answers. Hold the model fixed and vary only what context it receives: | Configuration (same model: GPT-4-Turbo) | Failure rate (incorrect or refused) | | --- | --- | | Standard retrieval system (vector store) | 81% | | Full documents provided in context | 21% | *Sources: [FinanceBench, arXiv:2311.11944](https://arxiv.org/abs/2311.11944); [Patronus AI announcement](https://www.patronus.ai/announcements/patronus-ai-launches-financebench-the-industrys-first-benchmark-for-llm-performance-on-financial-questions).* ![Bar chart: GPT-4-Turbo fails 81% of FinanceBench questions with a retrieval system but 21% with full documents in context](/images/blog/why-a-smarter-model-cant-fix-worse-data/fig-1.svg) *Figure 1: Nothing about the model changed between these two bars — only the data pipeline feeding it. Source: [FinanceBench](https://arxiv.org/abs/2311.11944) / [Patronus AI](https://www.patronus.ai/announcements/patronus-ai-launches-financebench-the-industrys-first-benchmark-for-llm-performance-on-financial-questions).* No model release in history has cut a task's failure rate by 60 points. A better retrieval and data layer did exactly that, for the *same* model, on the same questions. ## "Just wait for bigger context windows" doesn't work either The usual rebuttal is that context windows will get big enough to skip curation entirely: dump every filing in and let the model sort it out. Chroma's [Context Rot](https://www.trychroma.com/research/context-rot) study tested that premise across 18 frontier models — including o3, GPT-4.1, Claude Opus 4, and Gemini 2.5 Pro — and found performance degrades as input length grows, for every model tested, long before the advertised context limit. | Experiment | Comparison | Finding | | --- | --- | --- | | LongMemEval | ~300-token focused context vs ~113k-token full prompt | All models significantly better on the focused context | | Input scaling | Same task, growing input length | Performance degrades with length across all 18 models | | Haystack structure | Coherent documents vs shuffled sentences | Models score *better* on shuffled — structure itself adds load | *Source: [Context Rot — Chroma Research technical report, July 2025](https://www.trychroma.com/research/context-rot).* Newer models degrade more gracefully than older ones, but the direction never changes — which means the "dump everything in" strategy gets worse precisely as your coverage grows. Relevance filtering isn't a workaround for weak models; it's what keeps strong models strong. ## Frontier models still distort what they read Model capability also doesn't protect the step everyone treats as safe: summarizing the raw material. A June 2026 study of [LLM compression on real 10-Q MD&A sections and earnings calls](https://arxiv.org/abs/2606.29251) — using 2026-frontier compressors — found summaries that flip the downstream investment call far above the decision model's own variance: | Compression by 2026-frontier models (10-Q MD&A) | Decision-flip rate | | --- | --- | | No compression (baseline variance) | 11.0% | | Contextualized summary | 21.3% | | Naive summary | 33.0% | | Token pruning | 50.0–53.0% | *Source: [When Summaries Distort Decisions, arXiv:2606.29251](https://arxiv.org/html/2606.29251), Table 2. We cover this paper in depth in [its own post](/blog/when-ai-compresses-your-filings).* Worse for the "models will fix it" thesis: different frontier compressors [disagreed on roughly one filing in four](https://arxiv.org/html/2606.29251). If model capability were the binding constraint, better models would converge on the same reading of the same document. They don't. ## The market has started pricing data readiness This isn't only a research finding. Gartner's prediction is that [through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk), and its Q3 2024 survey found [63% of 248 data-management leaders either don't have or aren't sure they have the right data practices for AI](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk). The projects failing aren't failing for lack of model IQ. ## Better models increase the value of better data None of this is an argument against better models — we're building for them. It's an argument about what compounds. A weak model on clean, complete, cited data leaves evidence on the table. A strong model on fragmented, scraped, silently-summarized data produces something more dangerous: fluent, confident answers built on incomplete evidence. Strong models on structured, complete data is where the returns compound — better reasoning applied to information actually worth reasoning over. ![Two-by-two diagram: model capability vs data quality; strong model with weak data is confident but blind, strong model with strong data compounds](/images/blog/why-a-smarter-model-cant-fix-worse-data/fig-2.svg) *Figure 2: The failure quadrant that matters isn't weak-model/weak-data — it's strong-model/weak-data, because the output still sounds authoritative.* That's why our FAQ answer ends the way it does: **better models increase the value of better data.** Every capability gain makes the model better at exploiting the evidence you give it — and no capability gain recovers the evidence you didn't. ## FAQ ### Will better models make a financial data layer unnecessary? No. Better models improve reasoning, but they depend on the information they can access — if it's missing, fragmented, or inaccurate, [even the best model cannot fully compensate](https://www.focusalpha.ai/#faq). Empirically, data quality moves financial-task failure rates far more than model choice ([FinanceBench](https://arxiv.org/abs/2311.11944)). ### What's the strongest evidence that data, not the model, is the bottleneck? The same GPT-4-Turbo fails [81% of FinanceBench questions with a standard retrieval setup but 21% with the right documents in context](https://arxiv.org/abs/2311.11944) — a 4× difference with zero change to the model. ### Won't huge context windows remove the need for data curation? No — [Chroma's testing of 18 frontier models](https://www.trychroma.com/research/context-rot) shows performance degrades as input grows, well before advertised limits, and models answer better from ~300 focused tokens than from ~113k-token full prompts. More dumped context makes answers worse, not better. ### What is "AI-ready data"? Data that's complete, normalized, deduplicated, and structured so a model can retrieve exactly the relevant evidence with its qualifying context attached. Gartner predicts organizations will [abandon 60% of AI projects that lack it through 2026](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk). ### If models keep improving, what should an investment team invest in first? The data layer — it compounds with every model upgrade. Benchmark gaps from missing or noisy context ([4× on FinanceBench](https://arxiv.org/abs/2311.11944)) dwarf generation-over-generation model gains on financial tasks, and [summarization distortion persists even with 2026-frontier models](https://arxiv.org/abs/2606.29251). ### How does FocusAlpha fit into this? FocusAlpha is the data layer: filings, transcripts, and company communications normalized into structured, agent-ready fields where every value keeps its source citation — so whatever model you run, it reasons over complete, trusted evidence instead of scraped fragments. ### What is FocusAlpha? FocusAlpha is a [SEC filings API](/docs/api/filings) and agent-ready financial data layer: it turns SEC filings (10-K, 10-Q, 8-K, 13F), earnings-call transcripts, and other trusted company communications into structured, normalized data where every value keeps its citation back to the source document. AI agents connect via [API or MCP](/docs) to research public companies from complete, trusted information. ## Sources - [FocusAlpha FAQ — "Will better models make the data layer unnecessary?"](https://www.focusalpha.ai/#faq) - [FinanceBench: A New Benchmark for Financial Question Answering — arXiv:2311.11944](https://arxiv.org/abs/2311.11944) - [Patronus AI launches FinanceBench (long-context configuration numbers)](https://www.patronus.ai/announcements/patronus-ai-launches-financebench-the-industrys-first-benchmark-for-llm-performance-on-financial-questions) - [Context Rot: How Increasing Input Tokens Impacts LLM Performance — Chroma Research, July 2025](https://www.trychroma.com/research/context-rot) - [When Summaries Distort Decisions — arXiv:2606.29251](https://arxiv.org/abs/2606.29251) - [Gartner: Lack of AI-Ready Data Puts AI Projects at Risk — press release, Feb 26, 2025](https://www.gartner.com/en/newsroom/press-releases/2025-02-26-lack-of-ai-ready-data-puts-ai-projects-at-risk)