Every analyst agent compresses. A 10-Q's management discussion or a full earnings-call transcript doesn't fit comfortably in a working context, so somewhere in the pipeline an LLM summarizes it before anything reasons over it. A new working paper from an 18-author team spanning academia and industry, When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysis (arXiv, June 2026), tests what that step actually costs — using real SEC filings and earnings calls — and finds that summaries which contain no false statements still change the investment decision, in up to half of cases.
TLDR:
- On S&P 100 10-Q MD&A sections, one-shot LLM summaries flipped a downstream bear/neutral/bull call in 33% of cases — and aggressive token pruning flipped 50–53% — against an 11% no-compression baseline.
- The two failure patterns are decontextualization (the headline survives, the caveat doesn't) and model dependency (different compressors, different decisions — disagreeing on roughly one filing in four).
- The fix that works — auditing compressed claims against the original source — is what a structured data layer does by construction, instead of as an extra agentic step.
The experiment: real filings, a real decision task
The authors didn't test summarization quality in the abstract — they tested whether compression changes what you do. The setup pipes a source document, or a compressed version of it, into the same decision model and checks whether the bear/neutral/bull call changes (Lee et al. 2026).
| Component | What was used |
|---|---|
| Source documents | 300 10-Q MD&A sections + 297 earnings-call transcripts |
| Universe | S&P 100 firms, fiscal 2025 Q1–Q3, from SEC EDGAR |
| Compressors | GPT-5.4-Mini (naive + contextualized prompts), Finance-Llama3-8B with LLMLingua token pruning, DeepSeek-V4-Flash |
| Decision task | Bear / neutral / bull classification by a fixed decision model |
Source: When Summaries Distort Decisions, arXiv:2606.29251, experimental setup.
The headline metric is the decision-flip rate: how often the decision made on the compressed context differs from the decision made on the full source.
| Compression method | 10-Q MD&A | Earnings calls |
|---|---|---|
| No compression (baseline) | 11.0% | 8.8% |
| Contextualized summary | 21.3% | 22.2% |
| Naive summary | 33.0% | 23.9% |
| Token pruning (LLMLingua) | 50.0–53.0% | 39.1–50.8% |
| Agentic compression (ACC) | 20.3% | 18.5% |
Source: arXiv:2606.29251, Table 2. "Flip" = the rate at which compression changes the source-based decision; the baseline reflects the decision model's own run-to-run variance.
Figure 1: One-shot compression of MD&A sections flips the downstream call far above the 11% baseline. Source: arXiv:2606.29251, Table 2.
Two things stand out. First, even the careful prompt — one that explicitly asks the model to preserve context — still roughly doubles the flip rate over baseline. Second, token pruning, the cheapest and most popular compression family, is the most destructive: it flips the decision about as often as a coin toss.
Diagnostic 1: decontextualization — the caveat gets cut
The paper's first failure pattern is the most recognizable to anyone who reads filings for a living. Compression "may preserve headline claims while stripping away decision-relevant caveats, comparisons, or expectation framing" (Lee et al.). Revenue grew 12% — but the offsetting one-time item, the FX qualifier, or the "we expect this trend to moderate" sentence is gone. Nothing in the summary is false. The judgment it supports is.
The numbers are stark: in MD&A compression, context facts fall from 25% of all facts in the source to 9% in the one-shot summary. And the paper shows this omission is causal, not incidental:
| Add-back experiment (flipped MD&A cases) | Decision recovered |
|---|---|
| Restore the dropped context facts | 37% |
| Restore boilerplate / random facts from the same filing | 16–19% |
Source: arXiv:2606.29251, decontextualization diagnostics.
Figure 2: What compression drops is precisely the qualifying context. Source: arXiv:2606.29251.
Diagnostic 2: model dependency — change the model, change the view
The second pattern is quieter and, for anyone building agent pipelines, more unsettling: different compressors expose different views of the same source. Across compressor pairs, mean agreement was 0.75 — meaning two compressors disagree on roughly one filing in four even when their overall flip rates look similar. Your judgment inherits a hidden dependency on which model happened to do the summarizing, and a routine model upgrade can silently reshuffle which filings you read as bullish.
This is also where the paper's sharpest observation for multi-source research sits: when two sources conflict, a compressor tends to resolve the conflict silently rather than surface it. A summary hands you a single fluent narrative; the disagreement that should have been the finding never reaches you.
What actually helps — and what it implies for your stack
The authors' remedy, Agentic Context Compression, generates multiple candidate compressions and audits their disagreements against the original source. It works: flip rates drop to 20.3% on MD&A and 18.5% on calls, and an industry case study showed a 59.5% reduction in decision flips and a 23.8% improvement in forecasting IC over naive compression. But notice what the fix is doing: it re-anchors every compressed claim to the source document.
That is an argument for moving verification into the data layer rather than bolting it onto every agent. This is the design principle behind FocusAlpha: instead of agents free-text-summarizing filings they scraped, they query a structured SEC filings API where every value keeps its pointer back to the source document — cited by construction. Context isn't an optional garnish a summarizer may drop; qualifiers, comparatives, and period definitions travel with the number. And because every agent reads from the same single source of truth, the "which compressor did you get?" lottery — and the silently-resolved conflict between two sources — disappears.
Compression will always be part of agentic finance; context windows are a budget. The lesson of this paper is that unaudited free-text compression is a position you're taking without knowing it.
FAQ
What does the paper "When Summaries Distort Decisions" actually measure?
Information fidelity: whether an LLM-compressed version of a financial document leads a downstream model to a different investment decision (bear/neutral/bull) than the full document. It tests this on 300 10-Q MD&A sections and 297 earnings-call transcripts from S&P 100 firms.
How often do LLM summaries change an investment decision?
On 10-Q MD&A sections, 33.0% of naive one-shot summaries and 50.0–53.0% of token-pruned compressions flipped the decision, versus an 11.0% no-compression baseline. Earnings calls showed 23.9% and 39.1–50.8% respectively against an 8.8% baseline.
What is decontextualization in LLM summarization?
Compression that keeps a headline claim but strips the caveats, comparisons, or expectation framing needed to interpret it. In the study, context facts fell from 25% of the source to 9% of the summary, and restoring them recovered the original decision in 37% of flipped cases.
What is model dependency, and why does it matter?
Different compressor models produce materially different views of the same document — mean pairwise agreement was 0.75, i.e. disagreement on roughly one filing in four. A pipeline's conclusions can therefore change with a model swap, even if nothing about the underlying company changed.
Does this mean agents should never summarize filings?
No — compression is unavoidable, since model performance itself degrades as raw context grows. The finding is that summaries must be auditable against the source. The paper's fix re-checks compressed claims against the original document; a structured data layer builds that anchoring in from the start.
How does FocusAlpha address the problems in this paper?
FocusAlpha delivers filings, transcripts, and company data as structured fields that retain their source citations — cited by construction — so qualifiers travel with the numbers and every agent reads one consistent source of truth instead of a model-dependent free-text summary.
What is FocusAlpha?
FocusAlpha is a SEC filings API and agent-ready financial data layer: it turns SEC filings (10-K, 10-Q, 8-K, 13F), earnings-call transcripts, and other trusted company communications into structured, normalized data where every value keeps its citation back to the source document. AI agents connect via API or MCP to research public companies from complete, trusted information.