Retrieval finds the right passage half the time. The system knows when it hasn't.

A 10-K runs to a few hundred pages, most of it boilerplate that barely moves year to year. The signal is in what changed. A risk factor that gained a paragraph, a hedge that got firmer, a customer concentration disclosure that grew a sentence. Finding that by reading is possible and nobody does it.

The incumbent tools, AlphaSense and Hebbia and Rogo, solve this for institutions at institutional prices. Conviction is scoped to one person's watchlist, and to transparency. Every claim points at the sentence it came from, and the system declines rather than guessing when the filing doesn't support an answer.

I aimed it at credit questions specifically: liquidity, debt, risk factors. Those are the questions where the answer lives in language rather than in a number you could pull from a table, and where being pointed at the wrong paragraph costs you something.

Filings come from EDGAR, get parsed into Items 1, 1A, 7 and 8, and are chunked section-aware at roughly 800 tokens with 100 tokens of overlap. Chunks never span Item boundaries, because a passage straddling Item 1A and Item 7 is answerable to neither question. The store is ChromaDB with ticker and item metadata, so retrieval scopes to one company and one part of the filing.

Generation runs against GPT-4o under a prompt that refuses to answer outside the provided excerpts and requires a [Source N] marker on every claim. LangChain is in the project for document loaders and nothing else. Everything downstream talks to the SDKs directly, which keeps the failure modes legible.

It runs as two containers on a t3.small, with Caddy terminating HTTPS and the Chroma store on a mounted volume. A FastAPI layer sits in front: API keys stored as SHA-256 hashes, per-key hourly and daily limits, a global daily cap, and a server-held demo key so the public page works without anyone signing up. Each request writes one structured JSON line to CloudWatch, carrying no keys and no question text.

I chose EC2 over a managed runtime deliberately. The app needs a persistent disk, and I wanted to be able to explain every layer of the thing rather than point at a platform that handles it.

I built year-over-year risk-factor diffing on top of retrieval, because retrieval was the system I had. Ask for both years' risk factors, get the top-k chunks for each, compare them. It produced plausible output and it was wrong in a way that took an eval set to see.

Any retrieval step returns the passages most similar to a query. For two versions of the same document, those are the passages that didn't change. The parts that moved are exactly the parts least likely to survive a similarity ranking. I was sampling boilerplate and diffing it.

The fix was to stop retrieving. The diff path bypasses the vector store entirely and does whole-document comparison, with a model producing structured output on what changed. Character-level diffing was the other wrong answer I tried: filings get lightly reworded every year, so a text diff flags hundreds of edits and buries the three that matter. The judgment about which changes are material is the product, and that judgment needs a model reading both passages in full.

29 questions across the four filings, 23 answerable and 6 that should be refused, with 10 of the answerable ones tagged as credit questions. Every question was checked against the filing by hand before it entered the set. Every citation the system produced was labeled by hand afterwards. No model judged any part of this.

The first metric was useless

I started by measuring item-level hit@8: did any retrieved chunk come from the Item the answer lives in. It came back near-perfect, and that number meant nothing. Each ticker has only four Items indexed, so retrieving eight chunks from the right one is close to guaranteed whether the system works or not.

So I replaced it. Answer-text recall@8 asks whether the evidence sentence I recorded when writing the question actually appears in the eight chunks that came back. That is a much harder test and it produced a much more useful number.

Retrieval is the weak link, and the system knows it

Just over half the time, the passage containing the answer doesn't make it into the context window. On credit questions, the ones I built this for, it's worse at 40%. That is the headline problem and no amount of prompt work fixes it.

What held up is the behavior around that failure. It refused all 6 questions it should have refused. It falsely refused only twice, and both false refusals were cases where retrieval had genuinely missed the evidence. The system was not being timid. It was correctly reporting that it couldn't see the answer.

Citation precision at 81% is the number I like least. Roughly one citation in five doesn't support the claim attached to it, and the two worst cases are instructive: one cited the wrong passage while still reaching a correct conclusion, and one attached a citation to a statement about something the filing doesn't mention. Both look completely normal in the output. That is exactly the failure a fluent answer hides, and it is why the labeling is manual.

Next, in order: fix the chunker and re-label, since retrieval recall is the ceiling on everything else. Try hybrid retrieval, because filings repeat enough standard phrasing that lexical matching should help where embeddings are drifting. Extend diffing beyond Item 1A to MD&A, where the language moves more and means more. And benchmark grounded retrieval against frontier models answering from parameters alone, which is the question dart-rag takes up in Korean.

The transferable lesson from the whole build: the architecture should follow the question, not the other way round. I had a retrieval system, so I reached for retrieval. The diff only worked once I was willing to route around the thing I had just built, and I only found out it was broken because something was measuring it.