Some questions about filings aren't retrieval questions.

A 10-K runs to a few hundred pages, most of it boilerplate that barely moves year to year. The signal is in what changed — a risk factor that gained a paragraph, a hedge that got firmer, a customer concentration disclosure that grew a sentence. Finding that by reading is possible and nobody does it.

The incumbent tools — AlphaSense, Hebbia, Rogo — solve this for institutions at institutional prices. Conviction is scoped to one person’s watchlist, and to transparency: every claim points at the sentence it came from.

[Your turn: a paragraph on why you personally wanted this, from four years on an FICC desk. That framing is the part nobody else can write.]

Filings come from EDGAR via sec-edgar-downloader, get parsed into sections, and are chunked section-aware at roughly 800 tokens with 100 tokens of overlap. Chunks never span Item boundaries, because a passage straddling Item 1A and Item 7 is answerable to neither question. Embeddings are text-embedding-3-small; the store is ChromaDB, local and persistent.

Generation runs against GPT-4o under a prompt that refuses to answer outside the provided excerpts and requires a [Source N] marker on every claim. LangChain is in the project for document loaders and nothing else — everything downstream talks to the SDKs directly, which keeps the failure modes legible.

Two capabilities sit on top: a multi-ticker watchlist, and risk-factor diffing — comparing this year’s Item 1A against last year’s and reporting what changed in the language.

I built the diff feature on top of retrieval, because retrieval was the system I had. Ask for both years’ risk factors, get the top-k chunks for each, compare them. It produced plausible output and it was wrong in a way that took an eval set to see.

Any retrieval step returns the passages most similar to a query — which, for two versions of the same document, are the passages that didn’t change. The parts that moved are exactly the parts least likely to survive a similarity ranking. I was sampling the boilerplate and diffing that.

The fix was to stop retrieving. The diff module in src/analyze/ bypasses Chroma entirely and does whole-document comparison, with an LLM producing structured JSON on what changed rather than a character-level diff:

# src/analyze/ — no vector store in this path
prior, current = load_item_1a(ticker, y-1), load_item_1a(ticker, y)
diff = llm_json_diff(prior, current)   # not difflib
# difflib reports every reworded sentence as a change.
# The question is which changes carry meaning.

Character-level diffing was the other wrong answer I tried. Filings get lightly reworded every year, so difflib flags hundreds of edits and buries the three that matter. The judgment about which changes are material is the product, and that judgment needs a model reading both passages in full.

[Your turn: one concrete example. A ticker, a year, and a specific risk-factor change the system surfaced. One real example is worth the whole section.]

• Grow the eval set and split it by question type — factual lookup, synthesis, refusal — since those fail differently
• Measure citation fidelity directly: does the cited chunk actually contain the claim? That’s the failure mode a fluent answer hides
• Extend diffing beyond Item 1A to MD&A, where the language moves more and means more
• Benchmark grounded retrieval against frontier models answering from parameters alone — the question dart-rag takes up in Korean

The transferable lesson: the architecture should follow the question, not the other way round. I had a retrieval system, so I reached for retrieval. The feature only worked once I was willing to route around the thing I’d just built.