The engineering behind a PDF assistant you can verify

This is the engineer’s companion to From a scanned PDF to an answer you can trust . That post argued why a document assistant is trustworthy when every layer is measured or grounded. This one is about how you actually build each layer so that claim holds — the contracts, the failure modes, and the design decisions that survive contact with a real corpus.

I will name the stack I used (Docling , Astra DB hybrid search , Langflow on Cloudflare), but the engineering is portable. Swap any component and the same four design rules carry over. The running example is the same one: assistants over motorcycle service manuals at virago.edestudio.us , vfr750.edestudio.us , and hondamonkey.edestudio.us .

One framing note before the architecture: this is a second iteration. Version one of these sites used Cloudflare AI Search — a managed retrieval-augmented service you point at a bucket of documents — and it is still live on each site at /classic. The extraction layer below carried over from v1 unchanged; it was already scored and shipped. What v2 rebuilt is contracts C and D and the agent on top of them: retrieval and answering. So when the post weighs hand-built hybrid retrieval against a managed service, that is not a hypothetical — it is the exact before-and-after running side by side on the same domains.

The thesis in one sentence: treat each stage as an independently testable unit with a hard contract to the next, and put the trust in the contracts, not in the model.

The system as four contracts ¶

Most RAG diagrams draw boxes. The useful version draws the interfaces between the boxes, because that is where correctness lives:

  PDF ──▶ [extraction] ──▶ corpus + provenance ──▶ [index] ──▶ dual-index chunks ──▶ [retrieval] ──▶ ranked, cited passages ──▶ [agent] ──▶ answer + page links

   contract A: every value is scored against truth before it is allowed downstream
   contract B: every page carries "which manual, which chapter, which page" with it
   contract C: every chunk is searchable by meaning AND by exact string
   contract D: every stage is differential — re-running only touches what changed

Each contract is a thing you can test in isolation, and each one closes off a specific way the system could lie to you. The rest of this post is those four, in order.

1. Extraction as a scored, swappable stage ¶

The first design decision is to refuse to trust the extractor. OCR is the only stage whose errors nothing downstream can detect — a wrong digit just becomes the truth — so it is the stage that earns the most scaffolding.

The backend contract. Every extractor, whether a local OCR model or a hosted API, implements the same small interface: take a document, return per-page text, optional figures, and a metadata block carrying time and cost. Once extraction is behind a uniform contract, a hosted OCR API and an open-source engine are interchangeable inputs to the same scorer, and “which extractor” becomes a measurement instead of an opinion.

extract(document, page_range?) -> {
  pages:  { 1: "# ...md...", 2: "...", ... },
  images: { "page_003/fig_1.png": <bytes>, ... },
  meta:   { backend, model, seconds, cost_usd }
}

Ground truth and strict scoring. Pick a representative document and hand-verify a fixed set of values from across it — the things people actually look up (specifications, clearances, capacities). The scorer concatenates a backend’s output, normalises only what is safe to normalise (collapse whitespace, strip table-cell pipes), and checks each ground-truth value for an exact presence. Crucially the normaliser never touches a digit or a unit glyph. There is no partial credit: “23 N·m” is present or it is not. Strictness is the point — a metric that forgives a transposed digit is measuring the wrong thing for this domain.

The same code path that the scorer runs is the code that produces the production corpus, so the number you trust is the number you ship. The sweep is cached (skip-if-exists) and per-backend isolated, so one backend failing or one new backend being added does not disturb the rest.

On the reference manual this produced a 24-point spread across four local backends (61% to 85%), and the version that had originally been shipped was the worst of the four. The headline number matters less than the shape of the result: without the harness you cannot tell those backends apart by looking, and the difference between them is the difference between a tool and a liability.

The harness also makes the cost/quality trade explicit rather than implicit. A hosted OCR API scored a few points above the best local engine on the same test — but the local engine (RapidOCR, reading the page pixels) was chosen anyway, because it carries no per-page bill, sends no document off the machine, and works on a confidential corpus or offline. That is a defensible decision precisely because it is quantified: you are trading a known, small number of points for properties you can name, not guessing. When the contract is “any backend, scored on the same basis”, the build-versus-buy and local-versus-hosted arguments stop being vibes and become a column in the results table.

Corpus prep is part of extraction, not an afterthought. Two things happen before a page is allowed to become a chunk:

• Structure-aware splitting. Respect heading boundaries and keep specification blocks and code-like tables intact. A chunk that begins mid-table or mid-fence is useless context — the retriever will surface it and the model will reason over half a row. Page count, chunk count, and hit count are three different numbers; you tune and measure at the chunk level, not the page level.
• Provenance injection (contract B). Every page gets a self-describing header — which manual, which chapter, which page number — and its figure links are rewritten to absolute URLs. This is a few bytes per page and it is the single most important thing you do for the agent layer, because citations are impossible to retrofit. If the page does not carry its own identity into the index, no amount of cleverness at answer time can reconstruct where a fact came from. Carry it from the first stage or live without verifiable answers.

2. Hybrid retrieval: two indexes, one query, a reranker on top ¶

The second design decision is to stop making the retriever choose between meaning and exact strings, because a technical document needs both in the same query. Version one delegated retrieval to a managed search service: fast to stand up, but a single retrieval mode you cannot give two arms to, and its misses clustered precisely on the exact-string lookups a manual exists to answer. That is the motivation for the rebuild.

The chunk shape (contract C). Each chunk is stored carrying two parallel representations of the same text, plus its metadata:

{
  content:   "Valve clearance (intake): 0.07 - 0.12 mm ...",
  $vectorize: "<same text, embedded server-side into a vector>",
  $lexical:   "<same text, tokenised for keyword search>",
  metadata:  { manual, chapter, page }
}

The semantic side is embedded by the database on write using a hosted embedding model. That is a deliberate operational choice: there is no embedding API key to hold or rotate in your own code, and the vector dimension is fixed by the model so it can never drift between ingest and query. The lexical side is the classic keyword index — it finds “0.07 mm” because it is matching characters, not concepts.

Why both, concretely. A pure-vector store is strong on “how do I bleed the brakes” and weak on “front axle nut torque”, because a bare number embeds almost identically to every other small number in the book. A pure-keyword store is the exact inverse. The failure is structural, not tunable — you cannot embed your way to good exact-string recall. So you run both arms and fuse them.

Find-and-rerank. The query issues both a semantic sort and a lexical match, each capped by its own limit, then a reranking model scores the merged candidate set against the original question and reorders it. From the application’s side this is a single call:

results = collection.find_and_rerank(
  query        = user_question,     # drives both the vector and lexical arms
  hybrid_limits= { vector: 20, lexical: 20 },
  rerank_on    = "content",
  include_scores = true,            # keep scores while you are tuning
).limit(top_k)

The knobs that actually move quality: the per-arm hybrid_limits (how many candidates each side contributes before reranking), the final top_k after rerank (kept small to bound prompt size and latency), and an awareness that lexical tokenisation is whitespace-delimited all-keywords matching — so query phrasing affects the lexical arm more than the vector arm. Keep scores on while tuning and turn them off in production.

The reranker is what lets you combine two mediocre-on-their-own result lists into one good one. Without it you are concatenating ranked lists and praying; with it, a conceptual question and a one-digit lookup come back correct from the same endpoint.

3. Make every stage differential ¶

The third design decision is that re-running is the normal path, not a rebuild. A corpus that is expensive to correct will, in practice, stay wrong. So every stage records what it last produced and, on a re-run, touches only what changed (contract D).

• Extraction keys each page by a content hash. Re-extracting a source re-converts only the pages whose hash moved, leaves unchanged pages in place, and deletes pages that disappeared from the source.
• Indexing carries a checksum manifest: syncing the corpus into the vector store re-embeds only changed chunks and removes deleted ones. A no-op sync after no changes does nothing and costs nothing.
• Publishing (for the managed-search baseline described below) records what it last uploaded and, on re-run, PUTs the changed pages, DELETEs the removed ones, and skips the rest.

The pattern is the same at every stage: a small sidecar of “what I last did”, a diff against current state, and work proportional to the change. The payoff is not just cost — it is that correcting one page after spotting a bad value is a five-second operation, which means people actually do it. Cheap-to-fix is what keeps a corpus correct over time.

4. The agent, and how citations actually work ¶

The fourth layer turns ranked passages into an answer a human can verify. Two engineering details carry it.

Streaming behind a thin edge proxy. The answering flow runs on a private service; the public site never holds its credentials. A small edge function sits in front, takes the user’s question, calls the flow, and transforms its event stream into a token-and-thinking stream the browser renders incrementally. The API key lives only in the edge function’s environment, never in the static page or the client bundle. This split — a public “experience” tier and a private “brain” tier with a streaming transform between them — is what keeps the thing both responsive and safe to expose.

The citation parser is where contract B pays off. Because every chunk dragged its manual | chapter | page header all the way through extraction and indexing, the retrieved context the agent reasons over still carries those source markers. The agent is instructed to cite them; a parser then extracts each marker from the generated answer, maps it back to the exact manual page, and rewrites it into a deep link. The result is an answer where every claimed figure points at the page of the official manual it came from, and the reader can open it. None of that is possible if provenance was not injected at stage one — which is why it is a stage-one job, not an answer-time one.

Keep the previous version as a baseline. Alongside the agentic front door, each site keeps version one running at /classic — the managed-search build the rebuild replaced — over the same documents. It is partly a fallback for when the agent service is down, and partly an honesty check: it lets you compare the hand-built hybrid+rerank agent against the managed setup it superseded, in public, on the same questions. If your bespoke pipeline cannot beat the managed baseline, that is worth knowing before you commit to maintaining it. Here it does, on exactly the exact-string queries the rebuild targeted — which is the kind of claim you want to be able to demonstrate rather than assert.

One source, many sites. All three assistants are the same codebase. Everything that differs per site — branding, domain, which corpus, which flow — is config; the shared functions and front-end are written once and templated at build time, and each site’s secrets stay on its own deployment. Adding a fourth manual is a config file and a deploy, not a fork. This is the same “isolate the unit, define the interface” discipline as the rest of the system, applied to the sites themselves.

The contracts, restated ¶

Pull the four design rules back together, because the system’s trustworthiness is exactly the conjunction of them:

Each is independently testable. Each closes one specific failure mode. None of them is “use a better model” — the model is the most swappable part of the system and the worst place to put your trust.

What I would tell an engineer building this ¶

1. Score your extraction before anything else. Until you have a number for how much of your source survived, every downstream metric is built on sand. Hand-key a fixed ground truth once; it pays for itself the first time you compare two extractors.
2. Store two indexes, not one. If your documents contain exact strings people search for, semantic search alone will quietly fail on precisely those queries. Hybrid plus a reranker is the default, not an optimisation.
3. Inject provenance at extraction time. Citations cannot be reconstructed later. The few bytes of “which page” you add at the start are what make the whole thing verifiable at the end.
4. Make every stage differential. Build the diff in from day one. The goal is that re-running is boring and cheap, because that is what keeps the corpus correct.
5. Keep secrets at the edge and keep a baseline. A thin public tier in front of a private brain keeps you safe; a managed-search baseline keeps you honest.

Try it, and read the overview ¶

If you want the case for why this matters rather than how it is built, start with From a scanned PDF to an answer you can trust .

Then go stress-test the contracts directly. Open virago.edestudio.us , vfr750.edestudio.us , or hondamonkey.edestudio.us and ask a precise specification followed immediately by a “how do I” procedure. If both come back correct and the figure cites its page, you are watching all four contracts hold at once. That is the entire design, working.