# Reader brief — audit a retrieval pipeline by deleting stages

**Use this if:** you have a retrieval or ranking stack that has grown one reasonable-looking stage at a
time, and you cannot say which stage is actually carrying the answer. **Time:** one afternoon for the
harness, one day for the ablations. **You need:** a query set whose correct entry you know, the raw
ranking your system produces, and permission to delete code.

This is the method behind *A Forward Pass on the Graph*: the pipeline was not redesigned, it was
audited. Every stage was removed in turn and kept only if the number moved. The pack contains the
code that survived, the architecture document with the removed-stage table, and the figure script.

## The prompt

```text
You are a retrieval engineer auditing a ranking pipeline. Your job is NOT to improve the
pipeline. Your job is to find out which stages are carrying weight and delete the rest.

INPUTS
- PIPELINE: the ordered stages from query to ranked result (paste the list or point at the code).
- EVAL SET: N queries with a known correct entry, plus a separate set of OUT-OF-CORPUS queries
  whose correct behaviour is to decline.
- RAW ORDER: the ordering produced by the earliest stage that already returns candidates
  (usually plain similarity), with no bonuses, multipliers or re-ranks applied.
- CURRENT ORDER: the ordering the live configuration produces.
- Any existing ablation notes or receipts.

DO
1. Write down the pipeline as ordered stages, and for each stage: its input, its output, and the
   one number it is supposed to move (hit@10, grounded rate, decline rate, latency).
2. Establish two baselines on the eval set: CURRENT and RAW. Report both, per query set.
3. Ablate ONE stage at a time against a frozen store and a fixed seed. Record: hit@10, the delta
   against CURRENT, and whether the ordering is byte-identical, merely reordered, or actually
   different.
4. For every multiplicative term, test several values (e.g. 0.3, 0.5, 0.7, 1.0) and report whether
   the orderings differ at all. A value that produces identical orderings is a hard partition, not
   a prior.
5. For any stage that only reorders, prove it cannot change admission: check the set of candidates
   before and after, not just the order.
6. For any stage that only admits, prove it cannot evict: the score of an admitted entry must be
   computed exactly like every other entry, and the admission must be a union, never a max/rerank.
7. Re-run the winning configuration twice on the same queries and diff the top-k. Determinism is a
   requirement, not a nice-to-have.
8. Produce a decision table: stage | measured delta | identical ordering? | keep/delete | evidence.

RULES
- Do not add a stage to compensate for a stage you are testing.
- Do not tune two stages together; one variable at a time.
- A stage that cannot be shown to move a number is deleted, even if it is interesting and even if
  it took weeks.
- If the eval set cannot reach the behaviour a stage is meant for, say so: the instrument is
  pointed at the wrong distance and no conclusion is valid.

DELIVERABLE
A short report with: the pipeline table, the two baselines, the ablation table with raw numbers,
the list of deleted stages, the surviving pipeline in one paragraph, and a "what I did not verify"
section. No prose about the elegance of the design.
```

## Feed it these files

| File | What it gives the audit |
|---|---|
| `architecture.md` | the pipeline diagram, the removed-stage table with verdicts, and the phase-engine ablation |
| `forward-pass.ts` | the surviving pass: routing, geodesic similarity, floors, the two admission rules |
| `retrieve.ts` | what the retrieval layer actually returns to the caller |
| `reranker.ts` | the precision stage, its similarity band and its no-op fallback |
| `grounding-ladder.ts` | the decline gate and how grounding is computed |
| `make-figure.py` | the figure, reproducible with Pillow |

## What good looks like

- [ ] Both baselines are reported on the same eval set, with the query count and the store size.
- [ ] Every removed stage has a number attached, not an opinion.
- [ ] At least one "obviously sensible" term is shown to be inert or harmful — that is the finding.
- [ ] Removing a multiplicative partition leaves the ordering **byte-identical**, or the claim is
      withdrawn.
- [ ] The survivor is described in one direction: no stage reads the output of a later one.
- [ ] Out-of-corpus probes still decline at the same rate after every deletion.
- [ ] The same query returns the same top-k twice in a row.

## Failure modes this piece paid for

- **A confident plan beats a measurement.** The plan said the trained window was the ceiling and
  there was nothing above it. The ablation said every additive term lowered hit@10. Write the
  numbers down before the architecture diagram.
- **A multiplier that is really a partition.** The cross-node ×0.3 was tested at 0.3, 0.5 and 0.7 and
  produced identical orderings; it was never a soft prior. Test the ordering, not the value.
- **A re-rank that changes admission.** If the "precision" stage can drop or add candidates, it is
  doing recall's job, and the two can no longer be measured separately. Gate it to a band.
- **An admission stage that can evict.** Taking the better of two scores lets one key cancel a
  document match. Union only; score every candidate the same way.
- **A test that cannot reach the horizon.** A probe that stops at 6% of the window cannot tell you
  the window is wrong. State the reach of every instrument.
- **A walk that compounds.** Each stage reading the last means an error does not stay local. If you
  cannot flatten the pipeline, you cannot attribute the answer.

## Where to go next in the pack

`architecture.md` first, then `forward-pass.ts`. `retrieve.ts` and `reranker.ts` are the two stages
most likely to disagree about who owns admission; `grounding-ladder.ts` is the "or refuse" branch.
`make-figure.py` redraws the figure from the same numbers.

The two rules that keep the stages honest are in the piece: *a precision stage can only reorder what
recall surfaced*, and *an admission stage may only add*. They are testable; use them as the review
checklist.