# Reader brief — audit a grounded answer, and the refusal next to it

**Use this if:** you run a retrieval-augmented system (or depend on one) and want to know whether its
answers actually come from its sources, or whether the sources are decoration on a model's guess.
**Time:** one afternoon. **You need:** an endpoint that returns the answer *and* the retrieval record
— the passages, scores, threshold and verdict — or the ability to add that record to the response.
Without the record you can audit the prose, not the system.

This is the method behind *The engine, on the record*: one question the store should answer, one it
should not, and the same three numbers read from both. The pack contains the two raw response objects,
the live status and topology reads, the ladder code, and the architecture note.

## The prompt

```text
You are auditing a grounded-retrieval system. Your job is NOT to judge whether the answers sound
good. Your job is to prove where an answer came from — and what the system does when it has nothing.

INPUTS
- ENDPOINT: the public chat endpoint and the shape of its response (paste one real response object).
- STORE FACTS: the system's own status read: store size, index/cube counts, genesis/age, configured
  threshold or decline policy, and the calibration/audit record if one exists.
- ONE ANSWERABLE QUESTION whose correct source you know, and ONE OUT-OF-CORPUS QUESTION whose right
  behaviour is to decline (invent a mission, a paper, a person; do not use something merely obscure).
- The retrieval record: passages returned, their scores, the threshold, the verdict, whether a second
  reader was called, whether a web search ran and what it returned, and whether the writer was asked.

DO
1. Read the system's own status first and write down its claims: store size, units/cubes, threshold,
   decline policy, freshness. A number you cannot read from the system is not a receipt.
2. Ask the answerable question. Record: the route (which unit/cube/dictionary answered), the top-k
   passage scores, the threshold, the verdict, the source ids named, and the wall-clock time.
3. Ask the out-of-corpus question. Record the same fields, plus: did the engine search the web itself?
   How many results came back, what did the best one score, and what was the floor for counting it?
   Was the writer called at all, or did the engine answer with its own sentence?
4. Compare the two turns field by field. The refusal is the evidence that the answer path is a
   decision and not a style. If the writer is called on a refusal, the refusal is the model's, not
   the system's.
5. Check the routing number is not being used as confidence: compare the route score (resonance /
   centroid distance) of an answered turn with a refused one. A higher route score on the refused
   turn proves routing is a label, not a confidence.
6. Re-run both questions twice, back to back, and diff the top-k and the attribution. Determinism is a
   requirement, not a nicety.
7. Read the threshold from the code, not from the prose. Then read the calibration record: how many
   out-of-corpus probes were caught, how many in-corpus probes were falsely declined, on what date and
   at what store size.
8. Write the receipt: question | route | top score | threshold | verdict | rung | writer asked |
   latency. One row per turn, raw numbers only.

RULES
- Do not ask "does the answer look right?"; ask "which passage, at what score, and was it above the
  floor?" An answer can be right for the wrong reason.
- Do not accept a citation the system writes in prose. Accept only a source id the engine attached to
  the record, and then check that the source was in the retrieved window.
- Do not treat "the web was searched" as "the web had an answer". Require the result's own score
  against the question and the floor it had to clear.
- Do not audit only the refusals or only the answers. The pair is the instrument.
- If the system cannot return the retrieval record, say so: the audit is not possible at that surface.

DELIVERABLE
Two response objects (or trimmed records), a status capture, a two-row receipt table, the threshold
and calibration numbers with their date and store size, and a short "what I did not verify" section.
```

## Feed it these files

| File | What it gives the audit |
|---|---|
| `public-status.json` | the system's own status: 145,565 entries, 21 specialists, genesis, calibration record, decline policy |
| `cubes-topology.json` | the store's shape: 22 cubes, the 760-entry generalist, every specialist and its size |
| `chat-covered.json` | the answered turn, in full: route, 20 passages from one document at 0.7833–0.8278, attribution at 0.828 against 0.76 |
| `chat-declined.json` | the refused turn, in full: best 0.7171, web search 8 results at best 0.748 against 0.800, writer not asked |
| `grounding-ladder.ts` | the refusal mechanism: three verdicts, related margin, web floor, attribution as data, who gets which rung |
| `architecture-2026-10-18.md` | the whole pipeline and the constants, with the code map |
| `reproduce.md` | the exact commands and what to look for in each response |

## What good looks like

- [ ] The status read is captured before any question is asked, with a timestamp.
- [ ] The answered turn names a source id that is present in its own retrieved window.
- [ ] The refused turn shows `writer.asked: false` (or its equivalent), and the engine's own sentence.
- [ ] The web search, if any, has a **measured** best result and a floor — not just a result count.
- [ ] The calibration record has a date, a store size, and both error directions: missed out-of-corpus
      probes and falsely declined answerable ones.
- [ ] The routing/confidence number is shown to be a label: the refused turn can score *higher* on it.
- [ ] Both questions return the same top-k and the same attribution when repeated.
- [ ] The receipt table has raw numbers, not adjectives.

## Failure modes this piece paid for

- **A gate with no measured outcome.** The threshold is a number until the calibration record says how
  many out-of-corpus probes it caught and how many answerable ones it killed. Ask for both.
- **A "precision" stage that changes admission.** A re-ranker that can add or drop candidates is doing
  recall's job, and the two can no longer be measured separately. Gate it to a band and prove the
  candidate set is unchanged.
- **A refusal written by the model.** If the writer is asked and told to say "I don't know", the
  refusal is a style. The engine must decide before the writer is called.
- **A provenance line the model can forge.** The source note has to be data stored with the message, not
  a sentence in the prompt.
- **A routing score read as confidence.** The turn that could not be answered can have the higher
  routing amplitude. Check it; do not present it as certainty.
- **A store count that includes what chat cannot reach.** Ask what is excluded and why. The number a
  visitor should see is the one the answer path can reach.
- **A "learns continuously" claim with no learning event.** Ask for the field, the write path, and the
  measurement. The honest version is continuous *ingestion*.

## Where to go next in the pack

`chat-declined.json` first — the refusal is the shortest complete proof. Then `chat-covered.json` beside
it. `grounding-ladder.ts` is the mechanism; `architecture-2026-10-18.md` is the map; `reproduce.md`
re-runs both turns. The status and topology captures are what you compare your own system's numbers to.