# METATRON — Architecture

## Overview

METATRON is a **grounded retrieval engine over a validated dictionary**. Text is
validated chunk-by-chunk by an LLM rubric, embedded into 768-dimensional semantic
space (nomic-embed-text-v1.5) and written to a dictionary. A query is embedded the
same way and answered from the entries nearest to it, ordered by raw geodesic
similarity, with a decline gate that refuses to answer when nothing is close
enough. **It does not learn online** — nothing about the engine changes in
response to being queried.

> Rewritten 2026-09-06. Until then this document described "Kuramoto phase-locked
> oscillator dynamics for learning". That mechanism was audited over 2026-09-05/06
> and deleted: it could not lock an edge at any coupling within 10–20× its
> configured maximum, its state was erased within ~5 passes, and a frozen rotor
> produced retrieval indistinguishable from a live one. See
> [GEOMETRY-RESET-AUDIT-PLAN.md](GEOMETRY-RESET-AUDIT-PLAN.md) and
> [CONSTRAINTS-LEDGER.md](CONSTRAINTS-LEDGER.md).

## Core Principle

**Similarity is the ranking. Amplitude is a routing prior. Nothing accumulates.**

The forward pass computes 13 geodesic amplitudes against immutable anchors and
takes `winnerNode = argmax` — a label used for node affinity, density-based
spawning and interpretability. Retrieval itself is `finalScore := querySim`,
sorted descending. There is no learning law, no weight matrix, no phase state.

## 13 Node Labels — 6 Axes + a Residual

The 13 names below are **poles on six axes plus one residual direction, not 13
independent directions**. The generalist's anchors are built as differences
between scaffold centroids over `POLAR_PAIRS`, so `Anchor_b = −Anchor_a` exactly:
a query's amplitude on Changing is exactly `1 −` its amplitude on Knowing, and the
13×13 anchor Gram matrix has **rank 7**. Treat the labels as a routing and
interpretability scheme over a 7-dimensional prior — not as 13 separable "ways of
thinking".

**Node 12 = center (Existence). First ring (0-5). Second ring (6-11).**

| Index | Cognitive Mode | Antonym | Ring | Combines |
|-------|---------------|---------|------|----------|
| 0 | Knowing | Confusion | 1st | — |
| 1 | Feeling | Numbness | 1st | — |
| 2 | Reasoning | Chaos | 1st | — |
| 3 | Changing | Stasis | 1st | — |
| 4 | Sensing | Blindness | 1st | — |
| 5 | Connecting | Isolation | 1st | — |
| 6 | Judging | Amorality | 2nd | Knowing + Feeling |
| 7 | Creating | Sterility | 2nd | Changing + Sensing |
| 8 | Analyzing | Vagueness | 2nd | Reasoning + Knowing |
| 9 | Expressing | Muteness | 2nd | Feeling + Connecting |
| 10 | Discovering | Oblivion | 2nd | Sensing + Changing |
| 11 | Harmonizing | Discord | 2nd | Connecting + Reasoning |
| 12 | Existence (Center) | Void | Center | Ground of all modes |

### Anchor Derivation

**Generalist — difference vectors, NOT hand-crafted phrases.** `server/index.ts`
computes a centroid per label from the 520 scaffold entries, then for each of the
six `POLAR_PAIRS` sets `Anchor_a = normalize(centroid_a − centroid_b)` and
`Anchor_b = −Anchor_a`; N12 is the residual (existence centroid minus the mean of
the other twelve). This is "Approach B", chosen 2026-04-25 on evidence: raw
centroids gave an amplitude spread of 0.04, difference vectors 0.30–0.50, and a
raw-phrase construction made N12 a 100 % attractor. `COGNITIVE_MODE_PHRASES` is
the **non-default** `'raw-phrases'` mode and is not what the generalist uses. (The
13 mode words on their own are not separable in nomic space — mean pairwise
cosine 0.713; the separability comes entirely from the difference construction.)

**Specialists (depth 1)**: Inherit the parent's anchors. Routing is still
conceptual — the specialist differentiates via its filtered dictionary content.

**Specialists (depth 2+)**: `deriveSpecialistAnchors()` runs k-means over their
dictionary embeddings for domain-specific sub-anchors — the archetypal →
empirical gradient (contract law vs criminal law, not Reasoning vs Judging).
Graceful fallback to inherited anchors if cluster coherence or entry count is
insufficient.

### Initialization Protocol

3-layer boot sequence (`server/initialization.ts`):
1. **Identity**: **240** entries from `identity.json` (the "42" this doc carried for months was never true in this repo)
2. **Scaffold**: 520 entries from `generalist-scaffold.json` (40 per label) — the semantic gravity per label, AND the source of the anchors themselves. Measured routing accuracy: 469/520 = 90.2 % in-sample, 0 within-axis confusions (structurally impossible), ~10 % cross-axis.
3. **Baseline Snapshot**: Saved as `generalist-baseline` — the reset-to-baseline target

**Init validation gate — structural since 2026-09-06.** Six antipodal pairs at
cos ≤ −0.999, Gram rank == 7, average pairwise geodesic 1.69 ± 0.01 rad (also an
embedder-parity check: identical on x86 and aarch64), scaffold routing ≥ 85 %,
per-node cross-axis arrivals ≤ 15, and dict == identity + scaffold with 0
rejected. The previous gate read the Kuramoto order parameter, phase spread and
locked-pair ratio; it passed at init (R 0.066) and would have failed 2 of its 5
checks after a restart of the identical state (R 0.732).

### Post-Init: the supervisor (no training loop)

`server/training/continuous-trainer.ts` is a **supervisor**, not a learner. Three
states:

| State | What happens |
|---|---|
| **Executor has work** | The work-ledger executor (`server/work/`) is the ONLY dictionary writer: chunker → LLM validator → dictionary → specialist copy. The supervisor sweeps `./ingest` into the ledger, ticks the durable counter that counter-clocked jobs ride, and stays out of the way. |
| **Audit is due** | `server/training/retrieval-audit.ts` runs (wall-clocked, default 6 h): a passage-derived question per source document, then "is that document in the top-10?". Reports **hit@10 by document**, **grounded rate**, and **decline rate over ≥ 50 out-of-corpus probes**. readOnly throughout — it writes nothing — and is **never fed back**. |
| **Quiescent** | Executor idle and audit not due → announce rest, which hands the box to idle maintenance, Lance compaction and dreams. |

The supervisor holds **no engine-write lease in any state**. Measured baseline
before the rewrite: 13.94 min/h of engine-write on an idle box, 9.35 of it
per-tick bookkeeping containing no learning event.

**Deleted 2026-09-06, with receipts** (see the plan's §"Phase 5 — trainer after
the reset"): the Leitner spaced-retrieval ladder (box state carried no ranking
signal — AUC 0.46–0.51, chance, over 31k reviews with zero demotions), all four
assess cycles (one arm wrote 49 unvalidated LLM-authored rows into the pool), the
discrimination concern (0 fires in 24 h), self-reflection (the "test" was the
engine retrieving its own entry), the geometry scorer, the maturity signal, the
coherence target, and the R-based halts and cooldowns.

**Report card** (`server/training/training-grader.ts`): **two** sections —
Retrieval Audit (hit@10 / decline / grounded, read from the last run; N/A rather
than zero when none has happened) and Structural (the init checks re-measured
live). Composite renormalises over the sections that ran. Available via
`/metatron grade`.

**Auto-resume on restart**: server startup starts the supervisor if dict > 100.

API: `POST /api/continuous-training/start|stop`, `GET /api/continuous-training/status`,
`GET /api/training/grade`, `POST /api/admin/run-retrieval-audit`.

### Two-Tier Reset

- `POST /api/reset { mode: "full" }` — clean geometry, zero dictionary, clear specialists
- `POST /api/reset { mode: "baseline" }` — restore scaffold snapshot, clear specialists

Nodes 0-5 form the inner hexagon, 6-11 the outer hexagon, 12 is the center. Edge distances follow the sacred geometry of Metatron's Cube.

## Forward Pass Pipeline

```
Query
  |
  v
1. Embedding (nomic-embed-text-v1.5 768d — raw, NOT centered for routing)
  |
  v
2. Geodesic resonance against the 13 immutable anchors -> 13 amplitudes
  |       (centering off by default: nomic is already isotropic, and centering
  |        over-collapses the distribution)
  v
3. winnerNode = argmax(amplitudes)   <- a ROUTING LABEL, not a retrieval weight
  |
  v
4. Candidate set
  |       async path: Lance HNSW top-50 for this cube (RERANK_FETCH_K)
  |       sync path : the full in-RAM dictionary
  v
5. Score: finalScore := querySim (geodesic), sorted DESCENDING
  |       admission floor from getAdaptiveQuerySimFloor(), clamped [0.55, 0.65]
  v
6. Cross-cube blend: foreign matches merged on the SAME querySim (two-pointer)
  |
  v
7. Prompt window: querySim order + near-duplicate / per-parent diversity filter
  |
  v
8. Grounding gate: max ORGANIC querySim >= GROUNDING_DECLINE_THRESHOLD
  |       below it -> decline. Tools cannot rescue a decline.
  v
9. LLM voice (small or agentic harness) -> answer
  |
  v
10. Context buffer + winner tracking   (the ONLY per-query write)
```

**Removed from this pipeline on 2026-09-06.** Each one measurably LOWERED
known-entry hit@10 — the live configuration scored 0.750 where turning them off
scored 0.995–1.000. Receipts: `data/audit/forward/`, CONSTRAINTS-LEDGER
§Forward-pass audit.

| Removed stage | What it did | Verdict |
|---|---|---|
| Contrast sharpening | `amp ^ CONTRAST_SHARPENING_POWER` | exponent was 1.0 — inert |
| Ond/onde dual resonance | split-vector routing | removed earlier; full 768d throughout |
| Phase path walk | followed `K_ij · cos Δθ` to choose "walked" nodes | fed the partition below |
| Cascade weight | `0.4 + 0.6·(amp/maxAmp)` on walked nodes | 24–37 % of top-10 membership; harmful |
| Cross-node multiplier | **×0.3** on everything not walked | a hard partition; ×1.0 lifted hit@10 by +0.15 to +0.25 on every arm |
| Additive bonuses | affinity, confidence, type, lanceBoost, length, age | tiebreakers over dead calibration; three were byte-identical dead |
| Synth re-rank | `querySim × (1 + α·proximity)` | the live partition; α → 0 rescued 96 entries with 0 losses |
| Phase-displacement sim | 5-step forced Kuramoto → `phaseWinner` | computed every pass, never read |
| Coherence composite | 0.3 amp + 0.3 queryFit + 0.3 phaseDiscrim + 0.1 ctx | no control reader; now returns 0 |
| Density `capMap` | `dict.length / tierCap` | the caps are falsified |
| Phase handoff | `gatedKuramotoStep` / `scoreModulatedStep` | the phase engine is gone |

## Phase Engine — REMOVED (2026-09-06)

`server/engine/phase-engine.ts` and everything that read it are deleted. It ran
13 coupled oscillators with golden-ratio-spaced natural frequencies and stepped
them on every pass; the claim was that this constituted learning. Three
pre-registered probes falsified it:

- **Locking capacity** (offline simulation): the coupling cannot lock ANY edge at
  any strength within 10–20× the configured maximum. There is no regime in which
  the mechanism does what it was named for.
- **Persistence**: phase state is erased within roughly five passes by the
  natural-frequency rotation. Nothing it wrote survived to be read.
- **Ablation**: a frozen rotor, a static `K_ij`, resting phases and *random*
  phases all produced retrieval indistinguishable from the live engine — and the
  no-walk arm scored HIGHER (hit@10 0.995–1.000 vs 0.750).

What survives is a **static topology prior**: `K_ij = 1/(1+distance)` over
`METATRON_EDGES`, with exactly one reader — the dream walk's choice of next node.
No angles, no order parameter, no coupling updates.

`cube_snapshots.phase_state` remains in the schema, always NULL, so pre-cleanup
snapshots stay readable for archaeology.

## Dictionary Management

- **Capacity**: Hardware-tiered (5K-50K based on VRAM)
- **Response cap**: 50% of capacity
- **Dedup**: 0.85 centered resonance for facts, 0.75 centered for responses (response-vs-response uses centered to avoid personality-prefix anisotropy)
- **Tiered pruning**: meta contamination -> questions -> low-confidence -> monologue -> shortest
- **Adaptive floor**: querySim threshold dynamically computed from ring buffer (0.70-0.80)
- **Isotropic centering**: Mean vector subtraction corrects nomic-embed-text-v1.5 anisotropy

### Shutdown maintenance pass

Every clean shutdown ([server/index.ts](../server/index.ts) `shutdown()`)
runs three sweeps before the snapshot save:

1. **Garbage-can purge** — `detectGarbageCans()` ([server/multi-cube/garbage-can-detector.ts](../server/multi-cube/garbage-can-detector.ts)) finds specialists with malformed IDs (sentence fragments / OCR noise from imperfect spawn-pathway sanitization), archives their entries to `data/garbage-can-archive/<ISO>.jsonl`, propagate-deletes the same phrases from every surviving cube (`Dictionary.removeByPhrases`), then drops the cans via `cube-registry.removeCube()` — which cascades dict + retention + snapshots in one SQLite transaction AND clears memories from the LanceDB vector store via `vectorStore.deleteMemoriesForCube()`. Read-only audit endpoint: `POST /api/admin/garbage-can-audit`.
2. **Per-entry artifact purge** — `engine.purgeArtifactPatterns('likely')` on every cube using the regex catalog at [server/ingestion/artifact-patterns.ts](../server/ingestion/artifact-patterns.ts). Backstop for content the parser missed.
3. **Orphan-retention prune** — sweeps retention rows whose entry hash no longer matches a dict phrase.

The two cleanup mechanisms (clustering vs patterns) are complementary: the
garbage-can pass catches what doesn't fit a regex; the pattern pass catches
what doesn't cluster. Both share the same shutdown window so the snapshot
rolls forward already cleaned. Full discussion in
[docs/INGEST-ARTIFACT-CATALOG.md](INGEST-ARTIFACT-CATALOG.md).

## Calibrated Match Quality Bands

Empirically calibrated from document benchmark experiments (2026-04-02). The 70% nomic-embed-text-v1.5 semantic baseline means all discrimination happens in the remaining 30%.

| matchQuality | Band | Action | Notes |
|-------------|------|--------|-------|
| 0.000 | Empty | No dictionary entries exist | Only on fresh reset |
| < 0.55 | Unknown | Trigger auto-discovery (LLM researches topic) | Below centered anisotropy floor |
| 0.55 - 0.65 | Partial | LLM-assisted synthesis with dictionary context | Vaguely related content exists |
| 0.65 - 0.72 | Match | Template mode (dictionary retrieval, no LLM) | Genuine semantic alignment |
| > 0.72 | Strong | Template mode + confidence bump | Content well-established |

**rawQuerySim = matchQuality** — the scoring pipeline is transparent. No signal compression between raw centered cosine and reported match quality.

## Reporting System

**WebSocket push** (not polling):
- `metric_snapshot` — every 10th pass (every pass during anomaly)
- `anomaly_event` — resonance variance collapse, dictionary threshold, winner stagnation (the sync-onset and phase-death-spiral checks went with the phase engine)
- `training_log` — training events
- `training_tick` — supervisor state: tick, quiescent, executor counts, last audit
- `retrieval_audit` — hit@10 by document, grounded rate, decline rate
- `dictionary_event` — entry added/removed/pruned

(Geometric checkpoints were deleted 2026-09-06 — their verdict was computed from the R window and locked-pair counts.)

## LLM Providers

- **Ollama** (local): Fast training saturation, saves API costs
- **Grok API** (xAI): Teacher scoring, high-quality synthesis
- **Future**: Local geometry-trained LLM as "translator" of sparse geometric state into coherent speech

## Client Architecture

Three-column layout: Dashboard (left 360px) | Cube + Chat (center) | Reporting (right, toggleable)

### Data Flow
- **WebSocket** (`/ws`): Primary real-time channel. `use-websocket.ts` handles auto-reconnect with exponential backoff, routes messages by type (`metric_snapshot`, `training_tick`, `anomaly_event`, `training_log`, `dictionary_event`, `retrieval_audit`). The `phase_update` frame was removed 2026-09-06.
- **REST** (`/api/*`): Historical queries and health polling fallback. `use-metrics.ts` merges WS real-time with REST historical data.
- **Chat → Cube**: `ChatPanel.onResonanceUpdate` → `home.tsx` state → `CubeBackground` props (amplitudes drive node glow, winner drives color flash).

### Dashboard Panels (left sidebar, all WS-driven)
- `live-monitor.tsx` — Recharts sparkline: resonance (amber), coherence (indigo), density (teal), rolling 60-point window
- `node-heatmap.tsx` — 13-cell grid (5-3-5 layout), amplitude → intensity (the phase-angle → hue mapping went with the phase engine)
- `training-timeline.tsx` — Chronological WS training events with severity badges
- `dictionary-health.tsx` — Fact/response ratio bar, per-node amplitude distribution, contamination alert (>30%)
- `system-controls.tsx` — URL crawl, file import (.pdf/.txt/.md), dream/wake toggle, reset

### 3D Visualization
`cube-background.tsx` — Three.js via `@react-three/fiber`. 13 nodes (icosahedron shells + aura) connected by 78 edges. Node size scales with density, winner node glows white. Cascade sweep animates edges on forward pass. Specialist cubes orbit parent nodes. Post-processing: bloom, noise, vignette.

### Reporting Panel (right overlay, 3 tabs since 2026-09-06)
- **Metrics**: Historical match-quality/density/drift line charts + current state
- **Anomalies**: Severity-tagged event log (resonance variance collapse, dictionary thresholds). The sync-onset and phase-death-spiral checks went with the phase engine.
- **System**: Connection status, engine state, raw WS feed

(The **Phase** tab — order parameter, phase spread, locked/repelled pairs, per-node angles — is deleted.)

## Module Structure

```
server/
  engine/          Engine core (forward pass, dictionary) — no phase engine since 2026-09-06
    types.ts       DEFAULT_ARCHETYPAL_PAIRS (13 labels), METATRON_EDGES (static topology)
  embeddings/      nomic-embed-text-v1.5 semantic embedder, k-means anchor derivation (specialists only)
  synthesis/       Response generation (template/LLM/dictionary), query decomposer
  training/        supervisor (continuous-trainer), retrieval-audit, training-grader (2 sections), training-data-provider, training-pair-emitter
  reporting/       WebSocket broadcaster, anomaly detection, metrics collection
  storage/         SQLite + Drizzle ORM
  llm/             Ollama + Grok providers with task-based routing
  work/            Work ledger + single-flight step executor — the ONLY dictionary writer
  multi-cube/      Specialist spawning, routing, saturation detection, cube-initializer
  ingestion/       PDF/DOC/URL content ingestion pipeline
  datasets/        identity.json, generalist-scaffold.json (520 entries), domain datasets
  initialization.ts  3-layer boot protocol (identity → scaffold → baseline snapshot)
  plugins/         Future: external data feed plugin interface
  network/         Future: master/slave distributed architecture

client/src/
  pages/home.tsx           Three-column layout shell
  components/
    cube-background.tsx    Three.js 3D Metatron's Cube
    chat-panel.tsx         Chat interface with resonance badges
    reporting-panel.tsx    4-tab reporting overlay
    dashboard/             6 real-time monitoring panels
    ui/                    48 shadcn/Radix UI primitives
  hooks/
    use-websocket.ts       WS connection with auto-reconnect
    use-metrics.ts         Combined WS + REST metric aggregation
    use-mobile.tsx         Responsive breakpoint detection
    use-toast.ts           Toast notification system
    use-gpu-info.tsx       WebGPU capability detection
  lib/
    utils.ts               cn() class merge helper
    queryClient.ts         React Query client + API request helper
```