# Research: PoC Codebook Architecture Analysis (OQ-02)

**Date**: 2026-06-13
**Status**: Complete
**Question**: What is the minimum viable codebook? Can the 1,245-line PoC codebook be compressed, and what is essential vs. exploratory/dead code?

---

## 1. PoC Architecture Overview

### 1.1 File Structure & Role

The PoC codebook lives in `firewall_codebook.py` (1,245 lines) and depends on three metaspline core modules:

```
firewall_codebook.py (1,245 lines)
├── Imports from metaspline core:
│   ├── metaspline.spline.SplineDistribution    (spline.py, 378 lines)
│   ├── metaspline.spline.ensure_strictly_increasing (spline.py)
│   ├── metaspline.space.unfold / fold           (space.py, 46 lines)
│   └── metaspline.transform.simplex             (transform.py, 78 lines)
├── External imports:
│   ├── sklearn.linear_model.LogisticRegression
│   └── sklearn.mixture.GaussianMixture (imported but unused)
└── Internal definitions (see §1.2)
```

### 1.2 Major Sections of `firewall_codebook.py`

| Lines | Component | Description |
|-------|-----------|-------------|
| 1–50 | Module docstring + imports | Theory overview, imports |
| 53–75 | `reverse_bary3d()` | Simplex → barycentric (u,v) transform |
| 69–74 | `bary_to_simplex()` | Inverse: barycentric → simplex |
| 77–112 | `DirectionProfile` dataclass | Per-contrast statistical profile |
| 114–127 | `DirectionClassifier` dataclass | Per-contrast logistic regression weights |
| 129–146 | `HistogramClassifier` dataclass | 2×2×2 codebook-state histogram classifier |
| 148–165 | `DetectionResult` dataclass | Output of `detect()` |
| 167–596 | `FirewallCodebook.__init__` + `build()` | Codebook construction (429 lines!) |
| 598–629 | `FirewallCodebook.decompose()` | z → (sum, u, v) copula transform |
| 631–669 | `FirewallCodebook.classify()` | Per-contrast logistic classification |
| 671–729 | `FirewallCodebook.classify_histogram()` | 8-state histogram classification |
| 731–860 | `FirewallCodebook.detect()` | Main detection entry point |
| 862–884 | `FirewallCodebook.detect_from_perturbations()` | Convenience: P → z → detect |
| 886–945 | `FirewallCodebook.summary()` | Human-readable summary |
| 947–1041 | `FirewallCodebook.evaluate_auc()` | AUC evaluation on held-out data |
| 1044–1118 | `build_codebook_from_precomputed()` | Load from saved .pt files |
| 1121–1245 | `__main__` block | Script-mode evaluation + duplicated data loading |

### 1.3 Dependency Map

```
                     ┌──────────────────┐
                     │ FirewallCodebook  │
                     │  (main class)     │
                     └────────┬─────────┘
                              │
              ┌───────────────┼───────────────┐
              │               │               │
    ┌─────────▼──┐   ┌──────▼──────┐   ┌─────▼─────┐
    │ SplineDist  │   │ simplex()   │   │ bary3d()  │
    │ (CDF/ICDF) │   │ (transform) │   │ (local)   │
    └─────────┬──┘   └─────────────┘   └───────────┘
              │
    ┌─────────▼──────────────┐
    │ MonotonicCubicSpline   │
    │ (pchip interpolation)   │
    └────────────────────────┘
```

The `FirewallCodebook` has these hard dependencies at runtime:
1. **SplineDistribution** — CDF/ICDF transforms (population fitting + inference)
2. **simplex()** — normalize to simplex (x/sum(x))
3. **reverse_bary3d()** — project simplex to 2D barycentric coordinates
4. **torch** — tensor operations
5. **numpy** — sklearn bridge for training only

Training-time dependencies (not needed at inference):
- **sklearn.linear_model.LogisticRegression** — classifier training
- **sklearn.metrics.silhouette_score** — profile quality metric
- **sklearn.metrics.roc_auc_score** — evaluation metric

---

## 2. Essential vs. Exploratory vs. Dead Code Classification

### 2.1 Essential (Required for Production Codebook)

These are the core components that must be extracted into the production package:

| Component | Lines | Role | Production Mapping |
|-----------|-------|------|-------------------|
| `reverse_bary3d()` | 53–66 | z → (u,v) barycentric projection | `codebook/transforms.py` |
| `bary_to_simplex()` | 69–74 | Inverse barycentric (needed for reconstruction) | `codebook/transforms.py` |
| `SplineDistribution` | spline.py:200–261 | CDF/ICDF for copula transform | `codebook/splines.py` (adapted) |
| `MonotonicCubicSpline` | spline.py:80–197 | PCHIP interpolation engine | `codebook/splines.py` (adapted) |
| `ensure_strictly_increasing` | spline.py:43–73 | Knot sanitization | `codebook/splines.py` |
| `simplex()` | transform.py:34–36 | Normalize to unit simplex | `codebook/transforms.py` |
| `FirewallCodebook.__init__` | 182–203 | State initialization | `codebook/codebook.py` |
| `FirewallCodebook.decompose()` | 598–629 | z → (sum, u, v) copula space | `codebook/projection.py` |
| `FirewallCodebook.detect()` | 731–860 | Main detection logic | `codebook/detection.py` |
| `DetectionResult` | 148–165 | Output dataclass | `codebook/results.py` |
| `FirewallCodebook.build()` (core logic only) | 204–396 | SVD, spline fitting, profile computation | `training/compiler.py` |
| `DirectionProfile` | 77–112 | Per-direction statistical profile | `codebook/profiles.py` |
| `DirectionClassifier` | 114–127 | Per-direction linear classifier | `codebook/classifiers.py` |
| `FirewallCodebook.detect_from_perturbations()` | 862–884 | P → z convenience wrapper | `codebook/projection.py` |

**Total essential lines**: ~480 lines (including metaspline core)

### 2.2 Exploratory / Research Code

These were useful for research but are **not needed** in production:

| Component | Lines | Purpose | Disposition |
|-----------|-------|---------|-------------|
| `HistogramClassifier` dataclass | 129–146 | Alternative 2×2×2 discretized classifier | Keep as optional, not MVP |
| `classify_histogram()` | 671–729 | Histogram-based classification variant | Research variant, not MVP |
| `build()` histogram classifier section | 481–596 | Training histogram classifiers | Research variant |
| `evaluate_auc()` | 947–1041 | Offline AUC evaluation | Testing/benchmarking only |
| `summary()` | 886–945 | Human-readable codebook summary | Debugging/diagnostic tool |
| `classify()` | 631–669 | Per-position probability output | Subsumed by `detect()` |
| `build_codebook_from_precomputed()` | 1044–1118 | Load from .pt files | Training pipeline I/O |
| `build()` contrast_pairs default | 268–276 | Hardcoded 7-pair contrast list | Config, not code |
| `pooled_std()` inner function | 327–331 | Statistical utility | Extract to `training/stats.py` |
| `cohen_d()` inner function | 337–340 | Effect size utility | Extract to `training/stats.py` |
| `compute_silhouette()` inner function | 365–370 | Quality metric | Training diagnostic |

### 2.3 Dead Code

| Component | Lines | Issue |
|-----------|-------|-------|
| `sklearn.mixture.GaussianMixture` import | 44 | Imported but never used |
| `unfold()` / `fold()` from `space.py` | space.py:4–45 | Imported but never called in codebook |
| `dcs_norm()` from `transform.py` | transform.py:20–23 | Imported but never used |
| `__main__` block duplicated data loading | 1121–1245 | Lines 1203–1245 repeat 1126–1182 verbatim with different formatting — copy-paste artifact |
| `bary_to_simplex()` | 69–74 | Defined but never called in codebook |
| `DensitySpline` class | spline.py:315–378 | Legacy alternative, not used by codebook |
| `empirical_cdf()` / `empirical_density()` / `log_bins()` / `generate_asymmetric_knots()` | spline.py:268–313 | Utility functions not used by codebook |

### 2.4 Infrastructure (Training Pipeline, Not Runtime)

| Component | Lines | Purpose |
|-----------|-------|---------|
| `run_manifold_projection.py` (entire) | 823 | Model loading, data collection, SVD computation, saving artifacts |
| `analyzer.py` (entire) | 560 | Multi-layer direction analysis, residual extraction |
| `discover_directions.py` (entire) | 401 | Post-hoc direction discovery from trajectory data |
| `build()` SVD computation section | 229–233 | Population SVD → V3 basis |

---

## 3. Training Pipeline Analysis

### 3.1 `run_manifold_projection.py` — Step by Step

The training pipeline performs these operations:

1. **Model Loading** (L79–103): Load HuggingFace model + tokenizer. Configure for GPU/CPU.

2. **Condition Catalog Construction** (L106–153): Build contrastive prompt sets for 8 behavioral conditions:
   - self_ref / other_ref
   - violated / expected (semantic)
   - code_violated / code_expected
   - instruction / data
   - tool_call / natural_language
   - uncertain / confident
   - harmful / harmless
   - injection / benign_instruction

3. **Feature Extraction** (L156–213): For each condition, extract:
   - Hidden states across all layers → `residuals` (n_prompts, n_layers+1, hidden_dim)
   - ICDF perturbation vectors → `perturbations` (n_prompts, 64)
   - Last-layer hidden states → `hidden_last` (n_prompts, hidden_dim)

4. **SVD Computation** (L216–263):
   - Activation SVD: `H_all` (N, 2048) → principal components in hidden state space
   - Perturbation SVD: `P_all` (N, 64) → the **3D perturbation manifold** (this is the basis V3)

5. **Direction Vector Computation** (L393–434): Per-contrast mean-difference direction vectors at best layers.

6. **Projection Analysis** (L436–668): Extensive analysis of direction projections onto activation/perturbation subspaces. **This is research output, not needed for codebook compilation.**

7. **Save Results** (L670–755):
   - `.json`: Scalar metrics, SVD variance, separation stats
   - `.pt`: Tensors — **this is the key artifact**:
     - `perturbation_svd_Vh` → top-k right-singular vectors (the SVD basis)
     - `perturbation_mean` → population mean for centering
     - `condition_perturbations` → per-condition perturbation vectors
     - `condition_hidden_last` → last-layer hidden states per condition

### 3.2 Codebook Artifact Production

The `.pt` file from `run_manifold_projection.py` feeds directly into `build_codebook_from_precomputed()`, which:

1. Loads `.pt` file → extracts `perturbation_svd_Vh[:3]` (V3 basis) and `perturbation_mean` (P_mean)
2. Reconstructs z-coords: `z = (P - P_mean) @ V3.T`
3. Calls `FirewallCodebook.build()` which:
   - Fits SplineDistribution on each z dimension (population)
   - Fits SplineDistribution on sums (population)
   - Decomposes each condition via CDF → (sum, u, v)
   - Computes DirectionProfiles (pooled stats, Cohen's d, thresholds)
   - Trains DirectionClassifiers (logistic regression per contrast)
   - Trains HistogramClassifiers (8-state discrete classifiers)

**The produced codebook artifacts map to the production spec as:**

| PoC Artifact | Production Format | Notes |
|---|---|---|
| `FirewallCodebook.z_splines` (3× SplineDistribution) | `splines.json` (knot positions + coefficients) | Spline knots serialized as JSON arrays |
| `FirewallCodebook.svd_V3` (3×64 tensor) | `basis.safetensors` → `basis_vectors` | Reshaped for multi-layer format |
| `FirewallCodebook.population_mean_P` (64 tensor) | `basis.safetensors` → `mean` | Centering vector |
| `FirewallCodebook.direction_profiles` (dict) | `regions.safetensors` → centroids, scale | Per-direction statistical profiles |
| `FirewallCodebook.classifiers` (dict) | Part of `config.json` or `regions.safetensors` | Logistic weights (3 floats + intercept per direction) |
| `FirewallCodebook.sum_spline` (SplineDistribution) | `splines.json` | Sum distribution spline |
| `FirewallCodebook.population_stats` (dict) | `regions.safetensors` → centroids, scale | Population baselines |

---

## 4. Core Library Assessment

### 4.1 Metaspline Core Usage

The metaspline core (`spline.py` 378 lines, `transform.py` 78 lines, `space.py` 46 lines — 502 lines total) provides:

| Module | Lines | Used by Codebook | Lines Actually Used |
|--------|-------|-------------------|---------------------|
| `spline.py` | 378 | `SplineDistribution`, `ensure_strictly_increasing` | ~175 lines (SplineDistribution + MonotonicCubicSpline + ensure_strictly_increasing) |
| `transform.py` | 78 | `simplex()` only | 3 lines |
| `space.py` | 46 | None (imported but unused) | 0 lines |

**Actual dependency: ~178 lines out of 502.** The codebook uses only `SplineDistribution` (CDF/ICDF), `MonotonicCubicSpline` (its backbone), `ensure_strictly_increasing`, and `simplex()`. The following are unused:

- `DensitySpline` class (spline.py, 60 lines) — legacy CDF-based distribution, not used
- `empirical_cdf()`, `empirical_density()`, `log_bins()`, `generate_asymmetric_knots()` (spline.py, ~45 lines) — utility functions, unused
- `unfold()` / `fold()` (space.py, 46 lines) — digit expansion/contraction, unused
- `double_cumsum()`, `double_diff()`, `dcs_norm()`, `normalize_01()`, `column_cdf_normalize()`, `toBase()`, `numSymbols()`, `ndVec()` (transform.py, ~75 lines) — unused

### 4.2 How Much Is Inline vs. Library?

The `FirewallCodebook.build()` method has **significant inline reimplementation** of statistical operations that could be cleaner:

- **Lines 229–233**: SVD computation is inline (should use the pipeline's `compute_perturbation_svd()`)
- **Lines 236–246**: Spline fitting is inline but delegates to `SplineDistribution`
- **Lines 313–324**: CDF → decompose → barycentric is duplicated 3× (in `build()`, `classify()`, `classify_histogram()`)
- **Lines 327–340**: `pooled_std()` and `cohen_d()` are inner functions, not module-level
- **Lines 365–370**: `compute_silhouette()` is an inner function with sklearn import

The core decomposition pipeline (z → CDF → simplex → barycentric → (sum, u, v)) appears **verbatim** in:
1. `build()` lines 242–250 (population)
2. `build()` lines 313–324 (per-condition, profile computation)
3. `build()` lines 445–456 (per-condition, classifier computation)
4. `build()` lines 521–532 (per-condition, histogram computation)
5. `decompose()` lines 610–628 (runtime inference)

This is the **single most compressible pattern** — a 10-line decomposition sequence repeated 5 times.

---

## 5. Minimum Viable Codebook

### 5.1 Required Functions for Production

Based on the production spec (`codebook.md`), the minimum viable codebook needs:

1. **`project(activations) → z_coords`**: SVD projection (matrix multiply + centering)
2. **`decompose(z_coords) → (sum, u, v)`**: CDF → simplex → barycentric
3. **`score(z_coords) → list[DimensionSignal]`**: Per-direction scoring against profiles
4. **`detect(z_coords, threshold) → DetectionResult`**: Threshold comparison + flagging
5. **`load(path) → Codebook`**: Deserialize from safetensors + JSON
6. **SplineDistribution**: CDF evaluation for decompose

And for the **training pipeline** (not runtime):
7. **`build(population_data, direction_data) → Codebook`**: SVD, spline fitting, classifier training

### 5.2 Compression Estimate

| Source | Lines | Classification | Production Lines |
|--------|-------|----------------|------------------|
| `firewall_codebook.py` | 1,245 | Core + research + dead | ~350 |
| `spline.py` (used parts) | ~178 | Core library | ~180 |
| `transform.py` (used parts) | ~3 | Core library | ~5 |
| **Total PoC dependency** | **~426** | | **~535** |

**Target estimate: 400–500 lines for runtime codebook, 150–200 lines for training pipeline.**

Breakdown of production targets:

| Module | Target Lines | Contents |
|--------|-------------|----------|
| `codebook/transforms.py` | ~30 | `simplex()`, `reverse_bary3d()`, `bary_to_simplex()` |
| `codebook/splines.py` | ~180 | `MonotonicCubicSpline`, `SplineDistribution`, `ensure_strictly_increasing` |
| `codebook/profiles.py` | ~30 | `DirectionProfile` dataclass |
| `codebook/classifiers.py` | ~20 | `DirectionClassifier` dataclass |
| `codebook/results.py` | ~15 | `DetectionResult` dataclass |
| `codebook/projection.py` | ~30 | `project()` and `decompose()` |
| `codebook/detection.py` | ~50 | `detect()` with rolling window, threshold logic |
| `codebook/codebook.py` | ~40 | `Codebook` class (init, load, summary) |
| `training/compiler.py` | ~150 | `build()` — SVD, spline fitting, profile computation |
| `training/stats.py` | ~25 | `pooled_std()`, `cohen_d()`, silhouette |
| **Total** | **~570** | | |

This is **46% of the PoC's 1,245 lines**, or if including the used portion of metaspline core, **~35% of the total 1,745 lines** referenced in the overview.

### 5.3 What Gets Cut

| Lines Cut | Source | Reason |
|-----------|--------|--------|
| ~130 | `HistogramClassifier` + `classify_histogram()` + histogram training | Alternative approach, not MVP |
| ~95 | `evaluate_auc()` | Testing/benchmarking tool |
| ~60 | `summary()` | Debugging tool, not runtime |
| ~75 | `__main__` block (including duplicated code) | Script-mode evaluation |
| ~40 | `classify()` method | Subsumed by `detect()` |
| ~30 | `build_codebook_from_precomputed()` | Training I/O, not runtime |
| ~124 | Unused metaspline code (DensitySpline, unfold/fold, dcs_norm, etc.) | Dead code |
| ~50 | Repeated decomposition sequences | DRY refactoring |

---

## 6. Proposed Decomposition

Matching the production package structure from `codebook.md`:

```
src/alknet_firewall/
├── codebook/
│   ├── __init__.py            # Public exports
│   ├── codebook.py            # Codebook class (init, load, project, score)
│   ├── transforms.py          # simplex, reverse_bary3d, bary_to_simplex
│   ├── splines.py             # MonotonicCubicSpline, SplineDistribution
│   ├── profiles.py            # DirectionProfile, population stats
│   ├── classifiers.py          # DirectionClassifier (logistic weights)
│   ├── results.py             # DetectionResult, DimensionSignal, AlarmLevel
│   ├── projection.py          # project(), decompose()
│   └── detection.py           # detect(), threshold comparison, rolling window
├── training/
│   ├── __init__.py
│   ├── compiler.py            # build() — SVD, spline fitting, profile comp
│   ├── stats.py               # pooled_std, cohen_d, silhouette
│   └── data_loader.py         # Condition catalog, prompt sets, data loading
└── data/
    └── codebook/
        ├── basis.safetensors
        ├── regions.safetensors
        ├── splines.json
        └── config.json
```

### 6.1 Key Design Decisions for Extraction

1. **SplineDistribution stays in `codebook/splines.py`** — it's a general-purpose distribution class used at both training and inference time. No need for a separate package.

2. **`simplex()` moves to `codebook/transforms.py`** — it's a single pure function (3 lines), no need for the `transform.py` dependency chain.

3. **`unfold`/`fold` from `space.py` are dropped** — never used by the codebook.

4. **`DirectionProfile` and `DirectionClassifier` become separate dataclass modules** — clean separation of data from logic.

5. **`build()` moves entirely to `training/compiler.py`** — runtime codebook is read-only. This is the biggest architectural change: the codebook class should not have a `build()` classmethod.

6. **Decompose becomes a pure function** — `decompose(z, splines)` is a pure mathematical transform with no state dependencies beyond the splines. Making it a standalone function enables testing.

7. **Detection is separate from the codebook class** — `detect(z, classifiers, profiles, threshold)` is a stateless function given the codebook data. This enables swapping detection strategies without touching the codebook.

---

## 7. Testing Data

### 7.1 Saved Artifacts Referenced in Code

The PoC references these saved data files:

| File | Path | Contents | Reusable for Testing |
|------|------|----------|---------------------|
| Population precomputed | `saved_data/precomputed_seed42_qwen3_0.6b.pt` | z_coords, P_mean, perturbation_svd_Vh | Yes — basis for integration tests |
| Population precomputed | `saved_data/precomputed_seed42_qwen3_1.7b.pt` | Same for 1.7B model | Yes — multi-model test |
| Population precomputed | `saved_data/precomputed_seed42_qwen3_4b.pt` | Same for 4B model | Yes — multi-model test |
| Direction geometry | `experiments/direction_geometry/results/Qwen_Qwen3-0.6B_manifold_projection.pt` | Full condition data + SVD | Yes — golden data for codebook compilation |
| Direction geometry | `experiments/direction_geometry/results/Qwen_Qwen3-1.7B_manifold_projection.pt` | Same for 1.7B | Yes |
| Contrast pairs | Hardcoded in `build()` L268–276 and `run_manifold_projection.py` L139–148 | 7 behavioral contrasts | Yes — test fixture definition |

### 7.2 Validation Results Referenced

The `__main__` block (L1121–1245) contains:
- AUC evaluation at window sizes [1, 4, 8, 16]
- Per-direction AUC scores for both continuous and histogram classifiers
- Per-token AUC evaluation

These results should be captured as **golden test fixtures** for the production codebook:
- Build a codebook from the 0.6B precomputed data
- Verify that AUC scores match expected ranges
- Verify that detection decisions match expected flags

### 7.3 Calibration Data for Testing

For unit/integration tests, we need:

1. **Synthetic z-coord population**: Small N=1000 tensor for spline fitting tests
2. **Known-contrast z-coords**: Small pairs (harmful/harmless) for direction profile tests
3. **Expected spline parameters**: Known knot positions/coefficients for regression tests
4. **Expected detection results**: For a given input, what does `detect()` return?

The PoC's `build_codebook_from_precomputed()` provides a ready-made path to generate these fixtures from the saved `.pt` files.

---

## Summary

### Key Findings

1. **The 1,245-line PoC contains ~480 lines of essential code**. Including the metaspline core dependency (~178 lines used), the total essential code is ~658 lines. With dead code and research artifacts removed, the production codebook should target **400–500 lines** for runtime + **150–200 lines** for training.

2. **The decomposition pipeline (z → CDF → simplex → bary → (sum,u,v)) is repeated 5 times** in the PoC. Extracting it into a single `decompose()` function saves ~50 lines and eliminates a bug surface.

3. **The metaspline core has ~65% unused code** when viewed from the codebook's perspective. Only `SplineDistribution`, `MonotonicCubicSpline`, `ensure_strictly_increasing`, and `simplex()` are needed — the rest (DensitySpline, unfold/fold, dcs_norm, etc.) can be dropped entirely.

4. **The histogram classifier (2×2×2 discretized approach) is an exploratory alternative**, not the primary detection mechanism. The continuous logistic classifier is superior (higher AUC) and should be the MVP approach. The histogram classifier adds ~130 lines and can be deferred.

5. **The `build()` method is the largest single function (429 lines)** and mixes training with runtime state. It must be decomposed: training logic moves to `training/compiler.py`, runtime state becomes immutable serialized data.

6. **Saved `.pt` files from the PoC provide golden test data** — the manifold projection results for Qwen3-0.6B and 1.7B can be reused directly for integration tests.

### Recommendation

**Target: 500–600 lines total** for the production codebook (runtime + training), down from 1,245 lines in the PoC and 1,745 lines including metaspline core. This is a **~65% compression**.

The architecture should separate:
- **Runtime** (~400 lines): `Codebook`, transforms, splines, detection, results
- **Training** (~150 lines): compiler, stats, data loading
- **Data** (bundled): safetensors + JSON, no Python

### Next Steps

1. Create `src/alknet_firewall/codebook/` package structure
2. Extract `transforms.py` (simplex, barycentric) — trivial, ~30 lines
3. Port `splines.py` (MonotonicCubicSpline + SplineDistribution) — ~180 lines, mostly copy with cleanup
4. Implement `projection.py` (project, decompose) — thin wrappers, ~30 lines
5. Implement `detection.py` (detect with rolling window) — ~50 lines, port from PoC's detect()
6. Implement `codebook.py` (Codebook class with load) — ~40 lines
7. Extract `training/compiler.py` from `build()` — most complex extraction, ~150 lines
8. Create test fixtures from saved `.pt` data
9. Verify round-trip: build from .pt → serialize → load → detect matches PoC output