---
status: complete
last_updated: 2026-11-06
---

# alkgen: Reader POC Research Summary

**Status:** Research complete. The plan-driven handlebars codegen shape
validated end-to-end: a generated, self-contained Rust reader (no
`use alktype`) built at `cargo build` time from an alktype `ReadPlan`-
derived context, byte-parity-exact against alktype's own packed
materializer across a multi-buffer corpus, and ~1.9x faster than the
interpreter at identical output shape. 7 parity tests pass. The
remaining unknowns are spec-scope (Phase 1 ADRs), not feasibility.
**Date:** 2026-09-08
**Scope:** Validates OQ-GN-08 #1 (the reader POC), resolving the
OQ-GN-05/06/07 cluster (delegation boundary, dependency shape,
validation strategy) with executable evidence.

---

## Executive Summary

A standalone POC crate (`/workspace/alkgen-reader-poc`) validated the
alkgen branch-1 mechanism end-to-end:

1. **Plan-driven context projection works** (OQ-GN-02 resolved by
   construction): the `ReadPlan` carries everything the generated
   code needs: dispatch kinds, effective endianness (already
   propagated per node), array counts and true strides, union
   discriminators with the integer dispatch table, and compile-time
   `fixed_size` for every struct node. The BAST document is needed
   only for descriptive metadata the plan doesn't carry (nothing in
   the POC scope).
2. **Self-contained generated code works and is fast** (OQ-GN-05/06
   leaning (a) confirmed). The generated reader has zero alktype
   dependency, compiles clean, and runs 1.9x faster than
   `materialize_packed` with byte-parity on all parity fixtures.
3. **Parity discipline**, meaning three-way equality: generated reader vs
   alktype's `materialize_packed` vs a hand-rolled decoder, over a
   fixed corpus (typical buffer, max-value buffer, empty-strings
   buffer, binary-payload variant). All four fixtures agree exactly,
   including the interpreter's semantic quirks (see §Parity
   semantics).

### Benchmark (criterion, stable across 3 runs)

| Path | Time | Throughput |
|---|---|---|
| Hand-rolled raw decode (no Value building) | 72 ns | 1.67 GiB/s |
| Hand-rolled, **same `Value` output shape** | 2.4–2.7 µs | ~52 MiB/s |
| **Generated reader** | 3.0–3.3 µs | ~38 MiB/s |
| alktype interpreter (`materialize_packed`) | 5.5 µs | ~22 MiB/s |

**Typed output (contract (b), second render, same schema):**

| Path | Time |
|---|---|
| Hand-rolled raw decode (no allocs) | 72–80 ns |
| **Generated typed codec** (types + codecs, zero deps) | **291–322 ns** |

The typed generated codec is **~10x faster than the Value-shaped
generated reader** and **~15–20x faster than the interpreter**, at
~4x the raw no-allocation baseline (the residual gap is
`String`/`Vec` allocation for variable-length fields (label, tags,
points, payload), which the raw baseline does not perform). All
typed parity tests pass (struct equality incl. the TUnion enum, and
clean `GenError::OutOfBounds` on short buffers).

Interpretation:

- **Generated is ~1.9x faster than the interpreter** at identical
  output shape; the compile-time dispatch win is real, though far
  from the ~20x hand-rolled number cited in Phase 0 discussion.
- **The 20x gap is not dispatch overhead.** Hand-rolled code that
  builds the same `serde_json::Value` output runs 2.4–2.7 µs, only
  ~10–20% faster than generated. The 20x-vs-raw number is the cost of
  `Value`/`Map`/`String` allocation (paid equally by any
  Value-materializing path), not the interpreter's per-node walk.
- **Implication for alkgen's success metric (OQ-GN-03):** "recovering
  most of the 20x" is only meaningful *at equal output contract*. For
  the current materialize-to-`Value` contract, generated code is
  already at ~90% of what hand-written code can do; the remaining
  interpreter-vs-generated gap (~1.9x) is the win codegen delivers.
  A future typed-struct output (no `Value` materialization at all)
  is where the 20x-scale gains live; that is an output-contract
  decision for Phase 1 (see §Follow-ups).

**Typed output direction set 2026-09-08 (POC debrief): branch 1's
output contract should be typed Rust structs, in the russh-sftp
protocol-type shape.** The decomposition above made the `Value`
contract's ceiling visible; the next target is to generate the
hand-rolled shape itself: helpers + templates producing direct
wire codecs over typed structs, no `Value` anywhere. Conceptually
the alkminer-v1 kernel-fusion move: the schema compiles to native
code instead of driving a runtime walk.

Target shape (`/workspace/russh-sftp/src/protocol/*.rs` precedent):
per-type plain structs with named fields (`pub struct Close { pub
id: u32, pub handle: String }` + macro impls), field order = decode
order, one shared cursor-based codec core. BAST kinds map to Rust
types cleanly (`uint32`→`u32`, `string`→`String`, struct→struct,
TUnion→enum). russh-sftp rides serde derive with a generic wire
`Deserializer`; BAST's per-field endianness and TUnion byte-
discriminators don't fit serde's derive directly, so the typed path
forks:

- **(a) serde-derive architecture** (russh-sftp's): one generic
  `Deserializer` + per-type `Deserialize` impls, derived where the
  wire is homogeneous; generated impls (or endian newtype wrappers)
  where it isn't; TUnion → generated enum with a custom impl. Pro:
  ecosystem interop (`serde` consumers, `de::from_bytes::<T>`
  ergonomics). Con: serde machinery overhead vs direct code; the
  endian/union cases get wrapper-heavy.
- **(b) generated direct codecs**: `fn decode(buf: &mut Cursor)
  -> Result<Packet, GenError>` per type, literally the hand-rolled
  72 ns shape, template-generated. Pro: zero overhead, full
  control; the POC's error-returning cursor helpers carry over
  directly. Con: serde interop for typed output needs a bridge (or
  a later feature generating impls).

**Resolved 2026-09-08 (POC debrief): (b) generated direct codecs.**
Faster, smaller, and the wasm story is easiest: typed output with
zero dependencies (no serde, no serde_json, no alktype in generated
code; the cursor codec core is generated alongside). The
serde-interop con is accepted: bridge impls (or a serde feature) can
be added later if a consumer needs them; the inverse (ripping serde
out of generated code) would be a breaking change, so starting
dependency-free is the right one-way door.

**Typed output VALIDATED 2026-09-08 (second render, same POC):** the
typed template set emits struct/enum definitions *and* direct
`decode` codecs (`read_packet(&[u8]) -> ReadResult<Packet>`) with
zero dependencies. Parity tests pass (typed equality incl. TUnion
enum + clean errors). **Benchmark: 291–322 ns**, i.e. ~10x faster
than the Value-shaped generated reader, ~15–20x faster than the
interpreter, ~4x off the raw no-alloc baseline (residual gap =
String/Vec allocation, inherent to the output shape). The (b) path
is the branch-1 output contract; remaining Phase 1 work is template
coverage (records, field-disc unions, offset-indirect, BE, writers)
not feasibility.

**Typed-builder finding F-6: names come from the BAST doc, not the
plan.** The plan eagerly resolves `$ref`s and loses the def names
(`Point`, `Payload`, `Packet`) that typed output needs for type
identity. The typed context builder therefore walks the BAST typed
tree (`BastDoc`, where `BastType::Ref` retains the name), while the
plan remains the semantic source (sizes/strides/dispatch). This
refines OQ-GN-02: **plan for semantics, BAST doc for names.** Also
needed: a `$defs`-name-based type registry so shared defs map to
shared types; numeric discriminator keys get `V{key}` Rust variant
names (a digit is not an identifier).

Either way, the benchmark target is the raw-decode baseline (72 ns):
typed generated decode should land within striking distance,
recovering the ~20x claim at the typed contract. This is branch 1's
output-contract ADR (Phase 1's biggest), now with a concrete target
shape and two implementation strategies. A typed-output POC
(follow-up #1) can resolve (a) vs (b) cheaply by rendering the same
Packet schema both ways and re-benchmarking.

## Parity semantics the generated code must reproduce

Discovered by exercising the interpreter, all pinned in the parity
tests:

- **Bytes materialize as `Value::Array` of u32s** (not a binary
  representation); generated code must match (`bytes_value` helper).
- **Byte-disc unions**: discriminator stringified-integer lookup
  (the plan's `int_keys` gives compile-time dispatch), variant fields
  flattened into the union object under `"__discriminator"` (number),
  variant offset = discriminator end.
- **Fixed-stride arrays**: plan's `element_stride` (true stride for
  fixed elements, e.g. 4 for `Point`); variable-length elements get
  stride 0 → sequential walk.
- **Empty datagrams/negative cases**: short buffers must produce
  clean errors, not panics (see §Findings F-2).
- **Cursor-uniform addressing**: all fields (fixed and variable)
  read at the running cursor; constant cursor increments fold in
  LLVM, so source-level static offset expressions are unnecessary
  (and a first draft's `base + k`/`cur` split was buggy; see F-1).

## Findings

### F-1: Pure-cursor addressing beats source-level offset folding

First draft folded static offsets (`base + k` for fixed fields,
cursor only after the first variable field). Two bugs: fields after
variable fields mis-tracked, and nested-struct consumed-returns were
wrong. Rewriting to **uniform cursor addressing** (`cur + k` for
every field, `cur += n` always; nested fns start `cur = base`)
made the generator simpler (no static/dynamic phase tracking in the
context builder) and correctness obvious. LLVM folds the constant
chains to the same machine code; the compile-time win comes from
compiled dispatch, not source offset folding. **Carry into alkgen:
the template model should be pure-cursor; do not track static/dynamic
phases in context builders.**

### F-2: Generated code must return errors, not panic

The interpreter returns `AlkTypeError::Access` on short buffers;
first-draft generated code panicked on slice indexing (the
`range end index 35 out of range` failure). Fixed by bounds-checking
runtime helpers returning `ReadResult<T>` with `?` propagation, so
generated functions are `Result`-returning. **Carry into alkgen:
generated readers/writers must be error-returning end-to-end
(alkcall accepts BAST from arbitrary peers; AGENTS.md convention 2
applies to generated code too).** Open question for Phase 1: the
error type shape (POC used `String`; alkgen likely wants a typed
error mirroring `AlkTypeError::Access`'s shape without the alktype
dependency, or generated-code-local types).

### F-3: Recursive context, flat render (partials + per-field dispatch template)

The natural shape is: **context builder recursively emits one
`FnCtx` per struct/variant** (nested structs and union variants
become their own functions), and the template renders a uniform
function body: field statements in document order, each a
`{ read; advance }` statement from a small partial. Two template
anti-patterns from first drafts, worth encoding as alkgen guidance:

- Threading offset expressions through template context
  (string-built `offset_expr`) is fragile; cursor-based field
  statements need no offset math in templates at all.
- Partials with dynamic `{{> (lookup ...)}}` dispatch don't fit;
  a single field-statement partial with `{{#if}}` arms on the
  field's kind flags is sufficient and strict-mode-safe.

### F-4: The dev-tool integration shape (build.rs) works

The POC's `build.rs` = schema JSON + template + context builder →
`OUT_DIR/packet_reader.rs`, `include!`-ed in tests, is exactly the
OQ-GN-01 dev-tool model at crate-internal scale. It exercised:
strict-mode rendering (caught a stale `root.fields` context key as a
render error, strict mode paying off), template registration as
data (typed errors, no unwraps in render path), and deterministic
output. The generated file starts with the DO-NOT-EDIT banner.

### F-5: Coverage scope and what the POC did not exercise

POC scope (validated): fixed primitives all widths + bool + f32/f64,
length-prefixed strings/bytes, enum (u32), nested fixed struct,
fixed-stride primitive array, fixed-stride struct-element array,
variable-stride string array, byte-disc union with struct variants
(int dispatch). NOT exercised (Phase 1 residue): field-disc unions
(shared-field prefix walk), nested unions, records, offset-indirect
encoding, `maxLength` reservations, field-level endian overrides
(schema had none; the plan carries them but the template context
only used the effective endian, which is correct), big-endian
schemas, aligned mode (`OffsetMap` path; this POC was packed mode
only). None of these are expected to break the shape; they add
template arms.

## POC structure (for reference)

```
/workspace/alkgen-reader-poc/
├── schema/packet.json        # sample BAST document (Packet + Point + Payload)
├── templates/reader.hbs      # runtime helpers + function bodies
├── templates/partial_field_stmt.hbs  # per-field statement dispatch
├── build.rs                  # plan compile → context build → render
├── src/wire.rs               # known-good hand encoder (test corpus)
├── tests/parity.rs           # 3-way parity fixtures (7 tests)
├── benches/three_way.rs      # 4-way benchmark (raw + value-shaped baselines)
└── include/packet_reader.rs  # pinned copy of generated output
```

The `include/packet_reader.rs` copy exists because build-script
output lives in `OUT_DIR`; alkgen proper will emit into the
consumer's source tree (dev-tool model) or via a build-script helper
in alkgen; either way the include-file dance is a POC artifact, not
a design constraint.

## OQ resolution status after POC

- **OQ-GN-05 (delegation boundary)**: resolved. Self-contained
  generated code, no runtime alktype calls; alktype appears only in
  the generator (build script). The (a) posture is executable and
  faster.
- **OQ-GN-06 (dependency shape)**: resolved for branch 1. Generated
  output depends only on `serde_json` (the Value contract).
  **Caveat:** if branch 1 later wants typed-struct output (no Value),
  generated code would depend only on `serde` derive, or nothing.
  That is an output-contract decision (§Follow-ups).
- **OQ-GN-07 (validation strategy)**: materially informed. The POC
  validates reads (bounds checks, UTF-8, union dispatch) as generated
  error-returning code. Full BAST validation (jsonschema-derived
  semantic checks per `ValidationPlan`) was not exercised; still
  open, but the drift-risk mitigation is now demonstrated for the
  read path.
- **OQ-GN-02 (context source)**: plan-driven projection confirmed
  workable; the plan carried 100% of the semantic content in POC
  scope.

## Follow-ups (Phase 1 input)

1. **Output contract ADR**: typed-struct output (russh-sftp shape,
   §"Typed output direction") vs `Value`-materializing. The typed
   direction is set; the ADR decides the serde-derive (a) vs direct
   codec (b) fork. A typed-output POC resolves it by rendering the
   Packet schema both ways and re-benchmarking against the 72 ns
   raw-decode baseline.
2. **Typed error shape for generated code** (F-2): mirror
   `AlkTypeError` variants without the alktype dep, or a
   generated-code-local enum.
3. **Template asset organization**: the partial-per-statement
   pattern (F-3) scales; enumerate the full arm set (records,
   offset-indirect, unions, BE) as a template inventory task.
4. **Writer generation**: the write path mirrors the reader shape
   (cursor-based, `Result`); likely a template sibling with no new
   context projection (the plan's write side needs a small survey
   of alktype's write paths).
5. **Validation strategy** (OQ-GN-07): decide whether generated
   validators re-express `ValidationPlan` checks or call into
   jsonschema.

## Verification

- `cargo test` (POC crate): 7 passed (4 parity + 3 debug-intent
  tests pinned as executable documentation).
- `cargo bench --bench three_way`: stable numbers above.
- Generated code compiles warning-clean (allow-dead-code banner);
  parity tests run it against the interpreter's own output.

Re-validation 2026-11-06 (unpause): the POC crate was upgraded from
alktype 0.3.0 to 0.4.0 (reservation read/write API, maxLength reserved
encoding) along with a general dep refresh, and everything still holds:
all 7 parity tests pass, and both typed-parity suites pass (typed
equality incl. the TUnion enum, clean errors on short buffers). The
three-way bench re-run matches the recorded numbers within noise:
interpreter ~5.4 µs, generated Value-shaped ~3.0 µs (the recorded
~1.9x interpreter win holds), hand-rolled Value-shaped ~2.35 µs,
generated typed ~312 ns. No API
migrations were needed in the POC's plan-driven context builder.