alkgen/docs/research/reader-poc-summary.md

status, last_updated

status	last_updated
complete	2026-11-06

alkgen: Reader POC Research Summary

Status: Research complete. The plan-driven handlebars codegen shape validated end-to-end: a generated, self-contained Rust reader (no use alktype) built at cargo build time from an alktype ReadPlan- derived context, byte-parity-exact against alktype's own packed materializer across a multi-buffer corpus, and ~1.9x faster than the interpreter at identical output shape. 7 parity tests pass. The remaining unknowns are spec-scope (Phase 1 ADRs), not feasibility. Date: 2026-09-08 Scope: Validates OQ-GN-08 #1 (the reader POC), resolving the OQ-GN-05/06/07 cluster (delegation boundary, dependency shape, validation strategy) with executable evidence.

---

Executive Summary

A standalone POC crate (/workspace/alkgen-reader-poc) validated the alkgen branch-1 mechanism end-to-end:

1. Plan-driven context projection works (OQ-GN-02 resolved by construction): the ReadPlan carries everything the generated code needs: dispatch kinds, effective endianness (already propagated per node), array counts and true strides, union discriminators with the integer dispatch table, and compile-time fixed_size for every struct node. The BAST document is needed only for descriptive metadata the plan doesn't carry (nothing in the POC scope).
2. Self-contained generated code works and is fast (OQ-GN-05/06 leaning (a) confirmed). The generated reader has zero alktype dependency, compiles clean, and runs 1.9x faster than materialize_packed with byte-parity on all parity fixtures.
3. Parity discipline, meaning three-way equality: generated reader vs alktype's materialize_packed vs a hand-rolled decoder, over a fixed corpus (typical buffer, max-value buffer, empty-strings buffer, binary-payload variant). All four fixtures agree exactly, including the interpreter's semantic quirks (see §Parity semantics).

Benchmark (criterion, stable across 3 runs)

Path	Time	Throughput
Hand-rolled raw decode (no Value building)	72 ns	1.67 GiB/s
Hand-rolled, same Value output shape	2.4–2.7 µs	~52 MiB/s
Generated reader	3.0–3.3 µs	~38 MiB/s
alktype interpreter (materialize_packed)	5.5 µs	~22 MiB/s

Typed output (contract (b), second render, same schema):

Path	Time
Hand-rolled raw decode (no allocs)	72–80 ns
Generated typed codec (types + codecs, zero deps)	291–322 ns

The typed generated codec is ~10x faster than the Value-shaped generated reader and ~15–20x faster than the interpreter, at ~4x the raw no-allocation baseline (the residual gap is String/Vec allocation for variable-length fields (label, tags, points, payload), which the raw baseline does not perform). All typed parity tests pass (struct equality incl. the TUnion enum, and clean GenError::OutOfBounds on short buffers).

Interpretation:

• Generated is ~1.9x faster than the interpreter at identical output shape; the compile-time dispatch win is real, though far from the ~20x hand-rolled number cited in Phase 0 discussion.
• The 20x gap is not dispatch overhead. Hand-rolled code that builds the same serde_json::Value output runs 2.4–2.7 µs, only ~10–20% faster than generated. The 20x-vs-raw number is the cost of Value/Map/String allocation (paid equally by any Value-materializing path), not the interpreter's per-node walk.
• Implication for alkgen's success metric (OQ-GN-03): "recovering most of the 20x" is only meaningful at equal output contract. For the current materialize-to-Value contract, generated code is already at ~90% of what hand-written code can do; the remaining interpreter-vs-generated gap (~1.9x) is the win codegen delivers. A future typed-struct output (no Value materialization at all) is where the 20x-scale gains live; that is an output-contract decision for Phase 1 (see §Follow-ups).

Typed output direction set 2026-09-08 (POC debrief): branch 1's output contract should be typed Rust structs, in the russh-sftp protocol-type shape. The decomposition above made the Value contract's ceiling visible; the next target is to generate the hand-rolled shape itself: helpers + templates producing direct wire codecs over typed structs, no Value anywhere. Conceptually the alkminer-v1 kernel-fusion move: the schema compiles to native code instead of driving a runtime walk.

Target shape (/workspace/russh-sftp/src/protocol/*.rs precedent): per-type plain structs with named fields (pub struct Close { pub id: u32, pub handle: String } + macro impls), field order = decode order, one shared cursor-based codec core. BAST kinds map to Rust types cleanly (uint32→u32, string→String, struct→struct, TUnion→enum). russh-sftp rides serde derive with a generic wire Deserializer; BAST's per-field endianness and TUnion byte- discriminators don't fit serde's derive directly, so the typed path forks:

• (a) serde-derive architecture (russh-sftp's): one generic Deserializer + per-type Deserialize impls, derived where the wire is homogeneous; generated impls (or endian newtype wrappers) where it isn't; TUnion → generated enum with a custom impl. Pro: ecosystem interop (serde consumers, de::from_bytes::<T> ergonomics). Con: serde machinery overhead vs direct code; the endian/union cases get wrapper-heavy.
• (b) generated direct codecs: fn decode(buf: &mut Cursor) -> Result<Packet, GenError> per type, literally the hand-rolled 72 ns shape, template-generated. Pro: zero overhead, full control; the POC's error-returning cursor helpers carry over directly. Con: serde interop for typed output needs a bridge (or a later feature generating impls).

Resolved 2026-09-08 (POC debrief): (b) generated direct codecs. Faster, smaller, and the wasm story is easiest: typed output with zero dependencies (no serde, no serde_json, no alktype in generated code; the cursor codec core is generated alongside). The serde-interop con is accepted: bridge impls (or a serde feature) can be added later if a consumer needs them; the inverse (ripping serde out of generated code) would be a breaking change, so starting dependency-free is the right one-way door.

Typed output VALIDATED 2026-09-08 (second render, same POC): the typed template set emits struct/enum definitions and direct decode codecs (read_packet(&[u8]) -> ReadResult<Packet>) with zero dependencies. Parity tests pass (typed equality incl. TUnion enum + clean errors). Benchmark: 291–322 ns, i.e. ~10x faster than the Value-shaped generated reader, ~15–20x faster than the interpreter, ~4x off the raw no-alloc baseline (residual gap = String/Vec allocation, inherent to the output shape). The (b) path is the branch-1 output contract; remaining Phase 1 work is template coverage (records, field-disc unions, offset-indirect, BE, writers) not feasibility.

Typed-builder finding F-6: names come from the BAST doc, not the plan. The plan eagerly resolves $refs and loses the def names (Point, Payload, Packet) that typed output needs for type identity. The typed context builder therefore walks the BAST typed tree (BastDoc, where BastType::Ref retains the name), while the plan remains the semantic source (sizes/strides/dispatch). This refines OQ-GN-02: plan for semantics, BAST doc for names. Also needed: a $defs-name-based type registry so shared defs map to shared types; numeric discriminator keys get V{key} Rust variant names (a digit is not an identifier).

Either way, the benchmark target is the raw-decode baseline (72 ns): typed generated decode should land within striking distance, recovering the ~20x claim at the typed contract. This is branch 1's output-contract ADR (Phase 1's biggest), now with a concrete target shape and two implementation strategies. A typed-output POC (follow-up #1) can resolve (a) vs (b) cheaply by rendering the same Packet schema both ways and re-benchmarking.

Parity semantics the generated code must reproduce

Discovered by exercising the interpreter, all pinned in the parity tests:

• Bytes materialize as Value::Array of u32s (not a binary representation); generated code must match (bytes_value helper).
• Byte-disc unions: discriminator stringified-integer lookup (the plan's int_keys gives compile-time dispatch), variant fields flattened into the union object under "__discriminator" (number), variant offset = discriminator end.
• Fixed-stride arrays: plan's element_stride (true stride for fixed elements, e.g. 4 for Point); variable-length elements get stride 0 → sequential walk.
• Empty datagrams/negative cases: short buffers must produce clean errors, not panics (see §Findings F-2).
• Cursor-uniform addressing: all fields (fixed and variable) read at the running cursor; constant cursor increments fold in LLVM, so source-level static offset expressions are unnecessary (and a first draft's base + k/cur split was buggy; see F-1).

Findings

F-1: Pure-cursor addressing beats source-level offset folding

First draft folded static offsets (base + k for fixed fields, cursor only after the first variable field). Two bugs: fields after variable fields mis-tracked, and nested-struct consumed-returns were wrong. Rewriting to uniform cursor addressing (cur + k for every field, cur += n always; nested fns start cur = base) made the generator simpler (no static/dynamic phase tracking in the context builder) and correctness obvious. LLVM folds the constant chains to the same machine code; the compile-time win comes from compiled dispatch, not source offset folding. Carry into alkgen: the template model should be pure-cursor; do not track static/dynamic phases in context builders.

F-2: Generated code must return errors, not panic

The interpreter returns AlkTypeError::Access on short buffers; first-draft generated code panicked on slice indexing (the range end index 35 out of range failure). Fixed by bounds-checking runtime helpers returning ReadResult<T> with ? propagation, so generated functions are Result-returning. Carry into alkgen: generated readers/writers must be error-returning end-to-end (alkcall accepts BAST from arbitrary peers; AGENTS.md convention 2 applies to generated code too). Open question for Phase 1: the error type shape (POC used String; alkgen likely wants a typed error mirroring AlkTypeError::Access's shape without the alktype dependency, or generated-code-local types).

F-3: Recursive context, flat render (partials + per-field dispatch template)

The natural shape is: context builder recursively emits one FnCtx per struct/variant (nested structs and union variants become their own functions), and the template renders a uniform function body: field statements in document order, each a { read; advance } statement from a small partial. Two template anti-patterns from first drafts, worth encoding as alkgen guidance:

• Threading offset expressions through template context (string-built offset_expr) is fragile; cursor-based field statements need no offset math in templates at all.
• Partials with dynamic {{> (lookup ...)}} dispatch don't fit; a single field-statement partial with {{#if}} arms on the field's kind flags is sufficient and strict-mode-safe.

F-4: The dev-tool integration shape (build.rs) works

The POC's build.rs = schema JSON + template + context builder → OUT_DIR/packet_reader.rs, include!-ed in tests, is exactly the OQ-GN-01 dev-tool model at crate-internal scale. It exercised: strict-mode rendering (caught a stale root.fields context key as a render error, strict mode paying off), template registration as data (typed errors, no unwraps in render path), and deterministic output. The generated file starts with the DO-NOT-EDIT banner.

F-5: Coverage scope and what the POC did not exercise

POC scope (validated): fixed primitives all widths + bool + f32/f64, length-prefixed strings/bytes, enum (u32), nested fixed struct, fixed-stride primitive array, fixed-stride struct-element array, variable-stride string array, byte-disc union with struct variants (int dispatch). NOT exercised (Phase 1 residue): field-disc unions (shared-field prefix walk), nested unions, records, offset-indirect encoding, maxLength reservations, field-level endian overrides (schema had none; the plan carries them but the template context only used the effective endian, which is correct), big-endian schemas, aligned mode (OffsetMap path; this POC was packed mode only). None of these are expected to break the shape; they add template arms.

POC structure (for reference)

/workspace/alkgen-reader-poc/
├── schema/packet.json        # sample BAST document (Packet + Point + Payload)
├── templates/reader.hbs      # runtime helpers + function bodies
├── templates/partial_field_stmt.hbs  # per-field statement dispatch
├── build.rs                  # plan compile → context build → render
├── src/wire.rs               # known-good hand encoder (test corpus)
├── tests/parity.rs           # 3-way parity fixtures (7 tests)
├── benches/three_way.rs      # 4-way benchmark (raw + value-shaped baselines)
└── include/packet_reader.rs  # pinned copy of generated output

The include/packet_reader.rs copy exists because build-script output lives in OUT_DIR; alkgen proper will emit into the consumer's source tree (dev-tool model) or via a build-script helper in alkgen; either way the include-file dance is a POC artifact, not a design constraint.

OQ resolution status after POC

• OQ-GN-05 (delegation boundary): resolved. Self-contained generated code, no runtime alktype calls; alktype appears only in the generator (build script). The (a) posture is executable and faster.
• OQ-GN-06 (dependency shape): resolved for branch 1. Generated output depends only on serde_json (the Value contract). Caveat: if branch 1 later wants typed-struct output (no Value), generated code would depend only on serde derive, or nothing. That is an output-contract decision (§Follow-ups).
• OQ-GN-07 (validation strategy): materially informed. The POC validates reads (bounds checks, UTF-8, union dispatch) as generated error-returning code. Full BAST validation (jsonschema-derived semantic checks per ValidationPlan) was not exercised; still open, but the drift-risk mitigation is now demonstrated for the read path.
• OQ-GN-02 (context source): plan-driven projection confirmed workable; the plan carried 100% of the semantic content in POC scope.

Follow-ups (Phase 1 input)

1. Output contract ADR: typed-struct output (russh-sftp shape, §"Typed output direction") vs Value-materializing. The typed direction is set; the ADR decides the serde-derive (a) vs direct codec (b) fork. A typed-output POC resolves it by rendering the Packet schema both ways and re-benchmarking against the 72 ns raw-decode baseline.
2. Typed error shape for generated code (F-2): mirror AlkTypeError variants without the alktype dep, or a generated-code-local enum.
3. Template asset organization: the partial-per-statement pattern (F-3) scales; enumerate the full arm set (records, offset-indirect, unions, BE) as a template inventory task.
4. Writer generation: the write path mirrors the reader shape (cursor-based, Result); likely a template sibling with no new context projection (the plan's write side needs a small survey of alktype's write paths).
5. Validation strategy (OQ-GN-07): decide whether generated validators re-express ValidationPlan checks or call into jsonschema.

Verification

• cargo test (POC crate): 7 passed (4 parity + 3 debug-intent tests pinned as executable documentation).
• cargo bench --bench three_way: stable numbers above.
• Generated code compiles warning-clean (allow-dead-code banner); parity tests run it against the interpreter's own output.

Re-validation 2026-11-06 (unpause): the POC crate was upgraded from alktype 0.3.0 to 0.4.0 (reservation read/write API, maxLength reserved encoding) along with a general dep refresh, and everything still holds: all 7 parity tests pass, and both typed-parity suites pass (typed equality incl. the TUnion enum, clean errors on short buffers). The three-way bench re-run matches the recorded numbers within noise: interpreter ~5.4 µs, generated Value-shaped ~3.0 µs (the recorded ~1.9x interpreter win holds), hand-rolled Value-shaped ~2.35 µs, generated typed ~312 ns. No API migrations were needed in the POC's plan-driven context builder.