alkdev/alktype
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Compare commits

base: alkdev/alktype:v0.2.0
alkdev/alktype:main alkdev/alktype:readplan-poc alkdev/alktype:bast-validator-poc
alkdev/alktype:v0.4.0 alkdev/alktype:v0.3.0 alkdev/alktype:v0.2.0 alkdev/alktype:v0.1.0
...
compare: alkdev/alktype:v0.3.0
alkdev/alktype:main alkdev/alktype:readplan-poc alkdev/alktype:bast-validator-poc
alkdev/alktype:v0.4.0 alkdev/alktype:v0.3.0 alkdev/alktype:v0.2.0 alkdev/alktype:v0.1.0
34 Commits
v0.2.0 ... v0.3.0
Author SHA1 Message Date
glm-5.3-flash 9803d3b768 Pre-publish review #008: gate int_keys on canonical keys, restore no-prealloc array rule 
Review #008 (docs/reviews/008-pre-publish-review.md) audits the two
post-#007 unreviewed commits (dea96f0 bench port, d4635d2 perf) before
the first crates.io publish of 0.3.0.

- F1: the int_keys integer dispatch accepted non-canonical mapping
  keys ("01", "+1" parse as u64 1) — the reader dispatched disc 1
  while the materializer, validation plan, and tunion rejected the
  same buffer. compile_int_keys now builds the table only when every
  key is canonical (v.to_string() == key); otherwise the string
  fallback applies (agreement restored, perf kept for canonical
  mappings). Tests: r8_non_canonical_mapping_key_disables_int_dispatch,
  r8_canonical_mapping_keys_keep_int_dispatch.
- F2: d4635d2 reintroduced Vec::with_capacity(count) on both array
  materializers. Bounded per array by MAX_ARRAY_ELEMENTS but nesting
  compounds: probe (counting allocator) measured ~477 MB simultaneous
  allocation from a ~1 KB schema + empty buffer (100-level stride-0
  chain, all legal under the caps). H1's layer-1 rule restored:
  Vec::new() + push. Bench unchanged (packet read 220µs vs 246µs
  baseline). Test: r8_deeply_nested_stride0_array_rejects_before_bulk_prealloc.
- N3a disposition (review #007's deferred item): BastField::synthetic
  (pub(crate), zero callers, #[allow(dead_code)]) deleted; the seven
  source() accessors are public API and stay (semver decision —
  removal needs an explicit ask); resolve_typeref_as_def's inline
  struct/union/enum arms probe-verified reachable (inline struct
  union variants are legal) — kept.
- CHANGELOG: 0.3.0 entry (compiled forms, breaking surface, hardening
  fixes, coverage). README: ReadPlan/ValidationPlan roles, union
  conventions, untrusted-schema bounds.

Verification: 569 tests green (491 lib + 17 + 34 + 15 + 12, + 2
ignored doctests), clippy -D warnings clean, cargo doc 0 warnings,
wasm32 build green, cargo publish --dry-run clean.
 2026-09-07 10:58:49 +00:00
glm-5.3-flash d4635d28f0 perf: fixed-size struct fast path, integer union dispatch, zero-alloc read_next_borrowed 
Targets the bench gaps from the 0.3.0 port review (commit dea96f0):
packet read was ~111-189x hand-rolled, chunk read ~18x.

- ReadPlan gains compile-time fixed_size (cached field-size sum).
  Fixed structs skip the cursor size walk entirely (one bounds check
  instead); fixed-size union variants skip the plan_walk_variant_size
  pre-pass, eliminating the double walk of variant bytes for the
  common SFTP-shaped case.
- CompositePlan::Union gains an int_keys dispatch table (pre-parsed
  u64 mapping keys); byte-discriminator unions dispatch on the raw
  integer instead of stringifying per read. Returned discriminator
  String unchanged (public API). String-keyed fallback preserved.
- Additive SequentialReader::read_next_borrowed returns the field
  name borrowed from the plan — zero allocs per field for hot loops.
  read_next stays the owned-name form (single source of truth).
- plan_walk_struct_size / union shared walk: per-field format! moved
  to the error path only.
- materialize: with_capacity for bytes arrays, arrays, and struct
  objects.

Benches (1024 chunks/iter, criterion, pre-review baseline vs now):
- read_packet_stream: 600 -> 246 µs (~2.4x; gap to hand 189x -> ~74x)
- read_chunk_stream: 104 -> 67 µs (~1.6x; 18x -> ~11x)
- write/validate groups unchanged (within noise)
- engine_compile +8% (int_keys table + fixed-size precompute), still
  one-shot

Verification: 566 tests pass, clippy -D warnings clean, wasm32 build
green. Bench baselines saved as pre-review/post-review.
 2026-09-03 17:28:20 +00:00
glm-5.3-flash dea96f0195 bench: port wire_vs_bast from alktty, add union + validate_bytes groups 
The wire_vs_bast bench originated in alktty as the uncommitted curiosity
probe that surfaced review #004's 400x read gap (the driver for the 0.3.0
compiled-forms release). It now lives here so alktype owns its perf
story; the alktty-only async roundtrip group (tokio ChunkReader/
ChunkWriter over a duplex pipe) was dropped — that measures alktty's I/O
stack, not this engine. The alktty copy is deleted.

Groups:
- read_chunk_stream / write_chunk_stream — the original ChunkHeader
  shape, byte-identical methodology, so numbers stay comparable with the
  historical series (400x → ~18x on read p64).
- read_packet_stream (new) — SFTP-shaped byte-discriminator union
  (Read/Write variants, Write carries a length-prefixed bytes field):
  exercises CompositePlan::Union dispatch + variant walks + variable
  reads, the case ADR-011's framing argument was about. The alktype
  consumer pattern follows the documented FieldValue::Union contract;
  a pre-measurement parity check locks the pattern (variant walk size
  + disc size == packet size) so the stream loop can't drift silently.
- validate_stream (new) — engine.validate_bytes per buffer (materialize
  + ValidationPlan walk), the read+validate-on-untrusted-stream shape
  alkcall cares about; closes the phase-7/8 bench deferral.
- one-shots — engine_compile, sequential_reader_new, layout_build.

criterion 0.7 dev-dep (default-features off). Benches don't affect the
wasm gate (bench targets never compile under wasm32-unknown-unknown).

Numbers (1024 chunks/iter, Xeon D-1521, shared box — ±10% noise):
- read p64: hand 5.7 µs / alktype 104.8 µs (~18x; parity with the
  phase-2/8 record of 98-99 ns/chunk)
- read p4k: hand 12.1 µs / alktype 107.5 µs
- write p64: hand 14.0 µs / alktype 37.5 µs; p4k: 324/362 µs
- packet read p64: hand 3.3 µs / alktype 622.6 µs (~189x — dominated by
  per-field String allocs + variant reader construction; the read_next
  (String, FieldValue) signature is pinned by the semver contract)
- validate: header 458 ns/chunk, packet p64 2.23 µs, packet p4k 47 µs
- one-shots: compile 615 µs (meta-schema dominated), reader_new 15.7 ns,
  layout_build 343 ns

Verification: cargo test --release (566 tests green), clippy
--all-targets -D warnings, wasm32-unknown-unknown build clean.
 2026-09-03 08:45:13 +00:00
glm-5.3-flash 557a0d791e Review #007: fix YAML frontmatter — unquoted colon in reviewer value broke parsing 2026-09-03 06:56:38 +00:00
glm-5.3-flash 120c05cd60 Review #007: expand brace shorthand in reviewed_artifacts frontmatter 2026-09-03 06:54:27 +00:00
glm-5.3-flash cb952c9bf3 Review #007: record post-fix coverage (91.66% lines, +0.99) 2026-09-03 06:48:07 +00:00
glm-5.3-flash 7e5e58aa1b Close review #007 coverage holes C1-C3, L1, L2 (locking tests) 
- C1: packed validate_bytes now exercised over all eleven primitive
  kinds (LE battery + BE subset + corrupted-bool rejection) — the plan
  materializer's i16..bool arms had zero public-path executions.
- C2: aligned validate_bytes over the default inline length-prefixed
  encoding (string + bytes; ADR-006 last-position rule honored).
- C3: ReadPlan::compile cycle rejection through a union mapping entry
  (compile_variant's own cycle arm — field-level cycles were already
  covered; this shape reaches the variant path). Arm confirmed
  executed in the post-fix coverage run.
- L1: builder.rs standard JSON-Schema conveniences locked with exact-
  JSON table tests, plus an end-to-end build_validator compile test.
- L2: tunion::read_field_discriminator's enum arm (both endians) —
  the last untested arm of the documented kind set (N1 parity).

docs/reviews/007-coverage-audit.md updated with per-finding
resolution blocks.

Verification: 488 lib + 78 integration tests green, clippy -D
warnings clean, wasm32-unknown-unknown build green.
 2026-09-03 06:47:34 +00:00
glm-5.3-flash 844c199fb8 Fix F1/F2 from review #007: zero-progress guard parity + maxLength cap 
- F1: materialize_plan_array (validate_bytes' packed path) now carries
  the zero-progress array guard the reader and legacy walker already
  had; validate_bytes no longer accepts an empty buffer against a
  stride-0 empty-struct-element array that SequentialReader rejects.
  Cross-consumer agreement test added (review #007 probe transcript).
- F2: MAX_LENGTH = 2^26 cap on the maxLength annotation — the N2
  dual-layer pattern (clean Schema parse error naming value+maximum,
  meta-schema "maximum": 67108864 so the published contract matches).
  Also closes the silent usize-overflow drop in parse_max_length.
- docs/reviews/007-coverage-audit.md records the full audit: per-file
  numbers, all classifications, and the N3a dead-surface list deferred
  to the pre-release review.

Verification: 477 lib + 78 integration tests green, clippy -D warnings
clean, wasm32-unknown-unknown build green.
 2026-09-03 06:39:12 +00:00
glm-5.3-flash bb28ba3006 Resolve N3: maxLength is string/bytes-only (review #006) 
- Parse gate in BastField::parse: maxLength on any kind other than
  string/bytes is a clean Schema error (records, arrays, inline
  structs, refs, union shared fields all covered; the choke point
  needs no ref-following since $defs entries are struct/union/enum)
- Meta-schema FieldDef: if kind in {string, bytes} else maxLength
  forbidden — the published alk.dev/bast/v1 contract matches the
  parser (N2 dual-layer pattern)
- M5's compute-side record maxLength arm became unreachable and was
  deleted (offset-indirect arm stays); the two superseded M5
  maxLength tests rewritten as the n3_* parse-rejection family
- ADR-006 remedy message tailored per kind: for records both
  annotated remedies are dead ends, so the error text points at the
  last-position fix only
- Docs aligned: bast-format.md (FieldDef meta-schema + FieldDef/
  Variable-Length Encoding prose), layout-engine.md (Strategy 2 +
  ADR-006 paragraph), schema-layer.md, ADR-003 §2/§3a amended,
  builder .max_length() doc
- Review #006: N3 resolved (all findings now closed), M5 update
  note, test-count bookkeeping note (in-session probes vs static
  counts), status lines flipped to fully resolved

Verified: 547 tests green + 2 ignored doctests in BOTH release and
default profiles (a stale debug artifact from an earlier session
masked one H3 roundtrip test in debug; clean rebuild passes both),
clippy -D warnings clean, cargo doc --no-deps zero warnings, wasm
build green.
 2026-09-03 04:30:54 +00:00
glm-5.3-flash 0857ea1c23 Review #006: record L4/N1/M4 closure; add M5/M6 (fixed) and N3 (open) 
- Resolution blocks on L4 (BTreeMap index + first-occurrence-wins
  pinned), N1 (tunion extended), M4 items 1-4 (aligned family, reader
  arms, data-access guards, dead-arm verdict with the structural note
  that the legacy packed walker serves only aligned fallbacks now).
- New findings: M5 (aligned record maxLength/offset-indirect silent
  corruption, resolved same-day), M6 (legacy walker field-disc union
  skipped shared fields, resolved same-day), N3 (packed record
  maxLength unenforced in validate_bytes, open - posture decision).
- Stats, resolution log, recommended order, and notes updated.
 2026-09-02 20:28:05 +00:00
glm-5.3-flash 2eb086f400 Fix M6, extend M4 coverage over aligned/legacy-walk/reader paths (review #006) 
- M6 (new finding, fixed): the legacy packed BAST-walker's field-disc
  union arm materialized only the discriminator field and started the
  variant immediately after it — silently reading the remaining shared
  fields' bytes as variant data whenever the union had any (probe:
  record<union> values produced {"handle": 5} where 5 was seq's value).
  The arm now walks all shared fields in order and starts the variant
  after the whole shared walk, matching H3's convention and the plan
  materializer's object shape (__discriminator + typed disc value +
  shared + variant). Reachable via aligned record/leaf paths only.
- M4 item 1: aligned-materializer test family — nested-struct
  recursion (3-level, previously 0 executions), maxLength trim in
  nested structs, invalid-UTF-8 Access error, offset-indirect
  out-of-bounds + data-after-sibling roundtrip, and records with
  struct/array/byte-disc-union/field-disc-union/wide-primitive values
  driving the legacy walker's previously-dead arms.
- M4 item 2: reader coverage — field-disc uint16/uint32/enum arms,
  byte-disc uint16/uint32 arms, nested-union variant size walk, and
  the public schema()/plan() accessors (all previously 0-execution).
- M4 item 3: data_access indirect-write tests at nonzero pair offset +
  data-region bounds refusal (the u32-truncation guards themselves
  need >4GiB slices and stay documented as defensively unreachable
  on 64-bit).
- M4 item 4 follow-through: OQ-001 rejection for struct elements and
  endian propagation for fixed elements locked with offset-map tests;
  the dead composite arms were removed in the previous commit.

Coverage after: materialize.rs 64.48→85.72% lines, TOTAL 89.59→90.60%.
Verified: 567 tests green (463 crate + 17 + 34 + 15 + 12 + 2 ignored),
clippy -D warnings clean, wasm build green, cargo doc zero warnings.
 2026-09-02 20:26:21 +00:00
glm-5.3-flash 5f9793f9c0 Resolve L4/N1, fix M5, close M4 item 4 (review #006) 
- L4: BTreeMap path->index for OffsetMap::get and PackedLayout::get;
  the linear scans behind the "random access" doc claim are gone.
  First-occurrence-wins preserved (BastStruct::parse doesn't reject
  duplicate names); locking tests in both modules.
- N1: tunion::read_field_discriminator now accepts uint16/uint32 disc
  fields (matching the reader's plan_discriminator_string_value set);
  one answer to "which field kinds can discriminate a union".
- M5 (new finding, fixed): aligned Record fields accepted maxLength /
  offset-indirect annotations, but the materializer always walks the
  inline count-prefixed form from the entry start — probe-verified
  silent corruption (record data crossed the reservation into the next
  field's bytes; validate_bytes accepted the corrupt buffer).
  Both annotations now rejected at compute with clean Offset errors.
  Parity-preserved from 0.2.0.
- M4 item 4: field_endian_for_element's Struct/Union arms and
  element_alignment were dead — the OQ-001 gate rejects every
  non-fixed-size element kind before either runs, so the phase-5
  "element's own endian is consulted" divergence never existed on any
  reachable path. Dead arms deleted; OQ-001 rejection for struct
  elements and endian propagation for fixed elements locked with tests.

Verified: 546 tests green (446 crate + 17 + 34 + 15 + 12 + 2 ignored),
clippy -D warnings clean, wasm build green.
 2026-09-02 20:07:43 +00:00
glm-5.3-flash 0bc5a541ac Resolve N2: bound align annotations at 4096 (review #006) 
align: 2^62 compiled and reported total_size = 2^63 — meaningless
layout output the consumer may act on, and the reachable path to the
MAX_ARRAY_BYTES cap used exactly this knob.

- MAX_ALIGN = 4096 (page granularity) in schema.rs, documented with the
  probe arithmetic
- parse_align returns Result and rejects over-cap values with a clean
  Schema error naming path/value/maximum — a silent clamp was rejected
  (it would change layout semantics without telling the consumer);
  both call sites thread the path, so standalone BastDoc::new (which
  never runs the meta-schema) is covered
- Meta-schema: "maximum": 4096 on StructDef.align and FieldDef.align —
  the published alk.dev/bast/v1/schema contract now matches the parser
- H1's byte-cap test retuned to align 4096 x count 2^16 = 2^28 > 2^26
  (the byte cap stays reachable under the new align cap)

Tests: 3 new (struct align above cap, field align above cap, align at
cap accepted). 511 tests green, clippy -D warnings clean, wasm32 build
green, cargo doc zero warnings.
 2026-09-02 19:46:51 +00:00
glm-5.3-flash 8739d29550 Resolve L2/L3: real schema threaded through ReadPlan compile; dead locals dropped (review #006) 
L2: compile() builds Arc<Value> once and threads &Arc<Value> down the
compile walk; every real ReadPlan carries the document at construction
— the Value::Null-placeholder-then-map-overwrite dance is gone. The one
remaining Null in wrap_leaf is documented as correct-by-construction
(anonymous synthetic wrapper, never escapes).

L3: materialize_plan_field drops its plan parameter (taken solely to
discard) and the field-disc union arm's disc_field/let _ pair is
deleted — the order-walk + by-name capture is the materializer's
correct design, as the finding's parity note described.

508 tests green, clippy -D warnings clean, wasm32 build green,
cargo doc zero warnings.
 2026-09-02 19:43:51 +00:00
glm-5.3-flash dcfe9d16ff Fix M1/M2/M3: ADR-006 record gap + aligned record coverage + dead Struct arm (review #006) 
M1: field_variable_kind (offset_map) now matches Record — a non-final
inline length-prefixed record field in aligned mode hits the ADR-006
rejection instead of computing silently corrupt offsets (probe-verified
clobber in the review: counts prefix at 0, id at 4).

M2: aligned record path locked with public-path tests —
materialize_aligned roundtrip (record<uint16>, wire arithmetic
asserted) and engine validate_bytes roundtrip + corrupted-buffer
rejection (record<uint32>). Record-as-last-field is the only safe
inline position post-M1.

M3: read_field's unreachable Struct arm (no struct-path entry ever
exists in an OffsetMap) replaced with a documented defensive Offset
error; doc comment states struct paths have no entry and the Offset
miss is the reachable composite failure. FieldValue::Struct (public
API, constructed by the packed reader) untouched.

Tests: 7 new (2 offset_map, 2 materialize, 1 engine M3 lock, plus the
roundtrip pair). 508 tests green, clippy -D warnings clean, wasm32
build green, cargo doc zero warnings.
 2026-09-02 19:40:09 +00:00
glm-5.3-flash 5e74b991ac Fix H2: shared reference-graph guard for standalone walkers (review #006) 
Cyclic or over-deep $ref graphs stack-overflowed the three standalone
schema walkers (OffsetMap::compute, LayoutBuilder::new,
materialize_aligned) — SIGABRT on probe, parity-preserved from 0.2.0.

- New src/walk_guard.rs: check_ref_graph() — one bounded walk over the
  reachable reference graph (depth cap 128 matching the plan compilers,
  path-scoped cycle set; diamonds allowed, cycles and 201-def chains
  rejected with the plan compilers' error wording)
- All three walkers run the guard at entry, before any recursion;
  materialize_aligned's is defense-in-depth (a cyclic doc can no longer
  produce an OffsetMap, but mismatched doc/map inputs must still fail
  cleanly)
- Behavioral side effect, net-positive: the guard eagerly parses every
  reachable def, so an invalid non-root def now surfaces at
  LayoutBuilder::new instead of build() — four H3 tests updated to
  expect the same Schema error earlier
- Test family: 12 new tests (walk_guard, offset_map, layout_builder,
  materialize) covering self/two-def/composite-carrier cycles, deep
  chains, and diamond non-rejection; no stack-overflow reproducers
  in-tree per the review's Methodology warning
- Stale "walkers have no cycle guard" statements updated in
  validation.md, 030 plan, ADR-012, and the engine gate comment

Verified: 501 tests green (423 + 17 + 34 + 15 + 12 + 2 ignored),
clippy -D warnings clean, wasm32 build green, cargo doc zero warnings.
 2026-09-02 19:34:24 +00:00
glm-5.3-flash 05a2a42983 Fix H3: field-disc union wire convention — shared-then-variant (review #006) 
Decision (recorded as an ADR-011 addendum): the packed-mode wire layout
for a field-name-discriminator TUnion is shared-then-variant — the
union's declared `fields` (disc + shared fields) first, then the
variant's own fields. Reader and materializer already implemented this;
LayoutBuilder was corrected from variant-only layout.

Enforcement in BastUnion::parse (the choke point every consumer
inherits — union roots at BastDoc::new, referenced unions at
resolve_ref):
- discriminator field must be declared in `fields`
- `fields` must not contain duplicate names
- variants must not re-declare shared fields (checked inline and
  through $ref resolution — parse chain now threads the doc root)
- the discriminator field must be the FIRST entry in `fields` (the
  reader reads the disc at the union start; a later position made it
  dispatch on the wrong bytes — H3 item 2, probe-verified)

Schemas relying on the old variant-only builder convention (variants
re-declaring shared fields) are rejected with a clean Schema error
naming the convention. Breaking for 0.2.0-era re-declaring schemas;
announced with 0.3.x.

- L5: FieldValue::Union::variant_start doc now states per-kind
  semantics (byte-disc: union_start + disc.offset + disc.size;
  field-disc: after the shared walk).
- L6: roundtrip test added (poc_roundtrip.rs) — LayoutBuilder write →
  SequentialReader read → materialize_packed → validate_bytes over a
  field-disc union with a second shared field and non-redeclaring
  variant; pins event.type@0/seq@1/handle@5, total 10.
- ADR-011: Status-block addendum recording the convention decision,
  the no-re-declare rule, and the breaking-constraint note.
- Review #006 updated: H3/L5/L6 resolution blocks, resolution log,
  recommended order.

Verified: 488 tests green (410+17+34+15+12, 2 pre-existing ignored),
clippy -D warnings clean, wasm32 build green, cargo doc zero warnings.
 2026-09-02 18:37:51 +00:00
glm-5.3-flash 2d166f567b Fix H1: bound untrusted array counts; resolve L1 (review #006) 
- Replace the three Vec::with_capacity(count) sites in materialize.rs
  with Vec::new() — validate_bytes on an adversarial count no longer
  OOM-aborts the process (AGENTS.md §3).
- New compile-time caps in schema.rs: MAX_ARRAY_ELEMENTS (2^16,
  enforced at BastArray::parse — the choke point every consumer
  inherits, bounds the walkers' per-element entry loops) and
  MAX_ARRAY_BYTES (2^26, enforced per walker against the mode-specific
  stride: compile_array, walk_array, compute_array_field).
- L1: fixed_composite_size/fixed_plan_size now return
  Result<Option<usize>>; unwrap_or_default() gone, overflow is a clean
  Schema error instead of silent stride-0.
- Zero-progress guard: stride-0 arrays whose elements consume 0 bytes
  (legal empty-struct elements) now error in plan_walk_variable_array_
  size and materialize_array_packed instead of looping count times.
- Tests: 8 new (parse/build/compile rejections, cap boundary,
  short-buffer clean error) + array_count_large_u64_parses_on_64bit
  rewritten to assert the new cap rejection. In-tree tests assert only
  the safe (compile-time) half per review #006's Methodology warning.
- Review #006 updated: H1/L1 resolution blocks, new finding N2
  (unbounded align annotations, found while re-deriving the cap
  arithmetic), resolution log, recommended order.

Verified: 482 tests green (405+17+34+14+12, 2 pre-existing ignored),
clippy -D warnings clean, wasm32-unknown-unknown build green.
 2026-09-02 16:46:46 +00:00
glm-5.3-flash 27be01af93 Add review #006: 0.3.0 post-implementation review 
Audits the shipped 0.3.0 code (commits ff85258..9949f91) for
correctness, untrusted-input discipline, 0.2.0 parity, code smells,
and coverage. Gates re-run green in-session (474 tests, clippy -D
warnings, doc, wasm32, publish --dry-run); coverage measured with
cargo-llvm-cov (89.59% lines / 84.80% fn).

Findings:
- H1: huge declared array count OOM-aborts validate_bytes (three
  Vec::with_capacity(count) sites; release-blocking)
- H2: cyclic $ref stack-overflows OffsetMap::compute /
  LayoutBuilder::new / materialize_aligned when driven standalone
  (engine gated, public walkers not; parity-preserved)
- H3: field-disc unions — builder (variant-only layout), reader
  (disc at union start), and materializer (position-correct shared
  walk) disagree on layout and discriminator position; needs a
  convention decision
- M1: ADR-006 check misses non-final inline Record fields (probe-
  confirmed)
- M2: aligned-mode record fields untested end-to-end
- M3: read_field's aligned Struct arm is unreachable dead code
- M4: coverage weak spots (materialize.rs 64.5% lines; aligned
  nested-struct recursion 0 executions through any test)
- L1-L6, N1: stride unwrap_or_default conflation, Null-schema
  placeholder, dead locals, linear-scan get, variant_start doc gap,
  missing field-disc roundtrip test, tunion/reader disc-kind split

Includes an operational warning: the H1/H2 reproducers OOM/stack-
abort the test harness — reproduce in an isolated process only.

Verification: file-only change, no code touched; suite green before
commit.
 2026-09-02 15:06:26 +00:00
glm-5.3-flash 9949f914df Release v0.3.0: compiled forms — ReadPlan, owned BastDoc, LeafMeta, ValidationPlan, fingerprinting 
Public API bump 0.2.0 -> 0.3.0 (the 030-compiled-forms plan is now
fully implemented; all eight phases landed).

- Cargo.toml: version 0.3.0. lib.rs re-exports complete (ReadPlan +
  sub-types, LeafMeta, OffsetEntry, ValidationPlan + sub-types).
- ADR-007 "Cost" rewritten to the Arc<ReadPlan> cost (15.7 ns) with
  the 0.2.0 "re-parse on demand" framing as a historical note
  (review #004 L2, the last loose end from that review).
- ADR-011/012 status blocks flipped to implemented; architecture
  README ADR table rows updated; layout-engine.md rewritten for the
  0.3.0 surface (engine-factory reader construction, OffsetMap
  OffsetEntry/LeafMeta/fingerprint section, owned BastDoc compute
  signature); SequentialReader module doc points at the engine
  factory. Reviews #004 and #005 flipped to closed.
- Bench re-run (alktty wire_vs_bast, 0.3.0 tree): read p64 98
  ns/chunk (parity with phase 2; hand-rolled 5.7 us/stream),
  layout_build 180 ns (was ~1.2 us — the phase-4 owned-doc cache
  removed the per-build re-parse, ~7x), sequential_reader_new 15.7
  ns, write p64 -3%, engine_compile unchanged (meta-schema
  validation dominates). No dedicated validate_bytes-stream bench:
  the phase-7 spot check (~0.2 us plan-validate vs ~0.6 us
  compile-per-call) stands; a dedicated bench is a follow-up if
  alkcall profiling motivates it.
- Downstream: alktty compiles against the path dep unchanged; alkcall
  has no dependency yet.

Verification (full block, all green): 474 tests; clippy -D warnings
clean; cargo doc zero warnings; wasm32 release build green; cargo
publish --dry-run clean at 0.3.0.
 2026-09-02 09:24:30 +00:00
glm-5.3-flash 537a2170fb Fingerprint ReadPlan/OffsetMap: Hash + Eq + fingerprint() (ADR-012 §1/§4, plan phase 6) 
- #[derive(Hash, Eq)] on ReadPlan, FieldPlan, CompositePlan, ReadKind,
  DiscriminatorPlan (schema: Arc<Value> hashes via serde_json Value
  Hash + Eq under preserve_order), and on OffsetMap (+ Clone;
  LeafMeta/OffsetEntry/ByteRange payload already Hash from phase 5 /
  this phase).
- fingerprint() -> u64 on both via std DefaultHasher (deferred
  decision 3 resolved: no new dep, not hot, cross-version stability a
  non-goal per ADR-012).
- Contract tests both sides: equal schemas -> equal PartialEq +
  fingerprint; field-kind / field-order / endianness changes each
  break equality and fingerprint; different root names over the same
  document fingerprint differently (ReadPlan).
- ValidationPlan already carries its own Hash/Eq/fingerprint +
  contract test (phase 7 landed early).

Verification: 474 tests pass (9 new fingerprint contract tests);
clippy -D warnings clean; cargo doc zero warnings; wasm32 release
build green.
 2026-09-02 09:16:11 +00:00
glm-5.3-flash 255c8c493e OffsetMap carries LeafMeta; read/write_field dispatch on it (ADR-012 §2b, plan phase 5) 
Prerequisite (review #005 M2): Hash added to Endian/VariableEncoding
derives (additive; fieldless Eq enums), and to ByteRange.

- New public types LeafMeta { kind, encoding, endian } (Copy + Eq +
  Hash) and OffsetEntry { range, meta } (start()/end() accessors),
  re-exported from lib.rs. Deferred decision 2 resolved: struct —
  get(path) -> Option<&OffsetEntry>, iter() -> (&str, &OffsetEntry).
  Storage: Vec<(String, OffsetEntry)>.
- LeafMeta computed at compute time with effective endian threaded
  through the aligned walk (container default -> field override,
  propagated into nested-struct probes and array elements via the
  referring field, matching the aligned materializer).
- engine read_field/write_field dispatch on the entry's LeafMeta:
  the per-access BastDoc re-parse + lookup_leaf_field walk +
  LeafFieldInfo are gone — the last two review #004 M1 sites.
- Parity note: lookup_leaf_field computed nested-struct defaults from
  the nested struct's own endian annotation; the map now agrees with
  the aligned materializer and packed ReadPlan (referring-field
  propagation). The old divergence (nested struct declaring endian
  under a field that also declares one) is closed; no test pinned it.
- Behavior change: read_field on a map-absent path (whole-struct
  field) errors Offset ("field not found") instead of Access
  ("composite types"); the composite-path test accepted either.
- materialize_aligned's four offset_map.get call sites updated to
  .range.start. alktty/alkcall untouched (bench never uses
  OffsetMap::get; alkcall has no dependency yet).

Verification: 465 tests pass (offset_map tests updated to the
OffsetEntry shape with per-kind LeafMeta expectations; engine test
for the old lookup walk rewritten to assert map entries carry the
LeafMeta); clippy -D warnings clean; cargo doc zero warnings; wasm32
release build green.
 2026-09-02 09:13:48 +00:00
glm-5.3-flash b7c7dbe2a1 LayoutBuilder caches the owned BastDoc (ADR-012 §2a, plan phase 4) 
The builder stores doc: BastDoc + endian (the doc_value: Value +
root_name: String cache is gone); new parses the typed tree once and
build walks &self.doc — the per-build BastDoc::new re-parse
(layout_builder.rs M1) is retired.

- build's root-is-struct re-check replaces its unreachable!() with a
  clean Schema error (AGENTS.md §3 never-panic; invariant unchanged —
  new already rejects non-struct roots).
- Boxing fallout: the builder now holds the full owned tree, so
  Layout::Packed boxes it (Box<LayoutBuilder>) to keep the engine's
  Layout enum variant sizes balanced (clippy large_enum_variant).
  layout_builder() still returns Option<&LayoutBuilder> via
  auto-deref; public API unchanged.

Verification: 465 tests pass unchanged (layout_builder.rs suites
drive new/build through the public API); clippy -D warnings clean;
wasm32 release build green.
 2026-09-02 08:59:13 +00:00
glm-5.3-flash c583762352 Make BastDoc owned: drop Bast* lifetimes (ADR-012 §2a, plan phase 3) 
Every Bast* type drops <'a>: &'a str -> String, &'a Value -> Value
(deferred decision 1: plain String/Value — the tree is built once;
Arc<str> name-sharing needs a bench justification that doesn't exist).
BastDoc::new(&Value, &str) still takes references in and clones into
owned storage; the doc gains Clone. resolve_ref/resolve_typeref/
resolve_typeref_as_def return owned types.

- Engine ownership flip: AlkTypeEngine holds the owned BastDoc
  (replacing bast_doc: Value + root_name: String; root_name()
  delegates to the doc), killing its three per-call BastDoc::new
  re-parses (aligned validate_bytes, read_field, write_field — the
  review #004 M1 pattern removed by construction; phase 5 retires the
  lookup_leaf_field walk itself). New public accessor root_name()
  (additive). Engine Send + Sync with the owned doc, asserted in the
  existing thread-share test.
- Bonus cleanup: materialize_typeref_packed's dead _field param
  dropped (phase 2 left it dangling). Under ownership, keeping it
  would force a deep Value clone per array element / record value /
  union variant via dummy_field_for. The param, dummy_field_for, and
  ty_source are gone; no behavior change (the arg was already
  ignored). BastField::synthetic keeps an owned-signature
  #[allow(dead_code)] definition (no remaining callers today).
- Consumers adapted: OffsetMap::compute(&BastDoc),
  materialize_aligned(&BastDoc, ...) (no lifetime), BuildCtx/
  ComputeCtx hold &'d BastDoc, tunion/discriminator name borrows,
  lib.rs module doc. LayoutBuilder's doc_value re-parse cache is
  unchanged pending phase 4.

Verification: 465 tests pass with zero test-logic changes (bast.rs
suites exercise every parser path through the public API); clippy
-D warnings clean; cargo doc zero warnings; wasm32 release build
green.
 2026-09-02 08:31:58 +00:00
glm-5.3-flash 1641dab505 Document session-continuity rules in AGENTS.md 
opencode ends the turn when an assistant message contains no tool call,
including analysis-only messages. Document the working fix (end bursts
with a tool call or a final report, land work incrementally, resume
without re-deriving) so every session inherits it.
 2026-09-02 08:08:27 +00:00
glm-5.3-flash e5f1b9d825 Wire packed read path through ReadPlan (ADR-011 steps 2-4, plan phase 2) 
SequentialReader now walks Arc<ReadPlan> instead of re-parsing the
BAST typed tree per field (the 400x read-path gap, review #004 H1);
materialize_packed walks the same plan, unifying the two packed
read-side consumers on one compiled form.

- SequentialReader::new(Arc<ReadPlan>) -> Self, infallible: the
  fallible BastDoc parse moved to ReadPlan::compile (phase 1). The
  reader holds the plan Arc + cursor state only; schema() returns the
  Arc<Value> retained on the plan (review #005 H2 — no
  self-referential struct); new plan() accessor exposes the shared
  plan.
- ReadPlan carries schema: Arc<Value> (set at compile; sub-plans hold
  a Null placeholder — only the root plan is handed out).
- materialize_packed(&ReadPlan, &[u8]): plan-walking packed
  materializer. The aligned path keeps walking BastDoc with the
  retained dummy_field_for/ty_source/materialize_typeref_packed
  helpers (phase 5 Scope Boundary: aligned structure walk is the
  permanent 0.3.0 design).
- Engine: Layout::Packed stores Arc<ReadPlan> alongside the builder;
  sequential_reader() is an Arc::clone (was a full-document Value
  clone); packed validate_bytes calls materialize_packed(&self.plan).
- Stride (deferred decision 4): FieldValue::Array now reports the
  true stride for fixed-size struct/nested-array elements (0.2.0
  returned 0); doc comment documents the behavioral change; no
  existing test asserted the 0, so none needed changing.
- Two parity subtleties found and preserved:
  (a) materialize_plan_composite unwraps the plan's anonymous
      single-field wrapper for primitive array elements/record values
      — without it, materialized records nest each leaf under a
      synthetic object (caught by the record parity test);
  (b) field-disc unions keep 0.2.0's materialized key order
      (__discriminator first), observable under preserve_order.
  Both are now covered by plan-phase tests or construction.

Bench (alktty wire_vs_bast, 1024 chunks/stream): packed read
2.27 us/chunk (review #004) -> 98 ns/chunk p64 / 100 ns/chunk p4k
(~23x; the 400x gap closes to ~17x vs hand-rolled 5.6 ns/chunk).
Residual gap is the per-field String allocation mandated by the
unchanged (String, FieldValue) read_next signature (2 allocs/chunk)
plus data_access bounds checks. sequential_reader() construction:
15.7 ns (was a whole-document clone).

Verification: 465 tests pass unchanged (the existing reader/
materialize/engine suites drive the rewrite through the public API —
only constructor call sites moved to ReadPlan::compile); clippy
-D warnings clean; cargo doc zero warnings; wasm32 release build
green.
 2026-09-02 07:52:46 +00:00
glm-5.3-flash ff85258d03 Implement ReadPlan type + compile (ADR-011 step 1, plan phase 1) 
Pure addition: the packed read-side compiled form (src/read_plan.rs)
and lib.rs wiring (module + re-exports of ReadPlan, FieldPlan,
CompositePlan, ReadKind, DiscriminatorPlan). No existing engine code
touched — phases 2-5 wire the plan into the reader/materializer/engine.

- Refined union shape (ADR-011 as refined by review #005):
  CompositePlan::Union { disc, shared, variants } with
  shared: Option<Box<ReadPlan>> for field-disc unions and
  variants: Vec<(String, CompositePlan)> — no VariantPlan/VariantKind,
  nested-union variants work by ordinary CompositePlan recursion
  (restores the 0.2.0 capability the POC rejected).
- by_name is BTreeMap (ADR-012 §1 Hash-derive prerequisite).
- True array strides (deferred decision 4): fixed struct/nested-array
  elements compute their real stride via fixed_composite_size;
  variable-length elements stay 0. 0.2.0 returned 0 for fixed struct
  arrays; that behavioral change rides the 0.3.0 bump (phase 2 will
  surface it through SequentialReader).
- Endianness: effective endian baked at every node. Parity lock: the
  plan propagates the referring field's effective endian into nested
  structs/unions — what the 0.2.0 packed reader/materializer actually
  do — and ignores nested containers' own endian annotations (the POC
  baked s.endian() there; latent divergence, never exercised by its
  equivalence tests). Nested-annotation tests lock this in.
- Untrusted input: compile carries its own depth cap (128) +
  definition-level cycle set (mirrors ValidationPlan::compile), so
  standalone compile is safe on adversarial docs: cyclic refs, deep
  chains, dangling refs, non-struct roots, and non-struct/union
  variants all surface as AlkTypeError::Schema, never a panic.
  Overflow-safe stride arithmetic (checked_mul).

Verification: 388 tests pass (355 existing + 33 new: every BastType
arm coverage, field-disc shared/nested-union compile shape, stride
computation, endian parity, cycle/depth/malformed rejection,
Send + Sync static-bound assertion); clippy -D warnings clean;
cargo doc zero warnings; wasm32-unknown-unknown release build green.

Next: phase 2 (SequentialReader + materialize_packed consume the plan).
 2026-09-02 07:19:54 +00:00
glm-5.3-flashandopencode e4636e6a44 Implement ValidationPlan (ADR-012 §3, plan phase 7) 
The compiled value-domain validation form: replaces the interpretive
BastDoc walk in validate_bytes with a compile-once-walk-many
constraint tree built at engine-compile time. This was the design
session + implementation ADR-012 §3 delegated; the shape decisions
are recorded in new ADR-012 §3a.

- New src/validation_plan.rs: ValidationPlan + ValidNode/ValidField/
  ValidVariant (Debug+Clone+PartialEq+Eq+Hash+Send+Sync),
  compile(&BastDoc) with eager $ref resolution, and a per-buffer walk
  with deferred error-path rendering (zero happy-path allocation,
  byte-identical error messages vs the 0.2.0 walker).
  fingerprint() via DefaultHasher, same as the phase-6 pattern.
- Compile-time graph safety: definition-level cycle set + depth cap
  (128) reject cyclic $ref graphs with AlkTypeError::Schema. The
  interpretive walker resolved refs lazily with no guard (stack-
  overflow hazard); diamond (shared) refs still compile.
- bast_validation.rs: interpretive walker retired (deleted);
  validate_value survives as a one-shot wrapper (compile + validate)
  for callers holding a doc without an engine.
- engine: Arc<ValidationPlan> built at compile in BOTH modes; the
  plan compile runs before the layout build and doubles as the
  engine's cyclic-ref gate (LayoutBuilder/OffsetMap struct recursion
  has no cycle guard; a cyclic doc previously overflowed there).
  validate_bytes walks the plan; new accessor validation_plan().
  validate_bytes signature unchanged.
- lib.rs: pub mod validation_plan + re-exports (ValidationPlan,
  ValidNode, ValidField, ValidVariant).

Verification: cargo test --release (355 pass, incl. parity suite,
fingerprint contract, cycle/depth rejection, Send+Sync + thread-share
assertions); clippy --all-targets -D warnings clean; cargo doc
zero warnings; wasm32-unknown-unknown release build green.

Co-authored-by: opencode <noreply@alk.dev>
 2026-08-31 17:45:46 +00:00
glm-5.2 e461f01c97 Resolve review #005: refine 0.3.0 plan + ADR-011/012 
Resolve all 11 findings from the 0.3.0 plan review (#005) in one
docs-only pass. No source changes; the crate still builds/tests at
v0.2.0. The one substantive decision change is M3 (per user
direction: ship ValidationPlan in 0.3.0, no more hedging); the rest
are spec corrections or pre-implementation refinements to types that
do not yet exist on main.

- H1: refine ADR-011 CompositePlan::Union to carry
  shared: Option<Box<ReadPlan>> (field-disc shared fields) and
  variants: Vec<(String, CompositePlan)> (drop VariantPlan/
  VariantKind). Plan phase 1 implements the refined shape.
- H2: plan phase 2 specifies ReadPlan stores schema: Arc<Value>
  (not &Value), avoiding the self-referential struct ADR-011
  rejects. Verified serde_json::Value: Hash + Eq holds with
  preserve_order, so phase 6 derives are not blocked.
- M1: nested-union support falls out of the H1 shape refinement
  (a variant can be CompositePlan::Union) — option (a) from the
  review, no behavioral drop vs 0.2.0, no Semver regression row.
- M2: plan phase 5 adds an explicit first sub-step to derive Hash
  on Endian and VariableEncoding in src/schema.rs (additive,
  semver-safe prerequisite the original plan omitted).
- M3: reverse the ValidationPlan deferral. ADR-012's "Deferring
  ValidationPlan" becomes "ValidationPlan — in scope for 0.3.0";
  new ADR-012 §3 commits the decision (compiled form, no per-buffer
  BastDoc walk, Hash + Eq + fingerprint()) and defers only the
  concrete shape to a follow-on design session + the plan's new
  phase 7. Plan gains phase 7 (ValidationPlan); old phase 7 (bump)
  renumbered to phase 8. ADR-011's Out-of-scope and Scope
  Boundaries bullets updated to point at ADR-012 §3. The deferral
  black hole this review's methodology flagged is closed: the work
  is committed with a concrete reactivation trigger, not hedged
  into an unplanned future.
- L1: plan phase 2 corrects the dummy_field_for/ty_source removal
  claim — only packed-side call sites go away; the helpers stay
  for the aligned materialize_leaf_at path.
- L2: plan phase 2 states the packed-vs-aligned
  materialize_typeref_packed split (packed gets a new plan-walking
  function; the existing function stays for aligned).
- L3: plan phase 5 adds a Scope Boundary note — aligned
  materialize's BastDoc structure walk is the permanent 0.3.0
  design; an AlignedPlan is out of scope, tracked as an OQ.
- N1: fix "back-comat" -> "back-compat" typo.
- N2: plan phase 1 verification adds the read_plan_is_send_sync
  static-bound assertion test ADR-011 requires.
- N3: Semver Contract table notes the Result drop on
  SequentialReader::new (Result<Self, AlkTypeError> -> Self)
  alongside the argument-type change.

Also: ADR-012 title -> "Plan Fingerprinting, ValidationPlan, and
Closing the Deferred M1 Sites in 0.3.0"; §3 (Fingerprinting
OffsetMap) renumbered to §4; README ADR table updated; review #005
gets a Resolution section recording how each finding was closed.

Verification (docs-only change, v0.2.0 unchanged):
  cargo test --release                     ok (310 crate + 86 integration + 2 doctests)
  cargo clippy --all-targets -- -D warnings  ok
  cargo doc --no-deps                      ok
 2026-08-20 06:46:43 +00:00
glm-5.2 0e7921a02a Add review #005: 0.3.0 plan review 
Cross-checks docs/plans/030-compiled-forms.md against the codebase,
ADRs 011/012, the POC on readplan-poc, and review #004.

Findings:
- H1: field-disc union shape is in neither ADR-011 nor the POC
- H2: schema() &Value on Arc<ReadPlan> is the self-referential
  pattern ADR-011 rejects
- M1: nested-union silent behavioral drop (POC rejects what 0.2.0
  accepts); deferral-black-hole pattern
- M2: Endian/VariableEncoding missing Hash derive (phases 5/6 break)
- M3: ValidationPlan deferral flagged for re-evaluation — the
  read+validate-on-untrusted-input case may be hotter than
  ADR-012's 'not a hot loop' dismissal accounts for
- L1/L2/L3: dummy_field_for wording, materialize_packed split,
  materialize_aligned BastDoc walk silence
- N1/N2/N3: typo, Send+Sync assertion test, Result drop on new

Includes a deferral-pattern scan methodology section surfacing
M1/L3/H2 as black-hole instances and confirming the plan's four
explicit deferred decisions are the healthy pattern.

Verification: file-only change, no code touched.
 2026-08-20 06:09:57 +00:00
glm-5.2 2310f6cbd8 Propose ADR-012 + 0.3.0 implementation plan 
ADR-012 bundles two pieces of work into the 0.3.0 release so the
crate ships one round of breaking changes, not two:

- Fingerprinting: #[derive(Hash, Eq)] + fingerprint() -> u64 on
  ReadPlan and OffsetMap. BTreeMap for ReadPlan.by_name (HashMap
  blocks Hash derive). Fingerprint contract: equal hashes => identical
  reads over identical bytes. Enables cross-run plan caching, alkcall
  hub/spoke schema handshake, schema-version diagnostics.
- Closing the deferred M1 sites via owned BastDoc (lifetime removal,
  scoped to LayoutBuilder/bast_validation/materialize_aligned/
  OffsetMap::compute) + extending OffsetMap with LeafMeta
  {kind, encoding, endian} for the aligned read_field/write_field paths.

Reframes the 'WritePlan' candidate from ADR-011's Future capabilities
section: the packed write-side compiled form is PackedLayout; the
aligned R/W compiled form is OffsetMap; the M1 fixes are 'cache the
parse' and 'extend the compiled form with leaf metadata', not 'add a
third compiled form.' Serves minimal-public-API-changes better than
a literal WritePlan type. ValidationPlan deferred (different shape,
not a hot loop).

The plan (docs/plans/030-compiled-forms.md) is the execution entry
point: seven phases ordered by dependency, each phase a session
boundary. Phase 1-2: ReadPlan (ADR-011). Phase 3: owned BastDoc.
Phase 4: LayoutBuilder M1 fix. Phase 5: OffsetMap LeafMeta. Phase 6:
fingerprinting. Phase 7: version bump + docs + verification. Includes
a semver contract table, deferred decisions, cross-phase invariants,
and the verification block.

ADR-011's Future capabilities section updated to point at ADR-012 for
the items moving into 0.3.0 and record the WritePlan reframe. README
ADR table gets ADR-012 as Proposed.

Verification: docs-only change; cargo test --release, cargo clippy
--all-targets -- -D warnings, cargo doc --no-deps unchanged (no source
touched).
 2026-08-19 08:06:55 +00:00
glm-5.2 1037e68091 Accept ADR-011: compiled read plan for packed mode 
Tighten framing per pre-acceptance review (no decision changes):

- Be precise about M1 coverage: closes the packed-side site
  (engine.rs:284 validate_bytes); the aligned-side M1 sites
  (engine.rs:334,467, layout_builder.rs:190) are a deliberate
  reversible bet, not a non-issue.
- Replace the 'two type walkers is symmetric with OffsetMap/
  PackedLayout' spin with an honest 'parallel typed tree, permanent
  maintenance tax, justified by ~20-50x composite-dispatch win on
  SFTP-shaped union-with-$ref-variants packets.'
- Move the 'determinism enables fingerprinting/cache/handshake' future
  work out of the positives list into a dedicated 'Future
  capabilities' section — it is a forward reference, not a current win.
- Clarify bast_doc: Value is retained unconditionally (unused in
  packed mode, still needed in aligned mode); add a Send+Sync test
  note for Arc<ReadPlan> shared from the Send+Sync engine.
- Tighten the 'no format! allocations' claim to 'no resolve_typeref,
  no BastDef::parse, no JSON node access on the happy path' (error-
  path format! remains, and is not the cost being removed).
- Add a 'POC coverage' section recording that the readplan-poc branch
  walked every BastType arm and confirmed ReadPlan covers all cases,
  including the union Byte/Field split and the array variable-stride
  (element_stride = 0) case.

Status flipped Proposed -> Accepted. README ADR table updated.

Verification: docs-only change; cargo test --release, cargo clippy
--all-targets -- -D warnings, cargo doc --no-deps unchanged (no source
touched).
 2026-08-18 09:32:17 +00:00
glm-5.2 3184818c08 Propose ADR-011: compiled read plan for packed mode 
Review #004 measured the packed read path at ~400x slower per chunk
than a hand-rolled codec, root-caused to SequentialReader re-parsing
BastDoc::new on every field read (the 're-parse on demand' framing
from ADR-007). The write path is competitive because it has a compiled
form (PackedLayout); the read path is the only mode/side pair without
one.

ADR-011 proposes ReadPlan — the packed read-side compiled form,
symmetric to OffsetMap (aligned R/W) and PackedLayout (packed W).
Packed positions are data-dependent (variable-length fields shift
subsequent fields), so the compiled form is necessarily a read program
(a pre-resolved tree of read instructions), not a flat lookup table
like OffsetMap. ReadPlan::compile walks BastDoc once at engine
construction, resolves all $ref\s eagerly, computes effective
endianness at every node, and inlines union variants; the read loop
then indexes into a Vec, matches on ReadKind, and calls data_access
with a precomputed Endian — no BastDoc, no resolve_typeref, no JSON
node access at read time.

Scope: SequentialReader and materialize_packed consume the plan (one
walker, not two); bast_validation, LayoutBuilder, and aligned one-shot
paths stay on BastDoc (different concern, not hot loops). Includes a
7-step Recommended Order matching review #004's structure. Breaking
public-API change (0.2.0 -> 0.3.0): SequentialReader::new and
materialize_packed take ReadPlan; BastDoc and the Bast* types are
unchanged (smaller breakage than review #004's Option A).

Cross-links: ADR-007's 'Cost' section gets a Note pointing at review
#004 and ADR-011, marking the 're-parse on demand' framing as the root
cause slated for retirement (the factory decision itself is retained);
the actual Cost/doc-comment rewrite happens in the implementation
commit per ADR-011's recommended order. README ADR table updated.

Verification: docs-only change, no source touched.

Closes review #004 H1, M1 (packed side), L1, L2 (on implementation).
 2026-08-18 07:51:45 +00:00
glm-5.2 51cb552715 Add review #004: read-path performance review 
Traces the ~400x read-path gap (alktty wire_vs_bast bench) to
SequentialReader::read_field_at re-parsing BastDoc::new on every field
read, with the self-referential lifetime constraint as the root cause.

Findings:
- H1: per-field BastDoc::new re-parse in sequential_reader.rs:262
- M1: same re-parse in four one-shot paths (LayoutBuilder::build,
  validate_bytes, engine read_field/write_field)
- L1: dead _field_schema param + Value clones in SequentialReader
- L2: ADR-007 Cost section + engine doc comment understate the re-parse
- N1: carry-forward of review #003 N2 (no new action)

Lays out fix options: Option A (owned typed tree, recommended, closes
H1+M1, breaking), Option B (read-plan precompute, fallback, H1 only,
non-breaking), Option C (borrow-from-engine, rejected, contradicts
ADR-007).

Verification: docs-only review; alktype source unchanged.

cab4932
 2026-08-17 14:18:23 +00:00
40 changed files with 14558 additions and 1676 deletions
Expand all files Collapse all files

No files matched your search

AGENTS.md
+23
View File
@@ -5,6 +5,29 @@ auto-loads this file as instructions, overriding the built-in defaults for
this project. Custom agents in `.opencode/agents/` inherit these rules
unless their own prompts say otherwise.
## Session Continuity (keep the agent loop alive)
opencode ends the turn whenever an assistant message contains no tool
call — including messages that are pure analysis. Long reasoning bursts
are welcome in this repo (they pre-catch errors and self-correct), but a
burst that ends as analysis-only text silently stops the session
mid-task. Past sessions documented this repeatedly ("the session
stalled"; the working fix discovered there: "call tools frequently, keep
thinking bursts short"). Keep the depth; change where the burst ends:
1. **Never end a turn with analysis-only text.** Every visible message
must either issue a tool call or be a final report for a genuinely
completed phase/task. When a thinking burst converges on a decision,
act on it (read, edit, bash) in the same turn.
2. **Land work incrementally.** Once a design decision is settled, write
the code before analyzing the next one. Do not emit full-design
essays in a single chat message; reasoning belongs in thinking tokens
or committed docs, not in the transcript.
3. **On a silent turn end, resume without re-deriving.** If the turn
ended after an analysis-only message and the task is incomplete,
pick up from the last settled decision — do not redo the analysis
and do not ask the user whether to continue.
## Git Workflow
**Commit and push when reasonable.** When a change is complete and
CHANGELOG.md
+108
View File
@@ -4,6 +4,113 @@ All notable changes to this crate are documented here. The format is
based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and
this crate adheres to [Semantic Versioning](https://semver.org/).
## [0.3.0] - 2026-09-07
The compiled-forms release. The packed read path — the hot path for
stream parsing — is driven by a compile-once `ReadPlan` instead of a
per-call walk of the BAST typed tree; byte validation runs a compiled
`ValidationPlan`; plans and offset maps fingerprint to a stable hash
(ADR-011/ADR-012). Reads of SFTP-shaped packet streams went from
~189× hand-rolled Rust to ~74× (~2.4× faster), fixed-stride chunk
reads from ~18× to ~11×, and `SequentialReader::read_next_borrowed`
makes the per-field hot loop allocation-free.
### Breaking changes
- **`Bast*` types are owned.** `BastDoc`/`BastStruct`/`BastField`/… no
longer borrow from the source `serde_json::Value`; all v0.2.0
lifetimes are gone. `BastDoc::new` parses the root eagerly; `$ref`s
resolve lazily.
- **`OffsetMap::get` / `PackedLayout::get`** return `&OffsetEntry`
(was `Option<OffsetEntry>` by value), backed by an O(log n)
`BTreeMap` path→index (first-occurrence-wins for duplicate names).
- **`SequentialReader::new`** takes the compiled plan; construct via
`AlkTypeEngine::sequential_reader()` (packed mode only).
- **`materialize_packed` / `materialize_aligned`** take the compiled
plan / `(&BastDoc, &OffsetMap)` pair respectively.
- **Field-name-discriminator union wire convention** (ADR-011
addendum): the builder lays out the union's declared `fields`
(shared) first, then the variant's own fields. Variants must not
re-declare the discriminator or any shared field, and the
discriminator field must be the first entry in `fields` — all
enforced at parse with clean `Schema` errors. Schemas relying on
0.2.0's variant-only layout are rejected (they produced
reader↔builder-disagreeing bytes).
- **`maxLength` is string/bytes-only** — rejected at parse on every
other kind (it was silently unenforced there).
- **Aligned-mode `Record` fields reject `offset-indirect`** (the
materializer always walks the inline count-prefixed form — the
annotated shape was never readable).
- **Schema input bounds** (untrusted-schema hardening, AGENTS.md §3):
array `count` ≤ 2^16 and `count × stride` ≤ 2^26 bytes; `align` ≤
4096; `maxLength` ≤ 2^26; cyclic `$ref` graphs and >128-deep nesting
are rejected by every public walker (`OffsetMap::compute`,
`LayoutBuilder::new`, `materialize_aligned` included), not just the
engine.
### Additions
- **`ReadPlan`** (ADR-011) — the compiled packed-read plan, re-exported
with `CompositePlan`/`FieldPlan`/`ReadKind`/`DiscriminatorPlan`.
`ReadPlan::compile` is untrusted-input-safe standalone (depth cap +
cycle set). `fixed_size()` exposes the compile-time-known byte size
for fixed structs.
- **`ValidationPlan`** (ADR-012 §3) — the compiled `validate_bytes`
walker, with `ValidNode`/`ValidVariant` sub-types.
- **`fingerprint()`** on `ReadPlan`/`OffsetMap`/`ValidationPlan` +
`Hash`/`Eq` derives on the plan types (ADR-012 §1/§4) — plan
identity for cache-keying across processes.
- **`OffsetMap` `LeafMeta`** — each entry records whether it is
fixed/length-prefixed/offset-indirect so `read_field`/`write_field`
dispatch without re-walking the schema; `OffsetEntry` type re-exported.
- **`SequentialReader::read_next_borrowed`** — zero-allocation variant
of `read_next` (field name borrowed from the plan).
- **`AlkTypeEngine::validate_bytes`** now runs the compiled
`ValidationPlan` (was an interpretive BAST walk in 0.2.0).
### Fixes (post-release-commit hardening — reviews #006, #007, #008)
All found and fixed before the first crates.io publish of 0.3.0, so
no published version ever exhibited them.
- **Untrusted-input crashes removed.** A huge declared array count
OOM-aborted the process (`Vec::with_capacity(count)` before reading
a byte) — now compile-capped and walked with push-only growth.
Cyclic `$ref` graphs stack-overflowed the three standalone layout
walkers — now guarded by a shared reference-graph check. Deeply
nested stride-0 arrays briefly allowed ~477 MB of simultaneous
allocation from a ~1 KB schema — restored to incremental growth.
- **Cross-consumer divergences closed.** Builder, reader,
materializer, tunion, and the validation plan now agree on
field-disc union layout (shared-then-variant), on the discriminator
field's position (must be first), and on union mapping-key matching
(numeric fast-path dispatch only for canonical keys like `"2"`;
`"01"`/`"+1"` fall back to the string comparison all consumers
share). The legacy BAST walker's field-disc union arm walks shared
fields before the variant (it previously materialized variant fields
from shared fields' bytes).
- **Silently-corrupt layouts rejected.** Aligned record fields with
`maxLength`/`offset-indirect`; non-final inline length-prefixed
fields (records included — the ADR-006 check now sees them);
aligned-mode `maxLength`/`offset-indirect` on records; unions in
aligned mode (pre-existing, now tested).
- **Coverage**: 90.67% lines / 86.32% functions at review #007's
audit, 91.66% after its fixes; every uncovered region outside test
modules read and classified in-tree (docs/reviews/007).
### Non-breaking improvements
- Engine compile is one-shot and allocation-tidy; plans are
`Send + Sync` (statically asserted) and fingerprintable.
- Zero-progress array-element guard on all three array walkers (a
zero-size element makes the declared count unbounded on the wire).
- WASM-clean unchanged: two dependencies (`jsonschema`
default-features off, `serde_json` with `preserve_order`), no
`async`, no `unsafe`, no feature flags.
- Benches (`benches/wire_vs_bast.rs`): read/write chunk streams, an
SFTP-shaped union packet stream, and `validate_bytes` per buffer —
the numbers quoted above and in ADR-007/ADR-011.
## [0.2.0] - 2026-08-17
A breaking release that replaces the v0.1.0 `AlkType:*` custom-keyword
@@ -115,5 +222,6 @@ Initial crates.io release. Custom-keyword JSON Schema format
`AlkTypeEngine` with packed/aligned layout modes, builder API producing
`serde_json::Value`.
[0.3.0]: https://git.alk.dev/alkdev/alktype/releases/tag/v0.3.0
[0.2.0]: https://git.alk.dev/alkdev/alktype/releases/tag/v0.2.0
[0.1.0]: https://git.alk.dev/alkdev/alktype/releases/tag/v0.1.0
Cargo.lock
Generated
+188 -1
View File
@@ -27,8 +27,9 @@ dependencies = [
[[package]]
name = "alktype"
version = "0.2.0"
version = "0.3.0"
dependencies = [
"criterion",
"jsonschema",
"serde_json",
]
@@ -39,6 +40,18 @@ version = "0.2.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "683d7910e743518b0e34f1186f92494becacb047c7b6bf616c96772180fef923"
[[package]]
name = "anes"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4b46cbb362ab8752921c97e041f5e366ee6297bd428a31275b9fcf1e380f7299"
[[package]]
name = "anstyle"
version = "1.0.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "940b3a0ca603d1eade50a4846a2afffd5ef57a9feac2c0e2ec2e14f9ead76000"
[[package]]
name = "autocfg"
version = "1.5.1"
@@ -84,12 +97,107 @@ version = "0.6.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "175812e0be2bccb6abe50bb8d566126198344f707e304f45c648fd8f2cc0365e"
[[package]]
name = "cast"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "37b2a672a2cb129a2e41c10b1224bb368f9f37a2b16b612598138befd7b37eb5"
[[package]]
name = "cfg-if"
version = "1.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
[[package]]
name = "ciborium"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "42e69ffd6f0917f5c029256a24d0161db17cea3997d185db0d35926308770f0e"
dependencies = [
"ciborium-io",
"ciborium-ll",
"serde",
]
[[package]]
name = "ciborium-io"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "05afea1e0a06c9be33d539b876f1ce3692f4afea2cb41f740e7743225ed1c757"
[[package]]
name = "ciborium-ll"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "57663b653d948a338bfb3eeba9bb2fd5fcfaecb9e199e87e1eda4d9e8b240fd9"
dependencies = [
"ciborium-io",
"half",
]
[[package]]
name = "clap"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "473c7e07f409a8d772161724aa8db6a765a2532a70f9667eeb7b49d3d02fbdca"
dependencies = [
"clap_builder",
]
[[package]]
name = "clap_builder"
version = "4.6.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7b48fea5a88e9ae728a2dcbedbfc0e730f7d60da42e1cb049a83c9fb8b789889"
dependencies = [
"anstyle",
"clap_lex",
]
[[package]]
name = "clap_lex"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8d4a3bb8b1e0c1050499d1815f5ab16d04f0959b233085fb31653fbfc9d98f9"
[[package]]
name = "criterion"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e1c047a62b0cc3e145fa84415a3191f628e980b194c2755aa12300a4e6cbd928"
dependencies = [
"anes",
"cast",
"ciborium",
"clap",
"criterion-plot",
"itertools",
"num-traits",
"oorandom",
"regex",
"serde",
"serde_json",
"tinytemplate",
"walkdir",
]
[[package]]
name = "criterion-plot"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9b1bcc0dc7dfae599d84ad0b1a55f80cde8af3725da8313b528da95ef783e338"
dependencies = [
"cast",
"itertools",
]
[[package]]
name = "crunchy"
version = "0.2.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "460fbee9c2c2f33933d720630a6a0bac33ba7053db5344fac858d4b8952d77d5"
[[package]]
name = "data-encoding"
version = "2.11.0"
@@ -107,6 +215,12 @@ dependencies = [
"syn 2.0.119",
]
[[package]]
name = "either"
version = "1.18.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "252afb9ae5eaa683babdc6a068b3f5726eb19e05070c731f9b2a23a7c3e8ed34"
[[package]]
name = "email_address"
version = "0.2.9"
@@ -174,6 +288,17 @@ dependencies = [
"wasm-bindgen",
]
[[package]]
name = "half"
version = "2.7.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ea2d84b969582b4b1864a92dc5d27cd2b77b622a8d79306834f1be5ba20d84b"
dependencies = [
"cfg-if",
"crunchy",
"zerocopy",
]
[[package]]
name = "hashbrown"
version = "0.16.1"
@@ -304,6 +429,15 @@ dependencies = [
"hashbrown 0.17.1",
]
[[package]]
name = "itertools"
version = "0.13.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "413ee7dfc52ee1a4949ceeb7dbc8a33f2d6c088194d9f922fb8318faf1f01186"
dependencies = [
"either",
]
[[package]]
name = "itoa"
version = "1.0.18"
@@ -479,6 +613,12 @@ version = "1.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9f7c3e4beb33f85d45ae3e3a1792185706c8e16d043238c593331cc7cd313b50"
[[package]]
name = "oorandom"
version = "11.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6790f58c7ff633d8771f42965289203411a5e5c68388703c06e14f24770b41e"
[[package]]
name = "outref"
version = "0.5.2"
@@ -628,6 +768,15 @@ version = "1.0.23"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
[[package]]
name = "same-file"
version = "1.0.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "93fc1dc3aaa9bfed95e02e6eadabb4baf7e3078b0bd1b4d7b6b0b68378900502"
dependencies = [
"winapi-util",
]
[[package]]
name = "scopeguard"
version = "1.2.0"
@@ -733,6 +882,16 @@ dependencies = [
"zerovec",
]
[[package]]
name = "tinytemplate"
version = "1.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be4d6b5f19ff7664e8c98d03e2139cb510db9b0a60b55f8e8709b689d939b6bc"
dependencies = [
"serde",
"serde_json",
]
[[package]]
name = "unicode-general-category"
version = "1.1.0"
@@ -773,6 +932,16 @@ version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5c3082ca00d5a5ef149bb8b555a72ae84c9c59f7250f013ac822ac2e49b19c64"
[[package]]
name = "walkdir"
version = "2.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "29790946404f91d9c5d06f9874efddea1dc06c5efe94541a7d6863108e3a5e4b"
dependencies = [
"same-file",
"winapi-util",
]
[[package]]
name = "wasip2"
version = "1.0.4+wasi-0.2.12"
@@ -827,12 +996,30 @@ dependencies = [
"unicode-ident",
]
[[package]]
name = "winapi-util"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c2a7b1c03c876122aa43f3020e6c3c3ee5c05081c9a00739faf7503aeba10d22"
dependencies = [
"windows-sys",
]
[[package]]
name = "windows-link"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f0805222e57f7521d6a62e36fa9163bc891acd422f971defe97d64e70d0a4fe5"
[[package]]
name = "windows-sys"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ae137229bcbd6cdf0f7b80a31df61766145077ddf49416a728b02cb3921ff3fc"
dependencies = [
"windows-link",
]
[[package]]
name = "wit-bindgen"
version = "0.57.1"
Cargo.toml
+10 -2
View File
@@ -1,6 +1,6 @@
[package]
name = "alktype"
version = "0.2.0"
version = "0.3.0"
edition = "2021"
rust-version = "1.85"
license = "MIT OR Apache-2.0"
@@ -19,4 +19,12 @@ default = []
[dependencies]
jsonschema = { version = "0.46", default-features = false }
serde_json = { version = "1", features = ["preserve_order"] }
serde_json = { version = "1", features = ["preserve_order"] }
[dev-dependencies]
serde_json = "1"
criterion = { version = "0.7", default-features = false }
[[bench]]
name = "wire_vs_bast"
harness = false
README.md
+24 -8
View File
@@ -24,10 +24,11 @@ A BAST document serves three roles simultaneously:
| Role | Mechanism | When |
|------|-----------|------|
| **Validation spec (bytes)** | BAST-native validator (recursive walker over the BAST type tree) | Access time (`validate_bytes`) |
| **Validation spec (bytes)** | Compiled `ValidationPlan` walk over the materialized `Value` (ADR-012) | Access time (`validate_bytes`) |
| **Validation spec (JSON)** | Standard `jsonschema::Validator` from a consumer-provided JSON Schema | Load time (build validator), access time (`validate_json`) |
| **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map / packed layout) |
| **Data access** | Read/write at computed offsets | Access time (read field, write field) |
| **Wire access (packed)** | Compiled `ReadPlan` (ADR-011) — compile-once, no per-read schema walk | Access time (`SequentialReader`) |
No separate format definition, no separate parser, no separate
validator. The BAST document is the single source of truth for the
@@ -159,11 +160,16 @@ same BAST document can be compiled in either mode. Decided in ADR-002.
- **Byte-offset** — a fixed-size integer (`uint8`/`uint16`/`uint32`) at
a known byte offset. The SFTP `Packet` pattern: byte 0 is the type
byte, bytes 1..N are the variant struct. Mapping keys are stringified
integers.
integers. With all-canonical numeric keys the compiled reader
dispatches on the raw integer (no per-read stringification).
- **Field-name** — a named field within the union. The TypeBox
`typedef.ts` pattern. Mapping keys are string values matching the
discriminator field's value. The `fields` array declares the
discriminator field (D-BAST-005).
discriminator field (D-BAST-005), which must be its first entry; the
variant must not re-declare it or any shared field. The builder lays
out the declared `fields` first, then the variant's own fields
(ADR-011 addendum) — builder, reader, materializer, and validator all
agree on that convention.
Variant `$ref`s are resolved lazily — no compile-time inlining step.
@@ -182,10 +188,10 @@ types (ADR-VAL-SPLIT):
- `validate_bytes(&[u8])` — for raw byte buffers (channels' chunk
header, SFTP packets). Materializes a `Value` tree from the bytes via
the layout engine, then runs the **BAST-native validator** — a
recursive walker over the BAST type tree that checks the value-domain
constraints the materializer doesn't (integer ranges, `maxLength`,
enum index bounds, union variant constraints). No
the layout engine, then runs the compiled **`ValidationPlan`** (0.2.0
used an interpretive BAST walker; 0.3.0 compiles the value-domain
constraints — integer ranges, `maxLength`, enum index bounds, union
variant dispatch — once at compile time). No
`jsonschema` involvement; the BAST document is the complete
validation spec for bytes (D-BAST-006).
- `validate_json(&Value)` / `is_valid_json(&Value)` — for already-parsed
@@ -244,6 +250,15 @@ code was converted to `Err` ahead of v0.1.0 (review #002, L2); the
BAST parser preserves this invariant — overflow-safe arithmetic
(`checked_add`, `usize::try_from`) on all offset/count casts.
0.3.0 adds compile-time bounds for adversarial schemas: array counts
≤ 2^16 elements, computed array sizes ≤ 2^26 bytes, `align` ≤ 4096,
`maxLength` ≤ 2^26, and a shared reference-graph guard that rejects
cyclic `$ref`s and >128-deep nesting in every public schema walker.
Adversarial buffers fail with `Access` errors at read time — the
materializers never preallocate from declared counts. Reviews #006,
#007, and #008 document the audit trail
([docs/reviews/](docs/reviews/)).
## Documentation
Architecture documentation lives under [`docs/architecture/`](docs/architecture/):
@@ -269,7 +284,8 @@ Architecture documentation lives under [`docs/architecture/`](docs/architecture/
(ADR-003), error handling (ADR-004), int64/uint64 kinds (ADR-005),
non-final inline variable fields (ADR-006), packed-mode read factory
(ADR-007), TUnion in aligned mode (ADR-008), builder API (ADR-009),
`validate_bytes` (ADR-010)
`validate_bytes` (ADR-010), compiled read plan (ADR-011), plan
fingerprinting + `ValidationPlan` (ADR-012)
## License
benches/wire_vs_bast.rs
+645
View File
@@ -0,0 +1,645 @@
//! Informal speed comparison: hand-rolled codec logic vs alktype-driven
//! codec over the same wire shapes.
//!
//! History: this bench originated (uncommitted) in `alktty` as the
//! curiosity probe that surfaced review #004's 400x read gap — the
//! finding that drove the 0.3.0 compiled-forms release (ADR-011/012).
//! It now lives here so alktype owns its perf story. The alktty-only
//! async roundtrip group (tokio `ChunkReader`/`ChunkWriter` over a
//! duplex pipe) was dropped — that measures alktty's I/O stack, not
//! this engine.
//!
//! The alktype engine / layout / plans are built **once outside** the
//! measured routine, per the "build cost is paid once" framing.
//!
//! Shapes:
//!
//! - **ChunkHeader** — a 5-byte header (`stream_type: uint8`,
//! `length: uint32` big-endian). The original shape, kept so numbers
//! stay comparable with the historical series (review #004: 400x →
//! 0.3.0: ~18x on read p64).
//! - **Read** — the hand-rolled path mirrors
//! `ChunkReader::read_chunk_after_peek` minus the tokio I/O
//! (identical overhead on both sides): validate `stream_type <= 4`,
//! parse `u32::from_be_bytes`, slice the payload. The alktype path
//! drives `SequentialReader::read_next` over the `ChunkHeader`
//! struct, then slices the payload at the parsed length. Both
//! return a `&[u8]` payload view — no allocation in either measured
//! path.
//! - **Write** — serialize the 5-byte header. Hand-rolled mirrors
//! `ChunkWriter::write_chunk`'s header writes; alktype uses
//! `PackedLayout` offsets (built once) and
//! `data_access::write_u8`/`write_u32` at those offsets.
//! - **Packet** — a byte-offset-discriminator union
//! (`Read {handle, length}` / `Write {handle, length, data: bytes}`),
//! the SFTP-shaped case ADR-011's framing argument was about:
//! exercises `CompositePlan::Union` dispatch, variant walks, and
//! length-prefixed variable reads. The alktype consumer pattern is
//! the documented one: `read_next` on the root yields
//! `FieldValue::Union { discriminator, variant_start }`, the consumer
//! selects the pre-built reader for that variant and walks it over
//! `&buf[variant_start..]`.
//! - **validate_bytes** — `engine.validate_bytes` per buffer
//! (materialize + `ValidationPlan` walk, ADR-010/ADR-012 §3): the
//! read+validate-on-untrusted-stream shape `alkcall` cares about. No
//! hand comparator: a hand-rolled codec validates inline during the
//! (already measured) parse, while `validate_bytes` additionally
//! materializes a `Value` tree per buffer — the honest reading is the
//! absolute per-chunk cost.
//!
//! One-shots (paid once at startup, not per chunk):
//! `alktype_engine_compile` (dominated by BAST meta-schema
//! validation), `alktype_sequential_reader_new` (an `Arc::clone`),
//! `alktype_layout_build`.
//!
//! Two payload sizes (64 B, 4 KiB) so per-chunk fixed overhead is
//! visible separately from payload-copy cost.
//!
//! Run: `cargo bench --bench wire_vs_bast`
use criterion::{criterion_group, criterion_main, BenchmarkId, Criterion};
use std::hint::black_box;
use alktype::{
data_access, AlkTypeEngine, Endian, FieldValue, LayoutBuilder, LayoutMode, PackedLayout,
ReadPlan, SequentialReader,
};
/// Mirrors `alktty::wire::MAX_CHUNK_LEN` — the hand-rolled comparator
/// validates against the same cap the real codec enforces.
const MAX_CHUNK_LEN: u32 = 16 * 1024 * 1024;
/// The `ChunkHeader` BAST definition. The `StreamType` enum is
/// intentionally NOT used — BAST enums encode as `u32`, but the
/// on-wire `stream_type` is a `uint8`; both sides read it as `uint8`.
const CHUNK_HEADER_BAST: &str = r#"{
"$schema": "https://alk.dev/bast/v1/schema",
"$defs": {
"ChunkHeader": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "stream_type", "kind": "uint8" },
{ "name": "length", "kind": "uint32" }
]
}
}
}"#;
/// SFTP-shaped byte-discriminator union: one byte selects the variant,
/// `Write` carries a trailing length-prefixed `bytes` field. The root
/// struct wraps the union (`AlkTypeEngine::compile` requires a struct
/// root); mapping keys are the stringified `uint8` discriminator
/// values.
const PACKET_BAST: &str = r##"{
"$schema": "https://alk.dev/bast/v1/schema",
"$defs": {
"Packet": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "byte", "offset": 0, "type": "uint8" },
"mapping": {
"5": { "$ref": "#/$defs/Read" },
"6": { "$ref": "#/$defs/Write" }
}
},
"Read": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "handle", "kind": "uint32" },
{ "name": "length", "kind": "uint32" }
]
},
"Write": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "handle", "kind": "uint32" },
{ "name": "length", "kind": "uint32" },
{ "name": "data", "kind": "bytes" }
]
}
}
}"##;
// ---------------------------------------------------------------------------
// ChunkHeader fixtures
// ---------------------------------------------------------------------------
/// One chunk's worth of bytes on the wire: 5-byte header + payload.
fn make_chunk_bytes(stream_type: u8, payload: &[u8]) -> Vec<u8> {
let mut buf = Vec::with_capacity(5 + payload.len());
buf.push(stream_type);
buf.extend_from_slice(&(payload.len() as u32).to_be_bytes());
buf.extend_from_slice(payload);
buf
}
/// Concatenate `n` chunks into one buffer, each with `payload_len` bytes.
fn make_chunk_stream(n: usize, payload_len: usize) -> Vec<u8> {
let payload = vec![0xA5u8; payload_len];
let mut buf = Vec::with_capacity(n * (5 + payload_len));
for i in 0..n {
let st = (i % 5) as u8;
buf.extend_from_slice(&make_chunk_bytes(st, &payload));
}
buf
}
// ---------------------------------------------------------------------------
// Hand-rolled chunk read: mirrors ChunkReader::read_chunk_after_peek minus
// the tokio I/O. Returns (stream_type, payload) so the compiler can't
// elide the work. Validates stream_type <= 4 and length <= MAX_CHUNK_LEN.
// ---------------------------------------------------------------------------
#[inline]
fn hand_read_header(buf: &[u8]) -> Option<(u8, u32)> {
if buf.len() < 5 {
return None;
}
let stream_type = buf[0];
if stream_type > 4 {
return None;
}
let length = u32::from_be_bytes([buf[1], buf[2], buf[3], buf[4]]);
if length > MAX_CHUNK_LEN {
return None;
}
Some((stream_type, length))
}
#[inline]
fn hand_read_chunk(buf: &[u8]) -> Option<(u8, &[u8])> {
let (st, len) = hand_read_header(buf)?;
let end = 5usize.checked_add(len as usize)?;
if buf.len() < end {
return None;
}
Some((st, &buf[5..end]))
}
/// Drive `hand_read_chunk` across `n` contiguous chunks in `buf`.
/// Returns the total payload bytes consumed (so the loop body is
/// meaningfully used and not optimized away).
fn hand_read_stream(buf: &[u8], n: usize) -> usize {
let mut pos = 0usize;
let mut total = 0usize;
for _ in 0..n {
let (st, payload) = match hand_read_chunk(&buf[pos..]) {
Some(v) => v,
None => break,
};
total += payload.len();
pos += 5 + payload.len();
black_box(st);
}
black_box(total)
}
// ---------------------------------------------------------------------------
// alktype chunk read: SequentialReader over ChunkHeader. The reader is
// constructed once per benchmark group and reset() between chunks. After
// the header read, the payload is sliced at the parsed length — same as
// the hand-rolled path. We do NOT re-read a length prefix for the payload
// (that would be the double-prefix problem).
// ---------------------------------------------------------------------------
fn alktype_read_stream(buf: &[u8], n: usize, reader: &mut SequentialReader) -> usize {
let mut pos = 0usize;
let mut total = 0usize;
for _ in 0..n {
reader.reset();
let st = match reader.read_next_borrowed(&buf[pos..]) {
Ok(Some((_, FieldValue::U8(v)))) => v,
_ => break,
};
let len = match reader.read_next_borrowed(&buf[pos..]) {
Ok(Some((_, FieldValue::U32(v)))) => v,
_ => break,
};
if len > MAX_CHUNK_LEN {
break;
}
let end = match 5usize.checked_add(len as usize) {
Some(e) if e <= buf.len() - pos => e,
_ => break,
};
let payload = &buf[pos + 5..pos + end];
total += payload.len();
pos += end;
black_box(st);
black_box(payload.as_ptr());
}
black_box(total)
}
// ---------------------------------------------------------------------------
// Hand-rolled chunk write: mirrors ChunkWriter::write_chunk's header
// writes into a caller-provided buffer. Writes `n` contiguous chunks.
// ---------------------------------------------------------------------------
fn hand_write_stream(out: &mut Vec<u8>, n: usize, payload_len: usize) {
let payload = vec![0xA5u8; payload_len];
for i in 0..n {
let st = (i % 5) as u8;
let start = out.len();
out.resize(start + 5 + payload_len, 0);
out[start] = st;
out[start + 1..start + 5].copy_from_slice(&(payload_len as u32).to_be_bytes());
out[start + 5..start + 5 + payload_len].copy_from_slice(&payload);
}
black_box(out.len());
}
// ---------------------------------------------------------------------------
// alktype chunk write: data_access::write_u8 / write_u32 at the
// PackedLayout offsets. The layout is built once per group and reused.
// Payload bytes are copied with the same slice copy as the hand-rolled
// path so the comparison isolates the header-encoding overhead.
// ---------------------------------------------------------------------------
fn alktype_write_stream(out: &mut Vec<u8>, n: usize, payload_len: usize, layout: &PackedLayout) {
let payload = vec![0xA5u8; payload_len];
let st_pos = layout.get("stream_type").expect("stream_type field").offset;
let len_pos = layout.get("length").expect("length field").offset;
for i in 0..n {
let start = out.len();
out.resize(start + 5 + payload_len, 0);
let _ = data_access::write_u8(out, start + st_pos, (i % 5) as u8, "stream_type");
let _ = data_access::write_u32(
out,
start + len_pos,
payload_len as u32,
"length",
Endian::Big,
);
out[start + 5..start + 5 + payload_len].copy_from_slice(&payload);
}
black_box(out.len());
}
// ---------------------------------------------------------------------------
// Packet fixtures: byte-disc union stream, alternating Read/Write
// variants. Wire layout per packet (packed, big-endian):
// Read: disc(1) + handle(4) + length(4) = 9 bytes
// Write: disc(1) + handle(4) + length(4) + len(4)+data = 13 + payload
// ---------------------------------------------------------------------------
fn make_packet_bytes(disc: u8, payload: &[u8]) -> Vec<u8> {
let mut buf = Vec::with_capacity(13 + 4 + payload.len());
buf.push(disc);
buf.extend_from_slice(&0x0102_0304u32.to_be_bytes());
buf.extend_from_slice(&(payload.len() as u32).to_be_bytes());
if disc == 6 {
buf.extend_from_slice(&(payload.len() as u32).to_be_bytes());
buf.extend_from_slice(payload);
}
buf
}
fn make_packet_stream(n: usize, payload_len: usize) -> Vec<u8> {
let payload = vec![0xA5u8; payload_len];
let mut buf = Vec::new();
for i in 0..n {
let disc = if i % 2 == 0 { 5u8 } else { 6u8 };
buf.extend_from_slice(&make_packet_bytes(disc, &payload));
}
buf
}
// ---------------------------------------------------------------------------
// Hand-rolled packet read: read the discriminator byte, match the
// variant, parse its fields directly. Returns bytes consumed.
// ---------------------------------------------------------------------------
fn hand_read_packet(buf: &[u8]) -> Option<usize> {
let disc = *buf.first()?;
match disc {
5 => {
if buf.len() < 9 {
return None;
}
let handle = u32::from_be_bytes(buf[1..5].try_into().ok()?);
let length = u32::from_be_bytes(buf[5..9].try_into().ok()?);
black_box((handle, length));
Some(9)
}
6 => {
if buf.len() < 13 {
return None;
}
let handle = u32::from_be_bytes(buf[1..5].try_into().ok()?);
let length = u32::from_be_bytes(buf[5..9].try_into().ok()?);
let data_len = u32::from_be_bytes(buf[9..13].try_into().ok()?);
let end = 13usize.checked_add(data_len as usize)?;
if buf.len() < end {
return None;
}
black_box((handle, length));
black_box(&buf[13..end].as_ptr());
Some(end)
}
_ => None,
}
}
fn hand_read_packet_stream(buf: &[u8], n: usize) -> usize {
let mut pos = 0usize;
let mut total = 0usize;
for _ in 0..n {
let Some(consumed) = hand_read_packet(&buf[pos..]) else {
break;
};
total += consumed;
pos += consumed;
}
black_box(total)
}
// ---------------------------------------------------------------------------
// alktype packet read: the documented union consumer contract. The root
// reader walks the wrapping struct; `read_next` returns
// `FieldValue::Union { discriminator, variant_start }`; the consumer
// selects the pre-built reader for that variant and walks it over
// `&buf[pos + variant_start..]` until exhausted.
// ---------------------------------------------------------------------------
/// Walk one variant's fields to exhaustion; returns bytes consumed.
/// Uses `read_next_borrowed` — the zero-alloc hot-loop pattern for
/// consumers that match or discard the field name.
fn alktype_walk_variant(reader: &mut SequentialReader, buf: &[u8]) -> Option<usize> {
reader.reset();
loop {
match reader.read_next_borrowed(buf) {
Ok(Some((name, value))) => {
black_box(name);
black_box(&value);
}
Ok(None) => return Some(reader.position()),
Err(_) => return None,
}
}
}
fn alktype_read_packet_stream(
buf: &[u8],
n: usize,
packet: &mut SequentialReader,
read: &mut SequentialReader,
write: &mut SequentialReader,
) -> usize {
let mut pos = 0usize;
let mut total = 0usize;
for _ in 0..n {
packet.reset();
let disc = match packet.read_next_borrowed(&buf[pos..]) {
Ok(Some((_, FieldValue::Union {
discriminator,
variant_start,
}))) => {
pos += variant_start;
discriminator
}
_ => break,
};
let vbuf = &buf[pos..];
let consumed = match disc.as_str() {
"5" => alktype_walk_variant(read, vbuf),
"6" => alktype_walk_variant(write, vbuf),
_ => break,
};
let Some(consumed) = consumed else {
break;
};
total += consumed;
pos += consumed;
}
black_box(total)
}
/// One-time sanity check (outside the measured loops): the union
/// consumer pattern the stream loop relies on — root reader reports the
/// mapping key and the variant start; the variant reader's walk to
/// exhaustion reports exactly the variant's byte size, so
/// `variant_start + consumed` lands on the next packet.
fn assert_packet_reader_parity(
payload_len: usize,
packet: &mut SequentialReader,
read: &mut SequentialReader,
write: &mut SequentialReader,
) {
let payload = vec![0u8; payload_len];
let read_pkt = make_packet_bytes(5, &payload);
packet.reset();
match packet.read_next_borrowed(&read_pkt) {
Ok(Some((_, FieldValue::Union {
discriminator,
variant_start,
}))) => {
assert_eq!(discriminator, "5");
assert_eq!(variant_start, 1, "variant starts after the 1-byte disc");
}
_ => panic!("expected union value for Read packet"),
}
let consumed = alktype_walk_variant(read, &read_pkt[1..]).expect("read variant walk");
assert_eq!(consumed, 8, "Read = handle(4) + length(4)");
assert_eq!(1 + consumed, read_pkt.len(), "Read packet fully consumed");
let write_pkt = make_packet_bytes(6, &payload);
packet.reset();
match packet.read_next_borrowed(&write_pkt) {
Ok(Some((_, FieldValue::Union {
discriminator,
variant_start,
}))) => {
assert_eq!(discriminator, "6");
assert_eq!(variant_start, 1);
}
_ => panic!("expected union value for Write packet"),
}
let consumed = alktype_walk_variant(write, &write_pkt[1..]).expect("write variant walk");
assert_eq!(
consumed,
12 + payload_len,
"Write = handle(4) + length(4) + len-prefix(4) + data"
);
assert_eq!(1 + consumed, write_pkt.len(), "Write packet fully consumed");
}
// ---------------------------------------------------------------------------
// Benchmarks
// ---------------------------------------------------------------------------
fn bench_read(c: &mut Criterion) {
let bast: serde_json::Value = serde_json::from_str(CHUNK_HEADER_BAST).expect("bast json");
let engine =
AlkTypeEngine::compile(&bast, "ChunkHeader", LayoutMode::Packed, None).expect("compile");
let mut reader = engine.sequential_reader().expect("packed reader");
let mut group = c.benchmark_group("read_chunk_stream");
for (payload_len, label) in [(64usize, "p64"), (4096usize, "p4k")] {
let n = 1024;
let buf = make_chunk_stream(n, payload_len);
group.bench_with_input(BenchmarkId::new("hand_rolled", label), &n, |b, &n| {
b.iter(|| hand_read_stream(black_box(&buf), n));
});
group.bench_with_input(BenchmarkId::new("alktype", label), &n, |b, &n| {
b.iter(|| alktype_read_stream(black_box(&buf), n, &mut reader));
});
}
group.finish();
}
fn bench_write(c: &mut Criterion) {
let bast: serde_json::Value = serde_json::from_str(CHUNK_HEADER_BAST).expect("bast json");
let builder = LayoutBuilder::new(&bast, "ChunkHeader").expect("builder");
let layout = builder
.build(&std::collections::HashMap::new())
.expect("layout");
let mut group = c.benchmark_group("write_chunk_stream");
for (payload_len, label) in [(64usize, "p64"), (4096usize, "p4k")] {
let n = 1024;
group.bench_with_input(
BenchmarkId::new("hand_rolled", label),
&(n, payload_len),
|b, &(n, pl)| {
b.iter(|| {
let mut out = Vec::with_capacity(n * (5 + pl));
hand_write_stream(&mut out, n, pl);
});
},
);
group.bench_with_input(
BenchmarkId::new("alktype", label),
&(n, payload_len),
|b, &(n, pl)| {
b.iter(|| {
let mut out = Vec::with_capacity(n * (5 + pl));
alktype_write_stream(&mut out, n, pl, &layout);
});
},
);
}
group.finish();
}
fn bench_packet_read(c: &mut Criterion) {
let bast: serde_json::Value = serde_json::from_str(PACKET_BAST).expect("bast json");
let engine =
AlkTypeEngine::compile(&bast, "Packet", LayoutMode::Packed, None).expect("compile");
let mut packet_reader = engine.sequential_reader().expect("packed reader");
let read_plan = std::sync::Arc::new(ReadPlan::compile(&bast, "Read").expect("read plan"));
let write_plan = std::sync::Arc::new(ReadPlan::compile(&bast, "Write").expect("write plan"));
let mut read_reader = SequentialReader::new(read_plan);
let mut write_reader = SequentialReader::new(write_plan);
// One-time parity check of the union consumer pattern (not measured).
assert_packet_reader_parity(64, &mut packet_reader, &mut read_reader, &mut write_reader);
let mut group = c.benchmark_group("read_packet_stream");
for (payload_len, label) in [(64usize, "p64"), (4096usize, "p4k")] {
let n = 1024;
let buf = make_packet_stream(n, payload_len);
group.bench_with_input(BenchmarkId::new("hand_rolled", label), &n, |b, &n| {
b.iter(|| hand_read_packet_stream(black_box(&buf), n));
});
group.bench_with_input(BenchmarkId::new("alktype", label), &n, |b, &n| {
b.iter(|| {
alktype_read_packet_stream(
black_box(&buf),
n,
&mut packet_reader,
&mut read_reader,
&mut write_reader,
)
});
});
}
group.finish();
}
fn bench_validate(c: &mut Criterion) {
let header_bast: serde_json::Value =
serde_json::from_str(CHUNK_HEADER_BAST).expect("bast json");
let header_engine = AlkTypeEngine::compile(&header_bast, "ChunkHeader", LayoutMode::Packed, None)
.expect("compile");
let packet_bast: serde_json::Value = serde_json::from_str(PACKET_BAST).expect("bast json");
let packet_engine =
AlkTypeEngine::compile(&packet_bast, "Packet", LayoutMode::Packed, None).expect("compile");
let mut group = c.benchmark_group("validate_stream");
let n = 1024;
let headers: Vec<Vec<u8>> = (0..n)
.map(|i| make_chunk_bytes((i % 5) as u8, &[0xA5u8; 64]))
.collect();
group.bench_function("alktype_chunk_header", |b| {
b.iter(|| {
for h in &headers {
header_engine.validate_bytes(black_box(h)).expect("validate");
}
})
});
for (payload_len, label) in [(64usize, "p64"), (4096usize, "p4k")] {
let payload = vec![0xA5u8; payload_len];
let packets: Vec<Vec<u8>> = (0..n)
.map(|i| make_packet_bytes(if i % 2 == 0 { 5 } else { 6 }, &payload))
.collect();
group.bench_with_input(
BenchmarkId::new("alktype_packet", label),
&packets,
|b, packets| {
b.iter(|| {
for p in packets {
packet_engine.validate_bytes(black_box(p)).expect("validate");
}
})
},
);
}
group.finish();
}
/// One-shot costs paid once at startup, not per chunk.
fn bench_oneshot(c: &mut Criterion) {
let bast: serde_json::Value = serde_json::from_str(CHUNK_HEADER_BAST).expect("bast json");
c.bench_function("alktype_engine_compile", |b| {
b.iter(|| {
let _ =
AlkTypeEngine::compile(black_box(&bast), "ChunkHeader", LayoutMode::Packed, None)
.expect("compile");
});
});
c.bench_function("alktype_sequential_reader_new", |b| {
let engine = AlkTypeEngine::compile(&bast, "ChunkHeader", LayoutMode::Packed, None)
.expect("compile");
b.iter(|| engine.sequential_reader());
});
c.bench_function("alktype_layout_build", |b| {
let builder = LayoutBuilder::new(&bast, "ChunkHeader").expect("builder");
b.iter(|| builder.build(&std::collections::HashMap::new()));
});
}
criterion_group!(
benches,
bench_read,
bench_write,
bench_packet_read,
bench_validate,
bench_oneshot
);
criterion_main!(benches);
docs/architecture/README.md
+2
View File
@@ -45,6 +45,8 @@ format definition; the engine is generic.
| [008](decisions/008-reject-tunion-in-aligned-mode.md) | Reject TUnion in Aligned Mode for v1 | Unions are the protocol pattern; aligned-mode union semantics were broken |
| [009](decisions/009-builder-api.md) | Builder API for Schema Construction | Fluent Rust API producing `serde_json::Value`; covers BAST kinds + standard JSON Schema; resolves OQ-003. *Output format amended to BAST / standard JSON Schema by ADR-BAST.* |
| [010](decisions/010-generalized-validation-validate-bytes.md) | Generalized Validation — `validate_bytes` on `AlkTypeEngine` | Single-call binary-buffer validation; materialize `Value` from bytes, then validate. *Validation step amended to the BAST-native validator by ADR-VAL-SPLIT.* |
| [011](decisions/011-compiled-read-plan-for-packed-mode.md) | Compiled Read Plan for Packed Mode | `ReadPlan` — the packed read-side compiled form, symmetric to `OffsetMap` (aligned) and `PackedLayout` (packed write). Closes review #004's 400x read-path gap; retires ADR-007's "re-parse on demand" framing. *Accepted — implemented in 0.3.0 (phases 1–2).* |
| [012](decisions/012-plan-fingerprinting-and-m1-closure.md) | Plan Fingerprinting, ValidationPlan, and Closing the Deferred M1 Sites in 0.3.0 | `ReadPlan`/`OffsetMap`/`ValidationPlan` `Hash + Eq` + `fingerprint()`; owned `BastDoc` (lifetime removal); `OffsetMap` carries `LeafMeta` to close the aligned-side M1 sites; `ValidationPlan` retires the interpretive `bast_validation` walk (review #005 M3 reversed the original deferral). Bundles with ADR-011 into one 0.3.0 breaking release. *Accepted — fully implemented in 0.3.0 (fingerprinting, owned `BastDoc`, `LeafMeta`, `ValidationPlan`).* |
## Relevant Open Questions
docs/architecture/bast-format.md
+39 -22
View File
@@ -128,6 +128,12 @@ These are different validators for different inputs.
"encoding": { "enum": ["length-prefixed", "offset-indirect"] },
"maxLength": { "type": "integer", "minimum": 0 }
},
"if": {
"properties": {
"kind": { "enum": ["string", "bytes"] }
}
},
"else": { "properties": { "maxLength": false } },
"required": ["name", "kind"],
"additionalProperties": false
},
@@ -269,8 +275,11 @@ These are different validators for different inputs.
- `align` (optional): field-level alignment (aligned mode only).
- `encoding` (optional): `"length-prefixed"` (default) or
`"offset-indirect"`. See [Variable-length encoding](#variable-length-encoding).
- `maxLength` (optional): byte-length cap. See
[Variable-length encoding](#variable-length-encoding).
- `maxLength` (optional, `string`/`bytes` fields only): byte-length
cap. See [Variable-length encoding](#variable-length-encoding).
Rejected at parse on any other kind (review #006 N3: the annotation
was silently unenforced there — the validation plan bakes `maxLength`
into string/bytes leaves only).
### TypeRef
@@ -456,8 +465,11 @@ override). In little-endian mode, `u32::from_le_bytes`; in big-endian
mode, `u32::from_be_bytes`. Ensures SFTP consumers (big-endian) have
consistent byte order for field values and length prefixes.
Applies to all variable-length types: `string`, `bytes`,
`record`, and arrays of variable-length elements.
Applies to variable-length primitive types only: `string` and
`bytes`. The parser rejects `maxLength` (and the meta-schema forbids
it) on every other kind — including `record` (review #006 N3/M5: no
consumer honored it there, so the annotation was either silently
unenforced or, in aligned mode, silently corrupt).
## Endianness
@@ -528,24 +540,28 @@ materializer iterates the field list), types are correct (`read_u32`
produces `Value::Number`), bounds are checked (via `data_access::
check_bounds`), UTF-8 is valid (via `from_utf8`), the discriminator is
in the mapping, and the boolean byte is 0 or 1. What the materializer
does NOT check — and what the 19 v0.1.0 custom keyword validators check
afterward — are **value-domain constraints expressed in the BAST
document**. The BAST-native validator is a recursive walker over the
BAST type tree that checks exactly these:
does NOT check — and what the validation half checks afterward — are
**value-domain constraints expressed in the BAST document**. Under
ADR-012 §3 these constraints are compiled once into a `ValidationPlan`
at engine-compile time (eager `$ref` resolution, cyclic-graph
rejection); each `validate_bytes` call walks the compiled constraint
tree against the `Value`. The plan's nodes enforce exactly these
constraints (the set is normative; the walker that enforced it
interpretively in 0.2.0 is retired):
| Constraint | Validator arm |
|------------|---------------|
| Integer range (Int8..Uint64) | `validate_int`/`validate_uint` with `as_i64`/`as_u64` + range check |
| Int64/Uint64 (full range) | `validate_int64`/`validate_uint64` (JSON precision caveat per ADR-005) |
| Float finiteness (Float32/64) | `validate_float` with `as_f64().is_finite()` |
| String `maxLength` (byte length) | `check_string` reads the field-level `maxLength` |
| Bytes `maxLength` (array length) | `check_bytes` accepts the `Value::Array` form (the materializer emits bytes as an array of u8) |
| Enum index bounds | `validate_enum` checks `idx < values.len()` — **fixes the v0.1.0 dead constraint** |
| Union variant dispatch | `validate_union` reads `__discriminator`, looks up the variant, recurses via `validate_typeref` |
| Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses |
| Array count | `validate_array` checks `arr.len() == count` and recurses per element |
| Record values | `validate_record` recurses into each value's `values` type |
| Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) |
| Constraint | `ValidNode` arm |
|------------|-----------------|
| Integer range (Int8..Uint32) | `Int { min, max }` / `Uint { max }` with `as_i64`/`as_u64` + range check |
| Int64/Uint64 (full range) | `I64` / `U64` (JSON precision caveat per ADR-005) |
| Float finiteness (Float32/64) | `Float` with `as_f64().is_finite()` |
| String `maxLength` (byte length) | `Str { max_len }` — `maxLength` baked in from the owning field at compile time |
| Bytes `maxLength` (array length) | `Bytes { max_len }` — accepts the `Value::Array` form (the materializer emits bytes as an array of u8) |
| Enum index bounds | `Enum { count }` checks `idx < count` — **fixes the v0.1.0 dead constraint** |
| Union variant dispatch | `Union { variants }` reads `__discriminator`, dispatches on the compiled variant nodes |
| Struct fields | `Struct { fields }` requires each declared field present, recurses |
| Array count | `Array { count, element }` checks `arr.len() == count` and recurses per element |
| Record values | `Record { values }` recurses into each value |
| Boolean | `Bool` (materializer already rejects non-0/1 bytes) |
No external JSON Schema is required for `validate_bytes`. The BAST
document is the complete specification of the binary format — it
@@ -584,7 +600,8 @@ The `jsonschema` crate **remains a direct dependency** for
meta-schema. The only thing removed is the custom keyword integration
path. The `validate_bytes` path no longer touches `jsonschema` — a
small wasm binary-size win in addition to the architecture
simplification.
simplification. (Since ADR-012 §3, the interpretation step itself is
also compiled away: see the `ValidationPlan` above.)
### `AlkTypeError::Validation` payload shape
docs/architecture/decisions/003-schema-annotations.md
+10 -5
View File
@@ -136,9 +136,9 @@ reserving worst-case space.
- `true` is a shorthand for the default (length-prefixed). This keeps
the common case concise and the override explicit.
- The `encoding` annotation and `maxLength` apply to all variable-length
types: `AlkType:String`, `AlkType:Bytes`, `AlkType:Array`,
`AlkType:Record`, `AlkType:Timestamp`.
- The `encoding` annotation and `maxLength` apply to the variable-length
primitive types `AlkType:String` and `AlkType:Bytes`. (`maxLength` on
records was amended out by review #006 N3/M5 — see §3a.)
### 3a. TRecord value type
@@ -164,8 +164,13 @@ the `"values"` property in the schema:
the value's size is determined by its kind (fixed-size kinds have a
known size; variable-length kinds carry their own length prefix).
- The count and key-length prefixes respect the schema's endianness.
- In aligned static mode with `maxLength`, the entire record is reserved
at `maxLength` bytes (zero-padded).
- ~~In aligned static mode with `maxLength`, the entire record is
reserved at `maxLength` bytes (zero-padded).~~ **Amended (review #006
N3/M5, 2026-09-02):** `maxLength` is rejected at parse on record
fields. The aligned materializer walks the record's inline
count-prefixed form and never honors the reservation (M5: silent
cross-field corruption), and no packed consumer enforced it either
(N3: silently unenforced). `maxLength` is `string`/`bytes`-only.
### 4. TUnion discriminators
docs/architecture/decisions/007-packed-mode-read-factory.md
+20 -5
View File
@@ -63,11 +63,26 @@ write-side.
### Cost
`SequentialReader::new` clones the top-level struct's field schemas (a
`Vec<(String, Value)>` of the `properties` entries) and clones the
schema itself. This is cheap — a struct has a small number of fields
(SFTP's largest packet has 5). The construction cost is negligible
compared to the cost of reading a buffer.
`SequentialReader::new(Arc<ReadPlan>)` is a refcount bump — 15.7 ns
(measured, alktty `wire_vs_bast` bench, 0.3.0). The reader shares the
engine's compiled [`ReadPlan`](011-compiled-read-plan-for-packed-mode.md)
(the packed read-side compiled form) via `Arc` instead of cloning
schema data; construction cost is negligible compared to reading a
buffer. The engine holds the owned `BastDoc` (ADR-012 §2a) for the
aligned materialize path and the one-shot `*::compile` paths.
> **Historical note**: the original 0.2.0 framing here ("re-parse on
> demand" — the read loop re-parsing `BastDoc::new` per field) was the
> root cause of the 400x read-path gap measured in
> [review #004](../../reviews/004-performance-review.md).
> [ADR-011](011-compiled-read-plan-for-packed-mode.md) (implemented,
> 0.3.0) retired it: the packed read loop walks `Arc<ReadPlan>` (2.27
> µs/chunk → 98 ns/chunk), `sequential_reader()` is an `Arc::clone`,
> and the owned `BastDoc` (ADR-012 §2a) removed the remaining
> per-access re-parse sites in `read_field`/`write_field`/`validate_bytes`.
> The factory decision itself (`sequential_reader() ->
> Option<SequentialReader>`, owned fresh reader, consumer-driven
> cursor) was retained unchanged.
## Consequences
docs/architecture/decisions/011-compiled-read-plan-for-packed-mode.md
+584
View File
@@ -0,0 +1,584 @@
# ADR-011: Compiled Read Plan for Packed Mode
## Status
Accepted. Implemented in 0.3.0 (phases 1–2, 2026-09-02). Closes review
#004 H1 + M1 (packed side) + L1 + L2;
retires the "re-parse on demand" framing from ADR-007. A derisking
POC on branch `readplan-poc` confirmed the `ReadPlan` shape covers
every `BastType` arm in the current read loop before implementation
began (see "POC coverage" at the end).
**Refinements on ADR-012 acceptance (2026-08-20, review #005):** the
`CompositePlan::Union` shape was refined to carry `shared:
Option<Box<ReadPlan>>` (field-disc union shared fields, resolving POC
Finding 1 / review #005 H1) and to drop `VariantPlan`/`VariantKind`
in favor of `variants: Vec<(String, CompositePlan)>` (resolving review
#005 M1 — nested unions now work by `CompositePlan` recursion,
restoring the 0.2.0 capability the POC rejected). The "BastDoc
unchanged" scope statement stands as the ADR-011-only view; ADR-012
§2a subsequently makes `BastDoc` owned. The `ValidationPlan` this
ADR's "Out of scope" originally deferred indefinitely is now in
0.3.0 via ADR-012 §3 (review #005 M3 reversed the deferral). These
are pre-implementation refinements to types that do not yet exist on
`main`; the ADR-011 decision (a compiled `ReadPlan` for packed reads)
is unchanged.
**Addendum — field-disc union wire convention (2026-09-02, review #006
H3):** the packed-mode wire layout for a field-name-discriminator
TUnion is **shared-then-variant**: the union's declared `fields` (the
discriminator field + any shared fields) occupy the union's start
offset in declaration order, and the selected variant's fields follow
immediately after all shared fields. All three packed-mode consumers
now implement this one convention: the reader and materializer already
walked `shared` then the variant (the `shared` sub-plan shape above);
`LayoutBuilder` was corrected in the same pass — it previously laid out
only the selected variant, disagreeing with the read side on span and
field positions (review #006 H3 item 1). The convention requires that
a variant **must not re-declare** the discriminator field or any
shared field — `BastUnion::parse` enforces this at parse time (also:
the discriminator field must be declared in `fields`, and `fields`
must not contain duplicate names), so the shared walk and the variant
walk cover disjoint fields and the wire has exactly one copy of each
shared byte. Schemas whose variants redeclared shared fields were
ambiguous under the old split-convention behavior and are rejected
rather than given a silent meaning; this is a **breaking wire-format
constraint** for any 0.2.0-era schema that relied on re-declaration,
announced with the 0.3.x series. `DiscriminatorPlan::Field`'s disc
read is at the disc field's position within the shared walk (the
materializer's position-correct behavior, review #006 H3 item 2); the
reader's plan-walk reads it there too.
## Context
Review #004 (`docs/reviews/004-performance-review.md`) measured the
packed read path at **~400x slower per chunk** than a hand-rolled codec
(2.27 µs/chunk vs 5.6 ns/chunk), with the cost fixed across payload
sizes — the signature of per-field interpretive overhead, not
payload-copy overhead. The write path is competitive (~1.1x at 4 KiB)
because it uses a compiled form; the read path is not because it
doesn't.
### The three layout-side compiled forms and the one gap
ADR-002 defines two layout modes. Each mode has a write-side and a
read-side. Three of the four slots already have a **compiled form** —
a data structure built once from the schema, held by the engine, and
walked at access time without re-touching the schema:
| mode | write-side | read-side |
|------|-----------|-----------|
| aligned static | `OffsetMap` (used for both) | `OffsetMap` |
| packed sequential | `PackedLayout` (`LayoutBuilder::build`) | *(none)* |
- **`OffsetMap`** (aligned, both sides) — a flat table of
`(field_path, ByteRange)` pairs computed once via
`OffsetMap::compute(&BastDoc)`. Read and write both look up a field's
byte range and call `data_access::read_*`/`write_*` at the known
offset. No schema walk at access time.
- **`PackedLayout`** (packed, write-side) — a flat table of
`(field_path, FieldPosition)` pairs computed once via
`LayoutBuilder::build(&var_sizes)`. The write loop calls
`data_access::write_*` at the precomputed offsets. No schema walk at
write time.
- **packed read-side** — `SequentialReader` walks `BastDoc`
interpretively on every field read. There is no compiled form.
This is the structural reason the read path is 400x slow: it is the
only access path in the engine with no compiled form. Every other
mode/side pair compiles the schema once and reuses the result.
### Root cause: the `BastDoc<'a>` borrow constraint
`BastDoc<'a>` borrows `&'a Value` and `&'a str` throughout
(`src/bast.rs:51-55`). The owning structs that need a parsed tree at
read time — `SequentialReader` (owns a cloned `Value`),
`AlkTypeEngine` (owns `bast_doc: Value`), `LayoutBuilder` (owns
`doc_value: Value`) — cannot store a `BastDoc` that borrows from their
own `Value` field. That would be a self-referential struct, which safe
Rust cannot express.
The workaround chosen in ADR-007 was "re-parse on demand": the engine
and reader retain a clone of the raw `Value` and reconstruct the
`BastDoc` from it whenever the typed tree is needed. ADR-007's "Cost"
section argued this was cheap because construction is a small `Vec` of
field schemas. That is true for *construction* (once), but the decision
did not account for `read_field_at` re-parsing `BastDoc::new` **per
field read** — the cost that actually dominates. For an N-field struct,
reading all fields is O(N²) in parse work (each of N reads re-parses
all N fields).
### Why `OffsetMap` is a flat table but the packed read plan cannot be
`OffsetMap` works as a flat `(path, byte_range)` lookup table because
aligned positions are **data-independent** — field N's offset depends
only on the schema, not on the bytes of fields 0..N-1. Random access
by path is free.
Packed positions are **data-dependent** — a variable-length field's
extent is read from its length prefix at access time, and every
subsequent field's position shifts accordingly. You cannot look up
field N's offset without reading fields 0..N-1 first. So the compiled
form for packed reads cannot be a flat lookup table; it must be a
**read program** — a pre-resolved tree of read instructions that the
read loop walks in order, advancing a cursor. The schema is compiled
into the program once; the bytes are walked against it at read time.
This asymmetry is inherent to packed sequential layout (ADR-002) and is
not a flaw in `OffsetMap`. The two modes need different compiled-form
shapes because they have different position-computation semantics.
## Decision
**Introduce `ReadPlan` — the compiled read-side form for packed mode,
symmetric to `OffsetMap` (aligned read-side) and `PackedLayout` (packed
write-side).**
`AlkTypeEngine::compile` builds the `ReadPlan` once from the `BastDoc`
(in packed mode) and holds it for the life of the engine.
`sequential_reader()` hands out fresh `SequentialReader`s that share
the engine's `Arc<ReadPlan>` — the plan is immutable; only the cursor
state (`field_index`, `position`) is per-reader. The read loop walks
the plan, never touching `BastDoc` or the raw `Value`.
The same `ReadPlan` is consumed by `materialize_packed` (the other
byte-walking path), unifying the two packed read-side consumers on one
compiled form — mirroring how `OffsetMap` unifies the aligned read and
write sides.
### The `ReadPlan` shape
A pre-resolved tree of read instructions. Every `$ref` is resolved, every
endianness is computed (field override or container default), every
union variant is inlined. The read loop indexes into a `Vec`, matches
on a `ReadKind`, and calls `data_access::read_*` with a precomputed
`Endian` — no `resolve_typeref`, no `BastDef::parse`, no JSON node
access on the happy path. (`format!` for error-path attribution may
still occur on the error path; it does not run on the happy path and
is not the cost being removed here.)
```rust
pub struct ReadPlan {
endian: Endian,
fields: Vec<FieldPlan>,
by_name: HashMap<String, usize>,
}
pub struct FieldPlan {
name: String,
kind: ReadKind,
endian: Endian,
encoding: VariableEncoding,
max_length: Option<usize>,
body: Option<CompositePlan>,
}
pub enum ReadKind {
Primitive(AlkTypeKind),
Enum,
Struct,
Union,
Array,
Record,
}
pub enum CompositePlan {
Struct(ReadPlan),
Union {
disc: DiscriminatorPlan,
shared: Option<Box<ReadPlan>>,
variants: Vec<(String, CompositePlan)>,
},
Array {
element: Box<CompositePlan>,
count: usize,
element_stride: usize,
},
Record {
value: Box<CompositePlan>,
},
}
pub enum DiscriminatorPlan {
Byte { offset: usize, disc_type: AlkTypeKind },
Field { name: String, field_index: usize },
}
```
Nested structs share the `ReadPlan` shape (a struct field's `body` is
`CompositePlan::Struct(ReadPlan)`). Union variants are pre-resolved:
each `(key, CompositePlan)` entry carries the variant's compiled body,
so dispatch is a flat lookup + recurse — no `resolve_typeref_as_def` at
read time. A variant may itself be `CompositePlan::Union { ... }`, so
**nested unions** (a union variant that is itself a union, which the
0.2.0 reader supports via `resolve_and_walk_variant`'s `Union` arm) are
covered by ordinary recursion; no separate `VariantKind` enum is
needed. Array element strides are precomputed (`element_stride = 0`
signals variable-length elements, same convention as today's
`FieldValue::Array`).
`Union.shared` carries the union's declared `fields` (the discriminator
field + any shared fields) for the field-name-discriminator case —
`DiscriminatorPlan::Field.field_index` indexes into `shared`, and the
read loop walks `shared` first, then looks up and walks the selected
variant's `CompositePlan` starting after the shared fields. The
byte-offset-discriminator case has no shared fields (`shared: None`):
the discriminator byte is read at `disc.offset` and the variant starts
immediately after the discriminator size. The POC's `plan_read_union`
`Field` arm stub (Finding 1) is replaced by this `shared` sub-plan;
there is no separate `VariantPlan`/`VariantKind` type in the production
shape — the POC's `VariantPlan { kind, plan }` wrapper is dropped in
favor of recursing on `CompositePlan` directly, which is what makes
nested-union support fall out for free.
### Construction
```rust
impl ReadPlan {
pub fn compile(bast_doc: &Value, root_name: &str) -> Result<Self, AlkTypeError>;
}
```
`compile` walks `BastDoc` once, resolves all `$ref`s eagerly, computes
effective endianness at every node, inlines union variants, and
builds the `FieldPlan`/`CompositePlan` tree. Malformed schemas surface
as `AlkTypeError::Schema` — the same untrusted-input discipline
(AGENTS.md §3) and overflow-safe arithmetic (AGENTS.md §4) as
`BastDoc::new`.
### Engine integration
`AlkTypeEngine::compile` builds the `ReadPlan` in packed mode and
stores `Arc<ReadPlan>`. `sequential_reader()` returns
`SequentialReader { plan: Arc::clone(&self.plan), .. }` — an owned
reader (ADR-007's factory decision is retained; the reader owns its
cursor, shares the immutable plan).
`bast_doc: Value` (`src/engine.rs:85`) is retained on the engine
unconditionally. In packed mode it becomes unused by the read path
(both `sequential_reader` and `validate_bytes` consume the plan);
in aligned mode it is still needed for `read_field`/`write_field`/
`validate_bytes`. Keeping it always avoids a mode-conditional field
and costs only a `Value` clone paid once at `compile`. `ReadPlan`
must be `Send + Sync` so `Arc<ReadPlan>` can be shared from the
`Send + Sync` engine (ADR-007); this falls out naturally from the
plan being immutable owned data, but the implementation should add a
`static` bound assertion test to lock it in.
### Public API change (breaking — version bump to 0.3.0)
- `SequentialReader::new(&Value, &str)` → `SequentialReader::new(Arc<ReadPlan>)`.
The old constructor is replaced by `ReadPlan::compile(&Value, &str)`
followed by `SequentialReader::new(Arc::from(plan))`.
- `materialize_packed(&BastDoc<'_>, &[u8])` →
`materialize_packed(&ReadPlan, &[u8])`.
- `materialize_aligned` is unchanged (already takes `&OffsetMap`, a
compiled form).
- `ReadPlan` is a new public type, re-exported from `lib.rs`.
- `BastDoc` and the `Bast*` types are **unchanged by this ADR** — they
remain the validation-side typed tree, borrowed, as today. This is a
smaller breakage than review #004's Option A (which changed
`BastDoc<'a>` → `BastDoc` and every `Bast*` signature).
**Note (added on ADR-012 acceptance):** ADR-012 §2a subsequently
makes `BastDoc` owned, riding the same 0.3.0 bump. That is an
ADR-012 change, not an ADR-011 change; ADR-011's scope statement
stands as the ADR-011-only view. With ADR-012 §3 (ValidationPlan,
now in 0.3.0), `bast_validation` will also stop being the permanent
home of the `BastDoc` walk — see ADR-012.
The crate is pre-1.0 with two in-house downstream consumers
(`alktty`, `alkcall`), both of which will be updated with the bump.
## Scope
### In scope (consumes the `ReadPlan`)
- **`SequentialReader`** — the read loop walks `FieldPlan`/`CompositePlan`
instead of `&BastField`/`&BastType` + `&BastDoc`. The functions
`read_field_value`, `read_typeref_value`, `walk_struct_size`,
`read_union_value`, `read_array_value`, `read_record_value` are
rewritten to take plan nodes. One walker, not two — the review's
Option B concern ("duplicates the `BastType` matching logic") does
not apply because the plan *replaces* the `BastType` matching, not
parallels it.
- **`materialize` (packed mode)** — `materialize_packed` takes
`&ReadPlan` and walks it to produce `serde_json::Value`. Same read
logic, same `data_access` calls, different input type. Unifies the
two packed read-side consumers on one compiled form.
- **`AlkTypeEngine::validate_bytes` (packed mode)** — calls
`materialize_packed(&self.plan, buffer)` instead of reconstructing a
`BastDoc`. Closes M1's `engine.rs:284` re-parse.
### Out of scope (stays on `BastDoc`)
- **`bast_validation`** — the BAST-native value-domain validator walks
`BastDoc` to check constraints (`maxLength`, enum string values,
union variant keys, integer ranges). These are value-domain checks,
not byte-position walks; they don't benefit from a *read* plan and
would require a separate "validation plan" with a different shape.
**Not in scope for ADR-011** — but no longer deferred indefinitely:
ADR-012 §3 brings a `ValidationPlan` into 0.3.0. The "not a hot
loop" framing this paragraph originally relied on was re-evaluated
and rejected (see ADR-012 §3): read+validate on untrusted streams
makes validation hot in the same sense review #004 measured for
the read path. For 0.3.0 as accepted by ADR-011 alone,
`bast_validation` keeps walking `BastDoc`; ADR-012 §3 closes that.
- **`LayoutBuilder` / `PackedLayout`** — the packed write-side already
has a compiled form (`PackedLayout`). `LayoutBuilder::build`
(`src/layout_builder.rs:190`) re-parses `BastDoc::new` per `build()`
call (M1), but the typical pattern is build-once-reuse, so this is
not a hot loop. A future `WritePlan` that lets `LayoutBuilder` cache
the typed tree (review #004 Option A's territory) is additive and can
follow; it is not blocking the read-path fix.
- **`OffsetMap` / aligned mode** — already a compiled form; unchanged.
`materialize_aligned` already takes `&OffsetMap`.
- **Aligned-mode `read_field` / `write_field`** (`engine.rs:334,467`)
re-parse `BastDoc::new` per call (M1). These are one-shot paths, not
hot loops; they can adopt a compiled form later without affecting
the packed read-path decision. Left as-is for now.
## Consequences
### Positive
- **Closes the 400x read-path gap (review #004 H1).** The read loop no
longer touches `BastDoc` or the raw `Value`. Per-field work drops
from "re-parse the typed tree + resolve_typeref + match" to "index
into a `Vec` + match `ReadKind` + `data_access::read_*` with a
precomputed `Endian`." The expected per-chunk cost is in the
hand-rolled codec's ballpark (the `data_access` calls are the same
ones the hand-rolled codec makes).
- **Closes M1's packed-side re-parse (`engine.rs:284`, `validate_bytes`).**
The aligned-side M1 sites (`engine.rs:334,467`,
`layout_builder.rs:190`) are deliberately left as-is — they are not
hot for the packed-codec use case, and Option A remains available as
an additive later fix if an aligned-mode hot loop ever emerges. This
is a reversible bet, not a claim that the aligned-side M1 is a
non-issue.
- **Unifies the two packed read-side consumers on one compiled form.**
`SequentialReader` and `materialize_packed` walk the same `ReadPlan`,
mirroring how `OffsetMap` unifies the aligned read and write sides.
The "two parallel walkers" concern from review #004 Option B does
not apply — the plan replaces the `BastType` matching, not
duplicates it.
- **Symmetric with the other compiled forms.** The engine now has a
compiled form for every mode/side pair: `OffsetMap` (aligned R/W),
`PackedLayout` (packed W), `ReadPlan` (packed R). The "compiled form
of a BAST document" framing in ADR-004/validation.md becomes true for
the read path, not just the write path.
- **ADR-007's factory gets cheaper.** Today
`sequential_reader()` clones `doc_value: Value` (the whole BAST
document) per reader. With the plan, it clones an `Arc<ReadPlan>`
(refcount bump). The plan is immutable and shared across all readers
from one engine. ADR-007's "owned fresh reader" decision is retained;
the reader owns its cursor, shares the plan.
- **Deterministic compile.** `ReadPlan::compile` is a pure function of
the BAST document + root name — same input, same plan. This makes the
"compiled form" visibly deterministic, which is a prerequisite for
future capabilities (fingerprinting the plan for cross-run caching,
disk-cached compiled plans, or schema-version handshakes for
`alkcall`'s hub/spoke topology). Not implemented in this ADR and not
needed to justify the decision; listed here only so a future ADR
doesn't re-derive the prerequisite. See "Future capabilities" below.
- **Retires the "re-parse on demand" framing (L2).** ADR-007's "Cost"
section and `src/engine.rs:112-115`'s doc comment framed re-parse as
the intended design. With the plan, the read path never re-parses;
the framing is retired. ADR-007's "Cost" section and the doc comment
are updated in the same commit.
- **L1 falls out.** The dead `_field_schema: &Value` parameter and the
`Vec<(String, Value)>` field storage (where the `Value` half is
unused) are replaced by `Vec<FieldPlan>`. No dead `Value` clones.
### Negative
- **Breaking public-API change (0.2.0 → 0.3.0).** `SequentialReader::new`
and `materialize_packed` change signatures (take `ReadPlan` instead
of `&Value`/`&BastDoc`). `ReadPlan` is a new public type. Per
AGENTS.md, this is semver-relevant. The crate is pre-1.0 with two
in-house downstream consumers, both updated with the bump. The
breakage is smaller than review #004's Option A (no `Bast*` type
changes — `BastDoc` stays borrowed, stays the validation-side tree).
- **A parallel typed tree, not a flat lookup table.** This is the
honest cost. `OffsetMap` and `PackedLayout` are flat `(path, range)`/
`(path, position)` projections of `BastType`; `ReadPlan` is a full
parallel hierarchy (`CompositePlan` mirrors `BastType`'s
Struct/Union/Array/Record). The maintenance tax is real and higher
than those: when schema semantics change, `BastDoc`/`BastType` and
`ReadPlan`/`CompositePlan` move together. It is worth it because the
perf win on composite-heavy schemas (the SFTP-shaped union-with-`$ref`
-variants packet) justifies it — see the next bullet. This is a
permanent tax accepted in exchange for a ~20–50x composite-dispatch
win on top of the 400x re-parse fix, not a structural symmetry with
the flat compiled forms.
- **Eager `$ref` resolution at compile time.** `ReadPlan::compile`
resolves all `$ref`s eagerly, including union variant refs. This is
correct (the schema is fixed at compile time) and matches the
review's Option B design, but it means a schema with a `$ref` cycle
(which BAST forbids — refs are always `#/$defs/<name>`, no
recursion) would loop forever. The meta-schema (`bast_meta`)
already rejects recursive schemas; `ReadPlan::compile` inherits
that guard. No new failure mode.
- **Nested `Box<CompositePlan>` vs flat `Vec<Op>`.** The plan as
specified uses nested `Box`es — idiomatic, debuggable, easy to
build. A flat `Vec<Op>` with jump indices (true "bytecode") would be
more cache-friendly but harder to build and read. Protocol headers
are small N (SFTP's largest packet has 5 fields); the perf win is
eliminating the re-parse and `resolve_typeref`, not SoA cache
effects. Start nested; go flat only if a bench says otherwise (a
two-way door — the plan is a private internal type; its shape can
change without a semver bump as long as the public `ReadPlan` name
and `compile`/`SequentialReader::new` signatures are stable).
## Scope Boundaries (What This Is Not)
- **Not a `BastDoc` replacement.** `BastDoc` stays as the
validation-side typed tree within ADR-011's scope (borrowed from
`&Value`, unchanged). The `Bast*` types and their signatures are not
touched by ADR-011. Validation (`bast_validation`), aligned one-shot
reads/writes (`engine.rs:334,467`), and `LayoutBuilder::build`
continue to walk `BastDoc` within ADR-011's scope. **ADR-012
subsequently revises two of these:** `BastDoc` becomes owned (§2a)
and `bast_validation` adopts a `ValidationPlan` (§3), both riding
the same 0.3.0 bump. ADR-011's scope statement is the ADR-011-only
view and is not re-litigated here.
- **Not a flat lookup table.** Packed positions are data-dependent;
the plan is a read program (instructions to walk), not a `(path,
offset)` table. This is inherent to packed sequential layout
(ADR-002), not a limitation of this design.
- **Not a validation plan.** `bast_validation`'s value-domain checks
(maxLength, enum values, union variant keys, integer ranges) are a
different concern and a different shape from `ReadPlan`. They are
out of ADR-011's scope; ADR-012 §3 adds a `ValidationPlan` in 0.3.0
rather than leaving validation on an interpretive `BastDoc` walk
indefinitely.
- **Not the review's Option A or Option B.** It is the "compiled form"
path the review pointed at but did not name: Option A (make `BastDoc`
own its data) kills the re-parse but leaves the read loop as a
`BastDoc` tree walk; Option B (precompute an owned read plan in
`SequentialReader::new`) is the surgical subset that closes H1 only.
This ADR is the principled version of B — a public `ReadPlan` built
at `compile` time, shared across readers and `materialize`, symmetric
with `OffsetMap`/`PackedLayout` — and it closes H1 + M1 (packed side)
+ L1 + L2.
## Recommended Order
1. **`ReadPlan` type + `compile`** — the `ReadPlan`/`FieldPlan`/
`CompositePlan`/`ReadKind`/`DiscriminatorPlan` types and the
`ReadPlan::compile(&Value, &str)` constructor. Pure addition; no
existing code touched. Unit-tested against the same BAST fixtures
the `BastDoc` tests use.
2. **`SequentialReader` rewrite** — the read loop walks `&ReadPlan`
instead of reconstructing `BastDoc`. `read_field_value`,
`read_typeref_value`, `walk_struct_size`, `read_union_value`,
`read_array_value`, `read_record_value` take plan nodes. The
existing `sequential_reader.rs` tests (which drive `read_next`/
`read_field`/`reset` over real buffers) pass unchanged — they
exercise the read path through the public API, so they validate
the rewrite without modification.
3. **`materialize_packed` rewrite** — takes `&ReadPlan`, walks the
plan to produce `Value`. The existing `validate_bytes` (packed)
tests cover it end-to-end.
4. **`AlkTypeEngine::compile` integration** — builds `Arc<ReadPlan>`
in packed mode, stores it, `sequential_reader()` hands out
`Arc::clone(&self.plan)`. `validate_bytes` (packed) calls
`materialize_packed(&self.plan, buffer)`.
5. **L2 — retire the "re-parse on demand" framing.** Update
ADR-007's "Cost" section (replace the "re-parse on demand"
paragraph with the `Arc<ReadPlan>` cost) and the
`src/engine.rs:112-115` doc comment. ADR-007's status block stays
"Accepted" for the factory decision; only the cost framing changes.
6. **Public API bump (0.2.0 → 0.3.0).** `lib.rs` re-exports `ReadPlan`;
`SequentialReader::new` and `materialize_packed` signatures change.
Update `alktty`/`alkcall` in the same commit.
7. **Verification block.** `cargo test --release`,
`cargo clippy --all-targets -- -D warnings`, `cargo doc --no-deps`
(new public type), `cargo build --target wasm32-unknown-unknown
--release` (the plan touches `sequential_reader.rs` and
`materialize.rs`, both wasm-relevant). Re-run the `alktty`
`wire_vs_bast` bench to confirm the 400x gap closes.
Steps 1–2 close H1. Step 3 closes the `materialize` half of M1.
Step 4 closes the `validate_bytes` half of M1 (packed side). Step 5
closes L2. L1 falls out at step 2. The aligned-side M1 paths
(`engine.rs:334,467`, `layout_builder.rs:190`) are left as-is per
"Out of scope."
## References
- [Review #004](../../reviews/004-performance-review.md) — the
performance finding (H1, M1, L1, L2) and the three fix options this
ADR supersedes
- [ADR-002](002-two-layout-modes-packed-vs-aligned.md) — the two
layout modes; `ReadPlan` is the packed read-side compiled form that
this ADR adds to the table
- [ADR-007](007-packed-mode-read-factory.md) — the engine as
`SequentialReader` factory; retained (owned fresh reader), with the
"re-parse on demand" framing retired (L2)
- [ADR-004](004-error-handling-validation-strategy.md) — `AlkTypeError`,
load-time build / access-time check, field-path-carrying errors;
`ReadPlan::compile` is a load-time build, the read loop is an
access-time check
- [ADR-010](010-generalized-validation-validate-bytes.md) —
`validate_bytes` (packed) consumes the `ReadPlan` via
`materialize_packed`
## Future capabilities (in 0.3.0 via ADR-012)
The deterministic-compile property of `ReadPlan` is a prerequisite for
several capabilities. ADR-012 ("Plan Fingerprinting, ValidationPlan,
and Closing the Deferred M1 Sites in 0.3.0") picks up all three items
below into the 0.3.0 release so they ship with this ADR's breaking
changes in one round of downstream churn, not two or three:
- **Fingerprinting the plan** (`#[derive(Hash)]` + a `fingerprint()`
method) for cross-run caching of compiled plans, disk-cached plans,
and `alkcall` schema-version handshakes. → **In 0.3.0 (ADR-012 §1).**
- **Closing the deferred M1 sites** via an owned `BastDoc` (lifetime
removal) for `LayoutBuilder` + extending `OffsetMap` with leaf
metadata for the aligned `read_field`/`write_field` paths. → **In
0.3.0 (ADR-012 §2).** Note: ADR-012 reframes the earlier "WritePlan"
candidate listed here as "not a new type — extend the existing
compiled forms (`PackedLayout`/`OffsetMap`) and cache the parse."
- A `ValidationPlan` that follows the same compile-once-walk-many
pattern for `bast_validation`. → **In 0.3.0 (ADR-012 §3).** Different
shape (value-domain, not byte-position) but the same class of
per-buffer re-walk cost on the read+validate-on-untrusted-input
common case. ADR-012 owns the shape decision and the implementation
plan scopes it. (Originally deferred by ADR-012 as "not a hot loop";
review #005 M3 reversed the deferral — see ADR-012 §3.)
None of the in-0.3.0 items justify this ADR; the 400x read-path gap
does. They are listed here as forward references and to record that
the "WritePlan" candidate has been reframed out by ADR-012.
## POC coverage
Before this ADR was accepted, a derisking POC on branch `readplan-poc`
walked the read loop against every `BastType` arm in
`src/sequential_reader.rs:303-381` (and the parallel arms in
`materialize.rs`) and confirmed the `ReadPlan`/`CompositePlan`/
`ReadKind`/`DiscriminatorPlan` shape covers all cases, including the
two spots where a plan arm could subtly miss a case:
- **Union discriminator split (`Byte` vs `Field`).** Both are covered:
`DiscriminatorPlan::Byte { offset, disc_type }` and
`DiscriminatorPlan::Field { name, field_index }`. The field-name case
pre-resolves the discriminator field's `ReadKind` so dispatch reads
it from the plan, not from a re-parsed `BastField`. The
field-disc union's declared `fields` (discriminator + any shared
fields) are carried as a sub-`ReadPlan` on `CompositePlan::Union`'s
`shared` field (a refinement of the POC shape, which stubbed the
`Field` arm — Finding 1); the production read loop walks `shared`
first, then the selected variant's `CompositePlan`. Nested-union
variants (a variant that is itself a union) are covered by ordinary
`CompositePlan` recursion; the POC rejected them, the 0.2.0 reader
accepts them, and the production shape restores parity.
- **Array variable-element-stride (`element_stride = 0`).** Covered:
`CompositePlan::Array { element, count, element_stride }` preserves
the `0`-signals-variable convention, and the read loop walks
sequentially when `element_stride == 0` (matching today's
`walk_variable_array_size`).
The POC also confirmed `ReadPlan: Send + Sync` holds for the planned
shape (immutable owned data, no interior mutability, no lifetimes).
docs/architecture/decisions/012-plan-fingerprinting-and-m1-closure.md
+611
View File
@@ -0,0 +1,611 @@
# ADR-012: Plan Fingerprinting, ValidationPlan, and Closing the Deferred M1 Sites in 0.3.0
## Status
Accepted. Implemented in 0.3.0 — §3's `ValidationPlan` (phase 7,
2026-08-31), §1's fingerprinting (phase 6), §2a's owned `BastDoc`
(phases 3–4), §2b's `LeafMeta` (phase 5); all shipped 2026-09-02.
Bundles three pieces of work into the 0.3.0 release so the
crate ships one round of breaking changes, not two (or three). The
three pieces: (a) fingerprinting `ReadPlan`/`OffsetMap`, (b) closing
the deferred M1 sites via an owned `BastDoc` + `OffsetMap` `LeafMeta`,
and (c) a `ValidationPlan` that retires the interpretive
`bast_validation` walk (added by reversing the original "defer
`ValidationPlan`" decision — see "ValidationPlan — in scope for
0.3.0" below). Companion to [ADR-011](011-compiled-read-plan-for-packed-mode.md)
(the `ReadPlan`) and the [0.3.0 implementation plan](../../plans/030-compiled-forms.md).
§3's concrete shape was scoped by the follow-on design session and is
implemented in `src/validation_plan.rs` — see §3a below.
## Context
ADR-011 accepted the `ReadPlan` as the packed read-side compiled form
and deferred two things to "future capabilities":
1. **Fingerprinting the plan** for cross-run caching, disk-cached
compiled plans, and `alkcall` hub/spoke schema-version handshakes.
2. **A `WritePlan` and/or `ValidationPlan`** following the same
compile-once-walk-many pattern for the deferred M1 sites and the
validation walk.
ADR-011 also explicitly deferred the aligned-side M1 sites
(`engine.rs:334,467` `read_field`/`write_field`;
`layout_builder.rs:190` `LayoutBuilder::build`) as "a deliberate
reversible bet that an aligned-mode hot loop won't emerge."
This ADR retires the deferrals in one release. The reasoning is
timing: 0.3.0 is already a breaking bump (ADR-011 changes
`SequentialReader::new` and `materialize_packed` signatures), and the
crate has no real downstream consumers yet (only `alktty`/`alkcall`,
both in-house). Doing all three pieces now costs one round of
downstream churn instead of two or three, and the fingerprinting work
cuts across both `ReadPlan` and `OffsetMap` — splitting would create
a cross-release dependency that's cleaner in one release. The
`ValidationPlan` inclusion follows the same logic applied to the
validation walk: deferring it would create a *second* breaking change
to `validate_bytes`/`bast_validation` after 0.3.0, which is exactly
the round of downstream churn this release exists to retire.
### Reframing "WritePlan"
ADR-011's "Future capabilities" section listed a `WritePlan` as a
candidate. On inspection, a new public `WritePlan` type is the wrong
shape for the deferred M1 sites, for two reasons:
1. **The packed write-side already has a compiled form: `PackedLayout`.**
`LayoutBuilder::build`'s M1 re-parse is the *builder* re-parsing
`BastDoc::new` on each `build()` call to get the typed tree it
walks. The fix is to cache the parsed tree on the builder at `new()`
time — internal, non-breaking, no new public type. The compiled
form (`PackedLayout`) is unchanged; only its construction stops
re-parsing.
2. **The aligned R/W side already has a compiled form: `OffsetMap`.**
`read_field`/`write_field`'s M1 re-parse is `lookup_leaf_field`
walking `BastDoc` to get leaf metadata (`kind`, `encoding`,
`endian`) that `OffsetMap` doesn't carry. The fix is to extend
`OffsetMap`'s entries with that metadata at `compute` time —
additive fields on an existing public type (breaking, but we're
bumping anyway). No new public type.
A new `WritePlan` type would overlap with `PackedLayout` (packed
write) and `OffsetMap` (aligned R/W) without a clean distinguishing
shape. The honest picture: the packed write-side compiled form is
`PackedLayout`; the aligned R/W compiled form is `OffsetMap`; the M1
fixes are "cache the parse" and "extend the compiled form with leaf
metadata," not "add a third compiled form." This serves the
minimal-public-API-changes goal better than a literal `WritePlan`.
### `ValidationPlan` — in scope for 0.3.0 (no longer deferred)
The BAST-native validator (`bast_validation`) walks `BastDoc` to
check value-domain constraints (enum value sets, integer ranges,
`maxLength` caps, union variant keys). This is a different shape
from `ReadPlan`/`OffsetMap` (value-domain, not byte-position), and
the earlier framing deferred it as "not a hot loop — validation is
opt-in per operation per AGENTS.md."
**That deferral is reversed.** The "not a hot loop" dismissal
under-counted the common case: **read + validate together on
untrusted input.** The downstream `alkcall` consumer accepts schemas
from arbitrary internet peers in a hub/spoke topology (AGENTS.md §3);
the common operation on an incoming frame is "read it, then validate
it before acting." `validate_bytes` (ADR-010) is therefore called
once per incoming buffer, and each call re-walks `BastDoc` for
validation even after ADR-011 makes the *read* half plan-fast. That
is the same class of per-buffer interpretive cost review #004 measured
for the read path (400x per chunk), on a different code path, on the
operation the untrusted-input discipline actually requires.
The cost-of-inaction framing that the original deferral relied on was
also wrong: a `ValidationPlan` introduced *after* 0.3.0 would be a
breaking change to `validate_bytes`'s contract and to the
`bast_validation` public surface, forcing rework of `alktty`/`alkcall`
— the exact downstream-churn this release is supposed to retire, not
create a second round of. Shipping it in 0.3.0 pays the cost once,
alongside the other breaking changes, while there are zero real
consumers. The cost of action now is a static, known quantity; the
cost of action later is the same work plus a second round of
downstream churn plus the risk of the interpretive path being the
one that gets used in the meantime on untrusted bytes.
**Decision: a `ValidationPlan` ships in 0.3.0 as §3 below.** The
shape is a compile-once-walk-many compiled form over the BAST
document's value-domain constraints, symmetric to `ReadPlan` (packed
read-side) and `OffsetMap` (aligned R/W). The concrete shape,
construction, and `validate_bytes` integration are scoped in the
0.3.0 implementation plan (a dedicated phase) and detailed in a
follow-on design session before implementation; this ADR commits the
*decision* (in 0.3.0, not deferred) and the *scope* (a compiled
validation form that retires the interpretive `BastDoc` walk in
`bast_validation`), so the deferral black hole is closed.
## Decision
### 1. Fingerprinting — `ReadPlan: Hash + Eq`, `OffsetMap: Hash + Eq`
Add `#[derive(Hash, Eq)]` (alongside the existing `Debug, Clone, PartialEq`)
to `ReadPlan` and `OffsetMap`, plus their public sub-types
(`FieldPlan`, `CompositePlan`, `ReadKind`, `DiscriminatorPlan`,
`ByteRange`, and the new `LeafMeta` — see §2). `VariantPlan`/
`VariantKind` are not in the production `ReadPlan` shape (ADR-011
was refined on acceptance to drop them — see ADR-011 status), so
they are not derived. `ValidationPlan` (§3) gets `Hash + Eq` + its
own `fingerprint()` as part of its public surface.
**`by_name` representation change.** `ReadPlan.by_name` is currently
`HashMap<String, usize>`. `HashMap` iteration order is non-deterministic
and `HashMap` does not implement `Hash`, which blocks `#[derive(Hash)]`
on `ReadPlan`. Switch `by_name` to `BTreeMap<String, usize>`. Lookup
cost at protocol-header N (~5 fields) is negligible (the `BTreeMap` is
only used by `read_field`'s name→index lookup, not by the sequential
`read_next` hot path). This makes the derived `Hash` cover the full
structural state of the plan.
**Fingerprint contract.** Two plans with equal `Hash` (or equal under
`PartialEq`) produce identical reads over identical bytes. Formally:
`plan1 == plan2 ⟹ ∀ buffer. read(plan1, buffer) == read(plan2, buffer)`.
This is the contract the downstream uses rely on:
- **Cross-run disk cache.** A consumer can hash a `ReadPlan`/
`OffsetMap` and cache the compiled plan keyed by the hash, skipping
`compile` on warm starts. Safe because the contract guarantees a
cache hit produces identical read behavior.
- **`alkcall` hub/spoke schema handshake.** Peers exchange plan
fingerprints instead of full BAST documents. A peer that receives a
fingerprint it has already compiled can skip re-transmitting the
schema. The contract guarantees fingerprint equality implies
behavioral equivalence, so the handshake is sound.
- **Schema-version diagnostics.** A consumer can log a plan
fingerprint alongside read results for reproducibility — two runs
over "the same schema" that produce different fingerprints reveal a
silent schema drift.
The contract is a *behavioral* equivalence, not a structural identity:
two plans with different `by_name` insertion order but the same
`fields` Vec produce the same reads, and after the `BTreeMap` change
they also produce the same `Hash`. The contract is documented on the
`Hash` impl and tested by a property-style test (compile the same
schema twice, assert `plan1 == plan2` and `plan1.hash() ==
plan2.hash()`).
**Fingerprint API.** No new public method is strictly needed —
consumers call `std::hash::Hash` directly. For ergonomics and to make
the contract visible, add a convenience method:
```rust
impl ReadPlan {
/// A stable 64-bit fingerprint of this plan's read behavior.
///
/// Two plans with the same fingerprint produce identical reads
/// over identical bytes (the fingerprint contract).
pub fn fingerprint(&self) -> u64;
}
impl OffsetMap {
/// A stable 64-bit fingerprint of this offset map's read/write
/// behavior. Same contract as `ReadPlan::fingerprint`.
pub fn fingerprint(&self) -> u64;
}
```
Implemented via `std::hash::DefaultHasher` (or a stable hasher like
`FxHasher` if we want cross-version stability — decision belongs to
the implementation step, called out in the plan). The fingerprint is
additive API, not breaking.
### 2. Closing the deferred M1 sites
#### 2a. `LayoutBuilder` — cache the parsed `BastDoc` at `new()`
`LayoutBuilder` currently stores `doc_value: Value` + `root_name: String`
and re-parses `BastDoc::new(&self.doc_value, &self.root_name)` on every
`build()` call (`layout_builder.rs:190`). The fix: store the parsed
typed tree at `new()` time and reuse it in `build()`.
This requires `BastDoc` to be owned (no lifetime borrowing from
`doc_value`). Two options:
- **Option α (smaller):** keep `BastDoc<'a>` borrowing, store
`doc_value: Value` + a *pre-resolved, owned* representation of just
what `build` needs (the field tree with `$ref`s resolved). This is
essentially a `WritePlan` by another name — rejected per the
reframing above.
- **Option β (cleaner):** make `BastDoc` own its data. This is
review #004's Option A, scoped to `LayoutBuilder` only. It's a
larger refactor but eliminates the lifetime entanglement for the
builder and is the prerequisite for any future owning consumer that
wants to cache the parsed tree.
**Decision: Option β, scoped to `LayoutBuilder`.** The `BastDoc<'a>` →
`BastDoc` (owned) refactor is the principled fix and is already
breaking (the `Bast*` types are re-exported from `lib.rs`), so it
rides the 0.3.0 bump. This does *not* change `SequentialReader` or
`materialize_packed` (those consume `ReadPlan` per ADR-011, not
`BastDoc`). It changes `LayoutBuilder::new` to parse once and `build`
to reuse. The `doc_value: Value` field is removed; the builder holds
the owned `BastDoc` directly.
**Note on `BastDoc` ownership scope:** ADR-011 left `BastDoc` borrowed
and unchanged ("the validation-side typed tree"). This ADR changes
that: `BastDoc` becomes owned. The validation-side (`bast_validation`)
and aligned-side (`OffsetMap::compute`, `materialize_aligned`)
consumers adapt to the owned `BastDoc` — they no longer need a
borrowed `&Value` kept alive alongside. This is a net simplification:
one typed-tree type, owned, used by all non-`ReadPlan` consumers. The
POC on `readplan-poc` confirmed `ReadPlan` doesn't need `BastDoc` to
be borrowed (it compiles from `&Value` once and discards the
`BastDoc`), so making `BastDoc` owned doesn't regress the read path.
#### 2b. `OffsetMap` — carry leaf metadata
`OffsetMap` currently stores `Vec<(String, ByteRange)>`. The
`read_field`/`write_field` M1 re-parse is `lookup_leaf_field` walking
`BastDoc` to get `LeafFieldInfo { kind, encoding, endian }`
(`engine.rs:556-600`). The fix: extend `OffsetMap`'s entries to carry
that metadata at `compute` time.
```rust
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct LeafMeta {
pub kind: AlkTypeKind,
pub encoding: VariableEncoding,
pub endian: Endian,
}
pub struct OffsetMap {
fields: Vec<(String, ByteRange, LeafMeta)>, // was Vec<(String, ByteRange)>
total_size: usize,
}
```
`OffsetMap::compute` resolves each leaf field's `LeafMeta` during the
walk (it already walks the tree; it just doesn't currently record the
metadata). `read_field`/`write_field` drop the `BastDoc::new` +
`lookup_leaf_field` calls and read `LeafMeta` from the map. The
`LeafFieldInfo` struct in `engine.rs` is removed (replaced by
`OffsetMap`'s `LeafMeta`).
**Breaking changes:**
- `OffsetMap::get` return type: `Option<&ByteRange>` →
`Option<(&ByteRange, &LeafMeta)>` (or a small accessor struct).
Call sites in `alktty`/`alkcall` update with the bump.
- `ByteRange` is unchanged (still `Copy + Hash`).
- `LeafMeta` is a new public type, re-exported from `lib.rs`.
This is additive on the *capability* (the map now answers questions it
previously couldn't) but breaking on the *signature* (`get`'s return
type changes). Rides the 0.3.0 bump.
### 3. `ValidationPlan` — compile-once validation form
`bast_validation` currently walks `BastDoc` interpretively on every
`validate_bytes` call to check value-domain constraints (enum value
sets, integer ranges, `maxLength` caps, union variant keys). After
ADR-011, the *read* half of `validate_bytes` (packed) is plan-fast;
the *validation* half is still an interpretive `BastDoc` walk per
buffer. On the `alkcall` hub/spoke topology, `validate_bytes` is the
gate between "bytes arrived from an untrusted peer" and "act on the
decoded frame," so it runs once per incoming buffer and validation is
hot in the same sense review #004 measured for the read path.
**Decision: a `ValidationPlan` is a compiled form over the BAST
document's value-domain constraints, built once at `compile` time
(symmetric to `ReadPlan`/`OffsetMap`) and walked by
`bast_validation`/`validate_bytes` without re-touching `BastDoc`.**
The shape, construction, and `validate_bytes` integration are scoped
in the 0.3.0 implementation plan as a dedicated phase and detailed in
a follow-on design session before implementation begins. The
properties this ADR commits to (so the plan and any implementing agent
have a fixed contract):
- **Compile-once-walk-many.** `ValidationPlan::compile` walks `BastDoc`
once; `validate_bytes` (both modes) walks the `ValidationPlan` per
buffer, never `BastDoc`. This is the same pattern as `ReadPlan` and
`OffsetMap`; it is the structural reason the per-buffer
interpretive cost goes away.
- **Value-domain, not byte-position.** The plan carries constraint
descriptors (enum allowed-sets, integer range bounds, `maxLength`
caps, union variant keys, and any other value-domain checks
`bast_validation` performs today), keyed for dispatch against the
materialized `Value` tree, not byte offsets. The shape is therefore
different from `ReadPlan`/`OffsetMap`; the *pattern* (compiled form,
immutable, shared via `Arc`) is the same.
- **No new `BastDoc` walk in the hot path.** After this ADR, the only
consumers that walk `BastDoc` interpretively are the one-shot
`compile` paths (`ReadPlan::compile`, `OffsetMap::compute`,
`ValidationPlan::compile`, `LayoutBuilder::new`). The per-buffer
paths (`sequential_reader`, `materialize_packed`,
`materialize_aligned`, `validate_bytes`) all walk compiled forms.
This is the end state ADR-011 pointed at; this ADR closes it.
- **Semver.** `ValidationPlan` is a new public type, re-exported from
`lib.rs`. `validate_bytes`'s *signature* is unchanged (still
`(buffer) -> Result<(), AlkTypeError>`); the change is internal
(walks the plan instead of `BastDoc`). If the `ValidationPlan`
design surfaces a need to change `validate_bytes`'s signature, that
rides the 0.3.0 bump and is recorded in the plan's Semver Contract
table when the shape is scoped. `bast_validation`'s public surface
(`build_validator`, `validate_value`) is reviewed at shape-scope
time; additive changes ride the bump, removals/renames are avoided
unless the shape work shows they're necessary.
- **Fingerprinting.** `ValidationPlan` is `Hash + Eq` with a
`fingerprint()` method, same as `ReadPlan`/`OffsetMap` (§1/§4), so
the downstream uses (cross-run cache, `alkcall` handshake,
schema-version diagnostics) extend to the validation form without
new API. The fingerprint contract generalizes: two validation plans
with equal hashes accept/reject identical `(bytes)` identically.
**What this ADR does *not* decide** (left to the follow-on shape
session + plan phase): the concrete `ValidationPlan` struct/enum
shape, how `maxLength`/range/enum/union-key constraints are
represented, whether `bast_validation`'s `validate_value` is retired
or kept as a convenience wrapper over the plan, and whether the
`AlkTypeKind`-driven dispatch in `bast_validation` collapses into the
plan or stays a thin match over plan-carried descriptors. These are
shape questions, not decision questions; the decision (in 0.3.0,
compiled form, no per-buffer `BastDoc` walk) is fixed here.
### 3a. `ValidationPlan` shape — resolved by the design session
The follow-on design session (0.3.0 phase 7 predecessor) resolved the
open shape questions; implemented in `src/validation_plan.rs`:
- **Shape.** `ValidationPlan { root: ValidNode }`, a compiled
constraint tree — one `ValidNode` arm per value-domain check,
mirroring the interpretive walker's arms one-to-one:
`Int { min, max }` / `I64` / `Uint { max }` / `U64` / `Float` / `Bool`
/ `Str { max_len }` / `Bytes { max_len }` / `Enum { count }` /
`Struct { fields: Vec<ValidField> }` / `Union { variants:
Vec<ValidVariant> }` / `Array { count, element }` / `Record
{ values }`. `ValidField` carries `name + node`; `ValidVariant`
carries `key + node`. The nodes are public (diagnostics access via
`ValidationPlan::root()`); construction is only possible through
`compile`. Note the union node carries *only the variant nodes* — the
declared union `fields` (shared fields) are validated as part of the
variant walk, because the walker dispatches on the materialized
`__discriminator` and validates the whole object against the selected
variant (the `ValidNode::Union` doc comment records this; the
interpretive union arm recursed into the variant the same way).
- **Constraint representation.** Inline scalar fields on the node arms
(ranges as `i64`/`u64` pairs, `maxLength` as `Option<usize>`, enum
bound as `count: u64`, union keys as owned `String`s). `maxLength` is
resolved from the *owning field* at compile time and baked into the
`Str`/`Bytes` leaf — the walk never consults field annotations. It
never crosses a `$ref` (a `$ref` always targets a struct/union/enum
`$defs` entry, so the interpretive walk could never consult it
through one either).
- **`compile` signature.** `ValidationPlan::compile(&BastDoc) ->
Result<Self, AlkTypeError>` — the plan-table's `&str` root-name
parameter was vestigial (the doc already holds its root).
- **`validate_value` disposition.** Retained as a one-shot wrapper:
`compile(doc)` + `validate(value)`. The interpretive walker behind it
is *retired* (deleted) — the wrapper delegates to the plan, so there
is one constraint implementation, not two. `bast_validation.rs` keeps
the shared error helper (`validation_err`) and the
`__discriminator` key constant.
- **Error contract preserved.** The plan walk reproduces the
interpretive error messages byte-identically: a segment stack
(`field` / `[index]` / `[key]`) renders paths only on failure — zero
per-node allocation on the happy path. Numeric `__discriminator`
dispatch matches mapping keys without allocation for the u64/i64
forms (mapping keys are stringified integers; non-integer numbers
fall back to `Number::to_string`).
- **Compile-time rejection of adversarial graphs.** Eager `$ref`
resolution with a definition-level cycle set and a depth cap (128):
a cyclic or self-referential schema is `AlkTypeError::Schema` at
compile, not a stack overflow — the interpretive walker resolved
`$ref`s lazily with no guard and could overflow on recursion.
Diamond (shared, non-cyclic) refs compile fine; the cycle set is
path-scoped.
- **Engine integration.** `AlkTypeEngine` holds `Arc<ValidationPlan>`
built at `compile` time in *both* modes; the accessor
`validation_plan() -> &Arc<ValidationPlan>` is new public API.
`validate_bytes`'s signature is unchanged. The plan compile runs
*before* the layout build: it is the engine's reference-graph gate
(see Consequences).
- **Scope note.** phase-7's `Send + Sync` / `Hash + Eq` /
`fingerprint()` requirements are structural on the types above
(`#[derive(...)]` on plain owned data; the same `DefaultHasher`
fingerprint as phase 6).
### 4. Fingerprinting `OffsetMap` (bundled with §2b)
Since `OffsetMap` is getting new fields (`LeafMeta`) in §2b, its
`#[derive(Hash, Eq)]` (from §1) covers the new fields automatically.
The fingerprint contract for `OffsetMap` is the aligned-side analog
of `ReadPlan`'s: two offset maps with equal hashes produce identical
aligned reads/writes over identical bytes.
## Scope
### In scope
- `ReadPlan: Hash + Eq` + `fingerprint()` method (§1).
- `OffsetMap: Hash + Eq` + `fingerprint()` method (§1, §4).
- `BastDoc<'a>` → `BastDoc` (owned) refactor, scoped to the consumers
that currently hold `doc_value: Value` and re-parse: `LayoutBuilder`,
`bast_validation`, `materialize_aligned`, `OffsetMap::compute`
(§2a). `ReadPlan::compile` and the packed read path are unaffected
(they consume `&Value` once and discard `BastDoc`).
- `LayoutBuilder` caches the owned `BastDoc` at `new()`, `build()`
reuses it — no re-parse (§2a).
- `OffsetMap` carries `LeafMeta`; `read_field`/`write_field` drop
`BastDoc::new` + `lookup_leaf_field` (§2b).
- `LeafMeta` new public type (§2b).
- `BTreeMap` for `ReadPlan.by_name` (§1).
- `ValidationPlan` new public type + `compile` + `Hash + Eq` +
`fingerprint()` (§3). `validate_bytes` (both modes) walks the
`ValidationPlan` instead of re-walking `BastDoc`. `bast_validation`
adopts the plan; the public `validate_value`/`build_validator`
surface is reviewed at shape-scope time and rides the bump only if
the shape work shows a signature change is necessary.
- `ValidationPlan: Hash + Eq` + `fingerprint()` method (§3, §1) — the
fingerprint contract extends to the validation form.
### Out of scope
- Disk-cache or handshake *implementations* — the fingerprint
*contract* and method are in scope (§1, §3); the downstream uses
(cache format, wire protocol) are the consumers' problem, not this
ADR's.
- The `ValidationPlan` shape — **resolved** (§3a). Decided by the
design session and implemented in `src/validation_plan.rs`; the
decision (in 0.3.0, compiled form, no per-buffer `BastDoc` walk) was
fixed here.
- Cycle-guard hardening for the *layout* walkers' own recursion
(`LayoutBuilder`/`OffsetMap` struct recursion) beyond the engine-path
gate described in Consequences — if a non-engine entry point walking
those types on untrusted docs becomes a consumer pattern, the
guards get their own change (the `AlkTypeEngine::compile` gate
covers the supported path today).
- Cross-version fingerprint stability — the fingerprint is stable
within a crate version but may change across versions (a new
`AlkTypeKind` variant, for example, changes the hash). Cross-version
stability is a non-goal; consumers cache within a version. The
implementation step chooses a hasher and documents the stability
contract.
## Consequences
### Positive
- **One breaking release, not two (or three).** ADR-011's `ReadPlan`
+ this ADR's `BastDoc`-owned + `OffsetMap` extension +
`ValidationPlan` all ship together. The two in-house downstream
consumers (`alktty`, `alkcall`) update once.
- **Closes all deferred M1 sites.** `LayoutBuilder::build`
(`layout_builder.rs:190`), `read_field` (`engine.rs:334`),
`write_field` (`engine.rs:467`) all stop re-parsing. The packed-side
`validate_bytes` (`engine.rs:284`) was already closed by ADR-011;
this ADR closes the aligned-side equivalent.
- **Retires the interpretive validation walk.** `validate_bytes` on
untrusted streams (the `alkcall` common case) stops re-walking
`BastDoc` per buffer. This is the latent perf cliff review #005 M3
flagged: the read half was plan-fast after ADR-011, the validation
half was not. Closing it here — while there are zero real consumers
and one breaking bump already paying the downstream-churn cost —
avoids a second breaking change to `validate_bytes`/`bast_validation`
after 0.3.0. (Implemented: a spot benchmark of plan-validate on a
4-field mixed frame puts the validation half at ~0.2 µs/validate;
the compile-per-call one-shot it replaces runs ~2.7x slower before
the walk is even counted — and the full 0.2.0 per-buffer cost
included lazy `$ref` deep-clones that the one-shot no longer pays.
The materialize half, not validation, remains the dominant
`validate_bytes` cost.)
- **Compile-time rejection of cyclic `$ref` graphs.** A side effect of
eager plan compilation: a self-referential document is now a clean
`Schema` error instead of a stack overflow. The plan compile runs
*before* the layout build in `AlkTypeEngine::compile`, making it the
engine's reference-graph gate — `LayoutBuilder`/`OffsetMap`
struct-recursion had no cycle guard and previously could recurse
unboundedly on such a document (a pre-existing untrusted-schema
hazard, surfaced by the phase-7 `compile_rejects_cyclic_ref_graph`
test). (Resolved since: review #006 H2 added the shared
`walk_guard::check_ref_graph` guard at every standalone walker entry,
so the trust boundary no longer depends on the engine path.)
- **Fingerprinting enables downstream uses.** Cross-run plan caching,
`alkcall` schema handshake, and schema-version diagnostics all
become possible without further API work — across `ReadPlan`,
`OffsetMap`, and `ValidationPlan`.
- **`BastDoc` owned is a net simplification.** One typed-tree type,
owned, used by all `*::compile` paths. No more
lifetime-entanglement workarounds. The "re-parse on demand" framing
from ADR-007 is fully retired across read, write, and validation
paths.
- **`OffsetMap` extension is additive capability.** The map now
answers `kind`/`encoding`/`endian` questions it previously couldn't,
enabling future aligned-side tools without re-walking `BastDoc`.
### Negative
- **Breaking public-API changes (0.2.0 → 0.3.0).** `BastDoc<'a>` →
`BastDoc` (owned) changes every `Bast*` signature that took `&'a`.
`OffsetMap::get` return type changes. `LeafMeta` is new public.
`ReadPlan` is new public (from ADR-011). `ValidationPlan` (+ the
`ValidNode`/`ValidField`/`ValidVariant` node types) is new public
(§3a). All ride the bump.
- **`BastDoc` ownership refactor is broad.** Touches `bast.rs` (every
typed node: `&'a str` → `String`/`Arc<str>`, `&'a Value` →
`Value`/`Arc<Value>`) and every consumer (`layout_builder`,
`offset_map`, `materialize`, `bast_validation`, `engine`). This is
review #004's Option A, which ADR-011 deferred — this ADR picks it
up because the `LayoutBuilder` M1 fix requires it and we're bumping
anyway. The refactor is mechanical (lifetime removal, not logic
rewrites); the POC on `readplan-poc` confirmed the read path is
unaffected.
- **Interpretive `validate_value` is compile-per-call.** The retained
one-shot wrapper (`bast_validation::validate_value`) compiles a plan
then validates — fine for one-off/diagnostic use, wrong for per-
buffer use. Per-buffer callers must hold the engine (or a plan) —
the doc comments say so. The walker it replaced had the inverse
trade (no compile, but interpretive per call); the engine path
(compile once) is the one that matters.
- **Aligned-mode `maxLength`-reserved strings/bytes.** Materialization
emits the *full reserved* (zero-padded) data for these fields. A
`ValidationPlan` compiled from a document used in packed mode would
apply `maxLength` to trimmed length, matching packed semantics; the
aligned materializer's zero-padding means the value passed to
validation can carry trailing NULs. This is pre-existing
materialize behavior (not a plan artifact); consumers relying on
trimmed values already see it.
- **Fingerprint cross-version stability is not guaranteed.** A future
`AlkTypeKind` variant changes the hash. Documented as a within-
version contract. Consumers that need cross-version stability
serialize the BAST document and re-compile.
- **`BTreeMap` for `by_name` is a tiny lookup cost.** Negligible at
protocol-header N; irrelevant to the 400x fix.
## Scope Boundaries (What This Is Not)
- **Not a `WritePlan` type.** The packed write-side compiled form is
`PackedLayout`; the aligned R/W compiled form is `OffsetMap`. The
M1 fixes are "cache the parse" (§2a) and "extend the compiled form
with leaf metadata" (§2b), not "add a third compiled form."
- **Not a `ValidationPlan` deferral.** `ValidationPlan` is in scope
(§3) and implemented (§3a); the interpretive walker is retired.
- **Not cross-version fingerprint stability.** Within-version only.
- **Not a disk-cache or wire-protocol spec.** The fingerprint contract
and method are in scope; the downstream uses are the consumers'
concern.
## Recommended Order
See [the 0.3.0 implementation plan](../../plans/030-compiled-forms.md)
for the step-by-step execution order. The high-level grouping:
1. **`ReadPlan` (ADR-011 steps 1–5)** — the packed read-path fix. Closes
H1 + packed-side M1 + L1 + L2.
2. **`BastDoc` owned (§2a)** — the typed-tree ownership refactor. Prerequisite
for the `LayoutBuilder` M1 fix and for `ValidationPlan::compile`.
3. **`LayoutBuilder` M1 fix (§2a)** — cache the owned `BastDoc` at `new()`.
4. **`OffsetMap` extension (§2b)** — carry `LeafMeta`; close the
aligned-side `read_field`/`write_field` M1.
5. **`ValidationPlan` (§3)** — compiled validation form; `validate_bytes`
walks the plan instead of `BastDoc`. Requires the owned `BastDoc`
from step 2 for `ValidationPlan::compile`.
6. **Fingerprinting (§1, §4)** — `Hash + Eq` + `fingerprint()` on
`ReadPlan`, `OffsetMap`, and `ValidationPlan`. Rides on top of the
above.
7. **Public API bump (0.2.0 → 0.3.0)** — `lib.rs` re-exports, version
bump, update `alktty`/`alkcall`.
8. **Verification block** — full suite + wasm + bench.
## References
- [ADR-011](011-compiled-read-plan-for-packed-mode.md) — the
`ReadPlan` (packed read-side compiled form). This ADR extends the
0.3.0 release with fingerprinting, the deferred M1 fixes, and the
`ValidationPlan`.
- [Review #004](../../reviews/004-performance-review.md) — the
performance finding (H1, M1, L1, L2). ADR-011 closed H1 + packed
M1 + L1 + L2; this ADR closes the aligned-side M1.
- [Review #005](../../reviews/005-plan-review-030.md) — the 0.3.0 plan
review whose M3 finding reversed the `ValidationPlan` deferral.
- [ADR-007](007-packed-mode-read-factory.md) — the "re-parse on
demand" framing, retired across read, write, and validation paths
by ADR-011 + this ADR.
- [ADR-002](002-two-layout-modes-packed-vs-aligned.md) — the two
layout modes; `OffsetMap` is the aligned R/W compiled form extended
here with `LeafMeta`.
- [0.3.0 implementation plan](../../plans/030-compiled-forms.md) —
the step-by-step execution plan.
docs/architecture/layout-engine.md
+42 -8
View File
@@ -25,7 +25,7 @@ protocols.
**Components:**
- **`LayoutBuilder`** — constructed via `LayoutBuilder::new(bast_doc, root_name)` (requires a `struct` at the root), then `builder.build(&var_sizes) -> Result<PackedLayout, AlkTypeError>` where `var_sizes: &HashMap<String, usize>` maps variable-length field paths (and TUnion discriminator/variant keys) to their actual byte sizes. Used at write time when the consumer knows the data sizes upfront. The builder computes positions only; the consumer writes data via the [`data_access`](data-access.md) functions at the computed positions.
- **`SequentialReader`** — constructed via `SequentialReader::new(bast_doc, root_name)`, then driven by `reader.read_next(&buffer) -> Result<Option<(String, FieldValue)>, AlkTypeError>` until `Ok(None)`, or `reader.read_field(&buffer, path)` to seek a single field (which walks all preceding fields to reach the target). `reader.reset()` rewinds to the start. Used at read time when the consumer is parsing an incoming frame.
- **`SequentialReader`** — constructed via `engine.sequential_reader()` (shares the engine's compiled `ReadPlan` via `Arc` — see [ADR-011](decisions/011-compiled-read-plan-for-packed-mode.md)), then driven by `reader.read_next(&buffer) -> Result<Option<(String, FieldValue)>, AlkTypeError>` until `Ok(None)`, or `reader.read_field(&buffer, path)` to seek a single field (which walks all preceding fields to reach the target). `reader.reset()` rewinds to the start. Used at read time when the consumer is parsing an incoming frame.
**How it works:**
@@ -69,7 +69,7 @@ and safetensors.
**Component:**
- **`OffsetMap`** — constructed via `OffsetMap::compute(&doc) -> Result<Self, AlkTypeError>` (requires a `struct` at the root). Walks the BAST typed tree once, computes fixed byte positions for each field based on type sizes and alignment. The output is a flat table of `(field_path, byte_range)` pairs (see [Public Types](#public-types)). Used for both read and write at known offsets.
- **`OffsetMap`** — constructed via `OffsetMap::compute(&doc) -> Result<Self, AlkTypeError>` (requires a `struct` at the root). Walks the BAST typed tree once, computes fixed byte positions for each field based on type sizes and alignment, and resolves each leaf's `LeafMeta` (kind, encoding, effective endianness — ADR-012 §2b). The output is a flat table of `(field_path, OffsetEntry)` pairs (see [Public Types](#public-types)). Used for both read and write at known offsets.
**How it works:**
@@ -113,7 +113,11 @@ with a `AlkTypeError::Offset` — the `OffsetMap` reserves only 4 bytes
(the length prefix), but `data_access::write_string` writes prefix +
data inline, which would clobber subsequent fields. Non-final variable
fields must use `maxLength` (fixed-size reservation) or
`"encoding": "offset-indirect"`. See
`"encoding": "offset-indirect"` — except `record` fields, for which
neither remedy is available (`maxLength` is rejected at parse — review
#006 N3 — and `offset-indirect` is rejected for records in aligned
mode — review #006 M5), so a non-final record field cannot be repaired
and must move to the last position. See
[ADR-006](decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md).
## Offset Computation Algorithm
@@ -194,6 +198,12 @@ annotation shapes).
only. The engine uses strategy 1 (inline length-prefixing) because
protocols don't benefit from fixed-size reservation.
`maxLength` applies to `string` and `bytes` fields only. The parser
rejects it on any other kind (review #006 N3): the validation plan
bakes it into string/bytes leaves only, so on a record (or any other
kind) the annotation did nothing — and in aligned mode a record
reservation was silently corrupt (review #006 M5).
**Strategy 3: Offset indirection (`"encoding": "offset-indirect"`).**
1. The field is a struct `{offset: u32, length: u32}`.
2. The `OffsetMap` records the position of this struct.
@@ -312,22 +322,46 @@ discriminators, the discriminator is recorded under the synthetic path
(schema `properties` order, with nested struct fields appearing inline
under their parent's path prefix).
### `LeafMeta` / `OffsetEntry` (aligned mode, 0.3.0)
```rust
pub struct LeafMeta {
pub kind: AlkTypeKind,
pub encoding: VariableEncoding,
pub endian: Endian,
}
pub struct OffsetEntry {
pub range: ByteRange,
pub meta: LeafMeta,
}
```
`OffsetMap::compute` resolves each leaf's read/write metadata (kind,
variable-length encoding, effective endianness — field override else
container default, propagated the aligned-materializer way) alongside
its byte range, so `read_field`/`write_field` dispatch on the entry
without re-walking the BAST tree per access (ADR-012 §2b).
### `OffsetMap` (aligned mode)
A flat table of `(field_path, byte_range)` pairs computed from a schema.
A flat table of `(field_path, OffsetEntry)` pairs computed from a schema.
```rust
impl OffsetMap {
pub fn compute<'a>(doc: &'a BastDoc<'a>) -> Result<Self, AlkTypeError>;
pub fn get(&self, field_path: &str) -> Option<&ByteRange>;
pub fn compute(doc: &BastDoc) -> Result<Self, AlkTypeError>;
pub fn get(&self, field_path: &str) -> Option<&OffsetEntry>;
pub fn total_size(&self) -> usize;
pub fn iter(&self) -> impl Iterator<Item = &(String, ByteRange)>;
pub fn iter(&self) -> impl Iterator<Item = (&str, &OffsetEntry)>;
pub fn fingerprint(&self) -> u64;
}
```
`compute` requires a `struct` at the root. `total_size`
includes trailing alignment padding. `iter` yields fields in the BAST
`fields` array order (nested struct fields appearing inline).
`fields` array order (nested struct fields appearing inline). The map
carries `Hash + Eq` (ADR-012 §1); `fingerprint()` is the
stable-within-version hash for caching and schema handshakes.
## Design Decisions
docs/architecture/schema-layer.md
+4 -1
View File
@@ -230,7 +230,10 @@ type-level properties. The concrete BAST shapes are in
The `maxLength` keyword is *not* a BAST invention — it is the standard
JSON Schema `maxLength`, repurposed as a byte-length cap. In aligned
mode it reserves a fixed-size slot; in packed mode it is a validation
constraint only. See [bast-format.md §Variable-Length
constraint only. It applies to `string`/`bytes` fields only: the parser
rejects it on any other kind (review #006 N3 — elsewhere it was
silently unenforced), and in aligned mode a record reservation was
silently corrupt (review #006 M5). See [bast-format.md §Variable-Length
Encoding](bast-format.md#variable-length-encoding) and
[ADR-003](decisions/003-schema-annotations.md).
docs/architecture/validation.md
+69 -36
View File
@@ -1,6 +1,6 @@
---
status: accepted
last_updated: 2026-08-15
last_updated: 2026-08-31
---
# alktype — Validation
@@ -22,7 +22,7 @@ and recorded in [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md).
| Path | Input | Validator | Schema source |
|------|-------|-----------|---------------|
| `validate_bytes(&[u8])` | Raw bytes | BAST-native validator (`bast_validation`) | The BAST document (binary layout + value constraints) |
| `validate_bytes(&[u8])` | Raw bytes | Compiled `ValidationPlan` walk | The BAST document (binary layout + value constraints) |
| `validate_json(&Value)` | Parsed JSON `Value` | Standard `jsonschema::Validator` | A consumer-provided standard JSON Schema |
### `validate_bytes` — bytes in, BAST is the validator
@@ -35,25 +35,39 @@ produces `Value::Number`), bounds are checked (via
`data_access::check_bounds`), UTF-8 is valid (via `from_utf8`), the
discriminator is in the mapping, and the boolean byte is 0 or 1.
What the materializer does NOT check — and what the BAST-native
validator checks afterward — are **value-domain constraints expressed
in the BAST document**. The BAST-native validator
(`src/bast_validation.rs`) is a recursive walker over the BAST typed
tree ([`crate::bast::BastDoc`]/[`BastType`]) that checks exactly these:
What the materializer does NOT check — and what the validation half
checks afterward — are **value-domain constraints expressed in the BAST
document**. Since ADR-012 §3 (0.3.0), those constraints are not walked
interpretively per buffer: they are **compiled once** into a
`ValidationPlan` ([`src/validation_plan.rs`](../../src/validation_plan.rs))
at `AlkTypeEngine::compile` time, and each `validate_bytes` call walks
the compiled constraint tree against the materialized `Value` — no
`$ref` re-resolution, no schema re-parse, no per-node path formatting
(error paths render only on failure). The plan's constraint nodes
implement exactly the table below (the constraint set is unchanged from
the retired interpretive walker):
| Constraint | Validator arm |
|------------|---------------|
| Integer range (Int8..Uint64) | `validate_int`/`validate_uint` with `as_i64`/`as_u64` + range check |
| Int64/Uint64 (full range) | `validate_int64`/`validate_uint64` (JSON precision caveat per ADR-005) |
| Float finiteness (Float32/64) | `validate_float` with `as_f64().is_finite()` |
| String `maxLength` (byte length) | `check_string` reads the field-level `maxLength` |
| Bytes `maxLength` (array length) | `check_bytes` accepts the `Value::Array` form (the materializer emits bytes as an array of u8) |
| Enum index bounds | `validate_enum` checks `idx < values.len()` — **fixes the v0.1.0 dead constraint** |
| Union variant dispatch | `validate_union` reads `__discriminator`, looks up the variant, recurses via `validate_typeref` |
| Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses |
| Array count | `validate_array` checks `arr.len() == count` and recurses per element |
| Record values | `validate_record` recurses into each value's `values` type |
| Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) |
| Constraint | Plan node (`ValidNode`) |
|------------|-------------------------|
| Integer range (Int8..Uint32) | `Int { min, max }` / `Uint { max }` |
| Int64/Uint64 (full range) | `I64` / `U64` (JSON precision caveat per ADR-005) |
| Float finiteness (Float32/64) | `Float` with `as_f64().is_finite()` |
| String `maxLength` (byte length) | `Str { max_len }` — `maxLength` baked in from the owning field at compile time |
| Bytes `maxLength` (array length) | `Bytes { max_len }` — accepts the `Value::Array` form (the materializer emits bytes as an array of u8) |
| Enum index bounds | `Enum { count }` checks `idx < count` — **fixes the v0.1.0 dead constraint** |
| Union variant dispatch | `Union { variants }` reads `__discriminator`, dispatches on the compiled variant nodes |
| Struct fields | `Struct { fields }` requires each declared field present, recurses |
| Array count | `Array { count, element }` checks `arr.len() == count` and recurses per element |
| Record values | `Record { values }` recurses into each value |
| Boolean | `Bool` (materializer already rejects non-0/1 bytes) |
The plan is a public type (`ValidationPlan`, `Debug + Clone +
PartialEq + Eq + Hash + Send + Sync`): `engine.validation_plan()`
exposes it for consumers that validate their own materialized `Value`
trees or want its `fingerprint()` for caching / schema handshakes
(ADR-012 §1). The one-shot `bast_validation::validate_value(&doc,
&value)` remains as a convenience wrapper (compile + validate) for
callers holding a BAST document without an engine.
No external JSON Schema is required for `validate_bytes`. The BAST
document is the complete specification of the binary format — it
@@ -122,12 +136,14 @@ simplification.
The strategy is decided in [ADR-004](decisions/004-error-handling-validation-strategy.md)
and refined by [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md):
1. **Load time:** Parse the BAST document into the typed tree, compute
1. **Load time:** Parse the BAST document into the typed tree, compile
the `ValidationPlan` (the value-domain constraint tree), compute
the layout engine, and (optionally) build the standard
`jsonschema::Validator` for the JSON-validation path. This is the
`AlkTypeEngine::compile` constructor.
2. **Access time:** Use the compiled engine for repeated read/write
operations. Validation is opt-in per operation.
operations. Validation is opt-in per operation: the validation half
walks the compiled `ValidationPlan`, never the BAST document.
### The `AlkTypeEngine` struct
@@ -138,6 +154,7 @@ supports both layout modes (ADR-002) via an internal `Layout` enum:
pub struct AlkTypeEngine {
layout: Layout, // packed or aligned (private enum)
json_validator: Option<jsonschema::Validator>, // None when no JSON Schema supplied
validation_plan: Arc<ValidationPlan>, // compiled value-domain constraints (ADR-012 §3)
endian: Endian, // parsed from the root struct's "endian"
bast_doc: Value, // retained for sequential_reader/read_field
root_name: String, // the selected $defs entry
@@ -175,6 +192,7 @@ impl AlkTypeEngine {
pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>; // D-BAST-007
pub fn is_valid_json(&self, instance: &Value) -> bool; // D-BAST-007
pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError>; // D-BAST-006
pub fn validation_plan(&self) -> &Arc<ValidationPlan>; // compiled constraints (ADR-012 §3)
}
```
@@ -273,16 +291,23 @@ The expensive work happens once at schema load time:
1. Parse the BAST document into the typed tree (`BastDoc::new`).
2. Parse the root struct's `"endian"` annotation.
3. Compute the layout (`LayoutBuilder` for packed, `OffsetMap` for
3. Compile the `ValidationPlan` — the value-domain constraint tree,
with eager `$ref` resolution. Its compile walk rejects cyclic `$ref`
graphs with a clean `Schema` error *before* the layout computation.
(The layout walkers now also guard themselves — each standalone
entry point runs the shared reference-graph check
(`walk_guard::check_ref_graph`, review #006 H2) — so the plan-first
ordering is belt-and-suspenders at engine compile, and the trust
boundary no longer depends on the call path.)
4. Compute the layout (`LayoutBuilder` for packed, `OffsetMap` for
aligned).
4. If `json_schema` is `Some`, build the standard
5. If `json_schema` is `Some`, build the standard
`jsonschema::Validator` via `validation::build_validator`.
The result is an `AlkTypeEngine` that can be used for repeated
operations. The BAST-native validator is not pre-built — it is a
recursive walker that runs on the materialized `Value` at access time,
re-using the `BastDoc` (re-parsed on demand from the retained
`bast_doc`).
operations. The validation half is pre-built: the engine holds an
`Arc<ValidationPlan>` and walks it per buffer without re-touching the
BAST document (ADR-012 §3).
### Access time: `engine.validate_bytes(&[u8])`
@@ -297,10 +322,13 @@ sequence (D-BAST-006):
→ dispatch then recurse; `Record` → object of key/value entries).
The read phase reuses the existing data-access functions and returns
`AlkTypeError::Access` (with field paths) on read failures.
2. **Validate the `Value`.** The materialized `Value` is passed to
`bast_validation::validate_value(&doc, &value)`, producing
2. **Validate the `Value` against the `ValidationPlan`.** The
materialized `Value` is walked against the compiled constraint tree
(`engine.validation_plan().validate(&value)`), producing
`AlkTypeError::Validation` on the first violated value-domain
constraint.
constraint. This is the ADR-012 §3 end state: the only per-buffer
schema-touching step is the materialize half (the bytes must be
decoded against the tree); the validation half is plan-fast.
Mode dispatch:
@@ -308,7 +336,7 @@ Mode dispatch:
- **Aligned mode** — uses the `OffsetMap` to read fields at their
computed offsets.
Both modes produce the same `Value` form; the BAST-native validator is
Both modes produce the same `Value` form; the validation plan is
mode-agnostic.
### Access time: `engine.validate_json(&Value)` / `engine.is_valid_json(&Value)`
@@ -379,9 +407,10 @@ then access the binary buffer.
| Decision | ADR | Summary |
|----------|-----|---------|
| Two-validator model (BAST-native + standard jsonschema) | [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md) | `validate_bytes` uses the BAST-native validator; `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema; D-BAST-006/007/009 |
| Two-validator model (BAST-native + standard jsonschema) | [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md) | `validate_bytes` uses the compiled `ValidationPlan`; `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema; D-BAST-006/007/009 |
| Error handling and validation strategy | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Generalized validation — `validate_bytes` | [ADR-010](decisions/010-generalized-validation-validate-bytes.md) | Single-call binary-buffer validation (materialize `Value` from bytes, then validate); two methods on one struct, not a trait |
| Compiled `ValidationPlan` | [ADR-012](decisions/012-plan-fingerprinting-and-m1-closure.md) | The value-domain constraint tree is compiled once at `compile` (eager `$ref` resolution, cycle rejection) and walked per buffer; `Hash + Eq` + `fingerprint()`; retires the interpretive `BastDoc` walk |
| BAST format | [ADR-BAST](decisions/bast-bast-format.md) | The BAST document is the complete binary-format spec (layout + value constraints) |
## Open Questions
@@ -397,15 +426,19 @@ see [builder.md](builder.md).
— the normative validation model
- [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md) — the
two-validator decision
- [ADR-012](decisions/012-plan-fingerprinting-and-m1-closure.md) — the
`ValidationPlan` decision (§3)
- [ADR-004](decisions/004-error-handling-validation-strategy.md) —
error handling and validation strategy
- [ADR-010](decisions/010-generalized-validation-validate-bytes.md) —
`validate_bytes` (the collapsed two-step dance)
- [schema-layer.md](schema-layer.md) — the BAST parser that the
BAST-native validator walks
plan compiler consumes
- [data-access.md](data-access.md) — read/write functions and the
materializer that produce the `Value` the validator checks
materializer that produce the `Value` the plan checks
- [`src/validation_plan.rs`](../../src/validation_plan.rs) — the
compiled `ValidationPlan` implementation
- [`src/bast_validation.rs`](../../src/bast_validation.rs) — the
BAST-native validator implementation
one-shot wrapper (`validate_value`) and shared error helpers
- [`src/validation.rs`](../../src/validation.rs) — the `build_validator`
helper
docs/plans/030-compiled-forms.md
+983
View File
@@ -0,0 +1,983 @@
---
status: done
created: 2026-08-19
last_updated: 2026-09-02
adr: ADR-011, ADR-012
---
# 0.3.0 — Compiled Forms: ReadPlan, Owned BastDoc, OffsetMap LeafMeta, ValidationPlan, Fingerprinting
This is the execution plan for the 0.3.0 release: the compiled-form
rollup that closes review #004's 400x read-path gap (ADR-011) *and*
the deferred M1 sites (ADR-012) *and* retires the interpretive
validation walk via a `ValidationPlan` (ADR-012 §3, reversing the
original deferral per review #005 M3) *and* adds plan fingerprinting
(ADR-012 §1/§4) in one breaking bump. It is the **entry point** an
implementing agent reads first.
Companion documents:
- [ADR-011](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md)
— the `ReadPlan` decision (packed read-side compiled form).
- [ADR-012](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md)
— fingerprinting + owned `BastDoc` + `OffsetMap` `LeafMeta` +
`ValidationPlan` (this release's other three pieces).
- [Review #004](../reviews/004-performance-review.md) — the
performance finding being closed.
- [POC findings](../../poc/readplan/FINDINGS.md) (branch `readplan-poc`)
— the derisking POC that confirmed the `ReadPlan` shape and surfaced
two findings (field-disc union read shape; struct-array stride).
**Working order:** read this plan top-to-bottom. The Semver Contract
section is the scope-creep guardrail — consult it before each step.
Each step links to its ADR and lists its verification gate. Implement
phases in order; within a phase, steps are ordered by dependency.
## Phases vs sessions
This plan is deliberately larger than one session's work. The eight
phases are the session boundaries — each phase is a coherent unit
that leaves the tree building and tests green, so any one session
can pick up a phase without needing context from the previous one.
Phase boundaries are also commit boundaries (and push boundaries per
AGENTS.md). If a phase is large enough to span sessions, the steps
within it are the sub-session boundaries.
## Semver Contract
The crate is on crates.io at 0.2.0 with zero real consumers (only
`alktty`/`alkcall`, both in-house path dev-deps). A breaking bump to
0.3.0 is free but the contract is explicit so the implementation
doesn't drift. Per AGENTS.md, the public surface is the `lib.rs`
re-exports.
| Public item (from `lib.rs` re-exports) | Class | Change |
|---|---|---|
| `AlkTypeEngine::compile` | **Breaking (internal)** | Signature unchanged `(bast_doc: &Value, root_name: &str, mode, json_schema) -> Result<Self, AlkTypeError>`. Internally builds a `ReadPlan` (packed) or extended `OffsetMap` (aligned) and stores it. The `bast_doc: Value` clone is retained (ADR-011 §Engine integration). |
| `AlkTypeEngine::sequential_reader` | **Breaking (return type)** | Returns `Option<SequentialReader>` (unchanged type), but the reader is now constructed from `Arc<ReadPlan>`, not from `&bast_doc`. The reader's public methods (`read_next`/`read_field`/`reset`/`position`/`endian`/`schema`) keep their signatures. `schema()` returns the `&Value` the plan was compiled from (retained on the engine). |
| `AlkTypeEngine::read_field` / `write_field` | **Unchanged (signature)** | Still `(buffer, field_path) -> Result<FieldValue, AlkTypeError>`. Internally reads `LeafMeta` from the extended `OffsetMap` instead of re-parsing `BastDoc`. |
| `AlkTypeEngine::validate_bytes` | **Unchanged (signature)** | Packed mode calls `materialize_packed(&self.plan, buffer)` (ADR-011); aligned mode calls `materialize_aligned(&doc, buffer, &self.offset_map)` with the owned `BastDoc`. |
| `LayoutMode`, `AlkTypeEngine` | **Unchanged** | — |
| `BastDoc`, `BastDef`, `BastDefKind`, `BastStruct`, `BastField`, `BastType`, `BastUnion`, `BastDiscriminator`, `BastEnum`, `BastArray`, `BastRecord`, `BastRef` | **Breaking (lifetime removal)** | `BastDoc<'a>` → `BastDoc` (owned). Every `&'a str` → `String` (or `Arc<str>` — decision in phase 3). Every `&'a Value` → `Value` (or `Arc<Value>`). Every method signature that took/returned `&'a` changes. The `Bast*` types are re-exported from `lib.rs` so this is a public break. |
| `OffsetMap` | **Breaking (`get` return type)** | `get(field_path) -> Option<&ByteRange>` → `get(field_path) -> Option<&OffsetEntry>` where `OffsetEntry { range: ByteRange, meta: LeafMeta }` (or two accessors). Additive capability. |
| `ByteRange` | **Unchanged** | Still `Copy + PartialEq + Eq + Hash`. |
| `LeafMeta` | **New public type** | `{ kind: AlkTypeKind, encoding: VariableEncoding, endian: Endian }`, re-exported from `lib.rs`. `Copy + PartialEq + Eq + Hash`. |
| `ReadPlan` | **New public type** | From ADR-011. Re-exported from `lib.rs`. `Debug + Clone + PartialEq + Eq + Hash`. |
| `SequentialReader` | **Breaking (constructor + return type)** | `SequentialReader::new(&Value, &str) -> Result<Self, AlkTypeError>` → `SequentialReader::new(Arc<ReadPlan>) -> Self` (infallible — just stores the `Arc`; the `BastDoc` parse moved to `ReadPlan::compile`). Public methods (`read_next`/`read_field`/`reset`/`position`/`endian`/`schema`) unchanged. `schema()` returns `&Value` retained on the plan (see phase 2 — the plan stores `Arc<Value>`, not `&Value`, to avoid the self-referential struct ADR-011 rejects). `engine.rs`'s `.ok()` on the old `Result` correspondingly goes away. |
| `FieldValue` | **Unchanged** | — |
| `materialize_packed` | **Breaking (signature)** | `materialize_packed(&BastDoc<'_>, &[u8])` → `materialize_packed(&ReadPlan, &[u8])`. |
| `materialize_aligned` | **Breaking (signature)** | `materialize_aligned(&BastDoc<'_>, &[u8], &OffsetMap)` → `materialize_aligned(&BastDoc, &[u8], &OffsetMap)` (owned `BastDoc`, no lifetime). |
| `ValidationPlan` | **New public type** | From ADR-012 §3. Re-exported from `lib.rs`. `Debug + Clone + PartialEq + Eq + Hash`. `compile(&BastDoc, &str) -> Result<Self, AlkTypeError>`, `fingerprint() -> u64`. Shape scoped in phase 7 (and a preceding design session); the contract is fixed in ADR-012 §3. |
| `AlkTypeEngine::validate_bytes` | **Unchanged (signature)** | Still `(buffer) -> Result<(), AlkTypeError>`. Internally walks `&ValidationPlan` (both modes) instead of re-walking `BastDoc` for value-domain checks. The `materialize` half is unchanged from the ADR-011/§2b work (packed: `materialize_packed(&self.plan, ...)`; aligned: `materialize_aligned(&self.doc, ..., &self.offset_map)`). |
| `bast_validation` (`build_validator`, `validate_value`) | **Additive (reviewed at phase 7)** | `validate_value` is expected to become a thin wrapper over `ValidationPlan` (or be retired if the shape work shows it's redundant). Additive changes ride the bump; renames/removals are avoided unless phase 7's shape work shows they're necessary. The BAST meta-schema validator (`build_validator`/`BAST_META_SCHEMA`) used at `compile` time is unaffected. |
| `LayoutBuilder`, `PackedLayout`, `FieldPosition` | **Unchanged (signature)** | `LayoutBuilder::new`/`build` signatures unchanged. Internally caches the owned `BastDoc` instead of re-parsing. |
| `AlkTypeKind`, `Endian`, `VariableEncoding` | **Unchanged** | — |
| `AlkTypeError` | **Unchanged** | — |
| `Schema`, `Definitions`, `Discriminator` builders | **Unchanged** | — |
| `UnionDispatch`, `build_validator`, `BAST_META_SCHEMA` | **Unchanged** | — |
| `data_access::*` functions | **Unchanged** | — |
**Net breaking surface:** `BastDoc` and all `Bast*` types (lifetime
removal), `OffsetMap::get` (return type), `SequentialReader::new`
(constructor + `Result` drop), `materialize_packed`/`materialize_aligned`
(signatures). **Net additive:** `ReadPlan`, `LeafMeta`, `ValidationPlan`,
`fingerprint()` methods, `Hash + Eq` derives on `ReadPlan`/`OffsetMap`/
`ValidationPlan`. **Net unchanged:** the builder, `AlkTypeEngine`
accessors (signatures), `FieldValue`, `AlkTypeKind`, `AlkTypeError`,
`data_access`, the `Schema`/`Definitions` builders, `validate_bytes`
(signature).
### Decisions deferred to their implementation phases
1. **`Arc<str>` vs `String` for owned `BastDoc` names** (phase 3): `Arc<str>`
shares allocation for repeated names (e.g. union variant keys appearing
in multiple places); `String` is simpler. The POC used `String`. Lean:
`String` unless a bench shows `Arc<str>` matters — the typed tree is
built once, not hot. Decided in phase 3.
2. **`OffsetMap::get` return shape** (phase 5): `Option<&OffsetEntry>` (a
new accessor struct) vs two methods `range(path) -> Option<&ByteRange>`
+ `meta(path) -> Option<&LeafMeta>`. The struct is fewer calls; the two
methods preserve back-compat shape for callers that only want the range.
Lean: struct — it's a breaking bump anyway and the struct is cleaner.
Decided in phase 5.
3. **Fingerprint hasher** (phase 6): `DefaultHasher` (std, stable within a
version) vs `FxHasher` (faster, also stable). Cross-version stability
is a non-goal (ADR-012). Lean: `DefaultHasher` — no new dep, the
fingerprint isn't hot. Decided in phase 6.
4. **Struct-array stride** (phase 2): the POC found the existing reader
returns `element_stride: 0` for fixed-size struct arrays
(`sequential_reader.rs:567`). The `ReadPlan` correctly computes the
stride. Decision: preserve the existing `0` behavior in `ReadPlan` for
back-compat with `SequentialReader`'s consumer contract, *or* fix it
and document the behavioral change. Lean: fix it — the `0` is a latent
bug, the stride is behaviorally observable, and we're bumping. The
plan step calls this out explicitly. Decided in phase 2.
## Phase 1 — `ReadPlan` type + `compile` (ADR-011 step 1) — **DONE (2026-09-02)**
> **Status: implemented.** `src/read_plan.rs` builds the refined
> `CompositePlan::Union` shape (`shared: Option<Box<ReadPlan>>` +
> `variants: Vec<(String, CompositePlan)>`, no `VariantPlan`/`VariantKind`)
> with eager `$ref` resolution, `BTreeMap` `by_name`, field-disc `shared`
> sub-plans, nested-union variant support, and true array strides
> (deferred decision 4 resolved: fixed struct arrays compute their real
> stride, not the 0.2.0 reader's `0`). Two parity notes recorded as
> compile-behavior locks in tests: (a) the plan propagates the
> *referring field's* effective endianness into nested structs/unions —
> exactly what the 0.2.0 packed reader/materializer do — rather than
> consulting nested containers' own `endian` annotations (the POC baked
> `s.endian()` there; its equivalence tests never covered a nested
> annotation, so the divergence was latent); (b) `compile` carries its
> own depth cap (128) + definition-level cycle set, so standalone
> `ReadPlan::compile` is untrusted-input-safe independent of the
> meta-schema and the engine's `ValidationPlan` gate.
**Goal:** Add the `ReadPlan`/`FieldPlan`/`CompositePlan`/`ReadKind`/
**ADR reference:** [ADR-011 §The `ReadPlan` shape](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md#the-readplan-shape),
[ADR-011 §Construction](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md#construction).
The `CompositePlan::Union` shape in ADR-011 was refined (vs the
accepted-at-POC shape) to carry the field-disc union's shared fields
and to drop `VariantPlan`/`VariantKind`; this phase implements the
refined shape.
**POC reference:** `poc/readplan/src/lib.rs` (branch `readplan-poc`)
is the reference scaffold. The production version lives in `src/` and
adds doc comments, clippy cleanliness, the field-name-discriminator
union read shape the POC stubbed (POC Finding 1), and nested-union
variant support the POC rejected but 0.2.0 accepts (POC "What this
POC does not cover" → nested unions). Both are resolved by the
refined `CompositePlan::Union` shape — see below.
**Files:** New `src/read_plan.rs`. Update `src/lib.rs` to add
`pub mod read_plan;` and re-export `ReadPlan` (and the plan sub-types
that are part of the public surface — `ReadKind`, `CompositePlan`,
etc. if the ADR's public-API section calls for them; the ADR lists
`ReadPlan` as the public type, sub-types can stay `pub` in-module if
consumers don't need to name them).
**Implementation notes:**
- `compile` walks `BastDoc` once (via the existing borrowed `BastDoc`,
which still exists at this phase — the owned-`BastDoc` refactor is
phase 3). Resolves all `$ref`s eagerly, computes effective endianness
at every node, inlines union variants. Malformed schemas surface as
`AlkTypeError::Schema` (AGENTS.md §3); overflow-safe arithmetic
(AGENTS.md §4).
- `by_name: BTreeMap<String, usize>` (not `HashMap` — ADR-012 §1
requires `Hash` on `ReadPlan`, and `HashMap` blocks derive). The
POC used `HashMap`; swap to `BTreeMap`.
- **Union shape (ADR-011 refined — resolves POC Finding 1 and the
nested-union gap):** `CompositePlan::Union { disc, shared,
variants: Vec<(String, CompositePlan)> }`.
- `shared: Option<Box<ReadPlan>>` — the union's declared `fields`
(the discriminator field + any shared fields) for the
field-name-discriminator case. `DiscriminatorPlan::Field.field_index`
indexes into `shared`. The read loop walks `shared` first, then
looks up the selected variant and walks its `CompositePlan`
starting after the shared fields. The byte-offset-discriminator
case sets `shared: None` (no shared fields; the variant starts
immediately after the discriminator size). This replaces the
POC's stubbed `plan_read_union` `Field` arm.
- `variants: Vec<(String, CompositePlan)>` — **not** the POC's
`Vec<(String, VariantPlan)>`. Dropping `VariantPlan`/`VariantKind`
means a variant's body is just a `CompositePlan`, so **nested
unions** (a variant that is itself a union, which the 0.2.0
reader supports via `resolve_and_walk_variant`'s `Union` arm at
`sequential_reader.rs:800`) work by ordinary `CompositePlan`
recursion — a variant can be `CompositePlan::Union { ... }`. No
separate `VariantKind::Union` arm, no behavioral drop vs 0.2.0,
no Semver Contract entry for a capability regression. The POC's
`VariantPlan { kind, plan }` wrapper is not carried forward.
- `compile_union` must reject a variant that is neither a struct
nor a union with `AlkTypeError::Schema` (mirroring
`resolve_and_walk_variant`'s `other => Err(...)` arm), so the
eager-resolution path keeps the untrusted-input discipline.
- Do *not* wire `ReadPlan` into `SequentialReader` or `materialize` yet
— that's phase 2. Phase 1 is the type + `compile` only, unit-tested
against the same BAST fixtures `bast.rs` uses (the existing `bast.rs`
tests are a ready source of fixtures).
**Verification:** `cargo test --release` (new unit tests for `compile`
covering every `BastType` arm — port the `cov_*` tests from
`poc/readplan/tests/coverage.rs`, **plus** a nested-union-variant
test asserting `compile_union` produces `CompositePlan::Union` whose
variant body is itself `CompositePlan::Union`, restoring the 0.2.0
capability the POC rejected). `cargo clippy --all-targets -- -D
warnings`. `cargo doc --no-deps` (new public type). `cargo build
--target wasm32-unknown-unknown --release` (`read_plan.rs` is
wasm-relevant). **Add a `fn read_plan_is_send_sync()` assertion
test** (a `const _: fn() = || { fn assert_send_sync<T: Send + Sync>()
{}; assert_send_sync::<ReadPlan>(); };` static-bound assertion, as
ADR-011 §"Engine integration" requires) to lock in `ReadPlan: Send +
Sync` so a future change can't break it silently — mirror the POC's
`readplan_is_send_sync` test.
---
## Phase 2 — `SequentialReader` + `materialize_packed` consume `ReadPlan` (ADR-011 steps 2–4) — **DONE (2026-09-02)**
> **Status: implemented.** The packed read loop walks `Arc<ReadPlan>`:
> `SequentialReader::new(Arc<ReadPlan>) -> Self` (infallible; the old
> fallible constructor's work moved to `ReadPlan::compile`), the reader
> holds `plan: Arc<ReadPlan>` + cursor only, `schema()` returns the
> `Arc<Value>` retained on the plan (review #005 H2 closed — no
> self-referential struct), and a new `plan()` accessor exposes the
> shared plan. `materialize_packed(&ReadPlan, &[u8])` walks the same
> plan; the aligned materialize path keeps walking `BastDoc` with the
> retained `dummy_field_for`/`ty_source`/`materialize_typeref_packed`
> helpers (phase 5 Scope Boundary). Engine: `Layout::Packed` carries
> `Arc<ReadPlan>`; `sequential_reader()` is an `Arc::clone` (15.7 ns,
> was a whole-document `Value` clone); packed `validate_bytes` calls
> `materialize_packed(&self.plan, ...)`. The temporary validation
> bridge (reconstruct `BastDoc` for the validator) is still in place —
> phase 7 already retired it on `main`'s ValidationPlan; this phase's
> `validate_bytes` edit merged cleanly onto that state.
>
> **Stride (deferred decision 4):** fixed struct/nested-array elements
> now report their true stride through `FieldValue::Array`
> (0.2.0 returned `0`); doc comment updated; no existing test asserted
> the `0`, so no test needed changing — the plan-compile tests lock the
> new values.
>
> **Two parity subtleties found and preserved** (both invisible to the
> existing test suite, both now locked by tests or by construction):
> (a) the materializer unwraps the plan's anonymous single-field
> wrapper for primitive array elements/record values — without this,
> materialized records/arrays would nest each leaf under a synthetic
> object and `validate_bytes` would fail its own parity suite (caught
> by `materialize_record_packed_count_prefixed_pairs`); (b) the
> field-disc union's materialized key order keeps `__discriminator`
> first (matching 0.2.0's `Map` insertion order, observable under
> `preserve_order`).
>
> **Bench (alktty `wire_vs_bast`, 1024 chunks/stream):** read p64
> 2.27 µs/chunk (review #004) → **98 ns/chunk** (~23x; gap 400x →
> ~17x vs hand-rolled's 5.6 ns); read p4k → 100 ns/chunk. `engine.
> sequential_reader()` construction 15.7 ns (was a full `Value` clone).
> The residual gap is dominated by the per-field `String` allocation
> mandated by the unchanged `(String, FieldValue)` `read_next` return
> signature (2 allocs/chunk) plus `data_access` bounds checks — both
> outside this phase's scope (the signature is pinned by the Semver
> Contract).
**Goal:** Rewrite the packed read loop to walk `&ReadPlan` instead of
reconstructing `BastDoc`. `SequentialReader` stores `Arc<ReadPlan>` +
cursor state; `materialize_packed` takes `&ReadPlan`. Closes H1
(the 400x gap) + the packed-side M1 + L1.
**ADR reference:** [ADR-011 §Scope](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md#scope),
[ADR-011 §Recommended Order](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md#recommended-order)
steps 2–4.
**Files:** `src/sequential_reader.rs` (rewrite the read loop, change
`new`'s signature), `src/materialize.rs` (`materialize_packed` takes
`&ReadPlan`), `src/engine.rs` (`compile` builds `Arc<ReadPlan>` in
packed mode, `sequential_reader()` hands out `Arc::clone`, packed
`validate_bytes` calls `materialize_packed(&self.plan, ...)`).
**Implementation notes:**
- `SequentialReader::new(&Value, &str) -> Result<Self, AlkTypeError>` →
`SequentialReader::new(Arc<ReadPlan>) -> Self` (infallible — the
fallible `BastDoc` parse moved to `ReadPlan::compile` in phase 1;
`new` just stores the `Arc`). The reader stores `plan: Arc<ReadPlan>`,
`field_index: usize`, `position: usize`. `endian()` reads
`self.plan.endian()`.
- **`schema()` ownership (resolves review #005 H2):** `schema()`
returns `&Value` retained on the plan, but the plan stores an
**`Arc<Value>`**, not a `&Value`. ADR-011 §"Root cause" rejects the
self-referential struct pattern (a `ReadPlan` storing `&Value`
borrowing from the engine's `bast_doc: Value` would make the engine
self-referential — exactly the construction ADR-007 worked around
and ADR-011's `Arc<ReadPlan>` was meant to retire). The fix:
`ReadPlan` carries `schema: Arc<Value>`; `ReadPlan::compile` clones
the input `&Value` into `Arc<Value>` once; `schema()` returns
`&self.schema`. The engine stores `bast_doc: Arc<Value>` internally
(one allocation, shared via refcount between the engine and all
plans it builds) — this is an internal change, not a public
signature change (`compile` still takes `&Value`). `Arc<Value>`
implements `Hash + Eq` (`serde_json::Value: Hash + Eq` as of the
pinned `serde_json` with `preserve_order`; `Map::hash` sorts keys
for determinism), so phase 6's `#[derive(Hash)]` on `ReadPlan` is
not blocked by carrying the schema. **Note:** if a future
`serde_json` version regresses `Value: Hash`, phase 6 would need
`ReadPlan`'s hash to exclude the `schema` field; that is a phase-6
concern, not a phase-2 blocker.
- `read_field_at`/`read_field_value`/`read_typeref_value`/
`walk_struct_size`/`read_union_value`/`read_array_value`/
`read_record_value` are rewritten to take plan nodes
(`&FieldPlan`/`&CompositePlan`/`&ReadKind`) instead of
`&BastField`/`&BastType`/`&BastDoc`. Port `plan_read_field_at`/
`plan_walk_struct_size`/etc. from `poc/readplan/src/lib.rs` — they're
the reference implementations. The `read_union_value` rewrite
handles both discriminator kinds via the refined `CompositePlan::Union`
shape from phase 1: byte-disc reads the discriminator at
`disc.offset` then walks the variant (no `shared`); field-disc walks
`shared` first, reads the discriminator field at
`disc.field_index` within `shared`, looks up the variant, and walks
it starting after the shared fields. Nested unions (variant body is
itself `CompositePlan::Union`) recurse naturally — no special arm.
- **Struct-array stride (deferred decision 4):** the POC computes the
true fixed-struct stride; the existing reader returns `0`. The
production `ReadPlan::compile_array` should compute the true stride
(the POC's `fixed_struct_size` helper). Document the behavioral
change in the `FieldValue::Array` doc comment: `element_stride` is
now the true stride for fixed-size struct elements, not `0`. This is
a breaking behavioral change; rides the bump. Update the
`eq_array_ref_element`-style test to assert the new stride.
- **`materialize_packed` rewrite + packed/aligned split (resolves
review #005 L1 and L2):** `materialize_packed(&BastDoc<'_>, &[u8])`
→ `materialize_packed(&ReadPlan, &[u8])`. The materializer walks the
plan instead of `BastDoc`. **Scoped removal of helpers:** only the
*packed-side* call sites of `dummy_field_for`/`ty_source`
(`materialize.rs:249, 316, 351, 391` — the packed
`materialize_*_packed` paths) go away when packed-materialize moves
to the plan. The helpers themselves **stay**, because
`materialize_leaf_at` (`materialize.rs:631`, which calls
`dummy_field_for`) is on the **aligned** path — it's called by
`materialize_struct_aligned` (`:475`), `materialize_array_aligned`
(`:544`), `materialize_variable_aligned` (`:613`). Aligned
`materialize` keeps walking `BastDoc` through 0.3.0 (see the Scope
Boundary note in phase 5), so `dummy_field_for`/`ty_source` must
stay. **`materialize_typeref_packed` split:** this function is
currently shared by both packed and aligned paths (aligned's
`materialize_leaf_at` calls it to read leaves, and aligned's record
path at `:498-506` calls it directly). After phase 2,
packed-materialize gets a new plan-walking function;
`materialize_typeref_packed` stays for aligned's
`materialize_leaf_at` and the aligned record path (renamed or not —
implementer's choice; the function is private). This is two
mode-specific paths — the existing design — not a "parallel walker"
in the maintenance-tax sense ADR-011 §"Negative" cautions against
(that caution is about packed read-side `SequentialReader` +
`materialize_packed` sharing one plan, which this preserves).
- `AlkTypeEngine::compile` (packed branch): build `Arc<ReadPlan>` via
`ReadPlan::compile(bast_doc, root_name)`, store it in `Layout::Packed`.
`sequential_reader()` returns
`Some(SequentialReader::new(Arc::clone(&self.plan)))`.
`validate_bytes` (packed) calls
`materialize_packed(&self.plan, buffer)` then runs validation on the
materialized `Value`. **Validation path through phase 2:** until
phase 7 (ValidationPlan), `validate_bytes` reconstructs a `BastDoc`
for the validator only — the *read* path uses the plan, the
*validation* path uses `BastDoc`. This is a temporary bridge: phase 7
replaces it with a `ValidationPlan` walk (ADR-012 §3), retiring the
per-call `BastDoc` reconstruction. The bridge is acceptable for
phases 2–6 because the ValidationPlan work is committed in this
release (not deferred), so the bridge has a known removal point in
phase 7.
**Verification:** `cargo test --release` — the existing
`sequential_reader.rs` and `materialize.rs` tests drive `read_next`/
`read_field`/`reset`/`validate_bytes` through the public API, so they
validate the rewrite without modification. If any test breaks, the
rewrite diverged from the existing behavior — investigate before
patching the test. `cargo clippy --all-targets -- -D warnings`.
`cargo build --target wasm32-unknown-unknown --release`. **Re-run the
alktty `wire_vs_bast` bench** to confirm the 400x gap closes (this is
the headline result; record the before/after numbers in the commit
message).
---
## Phase 3 — Owned `BastDoc` (ADR-012 §2a) — **DONE (2026-09-02)**
> **Status: implemented.** Every `Bast*` type dropped its `<'a>`:
> `&'a str` → `String`, `&'a Value` → `Value` (deferred decision 1
> resolved: plain `String`/`Value` — the tree is built once, name
> sharing via `Arc<str>` needs a bench justification that doesn't
> exist). `BastDoc::new(&Value, &str)` still takes references in and
> clones into owned storage; `BastDoc` gained `Clone`. `resolve_*`
> return owned `BastDef`/`BastType`. Consumers adapted:
> `OffsetMap::compute(&BastDoc)`, `materialize_aligned(&BastDoc, ...)`
> (no lifetime), `BuildCtx`/`ComputeCtx` hold `&'d BastDoc`.
> **Engine ownership flip:** `AlkTypeEngine` now holds the owned
> `BastDoc` (replacing `bast_doc: Value` + `root_name: String` —
> `root_name()` delegates to the doc), killing its three per-call
> `BastDoc::new` re-parses (`validate_bytes` aligned path,
> `read_field`, `write_field` — review #004 M1 sites by construction;
> phase 5 removes the `lookup_leaf_field` walk itself). New public
> accessor `AlkTypeEngine::root_name()` (additive). **Bonus cleanup:**
> `materialize_typeref_packed`'s dead `_field` param dropped (phase 2
> left it dangling); under ownership it would have forced a deep
> `Value` clone per array element/record value/union variant via
> `dummy_field_for` — the param, `dummy_field_for`, and `ty_source`
> are gone (no behavior change; the `_field` arg was already
> ignored). `BastField::synthetic` retains an owned-signature
> `#[allow(dead_code)]` definition (no remaining callers; kept for the
> phase-4/5-aligned materializer helpers if they need it). Existing
> `BastDoc` consumers (`LayoutBuilder`'s `doc_value` re-parse cache)
> are unchanged pending phase 4. Engine stays `Send + Sync` with the
> owned doc — the engine's thread-share test now asserts it directly.
**Goal:** Make `BastDoc` own its data (drop the `<'a>` lifetime).
`&'a str` → `String`, `&'a Value` → `Value` (or `Arc<str>`/`Arc<Value>`
— deferred decision 1). This is the prerequisite for the `LayoutBuilder`
M1 fix (phase 4) and simplifies all owning consumers. Broad but
mechanical refactor.
**ADR reference:** [ADR-012 §2a](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#2a-layoutbuilder--cache-the-parsed-bastdoc-at-new).
**Files:** `src/bast.rs` (every `Bast*` type), every consumer:
`src/layout_builder.rs`, `src/offset_map.rs`, `src/materialize.rs`,
`src/bast_validation.rs`, `src/engine.rs`, `src/tunion.rs`,
`src/bast_meta.rs` (if it walks `BastDoc`), `src/builder.rs` (if it
consumes `Bast*`). `src/lib.rs` re-exports (signatures change but
names stay).
**Implementation notes:**
- `BastDoc<'a>` → `BastDoc`. Fields: `root: Value` (was `&'a Value`),
`root_name: String` (was `&'a str`), `root_def: BastDef` (was
`BastDef<'a>`). `new(root: &Value, root_name: &str)` takes references
*in* (the caller still owns the input `Value`) but clones into owned
storage. The `&Value` → `Value` clone is the cost of ownership; it
happens once at `compile`/`new`, not per-field.
- `BastDef<'a>` → `BastDef`: `name: String`, `kind: BastDefKind`,
`source: Value`.
- `BastStruct<'a>` → `BastStruct`: `endian: Endian`, `align: Option<usize>`,
`fields: Vec<BastField>`, `source: Value`.
- `BastField<'a>` → `BastField`: `name: String`, `ty: BastType`,
`endian: Option<Endian>`, `align: Option<usize>`,
`encoding: VariableEncoding`, `max_length: Option<usize>`,
`source: Value`. `synthetic` constructor takes owned `BastType` +
`Value`.
- `BastType<'a>` → `BastType`: `Primitive(AlkTypeKind)`,
`Ref(BastRef)`, `Array(BastArray)`, `Record(BastRecord)`,
`Struct(BastStruct)`, `Union(BastUnion)`, `Enum(BastEnum)`.
- `BastUnion<'a>` → `BastUnion`: `endian: Endian`,
`discriminator: BastDiscriminator`, `fields: Vec<BastField>`,
`mapping: Vec<(String, BastType)>` (was `Vec<(&'a str, BastType)>`),
`source: Value`.
- `BastDiscriminator::Field { name: String }` (was `name: &'a str`).
- `BastEnum<'a>` → `BastEnum`: `values: Vec<String>` (was
`Vec<&'a str>`), `source: Value`.
- `BastArray<'a>` → `BastArray`: `element: Box<BastType>`, `count: usize`,
`source: Value`.
- `BastRecord<'a>` → `BastRecord`: `values: Box<BastType>`,
`source: Value`.
- `BastRef<'a>` → `BastRef`: `name: String`.
- **`resolve_typeref` / `resolve_ref` / `lookup_def`** now return owned
`BastType`/`BastDef`/`Value` instead of borrowed. The `clone()` in
the current `resolve_typeref` passthrough (`other => Ok(other.clone())`)
is no longer needed for the borrow case (everything is owned) but
the logic is unchanged — `BastType` is `Clone` either way.
- **Consumers adapt:** any code that held `&'a Value` alongside a
`BastDoc<'a>` (e.g. `LayoutBuilder.doc_value`, `AlkTypeEngine.bast_doc`,
`SequentialReader.doc_value` — though the reader is already on
`ReadPlan` after phase 2) drops the separate `Value` and holds the
owned `BastDoc` directly. `materialize_packed` is already on
`ReadPlan` (phase 2) and doesn't need `BastDoc` — unaffected.
`materialize_aligned` takes `&BastDoc` (owned, no lifetime).
- **`Arc<str>` vs `String` (deferred decision 1):** default to
`String`. The typed tree is built once; name sharing via `Arc<str>`
is a micro-optimization not justified without a bench. If phase 4's
`LayoutBuilder` work shows name allocation is measurable, revisit.
**Verification:** `cargo test --release` — the existing `bast.rs` tests
are the primary validation (they exercise every parser path). All
`Bast*`-consuming tests must pass unchanged (they go through public
APIs that still take `&Value`/`&str` in, just return owned types out).
`cargo clippy --all-targets -- -D warnings`. `cargo doc --no-deps`.
`cargo build --target wasm32-unknown-unknown --release` (`bast.rs` is
wasm-relevant).
---
## Phase 4 — `LayoutBuilder` caches the owned `BastDoc` (ADR-012 §2a) — **DONE (2026-09-02)**
> **Status: implemented.** `LayoutBuilder` stores `doc: BastDoc` +
> `endian` (the `doc_value: Value` + `root_name: String` cache is
> gone); `new` parses once, `build` walks `&self.doc` — the
> `layout_builder.rs` re-parse (M1) is retired. The `build`-time
> root-is-struct re-check replaced its `unreachable!()` with a clean
> `Schema` error (AGENTS.md §3 never-panic; the invariant is
> unchanged — `new` already rejects non-struct roots). Boxing fallout:
> the builder now holds the full owned tree, so `Layout::Packed`
> boxes it (`builder: Box<LayoutBuilder>`) to keep the engine's
> `Layout` enum variant sizes balanced (clippy
> `large_enum_variant`); `layout_builder()` still returns
> `Option<&LayoutBuilder>` via auto-deref, public API unchanged.
**Goal:** `LayoutBuilder::new` parses the owned `BastDoc` once and
stores it; `build` reuses it. Removes the `layout_builder.rs:190`
re-parse (M1). Non-breaking from the public API perspective
(`new`/`build` signatures unchanged); the change is internal.
**ADR reference:** [ADR-012 §2a](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#2a-layoutbuilder--cache-the-parsed-bastdoc-at-new).
**Files:** `src/layout_builder.rs`.
**Implementation notes:**
- `LayoutBuilder` currently stores `doc_value: Value` + `root_name:
String` + `endian: Endian`. After phase 3, it stores `doc: BastDoc`
(owned) + `endian: Endian`. `new` calls `BastDoc::new` once;
`build(&self, var_sizes)` uses `&self.doc` directly — no
`BastDoc::new` call inside `build`.
- The `BuildCtx<'d>` struct (currently `doc: &'d BastDoc<'d>`) becomes
`doc: &BastDoc` (no lifetime, or a single lifetime for the borrow
from `&self`). The walk logic is unchanged.
- The `doc_value: Value` clone is removed; the builder holds the owned
`BastDoc` directly. `endian` is read from the doc at `new` time
(already is).
**Verification:** `cargo test --release` — the existing
`layout_builder.rs` tests pass unchanged (they go through
`LayoutBuilder::new` + `build`). `cargo clippy --all-targets -- -D
warnings`. `cargo build --target wasm32-unknown-unknown --release`.
---
## Phase 5 — `OffsetMap` carries `LeafMeta` (ADR-012 §2b) — **DONE (2026-09-02)**
> **Status: implemented.** Prerequisite first (review #005 M2):
> `Hash` added to `Endian`/`VariableEncoding` derives in `schema.rs`
> (additive; both are fieldless `Eq` enums). New public types
> `LeafMeta { kind, encoding, endian }` (`Copy + PartialEq + Eq +
> Hash`) and `OffsetEntry { range, meta }` (with `start()`/`end()`
> convenience accessors), both re-exported from `lib.rs`; `ByteRange`
> gained `Hash` (additive). Storage is `Vec<(String, OffsetEntry)>`
> (deferred decision 2 resolved: struct — `get` returns
> `Option<&OffsetEntry>`, `iter` yields `(&str, &OffsetEntry)`).
> `LeafMeta` is computed at `compute` time with **effective** endian
> threaded through the walk: container default → field override per
> field, propagated into nested-struct probes and array elements via
> the referring field (the same propagation the aligned materializer
> uses). **Parity note:** this replaces `engine.rs`'s
> `lookup_leaf_field` walk, which computed nested-struct defaults from
> the *nested struct's own* `endian` annotation — the two paths
> diverged whenever a nested struct declared `endian` and its
> referring field also declared one (the map now agrees with the
> aligned materializer and the packed `ReadPlan`; the old divergence
> was unreachable through `read_field` only when a nested annotation
> existed, and no test pinned it). `read_field`/`write_field` dispatch
> on the entry's `LeafMeta` — the `BastDoc` re-parse +
> `lookup_leaf_field`/`LeafFieldInfo` per access are gone (the last
> two M1 sites, engine.rs `read_field`/`write_field`). Behavior
> change: `read_field` on a path absent from the map (e.g. a
> whole-struct field path) now errors with `Offset` ("field not found
> in offset map") instead of `Access` ("does not support composite
> types") — the composite-path test already accepted either variant.
> `materialize_aligned`'s four `offset_map.get` call sites updated to
> `.range.start`. `alktty`/`alkcall` untouched (the bench never uses
> `OffsetMap::get`; alkcall has no dependency yet).
**Goal:** Extend `OffsetMap`'s entries with `LeafMeta { kind, encoding,
endian }` computed at `compute` time. `read_field`/`write_field` drop
the `BastDoc::new` + `lookup_leaf_field` calls (M1 aligned-side).
Closes the last two M1 sites (`engine.rs:334,467`).
**ADR reference:** [ADR-012 §2b](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#2b-offsetmap--carry-leaf-metadata).
**Files:** `src/offset_map.rs` (extend entries, compute `LeafMeta`),
`src/engine.rs` (rewrite `read_field`/`write_field` to use the map's
`LeafMeta`, remove `lookup_leaf_field` + `LeafFieldInfo`), `src/lib.rs`
(re-export `LeafMeta`).
**Implementation notes:**
- **`Hash` on `Endian`/`VariableEncoding` (resolves review #005 M2 —
do this first, it's a prerequisite):** `src/schema.rs:205` (`Endian`)
and `:212` (`VariableEncoding`) currently derive only `Debug, Clone,
Copy, PartialEq, Eq` — no `Hash`. `LeafMeta` (below) requires all
its fields to be `Hash` for `#[derive(Hash)]`, and phase 6's
`#[derive(Hash)]` on `ReadPlan`/`OffsetMap` requires `FieldPlan`'s
`endian: Endian` + `encoding: VariableEncoding` to be `Hash`. Add
`Hash` to both derives in `src/schema.rs`. Both are fieldless enums
already at `Eq + PartialEq`, so this is additive and semver-safe —
no behavioral change. Trivial, but it's an unstated prerequisite
the original plan omitted.
- New public type `LeafMeta { kind: AlkTypeKind, encoding:
VariableEncoding, endian: Endian }`. `Copy + PartialEq + Eq + Hash`
(all fields are `Copy + Hash` once the sub-step above is done —
`AlkTypeKind` already derives `Hash`; `Endian`/`VariableEncoding`
get it from the sub-step above).
- `OffsetMap` storage: `fields: Vec<(String, ByteRange, LeafMeta)>`
(was `Vec<(String, ByteRange)>`). The `compute` walk already resolves
each leaf's type; add the `LeafMeta` extraction at the point where
the leaf `ByteRange` is recorded.
- **`OffsetMap::get` return type (deferred decision 2):** change to
`get(field_path) -> Option<&OffsetEntry>` where `pub struct
OffsetEntry { range: ByteRange, meta: LeafMeta }`. Add
`OffsetEntry` to `lib.rs` re-exports. Callers that used
`map.get(path).unwrap().start` become
`map.get(path).unwrap().range.start`. Update `alktty`/`alkcall` call
sites (in-house).
- `engine.rs::read_field`/`write_field`: drop the
`BastDoc::new(&self.bast_doc, &self.root_name)?` +
`lookup_leaf_field(&doc, field_path)?` calls. Read `LeafMeta`
from `offset_map.get(field_path)?.meta`. The `kind`/`encoding`/
`endian` match arms in `read_field`/`write_field` are unchanged
(they already dispatch on `AlkTypeKind`/`VariableEncoding`/`Endian`).
- Remove `LeafFieldInfo` and `lookup_leaf_field` from `engine.rs`
(subsumed by `LeafMeta` on the map).
- `materialize_aligned` also uses `OffsetMap` — it currently calls
`offset_map.get(&path)?.start` for leaf reads. Update those call
sites to `.range.start`. The materializer's `resolve_typeref` calls
for composite walks stay (composites aren't in the offset map as
leaves; they're walked recursively). The `BastDoc` argument to
`materialize_aligned` is now owned (phase 3) — no signature change
beyond the lifetime drop.
- **Scope Boundary — aligned `materialize`'s `BastDoc` structure walk
(resolves review #005 L3):** `materialize_struct_aligned`
(`materialize.rs:451-521`) walks `BastDoc` to traverse
struct/array/record *structure*, using `OffsetMap` only for leaf
byte positions. This is the **permanent design for 0.3.0**, not a
deferral: after phase 3 the walk is over owned data (no re-parse,
not O(N²)), and aligned `validate_bytes` is one structure walk per
call (not per-field), so there is no perf driver analogous to review
#004's packed per-chunk gap. ADR-011 §"Out of scope" is half-true
here (aligned materialize takes `&OffsetMap` *and* `&BastDoc`) —
this note owns the decision: aligned materialize keeps walking owned
`BastDoc` for structure through 0.3.0. An `AlignedPlan` that
compiles the structure walk is **not** in scope; if a future bench
shows an aligned-mode hot loop, it gets its own ADR (tracked as an
open question, not a silent gap). Phase 7's `ValidationPlan` does
not change this — validation is value-domain, orthogonal to the
aligned structure walk.
**Verification:** `cargo test --release` — existing `offset_map.rs`
and `engine.rs` `read_field`/`write_field` tests pass (they go through
public APIs). `cargo clippy --all-targets -- -D warnings`. `cargo doc
--no-deps` (new public `LeafMeta`/`OffsetEntry`). `cargo build --target
wasm32-unknown-unknown --release`.
---
## Phase 6 — Fingerprinting `ReadPlan`/`OffsetMap` (ADR-012 §1, §4) — **DONE (2026-09-02)**
> **Status: implemented.** `#[derive(Hash, Eq)]` added to `ReadPlan`,
> `FieldPlan`, `CompositePlan`, `ReadKind`, `DiscriminatorPlan`
> (`ReadPlan`'s `schema: Arc<Value>` hashes fine — `serde_json::Value:
> Hash + Eq` under the pinned `preserve_order` serde_json) and to
> `OffsetMap` (`Clone` added alongside; its `LeafMeta`/`OffsetEntry`/
> `ByteRange` payload gained `Hash` in phase 5 / this phase). The
> POC's `VariantPlan`/`VariantKind` don't exist in the production
> shape (phase 1 dropped them). `fingerprint() -> u64` on both via
> `DefaultHasher` (deferred decision 3 resolved: std `DefaultHasher`,
> no new dep; the fingerprint isn't hot; cross-version stability is a
> non-goal per ADR-012). Fingerprint contract tests on both: same
> schema twice → equal `PartialEq` + equal fingerprint; field-kind
> change, field-order change, and endianness change each → different
> fingerprints; (ReadPlan) different root names over the same document
> → different fingerprints. `ValidationPlan` already carries its own
> `Hash + Eq` + `fingerprint` + contract test (phase 7).
**Goal:** Add `Hash + Eq` derives + `fingerprint() -> u64` to `ReadPlan`
and `OffsetMap`. Enables cross-run caching, `alkcall` schema handshake,
schema-version diagnostics. (`ValidationPlan` gets the same treatment
in phase 7, where it's built — it carries its own `Hash + Eq` +
`fingerprint()` as part of its public surface.)
**ADR reference:** [ADR-012 §1](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#1-fingerprinting--readplan-hash--eq-offsetmap-hash--eq),
[ADR-012 §4](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#4-fingerprinting-offsetmap-bundled-with-2b).
**Files:** `src/read_plan.rs` (derives + `fingerprint`), `src/offset_map.rs`
(derives + `fingerprint`), `src/lib.rs` (no new re-exports — `Hash`/`Eq`
are trait derives, `fingerprint` is an inherent method).
**Implementation notes:**
- `ReadPlan` already uses `BTreeMap` for `by_name` (phase 1), so
`#[derive(Hash, Eq)]` works. Add it alongside the existing
`Debug, Clone, PartialEq`. Same for `FieldPlan`, `CompositePlan`,
`ReadKind`, `DiscriminatorPlan`. (The POC's `VariantPlan`/`VariantKind`
are not in the production shape — phase 1 dropped them — so they are
not derived here.) `ReadPlan` also carries `schema: Arc<Value>` from
phase 2; `Arc<Value>: Hash + Eq` because `serde_json::Value: Hash +
Eq` (with `preserve_order`, `Map::hash` sorts keys deterministically),
so the `schema` field does not block the derive. If a future
`serde_json` version regresses `Value: Hash`, exclude `schema` from
the derived `Hash` via a manual `impl Hash for ReadPlan` that hashes
every field except `schema` — phase-6 concern, not a blocker.
- `OffsetMap` already carries `LeafMeta` (phase 5), and `LeafMeta` is
`Copy + Hash + Eq` (phase 5 added `Hash` to `Endian`/`VariableEncoding`).
Add `#[derive(Hash, Eq)]` to `OffsetMap`,
`OffsetEntry`, `ByteRange` (already `Eq + Hash`), `LeafMeta`.
- **Fingerprint hasher (deferred decision 3):** `DefaultHasher` (std,
no new dep). The fingerprint isn't hot; cross-version stability is a
non-goal. `fingerprint()`:
```rust
pub fn fingerprint(&self) -> u64 {
use std::hash::{Hash, Hasher};
let mut h = std::hash::DefaultHasher::new();
self.hash(&mut h);
h.finish()
}
```
- **Fingerprint contract test:** compile the same schema twice, assert
`plan1 == plan2` and `plan1.fingerprint() == plan2.fingerprint()`.
Compile a schema with one field changed, assert fingerprints differ.
This is the contract test for ADR-012 §1's "two plans with equal
hashes produce identical reads over identical bytes."
**Verification:** `cargo test --release` (new contract tests).
`cargo clippy --all-targets -- -D warnings`. `cargo doc --no-deps`.
---
## Phase 7 — `ValidationPlan` (ADR-012 §3) — **DONE (2026-08-31)**
> **Status: implemented.** The design session ran and the shape landed
> in `src/validation_plan.rs`. Summary of what was decided and built
> (full detail in ADR-012 §3a):
>
> - **Shape:** `ValidationPlan { root: ValidNode }` — a constraint tree
> with one arm per value-domain check (`Int`/`I64`/`Uint`/`U64`/
> `Float`/`Bool`/`Str`/`Bytes`/`Enum`/`Struct`/`Union`/`Array`/
> `Record`), `ValidField { name, node }`, `ValidVariant { key, node }`.
> `maxLength` baked into leaf nodes from the owning field at compile
> time. Unions compile to variant nodes only (the declared union
> fields are validated via the variant walk, matching the interpretive
> arm's dispatch-on-`__discriminator` semantics).
> - **`compile` signature:** `compile(&BastDoc) -> Result<Self,
> AlkTypeError>` (the plan table's `&str` param was vestigial).
> - **`bast_validation`:** the interpretive walker is *retired*
> (deleted, not just bypassed); `validate_value` survives as a
> compile-once-per-call wrapper over the plan (one-shot/diagnostic
> use); shared error helper retained.
> - **Engine:** `Arc<ValidationPlan>` built at `compile` in both modes;
> new accessor `validation_plan()`. `validate_bytes` walks the plan.
> **Bonus:** `ValidationPlan::compile` runs before the layout build and
> serves as the engine's cyclic-`$ref` gate. (As of the review #006 H2
> fix, the layout walkers also carry their own guard —
> `walk_guard::check_ref_graph` runs at each standalone entry — so this
> ordering is now belt-and-suspenders rather than the only defense; the
> plan text below predates that fix.)
> - **Remaining phase-7 bench work** (a `validate_bytes`-stream bench
> in alktty) moves with the bench work into phase 8; a spot check
> during development measured plan-validate at ~0.2 µs/call vs ~0.6
> µs for the compile-per-call one-shot it replaced.
**Goal:** Retire the interpretive `bast_validation` walk. Introduce a
`ValidationPlan` — a compile-once-walk-many compiled form over the
BAST document's value-domain constraints — built once at `compile`
time and walked by `validate_bytes` (both modes) per buffer instead
of re-walking `BastDoc`. Closes the latent perf cliff review #005 M3
flagged: after ADR-011 the *read* half of `validate_bytes` is
plan-fast, but the *validation* half still re-walks `BastDoc` per
buffer, which is hot on the `alkcall` read+validate-on-untrusted-stream
common case.
**ADR reference:** [ADR-012 §3](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md#3-validationplan--compile-once-validation-form).
**Predecessor for this phase:** a **design session** to scope the
concrete `ValidationPlan` shape (constraint representation,
`compile`/walk structure, `bast_validation` public-surface review)
**before** implementation begins. ADR-012 §3 fixes the decision (in
0.3.0, compiled form, no per-buffer `BastDoc` walk, `Hash + Eq` +
`fingerprint`) and lists what is *not* decided (the struct/enum
shape, the constraint descriptors, whether `validate_value` is
retired or kept as a wrapper). This phase implements whatever the
design session scopes; the contract below holds regardless of shape.
**Files:** New `src/validation_plan.rs` (the `ValidationPlan` type,
`compile`, walk entry points). `src/bast_validation.rs` (adopt the
plan; `validate_value` either becomes a thin wrapper over the plan
or is retired per the design session's call). `src/engine.rs`
(`compile` builds `Arc<ValidationPlan>` in both modes, stores it;
`validate_bytes` walks `&self.validation_plan` instead of
reconstructing a `BastDoc` for the validator — this removes the
temporary bridge from phase 2). `src/lib.rs` (re-export
`ValidationPlan`).
**Implementation contract (fixed by ADR-012 §3, independent of
shape):**
- **Compile-once-walk-many.** `ValidationPlan::compile` walks the
owned `BastDoc` once (phase 3 made it owned); `validate_bytes`
walks the `ValidationPlan` per buffer, never `BastDoc`. The only
consumers that walk `BastDoc` interpretively after this phase are
the one-shot `*::compile` paths (`ReadPlan::compile`,
`OffsetMap::compute`, `ValidationPlan::compile`,
`LayoutBuilder::new`).
- **Value-domain, not byte-position.** The plan carries constraint
descriptors (enum allowed-sets, integer range bounds, `maxLength`
caps, union variant keys, and any other value-domain checks
`bast_validation` performs today), keyed for dispatch against the
materialized `Value` tree. The shape is different from
`ReadPlan`/`OffsetMap`; the pattern (compiled form, immutable,
shared via `Arc`) is the same.
- **`Send + Sync`.** `ValidationPlan: Send + Sync` (immutable owned
data, no interior mutability) so `Arc<ValidationPlan>` shares from
the `Send + Sync` engine. Add a `static` bound assertion test
mirroring phase 1's `read_plan_is_send_sync`.
- **`Hash + Eq` + `fingerprint()`.** `ValidationPlan` derives
`Debug, Clone, PartialEq, Eq, Hash` and has
`fingerprint() -> u64` (same `DefaultHasher` implementation as
phase 6). The fingerprint contract generalizes: two validation
plans with equal hashes accept/reject identical `(bytes)`
identically. Add a fingerprint contract test (compile the same
schema twice, assert `plan1 == plan2` and `plan1.fingerprint() ==
plan2.fingerprint()`; change one constraint, assert fingerprints
differ).
- **Untrusted-input discipline.** `compile` surfaces malformed
schemas as `AlkTypeError::Schema` (AGENTS.md §3); overflow-safe
arithmetic (AGENTS.md §4). No `unsafe`, no `async`, no new deps,
wasm-clean (AGENTS.md §5–§11).
**What this phase does *not* include (shape-dependent, scoped by the
design session):** the concrete `ValidationPlan` struct/enum, the
constraint-descriptor representation, the `bast_validation`
public-surface decision (`validate_value` retire-vs-wrapper), and any
`AlkTypeError::Validation` variant changes. These are shape questions
the design session resolves; they are *not* a re-opening of the
"ship in 0.3.0" decision, which is fixed in ADR-012 §3.
**Verification:** `cargo test --release` — the existing
`bast_validation.rs` and `engine.rs` `validate_bytes` tests are the
primary validation (they drive validation through the public API and
must pass unchanged, confirming behavioral parity with the
interpretive walk). New unit tests for `ValidationPlan::compile`
covering every constraint kind. New `Send + Sync` assertion test.
New fingerprint contract tests. `cargo clippy --all-targets -- -D
warnings`. `cargo doc --no-deps` (new public type). `cargo build
--target wasm32-unknown-unknown --release` (`validation_plan.rs` is
wasm-relevant). **Re-run the alktty `wire_vs_bast` bench** and, if
the design session scopes one, a `validate_bytes`-on-untrusted-stream
bench alongside `wire_vs_bast` to confirm the validation half of
`validate_bytes` no longer dominates per-buffer.
---
## Phase 8 — Public API bump, docs, verification (ADR-011 step 6, ADR-012) — **DONE (2026-09-02)**
> **Status: implemented.** Version flipped 0.2.0 → 0.3.0; `lib.rs`
> re-exports complete (`ReadPlan` + sub-types, `LeafMeta`,
> `OffsetEntry`, `ValidationPlan` + sub-types from earlier phases).
> Docs: ADR-007 "Cost" section rewritten to the `Arc<ReadPlan>` cost
> (15.7 ns) with the old framing as a historical note (review #004 L2
> closed); ADR-011/012 status blocks flipped to implemented; the
> architecture README ADR table rows updated; `layout-engine.md`
> rewritten for the 0.3.0 surface (`SequentialReader` construction via
> the engine factory, `OffsetMap` `OffsetEntry`/`LeafMeta`/`fingerprint`
> public-types section, `OffsetMap::compute(&BastDoc)` owned signature);
> `SequentialReader` module doc now points at the engine factory.
> Reviews #004 and #005 status flipped to closed. Bench (alktty
> `wire_vs_bast`, re-run on the 0.3.0 tree): read p64 98 ns/chunk
> (hand-rolled 5.7 µs/stream — parity held from phase 2), read p4k
> unchanged, `alktype_layout_build` **180 ns** (was ~1.2 µs — the
> phase-4 owned-doc cache removed the per-build re-parse, ~7x),
> `sequential_reader_new` 15.7 ns (unchanged), write p64 −3%
> (37.9 µs), `engine_compile` 590 µs (unchanged; dominated by
> meta-schema validation). No `validate_bytes`-stream bench was added:
> the phase-7 spot check (~0.2 µs/call plan-validate vs ~0.6 µs
> compile-per-call) stands as the validation-half measurement; a
> dedicated bench remains a follow-up if `alkcall` profiling motivates
> it. Downstream: `alktty` compiles against the path dep unchanged
> (the bench uses `LayoutBuilder::new`/`build` and
> `engine.sequential_reader()` — no touched signatures);
> `alkcall` has no dependency yet.
**Goal:** Flip the version to 0.3.0, update `lib.rs` re-exports, update
the architecture docs (ADR-007 "Cost" rewrite, ADR-011/012 status flip
if not already, README ADR table), update in-house downstream
consumers, run the full verification block.
**ADR reference:** [ADR-011 §Public API change](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md#public-api-change-breaking--version-bump-to-030),
[ADR-012](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md).
**Files:** `Cargo.toml` (version 0.2.0 → 0.3.0), `src/lib.rs`
(re-export `ReadPlan`, `LeafMeta`, `OffsetEntry`, `ValidationPlan`),
`docs/architecture/` (README ADR table, ADR-007 "Cost" section,
ADR-011/012 status), `docs/architecture/validation.md` /
`layout-engine.md` (mention `ReadPlan`/`LeafMeta`/`ValidationPlan`
where relevant), in-house downstream repos (`alktty`, `alkcall` —
update call sites for `OffsetMap::get`, `SequentialReader::new`,
`materialize_packed`, `BastDoc` owned, `validate_bytes` internal
change if any signature change surfaced in phase 7's shape work).
**Implementation notes:**
- **ADR-007 "Cost" section (L2 from review #004):** rewrite the
"re-parse on demand" paragraph to describe the `Arc<ReadPlan>` cost
and the owned-`BastDoc` cache. The factory decision itself stays
"Accepted." This is the last loose end from review #004.
- **`src/engine.rs:112-115` doc comment (L2):** rewrite the "re-parse
the typed tree on demand" comment to describe the compiled-form
architecture (`ReadPlan` for packed reads, `OffsetMap`+`LeafMeta`
for aligned, `ValidationPlan` for validation, owned `BastDoc` for
the builder and the `*::compile` paths).
- **`lib.rs` re-exports:** add `ReadPlan`, `LeafMeta`, `OffsetEntry`,
`ValidationPlan`. `BastDoc` and `Bast*` stay re-exported (signatures
changed in phase 3, names unchanged). `materialize_packed`/
`materialize_aligned` stay re-exported (signatures changed).
`SequentialReader` stays re-exported (`new` signature changed).
- **Downstream updates:** `alktty`'s bench (`benches/wire_vs_bast.rs`)
updates `SequentialReader::new` call + any `OffsetMap::get` usage.
`alkcall` updates similarly. Both are in-house path dev-deps; the
updates ride this release's commits (or follow-on commits in those
repos — they're separate repos, but the path dev-dep means a local
update is immediate).
- **`Cargo.toml` version bump:** `0.2.0` → `0.3.0`. The workspace
section added for the POC (`[workspace] members = ["poc/readplan"]`)
stays on the `readplan-poc` branch and is *not* merged to main — the
POC branch is derisking-only, like `bast-validator-poc`. If the POC
files ever merge to main, drop the workspace section (the POC is
disposable).
**Verification block (run all, all must pass):**
```bash
cargo test --release # full suite
cargo clippy --all-targets -- -D warnings
cargo doc --no-deps # new public types
cargo build --target wasm32-unknown-unknown --release # wasm-clean
cargo publish --dry-run --allow-dirty # before publish
```
Plus: **re-run the alktty `wire_vs_bast` bench** and record the
before/after numbers in the release commit message. The 400x gap
should close to within ~2–5x of hand-rolled (the `data_access` calls
are the same; the remaining gap is the `match` dispatch + `Arc` refcount
vs hand-rolled's direct calls). The SFTP-shaped union case (the one
ADR-011's framing argument cared about) should close further because
eager `$ref` resolution removes the `resolve_typeref_as_def` per-
variant dispatch cost.
---
## Cross-phase invariants
- **The tree builds and tests pass at every phase boundary.** No phase
leaves the crate in a non-compiling state. Phases 1 (add `ReadPlan`),
6 (add `Hash`/`Eq` derives to `ReadPlan`/`OffsetMap`), and 7 (add
`ValidationPlan`) are pure additions; phases 2–5 are rewrites that
must leave tests green; phase 8 is the bump/docs.
- **The POC on `readplan-poc` is the reference scaffold for phases 1–2.**
It is *not* merged to main; it stays on the branch as the derisking
record, like `bast-validator-poc`. If a phase 1–2 implementation
question arises about the plan shape, consult the POC. (The POC's
`VariantPlan`/`VariantKind` and its nested-union rejection are
**not** carried forward — phase 1's refined `CompositePlan::Union`
shape supersedes both; see phase 1.)
- **Review #004 is the closure target.** H1 → phase 2; packed M1 →
phase 2; aligned M1 → phases 4–5; L1 → phase 2 (falls out); L2 →
phase 8 (doc rewrite). The review's status flips to "closed" in the
phase 8 commit.
- **Review #005 is the closure target for the plan-spec issues.** H1
→ phase 1 (refined union shape in ADR-011 + plan); H2 → phase 2
(`Arc<Value>` on the plan); M1 → phase 1 (nested unions via
`CompositePlan` recursion, no behavioral drop); M2 → phase 5 (`Hash`
on `Endian`/`VariableEncoding`); M3 → ADR-012 §3 + phase 7
(`ValidationPlan` shipped in 0.3.0, deferral reversed); L1/L2/L3 →
phase 2 / phase 5 Scope Boundary; N1 → typo; N2 → phase 1
`Send + Sync` assertion test; N3 → Semver Contract table row.
- **No `unsafe`, no `async`, no new deps, no feature flags** (AGENTS.md
§5–§11). The owned-`BastDoc` refactor uses `String`/`Value`, not
`unsafe` self-referential tricks. `DefaultHasher` is std. Wasm-clean
throughout. `ValidationPlan` follows the same constraints.
- **`preserve_order` stays load-bearing** (AGENTS.md §8). The owned-
`BastDoc` refactor must not sort schema object keys anywhere; field
order in the `Value` still determines byte order in packed mode and
iteration order in both modes. `ValidationPlan::compile` inherits
this — value-domain checks that depend on field ordering (e.g. union
discriminator field lookup) respect `preserve_order`.
## What this plan is *not*
- **Not a disk-cache or wire-protocol spec.** The fingerprint contract
and method are in scope (phase 6 for `ReadPlan`/`OffsetMap`, phase 7
for `ValidationPlan`); downstream uses are the consumers' concern.
- **Not cross-version fingerprint stability.** Within-version only
(ADR-012). The fingerprint may change across versions if a new
`AlkTypeKind` variant is added; consumers cache within a version.
- **Not a perf bench.** The bench lives in alktty; this plan re-runs it
at phase 2 and phase 8 to confirm the gap closes. The plan itself
only asserts correctness/coverage.
- **Not an `AlignedPlan`.** Aligned `materialize`'s `BastDoc` structure
walk is the permanent 0.3.0 design (phase 5 Scope Boundary). An
`AlignedPlan` is out of scope; if a future bench motivates one, it
gets its own ADR.
docs/reviews/004-performance-review.md
+412
View File
@@ -0,0 +1,412 @@
---
status: closed
last_updated: 2026-09-02
reviewed_artifacts:
- src/sequential_reader.rs
- src/bast.rs
- src/engine.rs
- src/layout_builder.rs
- src/materialize.rs
- src/offset_map.rs
- src/data_access.rs
- src/lib.rs
- docs/architecture/decisions/007-packed-mode-read-factory.md
- ../@alkdev/alktty/benches/wire_vs_bast.rs
tool: manual source read + downstream criterion bench (`cargo bench --bench wire_vs_bast` in alktty)
reviewer: read-path performance review (triggered by alktty wire_vs_bast bench)
---
# Review #004 — Read-Path Performance: the `BastDoc` Re-Parse Gap
## Purpose
A downstream bench in `alktty` (`benches/wire_vs_bast.rs`) compared a
hand-rolled `ChunkHeader` codec against an alktype-driven codec built
from the same 2-field BAST struct (`stream_type: uint8`,
`length: uint32`, big-endian). The bench builds the engine / layout /
reader **once** outside the measured loop, then measures per-chunk read
and write over 1024 contiguous chunks.
The write path is competitive (~2.8x at 64 B, ~1.1x at 4 KiB — the
per-chunk overhead is just `data_access::write_u8`/`write_u32` at
precomputed `PackedLayout` offsets, and the payload copy dominates at
4 KiB). The read path is **~400x slower per chunk** (2.27 µs/chunk vs
5.6 ns/chunk hand-rolled), and the cost is fixed — it dominates at 64 B
*and* at 4 KiB.
This review traces the gap to its source, confirms it is an
implementation gap (not inherent to the design), and lays out the fix
options. The bench itself is honest — its in-file note
(`wire_vs_bast.rs:36-41`) already points at the root cause; this review
formalizes the finding and the remediation plan.
## Methodology
- Full read of the read-path code (`sequential_reader.rs`, `bast.rs`,
`engine.rs`), the write path (`layout_builder.rs`, `data_access.rs`),
and ADR-007 (the packed read factory decision).
- Cross-reference every `BastDoc::new` call site in `src/` to map the
full re-parse surface.
- Trace the lifetime/ownership constraint that forces the re-parse
(`BastDoc<'a>` borrows `&'a Value`; `SequentialReader` owns its
`Value` — self-referential struct, cannot cache the parsed tree).
- Read the downstream bench to confirm the measurement is honest (the
re-parse is inside the measured routine; the engine/reader are built
once outside it).
- Read `docs/reviews/003-code-review.md` for prior context — review #003
did not flag the re-parse (it was a correctness/coverage pass, not a
performance pass).
## Verification Baseline
The bench numbers below are from the alktty downstream tree
(`benches/wire_vs_bast.rs`, criterion 0.7), run against alktype at
commit `cab4932` (v0.2.0). The alktype tree itself is unchanged — this
is a review of existing code, not a fix.
| path | payload=64 B | payload=4 KiB |
|---|---|---|
| hand-rolled read | 5.7 µs (5.6 ns/chunk) | 12.1 µs (11.8 ns/chunk) |
| alktype read | 2.32 ms (2.27 µs/chunk) | 2.38 ms (2.33 µs/chunk) |
| hand-rolled write | 14.3 µs | 321 µs |
| alktype write | 39.8 µs | 355 µs |
One-shot startup costs (paid once): `AlkTypeEngine::compile` 573 µs,
`engine.sequential_reader()` 2.84 µs, `LayoutBuilder::build` 1.30 µs.
The read gap is ~400x per chunk and is **fixed** (does not shrink as
the payload grows), which is the signature of per-field overhead, not
payload-copy overhead.
## Summary Statistics
| Severity | Count |
|----------|------:|
| High | 1 (H1) |
| Medium | 1 (M1) |
| Low | 2 (L1, L2) |
| Nit | 1 (N1) |
The High finding is the read-path re-parse — a severe performance
regression that blocks the primary intended use case (alktype as a
runtime codec for protocol wire formats). It is not a correctness bug;
the re-parse produces the same result every time, it is just
catastrophically wasteful. The Medium finding is the same root cause
manifesting in four one-shot paths. The Low/Nit findings are dead code
and a stale doc comment exposed while tracing the root cause.
---
## Findings
### H1. `SequentialReader::read_field_at` re-parses the BAST typed tree on every field read
**File**: `src/sequential_reader.rs:262`
**Problem**: `read_field_at` calls `BastDoc::new(&self.doc_value,
&self.root_name)?` on every field read. `BastDoc::new`
(`src/bast.rs:69`) recursively parses the root def: `lookup_def_raw`
(hash lookup into the `serde_json` Map), `BastDef::parse` →
`BastStruct::parse` → allocates a `Vec<BastField>`, iterates fields,
`BastField::parse` each (several `node.get().as_str()`/`as_u64()`
calls + a `format!` allocation for the field path), `BastType::parse`
each. For the 2-field `ChunkHeader`, that is ~1 µs per call.
`read_next` calls `read_field_at` once per field, so a 2-field header
incurs **two** `BastDoc::new` calls per chunk = ~2.27 µs/chunk, matching
the bench. For an N-field struct, reading all fields is O(N²) in parse
work (each of N reads re-parses all N fields) — the gap widens with
struct width.
The re-parse is purely redundant: `SequentialReader::new`
(`src/sequential_reader.rs:130`) already parsed the `BastDoc` once at
construction and extracted the field list. `read_field_at` re-derives
the same `field_node` (the `BastField` at `index`) and the same `doc`
that were already in hand at construction time.
**Root cause**: a lifetime/ownership tension, not a logic error.
`BastDoc<'a>` borrows `&'a Value` and `&'a str` throughout
(`src/bast.rs:51-55`). `SequentialReader` **owns** a cloned `Value`
(`doc_value`, `src/sequential_reader.rs:111`). To cache the parsed
`BastDoc` in the reader, the `BastDoc` would have to borrow from
`self.doc_value` — a self-referential struct, which Rust's borrow
checker forbids and safe Rust cannot express without a self-referential
crate (`self_cell`/`ouroboros`, both rejected: new dep per AGENTS.md
§7, and `self_cell`'s soundness relies on `unsafe` the crate avoids per
AGENTS.md §11). So the only way to get a `BastDoc` at read time is to
re-parse from the owned `Value`. The engine's own doc comment
(`src/engine.rs:112-115`) acknowledges this design explicitly:
> "The engine retains a clone of the BAST `Value` so that
> `sequential_reader` and `read_field` can re-parse the typed tree on
> demand without lifetime entanglement with the caller's `Value`."
The "re-parse on demand" framing was a lifetime-entanglement workaround
that did not anticipate the hot-loop cost. ADR-007's "Cost" section
(`decisions/007-packed-mode-read-factory.md:64-70`) argues construction
is cheap ("a `Vec<(String, Value)>` of the `properties` entries ... a
struct has a small number of fields") — true for *construction* (once),
but the decision did not account for a per-field re-parse inside the
read loop.
**Why the write path is fine**: `LayoutBuilder::build`
(`src/layout_builder.rs:190`) also re-parses `BastDoc::new`, but it is
called **once** per write — the resulting `PackedLayout` offsets are
cached and reused. The write loop then does only `data_access::write_*`
at fixed offsets. There is no per-field re-parse on the write hot path,
which is why write is ~1.1x at 4 KiB.
**Fix**: cache the parsed typed tree so the read path does not re-parse.
See "Fix Options" below for the three approaches and the
recommendation.
**Lift**: closes a ~400x read-path gap and unblocks alktype as a runtime
codec for protocol wire formats (its stated purpose per ADR-001). This
is the difference between "plausible runtime codec" and "not viable."
Large effort depending on the chosen option.
---
### M1. The same re-parse pattern exists in four one-shot paths
**Files**: `src/layout_builder.rs:190`, `src/engine.rs:284,334,467`
**Problem**: `BastDoc::new(&self.doc_value, &self.root_name)?` /
`BastDoc::new(&self.bast_doc, &self.root_name)?` is re-called in:
- `LayoutBuilder::build` (`src/layout_builder.rs:190`) — once per
`build()` call. Re-parsing on each build is wasteful if a builder is
reused across writes, but the typical pattern is build-once-reuse,
so this is mild.
- `AlkTypeEngine::validate_bytes` (`src/engine.rs:284`) — once per
validate call. For a stream of buffers, this is a per-buffer re-parse.
- `AlkTypeEngine::read_field` (`src/engine.rs:334`, aligned mode) — once
per field read. Same class as H1 but for aligned random access, and
one parse per field read (not the O(N²) of the sequential reader).
- `AlkTypeEngine::write_field` (`src/engine.rs:467`, aligned mode) —
once per field write.
These are less acute than H1 (one parse per operation, not per-field-
in-a-loop), but they share the same root cause: the engine/builder own
a `Value` and cannot cache a borrowing `BastDoc`. Any fix that makes
`BastDoc` cacheable on an owning struct (Fix Option A) closes these for
free; a read-path-only fix (Option B) leaves them as-is, which is
acceptable since they are not hot loops.
**Lift**: removes redundant parse work on the validate/aligned paths.
Free with Option A; deferred with Option B.
---
### L1. `read_field_value` carries a dead `_field_schema` parameter; `fields` stores dead `Value` clones
**Files**: `src/sequential_reader.rs:114,294`
**Problem**: `SequentialReader` stores `fields: Vec<(String, Value)>`
where the `Value` is `f.source().clone()` per field
(`src/sequential_reader.rs:145`). `read_field_at` passes this as
`_field_schema` to `read_field_value` (`src/sequential_reader.rs:294`),
where it is unused (prefixed `_`). The raw `Value` clone per field is
dead weight — only the `String` name is used (for `read_next`'s return
and `read_field`'s lookup). This is a minor allocation cost on top of
H1's re-parse, and it will be removed naturally when the read plan is
precomputed (Option B) or the `BastDoc` is cached (Option A), since
both replace `Vec<(String, Value)>` with typed/owned field data.
**Lift**: trivial; falls out of the H1 fix.
---
### L2. ADR-007 "Cost" section and the engine doc comment understate the re-parse
**Files**: `docs/architecture/decisions/007-packed-mode-read-factory.md:64-70`,
`src/engine.rs:112-115`
**Problem**: ADR-007's "Cost" section argues `sequential_reader()` is
cheap because it clones a small `Vec` of field schemas. That is true for
the factory call (once). But the decision did not anticipate that
`read_field_at` would re-parse `BastDoc::new` per field — the cost that
actually dominates. The engine doc comment at `src/engine.rs:112-115`
explicitly frames re-parse-on-demand as the intended design ("re-parse
the typed tree on demand without lifetime entanglement"), which is the
root cause H1 traces.
**Fix**: whichever fix option is chosen, update ADR-007's "Cost" /
"Consequences" section and the engine doc comment to reflect that the
parsed tree is now cached (Option A) or precomputed into a read plan
(Option B), and that the "re-parse on demand" framing is retired.
**Lift**: documentation accuracy; prevents the same framing from
misleading a future edit.
---
### N1. `BastType::alk_kind()` returns `Struct` for any `$ref` (carry-forward from review #003 N2)
**File**: `src/bast.rs:715-725`
**Problem**: flagged in review #003 N2 and left as "defer unless it
bites." It does not bite here — `read_field_value` always calls
`doc.resolve_typeref(ty)` before matching on `BastType`, so the
misreporting `alk_kind` is never consulted on a `Ref`. Noting it only
because this review re-read the same path; no new action beyond review
#003's deferral.
---
## Fix Options
The core constraint: `BastDoc<'a>` borrows `&'a Value` / `&'a str`; an
owning struct (`SequentialReader`, `AlkTypeEngine`, `LayoutBuilder`)
cannot store a `BastDoc` that borrows from its own `Value` field
(self-referential). Three ways to break the constraint:
### Option A — Make the typed tree own its data (principled fix)
Change `BastDoc<'a>` → `BastDoc` (no lifetime), `&'a str` → `Arc<str>`
(or `String`), `&'a Value` → `Arc<Value>` (or `Value`). Then
`SequentialReader`, `AlkTypeEngine`, and `LayoutBuilder` each hold a
`BastDoc` directly (built once at construction), and `read_field_at`
uses `&self.doc` — no re-parse, anywhere.
- **Closes**: H1, M1 (all four one-shot paths), and the engine/reader
lifetime entanglement that ADR-007 worked around. The engine's
`bast_doc: Value` clone (`src/engine.rs:85`) becomes redundant with
the owned `BastDoc`.
- **Tradeoff**: broad refactor. Touches `bast.rs` (every typed node)
and every consumer (`layout_builder`, `offset_map`, `materialize`,
`sequential_reader`, `tunion`, `bast_validation`, `engine`). The
`Bast*` types are re-exported in `lib.rs:57-60`, so this is a
**breaking public-API change** — `BastDoc<'a>` becomes `BastDoc`,
and every method signature that took `&'a` changes. Per AGENTS.md,
this is semver-relevant and would need a version bump (0.2.0 → 0.3.0).
- **Dependency cost**: `Arc<str>`/`Arc<Value>` add `alloc` (already in
use via `Vec`/`String`); no new external deps. Stays wasm-clean. The
`preserve_order` serde_json feature remains load-bearing (AGENTS.md
§8) — owning the `Value` does not change field-order semantics.
- **Effort**: large but mechanical. The borrow-based design was chosen
for "allocation-free beyond the small typed nodes" (`src/bast.rs:11-
18`), but the re-parse-per-field already defeats that goal by
allocating a fresh `Vec<BastField>` per read. Owning the data makes
the "parse once, walk many times" invariant actually hold.
### Option B — Precompute an owned read plan in `SequentialReader::new` (surgical fix)
Keep `BastDoc<'a>` borrowing for the other consumers. In
`SequentialReader::new`, parse the `BastDoc` once, resolve all `$ref`s
eagerly, and build a flat, owned tree of read instructions
(`Vec<FieldPlan>`) that the read loop walks with no `BastDoc`
involvement. Each `FieldPlan` carries the field name, the resolved
`AlkTypeKind`, the effective `Endian`, and for composites a nested
plan (struct → sub-plans; union → discriminator + per-variant plans;
array → element plan + count; record → value plan).
- **Closes**: H1 only. M1 (the one-shot re-parses) remains, which is
acceptable since they are not hot loops.
- **Tradeoff**: non-breaking (internal to `sequential_reader.rs`; the
public `SequentialReader` type and its methods keep their
signatures). Duplicates some of the `BastType` matching logic that
`read_field_value`/`read_union_value`/etc. already encode, so there
are two parallel walkers to maintain.
- **Effort**: medium. Self-contained in one file but non-trivial
(composites require recursively resolving and pre-flattening the
type tree, including `$ref` chains into `$defs`).
### Option C — Cache `BastDoc` on the engine, reader borrows (rejected)
Have the engine own the parsed `BastDoc` and return a
`SequentialReader<'_>` that borrows from `&self`. This requires
`BastDoc` to be owned (Option A prerequisite) *and* changes
`sequential_reader() -> Option<SequentialReader>` to
`-> Option<SequentialReader<'_>>` — a breaking public-API change that
also contradicts ADR-007's "owned fresh reader" decision. Strictly
worse than Option A (same refactor cost, more API churn, contradicts
an ADR). Rejected.
### Recommendation
**Option A**, given the publisher's stated willingness to make breaking
changes ("no one is using this except us yet; ... we can change things
now"). It is the only option that closes H1 *and* M1 and retires the
lifetime-entanglement workaround that caused both. The refactor is
broad but mechanical (lifetime removal, not logic rewrites), and the
crate is pre-1.0 with only two in-house downstream consumers
(`alktty`, `alkcall`), so the breakage cost is bounded and known.
Option B is the fallback if the Option A refactor is deferred — it
closes the acute H1 gap non-breakingly while leaving M1 for later. It
is not the recommended path because it leaves a second parallel type
walker in the crate and does not address the root cause (the borrow-
based `BastDoc` design), which will keep forcing re-parses anywhere a
new owning consumer wants to cache the parsed tree.
Regardless of the chosen option, ADR-007's "Cost"/"Consequences"
section and the `src/engine.rs:112-115` doc comment should be updated to
retire the "re-parse on demand" framing (L2).
---
## What's Good
- **The bench is honest.** The alktty bench builds the engine/reader
once outside the measured loop and correctly isolates the per-chunk
logic. Its in-file note (`wire_vs_bast.rs:36-41`) already points at
the `read_field_at` re-parse and labels it "the honest current cost of
the alktype read path, not a bench bug." This review confirms that
assessment.
- **The write path is already competitive.** Once `PackedLayout` is
built, the write loop is just `data_access::write_*` at fixed offsets
— no schema walk, no re-parse. This validates the "build once, reuse"
pattern that the read path should also adopt.
- **`resolve_typeref` for primitives is cheap.** `BastType::Primitive`
is `Copy` (`AlkTypeKind: Copy`, `src/schema.rs:25`), so
`resolve_typeref` (`src/bast.rs:117-125`) returns `other.clone()`
without allocation for the common case. The re-parse cost is entirely
in `BastDoc::new`, not in the per-field type resolution — so caching
the `BastDoc` alone closes the gap without restructuring
`resolve_typeref`.
- **Overflow safety and error attribution are unaffected.** The
`checked_add` / `usize::try_from` discipline (AGENTS.md §4) and the
`field_path`-carrying errors (review #003 "What's Good") are in the
read functions, not the parser — a caching fix preserves them.
---
## Recommended Order
1. **H1 + M1 (Option A)** — the owned-typed-tree refactor. Decide
first (this is a one-way door: breaking public-API change, version
bump to 0.3.0). If approved, this is one refactor that closes both.
2. **L2** — update ADR-007 and the engine doc comment in the same
commit as the H1 fix, since the "re-parse on demand" framing is
being retired.
3. **L1** — falls out of the H1 fix (the dead `Value` clones are
replaced by the cached/owned field data).
4. **N1** — remains deferred per review #003.
If Option A is deferred, **H1 (Option B)** is the standalone
alternative — non-breaking, closes the acute gap only.
---
## Notes
- All line numbers refer to the tree at commit `cab4932` (v0.2.0, the
BAST pivot release).
- The bench is in the `alktty` downstream repo
(`/workspace/@alkdev/alktty/benches/wire_vs_bast.rs`), not in alktype.
alktty depends on alktype as a path dev-dep for the bench only; it
does not use alktype at runtime. The path dep means
`cargo publish --dry-run` for alktty would complain (the bench is
exploratory and uncommitted in alktty; the alktype crate itself has
no bench dependency).
- This review does not cover the wasm build (`cargo build --target
wasm32-unknown-unknown`) because the fix is not yet implemented; the
verification block for the fix should include it per AGENTS.md, as
the typed-tree ownership change touches `bast.rs` which is
wasm-relevant.
- Review #003 (post-BAST-pivot correctness review) did not flag the
re-parse — its scope was correctness, coverage, and panic safety, not
performance. The re-parse is not a correctness regression; the parsed
tree is identical across calls. This review complements #003 by
adding the performance axis.
docs/reviews/005-plan-review-030.md
+659
View File
@@ -0,0 +1,659 @@
---
status: closed
last_updated: 2026-09-02
resolved_findings: 2026-08-20 (all 11 — see "Resolution" at the end)
reviewed_artifacts:
- docs/plans/030-compiled-forms.md
- docs/architecture/decisions/011-compiled-read-plan-for-packed-mode.md
- docs/architecture/decisions/012-plan-fingerprinting-and-m1-closure.md
- docs/reviews/004-performance-review.md
- src/lib.rs
- src/bast.rs
- src/engine.rs
- src/sequential_reader.rs
- src/materialize.rs
- src/offset_map.rs
- src/layout_builder.rs
- src/schema.rs
- poc/readplan/{src/lib.rs, FINDINGS.md} (branch readplan-poc)
tool: manual source read + plan-vs-codebase cross-check + POC branch inspection
reviewer: 0.3.0 implementation plan review (triggered before phase 1)
---
# Review #005 — 0.3.0 Plan Review: Compiled Forms
## Purpose
The 0.3.0 implementation plan
([`docs/plans/030-compiled-forms.md`](../plans/030-compiled-forms.md)) is
the entry point an implementing agent reads first. It rolls up
[ADR-011](../architecture/decisions/011-compiled-read-plan-for-packed-mode.md)
(the `ReadPlan` packed read-side compiled form),
[ADR-012](../architecture/decisions/012-plan-fingerprinting-and-m1-closure.md)
(fingerprinting + owned `BastDoc` + `OffsetMap` `LeafMeta`), and the
fingerprinting work into one breaking bump. The plan is deliberately
structured as seven phases so each can be picked up by a fresh session
without prior context.
This review's purpose is to find planning-spec mistakes — factual
errors, contradictions, undocumented behavioral changes, hedges into an
unplanned future — *before* a phase-by-phase implementation starts,
because fresh-session implementations are reliable precisely when the
spec is accurate. A spec that contradicts the code or an ADR forces the
agent to either reverse-engineer the actual intent or guess, and the
failure rate goes up.
The review explicitly scans for the "deferral black hole" pattern: a
plan or ADR puts work off into a "future version/phase/downstream" with
no concrete reactivation condition, the next agent inherits the gap,
and the gap festers until something forces an untangle. This is a
known LLM-planning quirk distinct from classic planning mistakes, and
a default scan for it is part of this review's methodology.
## Methodology
- Full read of the plan and its two companion ADRs (011, 012), the
performance review (#004) the plan closes, and the POC findings on
branch `readplan-poc`.
- Cross-check every line-number reference and `src/` claim in the plan
against the actual codebase at `main` (commit `2310f6c`, v0.2.0).
Verified: `engine.rs:112-115,284,334,467`; `sequential_reader.rs:567`;
`layout_builder.rs:190`; `bast.rs:51-55`; `lib.rs` re-export list;
`Cargo.toml` version; presence of `poc/` (absent on main, present on
`readplan-poc` as expected); presence of `alktty`/`alkcall` downstream
path dev-deps.
- Cross-check the POC's `ReadPlan`/`CompositePlan` shape against both
ADR-011's shape section and the plan's phase-1 description.
- Cross-check the plan's phase 2 rewrite claims (`dummy_field_for`/
`ty_source` "are removed") against `materialize.rs`'s actual call
sites across both packed and aligned paths.
- Verify the derives the plan relies on (`Hash` on `LeafMeta`,
`ReadPlan`, `OffsetMap`) are reachable from the derives on their
constituent types in `src/schema.rs`.
- Scan for the deferral pattern by flagging every "future/deferred/
later/downstream/if needed" occurrence and asking: (a) is there a
concrete reactivation trigger? (b) is the decision owned or silent?
(c) does inaction have a cost that the deferral framing hides?
## Verification Baseline
The plan and both ADRs were read at the tree state at commit `2310f6c`
("Propose ADR-012 + 0.3.0 implementation plan"), which is `main` HEAD.
The codebase is v0.2.0 (`Cargo.toml`); the POC lives on branch
`readplan-poc` and is not merged, as the plan states. All line-number
references in the plan were verified correct against this tree.
## Summary Statistics
| Severity | Count |
|----------|------:|
| High | 2 (H1, H2) |
| Medium | 3 (M1, M2, M3) |
| Low | 3 (L1, L2, L3) |
| Nit | 3 (N1, N2, N3) |
The two High findings are correctness/contradiction issues that would
block or mislead an implementing agent. The Mediums are either
undocumented behavioral drops, missing implementation prerequisites, or
a deferral worth re-evaluating. Lows and Nits are wording/typo-level.
---
## Findings
### H1. Phase 1's field-name-discriminator union shape exists in neither ADR-011 nor the POC
**File**: `docs/plans/030-compiled-forms.md:144-152`
**Problem**: The plan describes the field-name-discriminator union read
shape as:
> `CompositePlan::Union` carries the union's declared `fields` as a
> sub-`ReadPlan` (the discriminator field + any shared fields), and the
> variant plans are laid out *after* the shared fields.
But ADR-011 §"The `ReadPlan` shape"
(`011-compiled-read-plan-for-packed-mode.md:144-158`) defines:
```rust
pub enum CompositePlan {
Struct(ReadPlan),
Union {
disc: DiscriminatorPlan,
variants: Vec<(String, VariantPlan)>,
},
...
}
```
There is no field for shared/declared fields on the `Union` variant.
The POC (`readplan-poc:poc/readplan/src/lib.rs`) matches the ADR's
shape — `CompositePlan::Union { disc, variants }` only — and its
`compile_union` does not carry shared fields. The POC's `FINDINGS.md`
Finding 1 (the same one the plan cites at lines 143-152) explicitly
says:
> `plan_read_union`'s `Field` arm is a stub that returns an error.
> ... The plan needs a sub-struct for the union's declared fields,
> separate from the variant plans.
So the plan describes a shape that exists in **neither** the accepted
ADR **nor** the reference POC, and presents it as "the production
version must implement it" within the existing `CompositePlan::Union`
shape. An implementing agent reading ADR-011 + plan + POC gets three
different `CompositePlan::Union` shapes and no guidance on where the
shared-fields sub-`ReadPlan` goes (a new `shared: Option<Box<ReadPlan>>`
field? a wrapper enum? two-variant split?).
This is a shape extension to an accepted ADR's public type. The plan
either needs to flag it as an ADR-011 refinement (with the ADR updated
first) or specify the concrete shape the agent should build.
**Lift**: unblocks phase 1. Without resolution, the agent will either
guess the shape and likely diverge from intent, or stop and ask.
---
### H2. `schema()` returning `&Value` from a `&Value` "stored on the plan" is a self-referential struct
**File**: `docs/plans/030-compiled-forms.md:188-191`
**Problem**: Phase 2 says:
> `schema()` returns a `&Value` retained on the plan (the plan stores
> the `&Value` it was compiled from — see ADR-011 §Engine integration;
> the `&Value` outlives the plan because the engine owns both).
The "the plan stores the `&Value` it was compiled from" is the
self-referential struct pattern ADR-011 §"Root cause"
(`011-...md:50-58`) explicitly identifies as impossible in safe Rust
and rejects. The engine owns `bast_doc: Value` and `Arc<ReadPlan>`. If
`ReadPlan` stores `&Value` borrowing from the engine's `bast_doc`, the
engine is self-referential — exactly the construction ADR-007 worked
around with "re-parse on demand" and ADR-011's `Arc<ReadPlan>` was
meant to retire. ADR-011 line 204 specifies the plan is "immutable
**owned** data"; it does not say the plan stores a `&Value`.
`schema()`'s current contract (`src/sequential_reader.rs:247`) is to
return the raw BAST `Value` the reader was built from. To preserve
that contract on `Arc<ReadPlan>` without a self-referential borrow,
`ReadPlan` must store an `Arc<Value>` (engine builds `Arc<Value>` at
compile time, hands a clone to the plan) or an owned `Value`. Then
`schema()` returns `&self.plan.value`. The plan should specify which.
**Lift**: prevents an agent from getting stuck in phase 2 trying to
make `&Value` in `Arc<ReadPlan>` work, which the borrow checker will
reject.
---
### M1. Nested unions: POC rejects a schema 0.2.0 accepts — undocumented behavioral drop
**Files**: `poc/readplan/FINDINGS.md` ("What this POC does not cover"),
`docs/plans/030-compiled-forms.md` (silent), `src/sequential_reader.rs:789-815`
**Problem**: The POC's `compile_union` rejects a union variant that is
itself a union with `AlkTypeError::Schema`. The existing reader
supports this: `resolve_and_walk_variant` at
`src/sequential_reader.rs:800` has a live `BastDefKind::Union` arm
that recurses via `read_union_value`. So 0.2.0 accepts and reads
nested-union schemas; phase 1's `ReadPlan::compile` (per the POC the
plan cites as the reference scaffold) would reject the same schema.
The plan's phase 1 calls out two POC findings explicitly (field-disc
union shape → H1 above, struct-array stride → deferred decision 4)
and says "the production version must implement/decide these." It does
**not** call out the nested-union rejection. An agent following the
plan would inherit the POC's reject-nested-unions behavior by default,
silently dropping a 0.2.0 capability — a behavioral regression that
rides the 0.3.0 bump without being listed in the Semver Contract
table.
This is also the cleanest example of the deferral-black-hole pattern
in the plan: the POC says "if a real schema needs it, the
implementation step adds a `VariantKind::Union` read path. Not
blocking — no current schema exercises it." The "if needed" framing
has no trigger, no OQ, no tracking — it's a black hole. The next agent
inherits the gap.
**Lift**: either (a) add `VariantKind::Union` read path in phase 1
(small — mirrors the existing `resolve_and_walk_variant` Union arm,
~20 lines), or (b) list it in the Semver Contract table as a
behavioral drop with a one-line OQ tracking the deferral. Given the
plan says there are zero real consumers, (b) is defensible, but it
must be *stated*, not silent. (a) is cheap and avoids the regression.
---
### M2. `Endian` and `VariableEncoding` don't derive `Hash` — phases 5/6 will not compile
**Files**: `src/schema.rs:205,212`, `docs/plans/030-compiled-forms.md:62,357,411-415`
**Problem**: `src/schema.rs:205` (`Endian`) and `:212`
(`VariableEncoding`) both derive only `Debug, Clone, Copy, PartialEq,
Eq` — no `Hash`. The plan requires:
- Phase 5 (line 62, 357): `LeafMeta { kind, encoding, endian }` as
`Copy + PartialEq + Eq + Hash`.
- Phase 6 (lines 411-415): `#[derive(Hash, Eq)]` on `ReadPlan`/
`OffsetMap`, and `FieldPlan` carries `endian: Endian` + `encoding:
VariableEncoding`.
Both derives will fail to compile: `#[derive(Hash)]` on a struct
requires all fields to be `Hash`. The plan never mentions adding
`Hash` to these two enums. The fix is trivial (both are fieldless
enums, already `Eq + PartialEq`, so adding `Hash` is semver-safe —
additive, no behavioral change), but it's a prerequisite the plan
omits. An agent working phase 5 will hit a compile error and have to
diagnose why.
**Lift**: trivial. Add a sub-step to phase 5 (or 6): "Add `Hash` to
`Endian` and `VariableEncoding` derives in `src/schema.rs`." This is
additive and safe to do earlier if convenient.
---
### M3. `ValidationPlan` deferral worth re-evaluating — the read+validate common case
**Files**: `docs/architecture/decisions/012-...md:64-73` ("Deferring
`ValidationPlan`"), `docs/plans/030-compiled-forms.md:526-528`,
`docs/reviews/004-performance-review.md` (the read-path perf review)
**Problem**: ADR-012 defers a `ValidationPlan` as "different shape
(value-domain, not byte-position), not a hot loop, separate ADR if a
bench motivates it." The plan inherits this deferral ("Not a
`ValidationPlan`" at lines 526-528). The deferral framing is "if a
bench motivates it" — a concrete trigger exists, so this is not a
black-hole hedge in the M1 sense.
Flagged for re-evaluation, not because the shape argument is wrong
(it's correct — value-domain checks are structurally different from
byte-position walks), but because the *hot-loop* dismissal may under-
account a common case: **read + validate together on untrusted input.**
Review #004 found the packed read path was 400x slow per chunk due to
per-field `BastDoc` re-parse. ADR-011 closes that. But
`validate_bytes`'s packed path (ADR-010) is `materialize_packed` →
`bast_validation::validate_value` over the materialized `Value`.
After ADR-011, `materialize_packed` walks the `ReadPlan` (fast).
`bast_validation::validate_value` still walks `BastDoc` to check
value-domain constraints — once per `validate_bytes` call, over the
full tree, on every buffer.
For a stream of N untrusted buffers (the `alkcall` hub/spoke topology
accepts schemas from arbitrary internet peers — AGENTS.md §3 — and
the common case is "read incoming frame, validate it before acting"),
`validate_bytes` is called N times. Each call does one `BastDoc`
walk for validation. After ADR-011, the *read* half of `validate_bytes`
is plan-fast; the *validation* half is still a `BastDoc` walk per call.
If validation is the common companion to read on untrusted input,
then skipping validation is risky (accepting untrusted bytes
unchecked) and running it re-walks `BastDoc` per buffer — the same
class of cost review #004 measured for the read path, just on a
different code path.
The argument is not "ValidationPlan has the same shape as ReadPlan"
(it doesn't). The argument is: ADR-012's "not a hot loop" dismissal
may be incomplete, because read+validate on untrusted streams makes
validation hot in the same sense read was hot. The deferral's
trigger ("if a bench motivates it") should be sharpened: either (a)
add a `validate_bytes`-on-untrusted-stream bench to alktty alongside
`wire_vs_bast` and let the bench decide, or (b) reason from the
existing review #004 numbers that the validation walk is
non-trivial and should be planned, not deferred.
This is not a request to implement `ValidationPlan` in 0.3.0. It's a
request to *own the decision*: either the trigger fires (and a
follow-on ADR/phase is scoped, possibly 0.4.0) or it doesn't (and the
deferral stands with a sharper justification than "not a hot loop").
As written, the deferral leaves the cost in the superposition where
it can neither be confirmed nor dismissed.
**Lift**: removes a latent perf cliff for the read+validate-on-
untrusted-input case that 0.3.0 is supposed to make viable.
---
### L1. `dummy_field_for`/`ty_source` are used in aligned `materialize`, not just packed
**Files**: `docs/plans/030-compiled-forms.md:209-210`,
`src/materialize.rs:249,316,351,391,631,650-663`
**Problem**: Phase 2 says:
> The `dummy_field_for`/`ty_source` helpers in `materialize.rs` are
> removed (the plan carries everything).
This is factually wrong. `dummy_field_for` is called at
`src/materialize.rs:631` inside `materialize_leaf_at`, which is called
by the **aligned** path: `materialize_struct_aligned` (line 475),
`materialize_array_aligned` (line 544), `materialize_variable_aligned`
(line 613). Aligned `materialize` keeps walking `BastDoc` through
0.3.0 (plan lines 379-385 confirm), so `dummy_field_for`/`ty_source`
must stay. Only the packed-side call sites (lines 249, 316, 351, 391)
go away when packed-materialize moves to the plan.
**Lift**: doc accuracy. An agent following the plan literally would
remove the helpers and break aligned `materialize`.
---
### L2. `materialize_packed` rewrite scope underspecified — packed-vs-aligned split of `materialize_typeref_packed`
**Files**: `docs/plans/030-compiled-forms.md:207-210`,
`src/materialize.rs:122-200, 498-506, 619-637`
**Problem**: `materialize_typeref_packed` is shared by both packed and
aligned paths — aligned's `materialize_leaf_at` (line 619-637) calls
`materialize_typeref_packed` to read leaves, and aligned's record path
(line 498-506) calls it directly. Phase 2 says
`materialize_packed(&ReadPlan, &[u8])` walks the plan instead of
`BastDoc` but does not state what happens to
`materialize_typeref_packed`.
The honest resolution: packed-materialize gets a new plan-walking
function; aligned keeps `materialize_typeref_packed` via
`materialize_leaf_at`; the function stays (renamed or not) for aligned.
This is two mode-specific paths — the existing design — not a
"parallel walker" in the maintenance-tax sense ADR-011 §"Negative"
(cautioning against) discusses. ADR-011's "one walker" claim (lines
234-237) is specifically about packed read-side (`SequentialReader` +
`materialize_packed` sharing the plan), not packed-vs-aligned, so
there's no ADR contradiction — just an underspecification in the plan.
**Lift**: prevents the agent from having to discover the split
mid-rewrite. Add one line to phase 2: "packed-materialize gets a new
plan-walking function; `materialize_typeref_packed` stays for
aligned's `materialize_leaf_at` and the aligned record path."
---
### L3. `materialize_aligned`'s `BastDoc` structure walk is silent in the plan
**Files**: `docs/plans/030-compiled-forms.md` (silent on this),
`src/materialize.rs:451-521`, `docs/architecture/decisions/011-...md:264`
**Problem**: `materialize_struct_aligned` walks `BastDoc` to traverse
struct/array/record structure, using `OffsetMap` only for leaf byte
positions. ADR-011 §"Out of scope" says "aligned mode is unchanged;
`materialize_aligned` already takes `&OffsetMap`" — which is half
true: it takes `&OffsetMap` for positions but also `&BastDoc` for
structure. The plan inherits the half-truth silently: there's no
statement anywhere that aligned materialize keeps walking `BastDoc`
for structure.
After phase 3 (owned `BastDoc`) + phase 5 (`LeafMeta`), the walk is
over owned data, no re-parse, not O(N²), and aligned `validate_bytes`
is one walk per call (not per-field). There's no perf driver
analogous to review #004's packed per-chunk gap. But the absence of
a driver is not the same as a decision: leaving it silent is a
deferral-by-omission. An implementing agent or future reader can't
tell whether the silence is "this is the permanent design" or "we'll
fix this later."
The decision should be owned. Either (a) add a "Scope Boundary" note
that aligned materialize keeps walking owned `BastDoc` for structure
as the permanent design (with an OQ if a future bench motivates an
`AlignedPlan`), or (b) if a bench motivation is plausible, scope an
OQ to track it. (a) is recommended — no perf driver, and after phase
3 the walk is over owned data, so it's not the re-parse pattern.
**Lift**: removes a silent gap that future agents would otherwise
have to reverse-engineer.
---
### N1. Typo: "back-comat" → "back-compat"
**File**: `docs/plans/030-compiled-forms.md:104-105`
**Problem**: "back-comat" in deferred decision 4.
**Lift**: trivial.
---
### N2. Phase 1 verification omits the `Send + Sync` assertion test ADR-011 requires
**Files**: `docs/plans/030-compiled-forms.md:158-163`,
`docs/architecture/decisions/011-...md:204-206`
**Problem**: ADR-011 §"Engine integration" says "the implementation
should add a `static` bound assertion test to lock it in" for
`ReadPlan: Send + Sync`. Phase 1's verification block lists `cargo
test`, `clippy`, `doc`, `wasm` but no mention of adding the assertion
test. An agent following the plan literally won't add it; the
property is currently true by construction but not asserted, so a
future change could break it silently.
**Lift**: add "add a `fn read_plan_is_send_sync()` assertion test" to
phase 1's verification, mirroring the POC's
`readplan_is_send_sync` test.
---
### N3. `SequentialReader::new` return-type change (`Result` drop) undocumented
**Files**: `docs/plans/030-compiled-forms.md:64`,
`src/sequential_reader.rs:129`, `src/engine.rs:205`
**Problem**: Currently `new(&Value, &str) -> Result<Self,
AlkTypeError>` — fallible (BastDoc parse). After phase 2,
`new(Arc<ReadPlan>)` is infallible (just stores the Arc) → returns
`Self`, not `Result<Self>`. The Semver Contract table (line 64) lists
only the argument-type change, not the `Result` drop.
`engine.rs:205`'s `.ok()` call correspondingly goes away. Minor, but
it's a signature change beyond what's listed.
**Lift**: add a row to the Semver Contract table noting the `Result`
drop.
---
## Deferral-pattern scan (LLM-planning quirk)
As part of the methodology, every "future/deferred/later/downstream/if
needed" occurrence in the plan and its ADRs was flagged and tested
for: (a) concrete reactivation trigger, (b) decision owned or silent,
(c) hidden cost of inaction.
| Item | Trigger? | Owned? | Cost of inaction | Finding |
|---|---|---|---|---|
| `ValidationPlan` (ADR-012) | "if a bench motivates it" | Yes (ADR + plan "What this is not") | Possible perf cliff on read+validate untrusted streams | M3 above — sharpen the trigger |
| Nested-union `ReadPlan` support | "if a real schema needs it" (POC) | No (POC only, plan silent) | Silent 0.2.0 capability drop | M1 above — state it |
| `materialize_aligned` structure walk | None — silent | No (silent) | Future agent ambiguity | L3 above — own the decision |
| `Arc<str>` vs `String` (decision 1) | "if phase 4 shows it's measurable" | Yes (deferred decision 1) | None | OK — has trigger, decided in phase 3 |
| `OffsetMap::get` shape (decision 2) | "decided in phase 5" | Yes (deferred decision 2) | None | OK |
| Fingerprint hasher (decision 3) | "decided in phase 6" | Yes (deferred decision 3) | None | OK |
| Struct-array stride (decision 4) | "decided in phase 2" | Yes (deferred decision 4) | None | OK |
| `BastDoc` `Arc<Value>` vs `Value` | None — silent | No (plan doesn't address) | Agent gets stuck (H2) | H2 above |
| Field-disc union shape (POC Finding 1) | "production version must implement" | Yes (plan phase 1) | None, but shape is undefined | H1 above — shape not in ADR |
The four explicit "deferred decisions" in the plan (items 4-7) all
have concrete triggers and decision points — these are the *good*
pattern. The black-hole pattern appears where deferrals lack triggers
(items 1-3, 8-9): three of those became findings (M1, L3, H2), and M3
is a deferral worth sharpening even though it has a trigger.
The general signal: a deferral is healthy when it has a concrete
reactivation condition and is tracked (OQ, ADR, or in-plan deferred
decision). A deferral is a black hole when it has no trigger, no
tracking, and the next agent inherits the gap by default.
---
## What's Good
- **Line-number accuracy is perfect.** Every `src/` reference in the
plan (`engine.rs:112-115,284,334,467`;
`sequential_reader.rs:567`; `layout_builder.rs:190`;
`bast.rs:51-55`; `lib.rs` re-exports) checks out against the v0.2.0
tree. This is unusual for a plan of this length and worth noting.
- **The Semver Contract table is a strong scope-creep guardrail.**
Walking every public `lib.rs` re-export against the table, the
classifications (Breaking / Unchanged / New) are correct for every
item, with the exceptions noted in N3 (the `Result` drop on `new`)
and M1 (the nested-union behavioral drop not listed).
- **The four explicit "deferred decisions" are the right pattern.**
Each has a trigger and a decision point in a named phase. This is
what deferrals should look like.
- **Phases are coherent session boundaries.** Phases 1 (pure
addition), 6 (pure addition), 7 (docs/bump) are small and clean.
Phases 3 (broad but mechanical), 4 (single file), 5 (single file +
engine) are well-scoped. Phase 2 is the largest and the plan
sanctions sub-session splits at the step level (lines 40-42), which
is the right escape valve.
- **Cross-phase invariants are stated and checkable.** "Tree builds
and tests pass at every phase boundary" is the right invariant; the
POC-on-`readplan-poc`-only convention is clearly separated from
production code; AGENTS.md §5-§11 constraints (no `unsafe`, no
`async`, no new deps, `preserve_order` load-bearing) are
reaffirmed.
- **The plan honestly scopes what it is not.** "Not a `ValidationPlan`",
"Not cross-version fingerprint stability", "Not a perf bench" —
these boundaries are stated rather than left implicit, which helps
an implementing agent resist scope creep. (M3 above is about
sharpening one of these, not removing the boundary.)
- **The POC reference is disciplined.** The plan is explicit that the
POC is "not production code," lives only on the branch, and is the
reference scaffold for phases 1-2 only. This matches how
`bast-validator-poc` was handled and avoids the POC leaking into
`main`.
---
## Recommended Order
1. **H1 (field-disc union shape)** — update ADR-011's
`CompositePlan::Union` to include the shared-fields sub-`ReadPlan`
(or document the wrapper shape), then update the plan's phase 1 to
reference the corrected ADR shape. Do this before phase 1 starts;
otherwise the implementing agent has to guess.
2. **H2 (`schema()` `&Value` on `Arc<ReadPlan>`)** — edit the plan's
phase 2 to specify `ReadPlan` stores `Arc<Value>` (or owned
`Value`), and `schema()` borrows from that. One-line edit to the
plan; avoid a phase-2 stuck point.
3. **M1 (nested unions)** — decide (a) implement `VariantKind::Union`
in phase 1, or (b) list as behavioral drop + OQ. Edit the plan and
(if b) the Semver Contract table accordingly. Decide before phase
1.
4. **M2 (`Hash` on `Endian`/`VariableEncoding`)** — add a sub-step
to phase 5 or 6. Trivial.
5. **M3 (`ValidationPlan` re-evaluation)** — either add a
`validate_bytes`-on-untrusted-stream bench to alktty (alongside
`wire_vs_bast`) and let the bench decide, or sharpen ADR-012's
"not a hot loop" justification. Does not block 0.3.0; can be
resolved in parallel with phase 1-7 work. **Flagged for
re-evaluation, not for implementation in 0.3.0.**
6. **L1, L2, L3** — edit the plan's phase 2 to fix the
`dummy_field_for` wording (L1), state the packed-vs-aligned
materialize split (L2), and add a Scope Boundary note for
aligned-materialize's `BastDoc` structure walk (L3). All three are
phase-2 doc edits.
7. **N1, N2, N3** — typo, `Send + Sync` assertion test, `Result`-drop
Semver row. Minor plan edits.
Items 1-3 must be resolved before phase 1 starts (they affect the
`ReadPlan` shape or 0.2.0 behavioral surface). Items 4-7 can be
resolved any time before their phase begins. Item 5 (M3) is
non-blocking and can run in parallel.
---
## Notes
- All line numbers refer to the tree at commit `2310f6c` (the plan's
commit) for `src/` files, and to the plan/ADR markdown as committed
at the same tree.
- The POC on `readplan-poc` was inspected via
`git show readplan-poc:poc/readplan/{src/lib.rs,FINDINGS.md}`; it is
not merged to `main` and the plan correctly states this.
- `alktty` and `alkcall` downstream repos exist as path dev-deps
(`/workspace/@alkdev/alktty`, `/workspace/@alkdev/alkcall`); the
plan's claim that they're in-house and updated with the bump is
verifiable, though this review did not inspect their call sites
in detail.
- This review does not re-litigate ADR-011 or ADR-012's accepted
decisions. H1 and H2 are about the plan *contradicting* the ADRs or
being unsound, not about the ADR decisions themselves; M3 is about
sharpening a deferral, not about re-deciding it.
- The deferral-pattern scan is a methodology experiment: a
pre-declared scan for LLM-specific planning quirks (deferral black
holes) alongside classic planning mistakes. It surfaced M1 and L3
that a conventional severity-only review would have missed or
under-weighted. Worth retaining as a default scan for future plan
reviews.
---
## Resolution (2026-08-20)
All 11 findings resolved in one docs-only edit pass to ADR-011,
ADR-012, and the 0.3.0 plan. No source changed; the crate still
builds/tests at v0.2.0. The M3 deferral reversal is the one
substantive decision change (per user direction: ship ValidationPlan
in 0.3.0, no more hedging); the rest are spec corrections or
pre-implementation refinements to types that do not yet exist on
`main`.
- **H1 (union shape):** ADR-011 §"The `ReadPlan` shape" refined —
`CompositePlan::Union` now carries `shared: Option<Box<ReadPlan>>`
(field-disc shared fields) and `variants: Vec<(String,
CompositePlan)>` (dropping `VariantPlan`/`VariantKind`). Plan
phase 1 rewritten to implement the refined shape. The shape
refinement is pre-implementation (the types don't exist on `main`).
- **H2 (`schema()` `&Value`):** plan phase 2 rewritten — `ReadPlan`
stores `schema: Arc<Value>` (not `&Value`); `schema()` returns
`&self.schema`. Verified `serde_json::Value: Hash + Eq` holds with
`preserve_order` (`Map::hash` sorts keys deterministically), so
phase 6's `#[derive(Hash)]` on `ReadPlan` is not blocked.
- **M1 (nested unions):** resolved as the review's option (a) —
nested-union support falls out of the H1 shape refinement (a
variant can be `CompositePlan::Union`), so no behavioral drop vs
0.2.0 and no Semver Contract entry for a capability regression.
Plan phase 1 adds a nested-union-variant test.
- **M2 (`Hash` on `Endian`/`VariableEncoding`):** plan phase 5
rewritten with an explicit first sub-step to add `Hash` to both
derives in `src/schema.rs` (additive, semver-safe). The inaccurate
"all fields are `Copy + Hash`" parenthetical on `LeafMeta` is
corrected.
- **M3 (`ValidationPlan`):** deferral **reversed** per user
direction. ADR-012 §"Deferring `ValidationPlan`" rewritten as
"ValidationPlan — in scope for 0.3.0"; new ADR-012 §3 commits the
decision (compiled form, no per-buffer `BastDoc` walk, `Hash + Eq`
+ `fingerprint()`) and lists the shape questions deferred to a
follow-on design session + the plan's new phase 7. Plan gains a
new phase 7 (ValidationPlan); old phase 7 (bump) renumbered to
phase 8. ADR-011's "Out of scope" `bast_validation` bullet and
"Scope Boundaries" `Not a validation plan` bullet updated to point
at ADR-012 §3. Plan's "What this plan is *not*" first bullet
removed. The deferral-black-hole pattern this review's methodology
flagged is closed: the work is committed in the plan with a
concrete reactivation trigger (the shape session before phase 7),
not hedged into an unplanned future.
- **L1 (`dummy_field_for`/`ty_source`):** plan phase 2 rewritten —
only the packed-side call sites go away; the helpers stay for the
aligned `materialize_leaf_at` path.
- **L2 (`materialize_typeref_packed` split):** plan phase 2
rewritten — packed-materialize gets a new plan-walking function;
`materialize_typeref_packed` stays for aligned's
`materialize_leaf_at` and the aligned record path.
- **L3 (aligned-materialize `BastDoc` structure walk):** plan phase 5
gains a Scope Boundary note — the walk is the permanent 0.3.0
design; an `AlignedPlan` is out of scope, tracked as an open
question if a future bench motivates it.
- **N1 (typo):** "back-comat" → "back-compat" in deferred decision 4.
- **N2 (`Send + Sync` assertion test):** plan phase 1 verification
rewritten to add the `read_plan_is_send_sync` static-bound
assertion test ADR-011 §"Engine integration" requires.
- **N3 (`Result` drop on `SequentialReader::new`):** Semver Contract
table row updated to note the constructor return-type change
(`Result<Self, AlkTypeError>` → `Self`) alongside the argument-type
change.
The deferral-pattern scan's general signal (healthy deferrals have a
concrete reactivation condition + tracking; black holes have neither)
is reaffirmed by the M3 reversal: the original "if a bench motivates
it" trigger was a black hole because no bench was ever going to be
run against a path that didn't exist yet, and the cost of inaction
(a second breaking change to `validate_bytes`/`bast_validation` after
0.3.0) was hidden by the "not a hot loop" framing.
docs/reviews/006-implementation-review-030.md
+1466
View File
File diff suppressed because it is too large. Load diff
docs/reviews/007-coverage-audit.md
+454
View File
@@ -0,0 +1,454 @@
---
status: resolved (F1, F2, C1, C2, C3, L1, L2 resolved 2026-09-02; N1/N2a/N3a/N4a are classified-no-action / deferred-by-design)
last_updated: 2026-09-02
reviewed_artifacts:
- src/materialize.rs
- src/sequential_reader.rs
- src/read_plan.rs
- src/offset_map.rs
- src/layout_builder.rs
- src/engine.rs
- src/data_access.rs
- src/bast.rs
- src/builder.rs
- src/tunion.rs
- src/validation_plan.rs
- src/walk_guard.rs
- src/bast_meta.rs
- tests/poc_roundtrip.rs
- tests/tunion_dispatch.rs
- tests/error_paths.rs
- tests/engine_integration.rs
- docs/reviews/006-implementation-review-030.md (post-fix coverage re-check)
tool: cargo-llvm-cov 0.8.4 (--release, per-line text) + manual classification of every uncovered production line + disposable probe tests (run in-session, then deleted)
reviewer: post-review-#006 coverage audit (session request — check test coverage for weak spots, meaningful tests, non-happy-path posture)
---
# Review #007 — Post-#006 Coverage Audit
## Purpose
Review #006 closed every finding and its M4 coverage map, but the
session-level posture (M4's item: "fold a coverage check into each fix
session") had never been run as a *whole-tree* pass after all those
fixes landed. This audit re-measures coverage after the eleven #006
commits, reads every uncovered production line, and classifies it —
the same "untested-but-fine / load-bearing / unreachable" discipline
M4's map used. Two probes were run in disposable tests (deleted after
the session, per #006's no-reproducer rule; neither was a crash
hazard — both reproduce cleanly inside the default harness).
## Methodology
- `cargo llvm-cov --release` (0.8.4, same tool as #006): summary +
per-line text. TOTAL **90.67% lines / 86.32% functions** — stable
with #006's post-M4 numbers (90.60%), the expected drift after the
N3 fix sessions added parse gates + tests.
- Per-file (worst first): `materialize.rs` 85.72, `data_access.rs`
80.32, `sequential_reader.rs` 86.19, `bast.rs` 87.41,
`builder.rs` 91.28, `layout_builder.rs` 91.42, `tunion.rs` 92.02,
`offset_map.rs` 92.93, `read_plan.rs` 90.69, `engine.rs` 96.44,
`validation_plan.rs` 93.97, `walk_guard.rs` 98.04,
`bast_meta.rs` 98.92, `bast_validation.rs`/`error.rs`/`schema.rs`/
`macros.rs`/`validation.rs` 100.
- Every uncovered line *outside* `#[cfg(test)]` modules (806 raw
lines) was read and classified. Lines inside test modules (the
`panic!("expected X, got {other:?}")` helpers) were excluded — they
distort per-file numbers (e.g. `bast.rs`'s 87.41% is really ~96%
production once its 60 helper lines are excluded).
- Two suspicions were probe-verified with disposable tests:
the F1 cross-consumer divergence and the F2 unbounded-`maxLength`
compile. Probe transcripts quoted verbatim in the findings.
- Happy-path posture audit: cross-checked which *error arms* adjacent
to covered code are 0-execution, and which public surfaces have only
success-path tests.
## Baseline
Audited at `main` HEAD `bb28ba3` ("Resolve N3"), 0.3.0, working tree
clean. Full suite green (548 tests static + 2 ignored doctests, per
#006's bookkeeping).
## Summary Statistics
| Severity | Count | Status |
|----------|------:|--------|
| High | 2 (F1, F2) | both resolved 2026-09-02 |
| Medium | 3 (C1, C2, C3) | all resolved 2026-09-02 |
| Low | 2 (L1, L2) | all resolved 2026-09-02 |
| Info | 4 (N1, N2a, N3a, N4a) | classified: N1 artifact, N2a/N4a no-action, N3a deferred to pre-release review |
**Resolution log:**
- **F1 + F2 (2026-09-02):** resolved in one commit — see the
resolution blocks on each finding. 477 lib tests green (511
static + 2 ignored doctests across all targets), clippy
`-D warnings` clean, wasm build green.
- **C1 + C2 + C3 (2026-09-02):** resolved in one commit — see the
resolution blocks. 481 lib tests green, clippy `-D warnings` clean,
wasm build green; `compile_variant`'s cycle arm confirmed executed
in the post-fix coverage run.
- **L1 + L2 (2026-09-02):** resolved in one commit — see the
resolution blocks. 488 lib tests green, clippy `-D warnings` clean,
wasm build green.
---
## Findings
### F1. Zero-progress guard missing in the plan materializer — `validate_bytes` accepts what `SequentialReader` rejects (cross-consumer divergence)
**Files**: `src/materialize.rs:227-253` (`materialize_plan_array` — no
guard), contrast `src/sequential_reader.rs:772-795`
(`plan_walk_variable_array_size` — has the guard) and
`src/materialize.rs:636-663` (`materialize_array_packed` — has the
guard)
**Problem**: The H1 fix session added the zero-progress runtime guard
("array element consumed 0 bytes") to two of the three array walkers:
the compiled reader's variable-array size walk and the legacy BAST
walker's packed array arm. The *plan-based* packed materializer —
`materialize_plan_array`, the walker `validate_bytes` actually uses in
packed mode (engine.rs:310-312) — got no guard.
A stride-0 array whose elements consume 0 bytes (empty-struct elements
are legal: the meta-schema's `StructDef` has no `minItems` on
`fields`) compiles with `element_stride: 0` and loops `count` times
materializing empty objects without reading a single buffer byte:
```
PROBE validate_bytes([]): OK — zero-progress guard MISSING in plan materializer
PROBE reader.read_next([]): Err(access error at items[0]: array element 0 consumed 0 bytes;
a zero-size element makes the declared count unbounded on the wire)
```
Schema: `{ "items": { "kind": "array", "element": { "kind":
"struct", "fields": [] }, "count": 8 } }`, packed mode, empty buffer.
Same schema, same buffer, opposite verdicts — the exact
cross-consumer-disagreement shape review #006 existed for (H3, M6).
Severity High by #006's own keying (AGENTS.md §3): `validate_bytes`
is the flagship untrusted-input path, and it silently accepts a
buffer the same engine's reader rejects. The H1 resolution text
("plan_walk_variable_array_size (reader) and materialize_array_packed
(materializer) now error") lists only two of the three walkers — the
plan materializer was missed because it is *not* the legacy walker
that finding named.
**Not a #006 regression**: the H1 fix text itself specified only the
reader and legacy-walker sites; the plan materializer predates the
guard and was outside that fix's blast radius. But the divergence is
new information — the guard's *invariant* ("a zero-progress element
makes the declared count unbounded") belongs to the array-walk
concept, not to two specific functions.
**Fix**: hoist the same guard into `materialize_plan_array`'s loop
(compare `*offset` before/after `materialize_plan_composite`; error
with the same wording the other two walkers use so downstream
matching sees one shape). Add a locking test driving the same schema
through BOTH paths asserting the verdicts agree (both reject an empty
buffer; both accept a buffer where the elements make progress —
empty-struct elements never do, so the acceptance half needs a
non-empty variant struct alongside).
**Resolution (2026-09-02):** the guard, hoisted verbatim from the two
existing sites (`*offset == before` after the element walk, same
"array element {i} consumed 0 bytes…" wording so downstream matching
sees one shape). Tests (3, in `materialize.rs`):
`f1_zero_progress_array_rejected_by_all_three_walkers` (plan
materializer + the record-value fallback path, both asserting the
`Access` error with the guard's wording),
`f1_validate_bytes_and_reader_agree_on_zero_progress_array` (the
cross-consumer agreement the probe showed was missing —
`validate_bytes` and `SequentialReader::read_next` both reject the
same schema+buffer with the same error class),
`f1_nonempty_variant_struct_array_still_materializes` (the
false-positive check: elements that consume bytes still walk).
Verified: 477 lib tests green, clippy `-D warnings` clean, wasm build
green.
### F2. `maxLength` is unbounded — the N2 analog
**Files**: `src/bast_meta.rs:81` (`"maxLength": { "type": "integer",
"minimum": 0 }` — no maximum), `src/bast.rs:1038-1043`
(`parse_max_length` — no cap, and silently drops non-`usize` values),
contrast `src/schema.rs` `MAX_ALIGN`/`parse_align` (the N2 pattern)
**Problem**: N2 bounded `align` at 4096 with a clean parse error plus
a meta-schema `"maximum"`. `maxLength` has the identical shape and
was not covered by that fix:
```
PROBE aligned maxLength 1e12 compiles; total_size = 1099511627776
```
A one-field schema declares a 1 TiB reservation; `total_size` in that
range is meaningless output the consumer may act on (N2's argument
(a)). Unlike align, no `Access` error follows at read time (an empty
buffer still fails buffer bounds first), so this is layout-meaningless
output, not a crash — the exact severity N2 recorded. Additionally,
`parse_max_length` returns `Option` and silently *drops* values that
overflow `usize` (`.and_then(|n| usize::try_from(n).ok())`) — on a
32-bit target a 5 GiB `maxLength` becomes "no maxLength", changing
layout semantics without telling the consumer.
**Fix**: the N2 playbook verbatim. A `MAX_LENGTH` cap in
`schema.rs` (value TBD — `align`'s 4096 is page granularity; a
reservation cap in the tens-of-megabytes range fits honest layouts;
suggest `2^26 = 67_108_864`, matching `MAX_ARRAY_BYTES`'s rationale),
enforced in `parse_max_length` (converted to `Result<Option<usize>>`,
clean `Schema` error naming the path/value/maximum — no silent drop),
plus `"maximum": 67108864` in the meta-schema's `maxLength` property
so the published contract matches the parser (the N2 dual-layer
pattern).
**Resolution (2026-09-02):** the N2 playbook, cap = `MAX_LENGTH`
(2^26 = 67_108_864, matching `MAX_ARRAY_BYTES`'s rationale: a single
fixed reservation no larger than the largest legal array):
1. `MAX_LENGTH` added to `schema.rs`, documented with the F2 probe
arithmetic.
2. `parse_max_length` converted to `Result<Option<usize>>`: non-integer
→ clean `Schema` error; `usize` overflow → clean `Schema` error (the
silent `.and_then(try_from().ok())` drop is gone); over-cap → clean
`Schema` error naming the path, value, and maximum.
3. Meta-schema `maxLength` property gains `"maximum": 67108864` — the
published contract matches the parser.
4. Tests (5, in `offset_map.rs`, mirroring the `n2_` family):
above-cap rejection naming value+maximum (bytes and string),
at-cap acceptance (`total_size == 67108864`), u64::MAX-scale value
rejected-not-silently-dropped (cap arm on 64-bit, overflow arm on
32-bit — one test covers whichever fires), and the meta-schema
dual-layer check (above-cap rejected, at-cap accepted).
Verified with F1's commit: 477 lib tests green, clippy clean, wasm
green.
### C1. Packed `validate_bytes` has never decoded a wide primitive
**Files**: `src/materialize.rs:121-169` (`materialize_plan_primitive`'s
Int16/Int32/Int64/Uint64/Float64/Boolean arms — all 0-execution),
`src/sequential_reader.rs:345-383` (the reader's same arms are covered
via `read_next` tests, but the materializer's are not)
**Problem**: every packed `validate_bytes` test feeds u8/uint32/
string-shaped data. The i16/i32/i64/u64/f64/bool arms of the plan
materializer — the code every untrusted packed wire buffer flows
through — have never executed through any test. Probe (in-session)
confirmed the BE i16/bool path works; the arms are correct, just
unexercised. This is the flagship decode path for `alkcall`'s packed
frames; one battery test closes it (mirror the aligned
`read_field` battery, tests/engine_integration.rs:130-190, which
already covers all twelve primitive kinds on the aligned side).
**Resolution (2026-09-02):** two tests in `engine.rs`:
`c1_validate_bytes_packed_decodes_all_twelve_primitives_le` (the full
eleven-field battery — i8..bool — plus a corrupted-bool rejection arm)
and `c1_validate_bytes_packed_decodes_big_endian_subset` (BE i16/u64/
f64 through the same public path). Both green.
### C2. Aligned `validate_bytes` never exercises the default inline encoding for string/bytes
**Files**: `src/materialize.rs:1037-1039`
(`materialize_variable_aligned`'s `LengthPrefixed`-else branch —
0-exec through the public path)
**Problem**: the aligned `validate_bytes` tests use records, unions,
maxLength reservations, and offset-indirect encodings. The *default*
encoding — an inline length-prefixed string or bytes field, the most
common real shape — reaches `read_field` (engine_integration.rs:192)
but never `validate_bytes`. The aligned `validate_bytes` surface has
thus never decoded the single most likely field kind through its
public path.
**Fix**: one aligned `validate_bytes` test with a trailing inline
string (and a bytes variant or arm), asserting acceptance plus a
short-buffer rejection.
**Resolution (2026-09-02):**
`c2_validate_bytes_aligned_inline_string_and_bytes_default_encoding`
in `engine.rs`. One wrinkle the test wrote itself into: ADR-006
allows an inline length-prefixed variable field only in the final
position, so the string and bytes shapes get separate one-field
schemas (string after a fixed `id`; bytes as a lone field). Asserts
acceptance for both plus a short-buffer rejection for the string.
### C3. `ReadPlan::compile`'s union-variant cycle arm is untested standalone
**Files**: `src/read_plan.rs:509-514` (`compile_variant`'s
`cycle_err` arm — 0-exec)
**Problem**: the H2 test family exercises `check_ref_graph` (walk
guard) via `OffsetMap::compute`/`LayoutBuilder::new`/
`materialize_aligned`, and `ValidationPlan::compile`'s cycle arm is
covered (`validation_plan.rs:297` shows executions, via the
`shared_refs_compile_without_false_cycle`/cycle tests). But
`ReadPlan::compile`'s own cycle rejection — the defense the *packed
read plan* relies on when driven standalone (its doc explicitly
promises untrusted-input safety) — has no test driving a two-def
cycle through it. The depth cap is tested
(`deep_nesting_beyond_depth_cap_is_schema_error`); the cycle arm is
shadowed in every engine-path test by the ValidationPlan gate running
first (engine.rs:153).
**Fix**: a `read_plan_compile_two_def_cycle_rejected` test calling
`ReadPlan::compile` directly on a two-def cycle, mirroring
`validation_plan.rs`'s existing standalone cycle test.
**Resolution (2026-09-02):**
`c3_cycle_through_union_mapping_variant_is_schema_error` in
`read_plan.rs`. Writing the test sharpened the finding: the
*field-level* cycle arm (`compile_typeref`, :362) was already covered
(2 execs) by `cyclic_ref_through_two_defs_is_schema_error`; the
0-exec arm was `compile_variant`'s own check (:513), reachable only
when the cycle closes through a **union mapping entry**. The new
test's shape (`A → B → U(mapping: "1" → $ref B)`) trips exactly that
arm — verified post-fix at the line level (1 execution).
### L1. `builder.rs`'s JSON-Schema conveniences are entirely untested
**Files**: `src/builder.rs:268-290` (`array()`, `number()`,
`boolean_()`, `null()`), `:465-479` (`items()`,
`additional_properties()`), `:508-565` (`maximum()`, `min_length()`,
`min_items()`, `max_items()`, `format()`, `title()`,
`description()`), `:440` (the `field()`-on-standard-repr path)
**Problem**: only the BAST-side builders have tests. The standard
JSON-Schema side feeds `jsonschema::build_validator` (the
`json_schema` parameter of `AlkTypeEngine::compile`), so a typo'd or
misplaced keyword would ship silently — the builder emits the JSON,
`jsonschema` interprets it, and nothing checks the translation. One
table-style test asserting each convenience produces the expected
JSON key/value closes the surface cheaply.
**Resolution (2026-09-02):** five tests in `builder.rs`:
`l1_standard_type_constructors_produce_type_keyword` (all eight
standard constructors, exact-JSON assertions),
`l1_field_on_standard_object_builds_properties` (`field()` on the
standard repr + `required()`),
`l1_items_and_additional_properties_on_standard_types`,
`l1_constraint_keywords_emit_expected_json_keys` (minimum/maximum/
minLength/minItems/maxItems/format/title/description, each asserted
on its exact keyword), and
`l1_standard_built_schema_compiles_as_json_validator` (the end of the
translation chain: the emitted JSON builds a `jsonschema` validator
and the constraints actually bite — valid passes, over-maximum/
missing-required/below-minimum fail).
### L2. `tunion::read_field_discriminator`'s enum arm is 0-exec
**Files**: `src/tunion.rs:166-169`
**Problem**: N1's resolution extended tunion to match the reader's
kind set and added uint16/uint32 tests both endians — but skipped the
enum arm, which is in the documented kind set
(tunion.rs:106-112 names "string / uint8 / uint16 / uint32 / enum").
The reader's enum arm is tested (`m4_field_disc_enum_dispatches_on_index`);
tunion's is not. One test locks parity on the last arm.
**Resolution (2026-09-02):** `l2_read_field_discriminator_enum_
dispatches_on_index` and `l2_read_field_discriminator_enum_big_endian`
in `tunion.rs` — enum index 0 (LE) and 1 (BE) dispatch with
`variant_offset == 4` / `discriminator_size == 4`.
### N1. `OffsetEntry::start()`/`end()` 0-execution in the combined run is a merge artifact, not a hole
**Files**: `src/offset_map.rs:79-86`
The combined `cargo llvm-cov --release` run reports these 0-exec;
`tests/poc_roundtrip.rs:187-189` calls `start()` (and the
`big_endian_round_trip_via_offset_map` test calls `end()`). Per-test
coverage confirms both execute (32/2 calls respectively in a
poc_roundtrip-only run). llvm-cov's profile merge does not attribute
integration-test-binary executions to the library in every run
configuration. Recorded so nobody "fixes" this by deleting the
accessors or writing a redundant in-module test. (Caveat for future
audits: when a combined run shows 0-exec on something an integration
test visibly calls, re-run per-test-target before classifying.)
### N2a. `data_access.rs`'s remaining uncovered lines are the documented >4 GiB guards — fine to leave
**Files**: `src/data_access.rs:54-95, 223-291, 336-414`
All are `checked_add` overflow arms and u32-truncation guards needing
multi-GiB slices or near-`usize::MAX` offsets — already documented as
defensively-unreachable on 64-bit test hardware in #006 M4 item 3's
resolution. (The `read_array`/`write_array` arms at :54-95 are
additionally unreachable-after-`check_bounds` belt-and-suspenders.)
No action.
### N3a. `bast.rs` dead-or-orphaned surface — flag for the pre-release review
**Files**: `src/bast.rs:212-214, 305-307, 404-406, 506-508, 741-743,
924-926, 973-975` (`source()` accessors — zero callers anywhere in
src or tests), `:353-363` (`BastField::synthetic`,
`#[allow(dead_code)]`, zero callers), `:149-167`
(`resolve_typeref_as_def`'s inline struct/union/enum arms — both call
sites pass `$ref`-only variants since H3's parse rules forbid
re-declaration; plausibly dead now)
Three small deletions-or-justifications. Not fixed this session (the
`source()` accessors are public API — removal is a semver decision
for the pre-release review, and AGENTS.md's semver exception list
says renames/removals need an explicit ask). Recorded so the
pre-release review session has the list.
### N4a. Internal-shape error arms are structurally unreachable — fine to leave
**Files**: `src/materialize.rs:241,309,422,858`,
`src/sequential_reader.rs:295,473,533,832`, `src/offset_map.rs:352`,
`src/layout_builder.rs:194,282`, `src/read_plan.rs:402`
The `"internal: …"` arms that dispatch on a `match` the caller
already narrowed (e.g. "union body at X is not CompositePlan::Union"
inside a function only reachable from a `Union` match arm). They are
honest defensive code — deleting them would force `unwrap()` — and
forcing them in tests would require constructing mid-walk corruption.
Leave uncovered; the pattern is consistent across the codebase.
---
## What's Good
- The #006 fix sessions left the tree in genuinely good shape: 90.67%
lines with every high-traffic wire path (reader dispatch, plan
compiler, offset map, walk guard) in the mid-90s or better.
- The untrusted-input discipline is visible in the coverage: every
parse-level gate added in #006 (H1 caps, N2 align cap, N3
string/bytes-only maxLength, H2 cycle rejections at all three
standalone walkers) has both rejection and boundary tests.
- The `#[cfg(test)]` helper noise is the only thing making
`bast.rs`/`data_access.rs` look worse than they are — the
production coverage of both is materially higher than the raw
per-file number.
## Recommended Order
1. ~~**F1** — guard hoist + cross-consumer agreement test~~
**resolved 2026-09-02** (with F2).
2. ~~**F2** — `MAX_LENGTH` cap, N2's dual-layer playbook verbatim~~
**resolved 2026-09-02** (with F1).
3. ~~**C1 + C2 + C3** — one locking test each~~ **resolved 2026-09-02**.
4. ~~**L1 + L2** — posture tests~~ **resolved 2026-09-02**.
5. **N3a** — defer to the pre-release review (semver decision).
## Notes
- Probe tests were run as `tests/zzz_probe.rs` in-tree during the
session and deleted before any commit (the #006 pattern). Neither
probe was a crash hazard; both reproduce safely in the default
harness.
- Per-file numbers are from a single `cargo llvm-cov --release`
run; the N1 merge artifact means integration-test-only calls
(e.g. `OffsetEntry::start()`) can show 0-exec in the combined
report — the classification above already accounts for that.
- The coverage holes fixed this session (C1-C3, L1, L2) were chosen
because each is load-bearing *and* one-test-cheap; the remaining
uncovered mass is dominated by N2a/N3a/N4a, which are documented
rather than forced.
- Static test counts at the review-#007 commits: 477 (F1/F2),
481 (C1-C3), 488 (L1/L2) — +14 net from the pre-audit 474.
- Post-fix coverage (same tool, full run): TOTAL **91.66% lines /
87.64% functions** (from 90.67/86.32). Per-file movement:
`builder.rs` 91.28→99.33, `engine.rs` 96.44→96.76,
`materialize.rs` 85.72→87.74, `read_plan.rs` 90.69→90.94,
`tunion.rs` 92.02→92.86. The remaining mass is the documented
N2a/N3a/N4a classes.
docs/reviews/008-pre-publish-review.md
+296
View File
@@ -0,0 +1,296 @@
---
status: resolved (F1, F2 fixed 2026-09-07; N1, N2, N3 classified; N4 fixed 2026-09-07)
last_updated: 2026-09-07
reviewed_artifacts:
- src/read_plan.rs
- src/sequential_reader.rs
- src/materialize.rs
- src/bast.rs
- benches/wire_vs_bast.rs
- docs/reviews/007-coverage-audit.md (N3a disposition)
tool: manual diff review of post-#007 commits (dea96f0, d4635d2) + disposable probe tests (run in-session, then deleted) + cargo bench + counting-allocator peak-RSS probe
reviewer: pre-publish review #008 (session request — audit the two post-#007 perf/bench commits, then gate the 0.3.0 publish)
---
# Review #008 — Pre-Publish Review: Post-#007 Perf Commits
## Purpose
0.3.0's release commit (`9949f91`) predates review #006 entirely; the
fix sessions for #006 and #007 landed eleven more commits, and *after*
#007 closed, two more commits landed unreviewed: `dea96f0` (bench port
from alktty) and `d4635d2` (the perf commit — fixed-size struct fast
path, integer union dispatch, `read_next_borrowed`). The perf commit
touches the flagship packed read path, which every untrusted wire
buffer flows through. This review audits those two commits before the
first crates.io publish of the 0.3.x line (0.1.0 and 0.2.0 are
published; 0.3.0 never was — every post-release fix can legally ride
inside the first published 0.3.0, no semver conflict).
It also disposes of review #007's N3a — the one finding explicitly
deferred to "the pre-release review", which this session is.
## Methodology
- Full diff read of `d4635d2` (perf) and `dea96f0` (bench port),
cross-checked against the invariants the earlier reviews established:
cross-consumer dispatch agreement (#006 H3, #007 F1), the H1
no-count-sized-prealloc rule, and the N2/F2 dual-layer cap pattern.
- Disposable probe tests (`tests/zzz_probe*.rs`, deleted after the
session; none was a crash hazard) to confirm/deny the three
behaviors code reading flagged: the `int_keys` non-canonical-key
divergence, the engine's nested-array acceptance envelope, and the
`with_capacity` amplification.
- A counting-`GlobalAlloc` probe (peak-bytes metric) to measure the
worst-case simultaneous allocation of the amplification shape
precisely — RSS timing proved too noisy to separate the two test
cases.
- `cargo bench --quick` before/after the fixes to confirm the perf
commit's wins survive.
- N3a dispositions probed where cheap (inline-struct union variants).
## Baseline
Audited at `main` HEAD `d4635d2`, 0.3.0, working tree clean. 566
tests green (488 lib + 17 + 34 + 15 + 12, + 2 ignored doctests),
clippy `-D warnings` clean, wasm build green — per the perf commit's
verification block.
## Summary Statistics
| Severity | Count | Status |
|----------|------:|--------|
| High | 0 | — |
| Medium | 2 (F1, F2) | both fixed 2026-09-07 |
| Info | 3 (N1, N2, N3) | classified |
| Fix | 1 (N4) | fixed 2026-09-07 |
No Highs: both Mediums are probe-verified cross-consumer divergences
and resource-bound violations, but neither aborts the process
(`with_capacity` is now bounded per array by `MAX_ARRAY_ELEMENTS`, so
H1's 1 TB SIGABRT class does not return). Both were fixed in-session
because they violate AGENTS.md §3 (divergent verdicts on untrusted
input; unbounded-count-shaped allocation) — the publish gate.
**Resolution log:**
- **F1 + F2 (2026-09-07):** fixed in one commit — see the resolution
blocks. 569 tests green (491 lib + 17 + 34 + 15 + 12, + 2 ignored),
clippy `-D warnings` clean, doc 0 warnings, wasm green.
- **N4 (2026-09-07):** fixed with F1/F2 — see the block.
---
## Findings
### F1. `int_keys` integer dispatch breaks cross-consumer agreement on non-canonical mapping keys
**Files**: `src/read_plan.rs` (`compile_int_keys`, introduced by
`d4635d2`), contrast `src/materialize.rs` (`materialize_plan_union`'s
byte-disc arm — stringifies), `src/validation_plan.rs`
(`validate_union_numeric` — stringifies), `src/tunion.rs`
(`read_byte_discriminator` — stringifies)
**Problem**: `d4635d2` added a pre-parsed `(u64, variant_index)`
dispatch table for byte-discriminator unions: when every mapping key
parses as `u64`, the reader matches the raw discriminator integer
instead of stringifying per read. But the meta-schema does not
constrain mapping-key shape beyond "object property name", and
`key.parse::<u64>()` accepts **non-canonical** decimal strings:
```
PROBE1 reader: field=msg disc="01" (DISPATCHED)
PROBE1 validate_bytes: Err(access error at msg: union discriminator value 1 not in mapping)
```
With mapping key `"01"` (and discriminator `1` on the wire): the
reader's numeric dispatch **matches** (`"01".parse::<u64>() == 1`) and
dispatches — returning `discriminator == "01"` — while the
materializer (`1.to_string() == "1" ≠ "01"`), the validation plan, and
tunion all **reject** the identical buffer. Pre-`d4635d2`, all four
consumers stringified and all four rejected — agreement held (both
verdicts "reject", same error class). The perf commit flipped the
reader to accept-while-everyone-else-rejects: the exact
cross-consumer-divergence shape #006 H3 and #007 F1 exist for, on the
flagship path. `"+1"` parses as `u64` too (Rust's `from_str_radix`
accepts a leading `+`) — same class. The returned key string also
became schema-quirk-dependent: the reader reports `"01"` where the
materializer's `__discriminator` for a *matched* key would report the
stringified form.
**Not a #007 regression**: `int_keys` did not exist before `d4635d2`.
But `d4635d2` postdates #007's close and was unreviewed — this is the
audit catching it.
**Fix**: build the integer table only from **canonical** keys — a key
qualifies iff `key.parse::<u64>()` succeeds *and*
`parsed.to_string() == key` (i.e. the key is exactly what
stringification would produce). Any non-canonical or non-numeric key
falls back to the string path (`int_keys: None`), which every consumer
already agrees on. No accepted schema's *reachable* behavior changed:
for fully-canonical mappings the numeric dispatch behaves identically
to stringified matching (the numeric value's `to_string()` equals the
key), and for non-canonical keys all consumers now reject exactly as
before `d4635d2`. The perf win (no per-read stringify) is preserved
for every mapping that was unambiguous to begin with.
**Resolution (2026-09-07):** exactly that — `compile_int_keys` now
requires `v.to_string() == *key` for the table to carry the entry;
any miss returns `Ok(None)` (string fallback). Doc comment states the
canonicality rule and why. Tests in `read_plan.rs`:
`r8_non_canonical_mapping_key_disables_int_dispatch` (key `"01"` →
`int_keys` is `None`) and `r8_canonical_mapping_keys_keep_int_dispatch`
(keys `"1"`,`"2"` → table `[(1,0),(2,1)]`). Probe output after the fix:
both `validate_bytes` and the reader reject disc 1 under key `"01"`
with the same error class — agreement restored.
### F2. `Vec::with_capacity(count)` reintroduced on both array materializers — ~477 MB simultaneous allocation from a ~1 KB schema
**Files**: `src/materialize.rs:247` (`materialize_plan_array` — the
`validate_bytes` packed path), `src/materialize.rs:650`
(`materialize_array_packed` — the legacy walker, reachable via the
aligned record arm)
**Problem**: `d4635d2`'s "materialize: with_capacity for bytes arrays,
arrays, and struct objects" item reintroduced
`Vec::with_capacity(count)` at two of the three sites H1's layer-1 fix
had converted to `Vec::new()` + push. `count` is now compile-capped at
`MAX_ARRAY_ELEMENTS` (2^16), so H1's 1 TB `SIGABRT` does not return —
but the per-array cap does not bound *nesting*:
```
PROBE validate peak bytes allocated simultaneously: 476780249
```
A schema of one 100-level nested array chain (each `count: 65535`,
innermost elements empty structs — all legal: the depth cap is 128 and
stride-0 chains evade `MAX_ARRAY_BYTES`, which only checks stride
products) peaks at **~477 MB of simultaneous allocation** on
`validate_bytes(&[])` from a ~1 KB schema and an *empty* buffer. Each
level's `with_capacity(65535 × sizeof(Value))` stays live across its
element walk, so the sizes multiply across ~127 legal depth levels
(the innermost zero-progress rejection fires only after the whole
chain has descended). On wasm32 — which this crate explicitly targets
— the same shape aborts the wasm heap well below 477 MB. H1's layer-1
rule ("no count-sized prealloc on untrusted input; the per-element
walk dominates") is exactly the invariant this violates; the perf
commit's own bench evidence doesn't need the prealloc either (see
below).
The other `with_capacity` additions in the commit are fine: byte-array
capacity from `b.len()` (a read slice), struct-object capacity from
`plan.fields().len()`, and the plan-compiler's from `fields.len()` —
all bounded by data/plan already in hand, not by declared counts.
**Fix**: restore H1's layer-1 shape at both sites — `Vec::new()` +
push loop (the loops already push `count` elements; the zero-progress
guard bounds honest progress per element). Optionally cap
preallocation at a small constant, but plain `Vec::new()` matches H1's
shipped behavior.
**Resolution (2026-09-07):** both sites restored to `Vec::new()` +
push. Bench before/after (criterion `--quick`, this session): packet
read 246 → 220 µs, chunk read 76/68 µs — the revert costs nothing
measurable on the bench shapes (small arrays; the materializer's
per-element work dominates), and the union/struct preallocs stay.
Locking test in `materialize.rs`:
`r8_deeply_nested_stride0_array_rejects_before_bulk_prealloc` (the
100-level chain still rejects cleanly with the zero-progress error at
the innermost level; the allocation shape itself is documented here —
in-tree cannot cheaply assert peak allocation, and the #008 probe was
deleted per the no-reproducer rule).
### N1. `d4635d2`'s fixed-size fast paths are sound (classified, no action)
**Files**: `src/read_plan.rs` (`fixed_size`, `fixed_plan_size`), `src/sequential_reader.rs`
The fixed-size struct fast path replaces the cursor size walk with one
bounds check; `fixed_plan_size` already returned `Result<Option>` with
clean overflow errors (L1's shape), and every new error arm formats
paths lazily on the error path only. The union-variant fast path
(`plan_variant_fixed_size`) applies only to struct variants and checks
bounds before use. No issue found.
### N2. Bench port (`dea96f0`) is methodology-honest (classified, no action)
**Files**: `benches/wire_vs_bast.rs`
The port drops alktty's async I/O group (correctly — it measured a
different stack) and adds a parity check before measurement so the
stream loop can't drift. The historical `read_chunk_stream` numbers
stay comparable by construction. No issue found.
### N3. N3a dispositions (review #007's deferred items)
**Files**: `src/bast.rs` (`source()` accessors, `BastField::synthetic`,
`resolve_typeref_as_def`'s inline arms)
- **`source()` accessors (7 sites)**: public API on `BastStruct`/
`BastUnion`/`BastField`/etc. Removal is a semver decision and
AGENTS.md's semver exception requires an explicit ask — **kept**.
They are one-line accessors over parsed source nodes, harmless, and
plausibly useful to downstream codegen (the announced consumer).
- **`BastField::synthetic` (`#[allow(dead_code)]`, zero callers)**:
`pub(crate)`, not public API — **deleted** (2026-09-07). No semver
impact; the `#[allow(dead_code)]` suppression is gone with it.
- **`resolve_typeref_as_def`'s inline struct/union/enum arms**: the
review-#007 suspicion ("plausibly dead after H3") was wrong —
probe-verified reachable: the meta-schema's
`mapping.additionalProperties: TypeRef` accepts inline struct
variants, and `LayoutBuilder`'s byte-disc and field-disc arms call
`resolve_typeref_as_def` on every union variant. The H3 parse rules
forbid variant *re-declaration of shared fields*, not inline variant
bodies. **Kept**, reachable.
### N4. Stale test-count references in review #006's resolution log
**Files**: `docs/reviews/006-implementation-review-030.md`
The bookkeeping note ("static count at `2eb086f` is 542 + 2 ignored")
and per-commit counts are accurate as written; no fix needed. Recorded
here so the review trail stays honest about what was re-checked
during this session's doc sweep. **Resolution (2026-09-07):** no code
change; superseded the "Fix" entry — this is the classification
record.
---
## What's Good
- The perf commit's core ideas are sound and survived review: the
compile-time `fixed_size` cache is computed through the existing
`Result`-returning sizer (no `unwrap_or_default` regression), and
the int-dispatch table's design was right — it just needed the
canonicality gate.
- The counting-allocator probe took 15 minutes and converted a
"probably too big" into a precise number (476,780,249 bytes) — the
same probe pattern the earlier reviews used, applied to allocation
instead of verdicts.
- `cargo bench --quick` before/after the fixes is the right tool for
guarding perf-fix reverts: packet read 220 µs post-fix vs 246 µs
baseline confirms the `Vec::new()` restore is free.
## Recommended Order
1. ~~**F1** — canonical-key gate on `compile_int_keys`~~ **fixed
2026-09-07**.
2. ~~**F2** — restore H1's no-prealloc rule at both array sites~~
**fixed 2026-09-07**.
3. ~~**N4** — `BastField::synthetic` deletion~~ **fixed 2026-09-07**.
4. **N3 source() accessors** — revisit only if/when the codegen
consumer confirms it does not want them (removal needs an explicit
ask per AGENTS.md).
## Notes
- Probe tests were run as `tests/zzz_probe*.rs` in-tree during the
session and deleted before any commit (the #006 pattern). None was a
crash hazard; the amplification probe allocates ~477 MB transiently
and completes in ~40 ms.
- Benches are not run in CI and are excluded from the publish (the
`[bench]` target ships — that is fine; benches don't affect the
library's API or its wasm compatibility).
- The 0.3.0 publish proceeds after these fixes: 0.1.0 and 0.2.0 are
on crates.io; this is the first 0.3.0 publish, so F1/F2's
behavior changes (both "previously-divergent, now-agreed" shapes)
land inside the version's first release — no semver bump implied.
src/bast.rs
+460 -202
View File
File diff suppressed because it is too large. Load diff
src/bast_meta.rs
+52 -3
View File
@@ -61,7 +61,7 @@ pub static BAST_META_SCHEMA: LazyLock<Value> = LazyLock::new(|| {
"properties": {
"kind": { "const": "struct" },
"endian": { "enum": ["little", "big"] },
"align": { "type": "integer", "minimum": 1 },
"align": { "type": "integer", "minimum": 1, "maximum": 4096 },
"fields": {
"type": "array",
"items": { "$ref": "#/$defs/FieldDef" }
@@ -76,10 +76,16 @@ pub static BAST_META_SCHEMA: LazyLock<Value> = LazyLock::new(|| {
"name": { "type": "string", "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$" },
"kind": { "$ref": "#/$defs/TypeRef" },
"endian": { "enum": ["little", "big"] },
"align": { "type": "integer", "minimum": 1 },
"align": { "type": "integer", "minimum": 1, "maximum": 4096 },
"encoding": { "enum": ["length-prefixed", "offset-indirect"] },
"maxLength": { "type": "integer", "minimum": 0 }
"maxLength": { "type": "integer", "minimum": 0, "maximum": 67108864 }
},
"if": {
"properties": {
"kind": { "enum": ["string", "bytes"] }
}
},
"else": { "properties": { "maxLength": false } },
"required": ["name", "kind"],
"additionalProperties": false
},
@@ -518,4 +524,47 @@ mod tests {
});
assert!(validator.validate(&doc).is_ok(), "inline struct in union mapping rejected");
}
#[test]
fn meta_schema_rejects_max_length_on_non_string_bytes_field() {
let validator = jsonschema::options()
.build(&BAST_META_SCHEMA)
.expect("meta-schema compiles");
let doc = serde_json::json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "counts", "kind": { "kind": "record", "values": "uint16" }, "maxLength": 8 }
]
}
}
});
assert!(
validator.validate(&doc).is_err(),
"maxLength on a record field accepted by the meta-schema"
);
}
#[test]
fn meta_schema_accepts_max_length_on_string_and_bytes_field() {
let validator = jsonschema::options()
.build(&BAST_META_SCHEMA)
.expect("meta-schema compiles");
let doc = serde_json::json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "name", "kind": "string", "maxLength": 256 },
{ "name": "blob", "kind": "bytes", "maxLength": 64 }
]
}
}
});
assert!(
validator.validate(&doc).is_ok(),
"maxLength on string/bytes fields rejected"
);
}
}
src/bast_validation.rs
+89 -357
View File
@@ -1,44 +1,34 @@
//! BAST-native validator — the `validate_bytes` validation step.
//! BAST-native validation — the `validate_bytes` validation step,
//! now a thin entry over the compiled [`crate::validation_plan::
//! ValidationPlan`] (ADR-012 §3, 0.3.0 phase 7).
//!
//! A recursive walker over the BAST typed tree ([`crate::bast::BastDoc`]/
//! [`crate::bast::BastType`]) that checks the value-domain constraints the
//! materializer ([`crate::materialize`]) does NOT check. This replaces the
//! v0.1.0 `jsonschema` custom-keyword validators on the bytes path
//! (D-BAST-006) with a flat match — no factories, no trait objects, no
//! sub-validator pre-computation.
//! ## History
//!
//! ## What the materializer already guarantees
//! Through 0.2.0 this module *was* the validator: a recursive walker
//! over the BAST typed tree ([`crate::bast::BastDoc`]) that resolved
//! `$ref`s lazily and rebuilt path strings per node, on every
//! `validate_bytes` call. That interpretive walk is the same class of
//! per-buffer cost review #004 measured on the read path (400x), so
//! ADR-012 §3 (per review #005 M3) committed the `ValidationPlan`: the
//! constraint tree (enum allowed-sets, integer ranges, `maxLength` caps,
//! union variant keys, array counts, record value types) is compiled
//! once at engine-compile time and walked per buffer with no `BastDoc`
//! touch. The interpretive walker's checks moved verbatim into the plan
//! (see [`crate::validation_plan`]'s constraint table); the last
//! interpretive consumers of this module are gone.
//!
//! By construction, a materialized [`serde_json::Value`] is structurally
//! correct: all declared fields are present, types are correct, bounds
//! are checked (via [`crate::data_access`]), UTF-8 is valid, the boolean
//! byte is 0 or 1, and the union discriminator is in the mapping. The
//! validator only needs to enforce the **value-domain constraints
//! expressed in the BAST document** — the ones the materializer can't
//! see from the bytes alone.
//! ## What remains here
//!
//! ## Constraint table
//!
//! See [bast-format.md §Validation Model](../../docs/architecture/bast-format.md#validation-model).
//! The arms of the `validate_typeref` walker implement the table:
//!
//! | Constraint | Validator arm |
//! |---|---|
//! | Integer range (Int8..Uint64) | `validate_int`/`validate_uint` |
//! | Int64/Uint64 (full range) | `validate_int64`/`validate_uint64` |
//! | Float finiteness (Float32/64) | `validate_float` |
//! | String `maxLength` (byte length) | `check_string` reads [`BastField::max_length`](crate::bast::BastField::max_length) |
//! | Bytes `maxLength` (array length) | `check_bytes` |
//! | Enum index bounds | `validate_enum` — **fixes the v0.1.0 dead constraint** |
//! | Union variant dispatch | `validate_union` reads `__discriminator`, resolves the variant, recurses |
//! | Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses |
//! | Array count | `validate_array` checks `arr.len() == count` and recurses per element |
//! | Record values | `validate_record` recurses into each value's `values` type |
//! | Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) |
//! - [`validate_value`] — the public one-shot entry (`compile` +
//! `validate`), retained for API compatibility and for callers holding
//! a `BastDoc` without an engine. The engine does not use it per
//! buffer; it holds a compiled plan (ADR-012 §3).
//! - `validation_err` / `DISCRIMINATOR_KEY` — shared helpers, also used
//! by the plan walk.
//!
//! ## Error payload (D-BAST-009)
//!
//! The bytes path no longer touches `jsonschema` for validation, but the
//! The bytes path does not touch `jsonschema` for validation, but the
//! error variant retains the `jsonschema::ValidationError<'static>` type
//! for uniformity with the `validate_json` path. Errors are constructed
//! via [`jsonschema::ValidationError::custom`] so consumers handle one
@@ -46,324 +36,40 @@
//!
//! ## Untrusted input
//!
//! Every walk path returns [`AlkTypeError::Schema`] on a malformed BAST
//! document, never `panic!`/`unreachable!`/`unwrap` (AGENTS.md §3). The
//! typed tree is parsed once by [`BastDoc::new`](crate::bast::BastDoc::new);
//! lazy `$ref` resolution via [`BastDoc::resolve_typeref`] surfaces a
//! `Schema` error for dangling refs.
//! Malformed BAST documents surface as [`AlkTypeError::Schema`] from
//! [`ValidationPlan::compile`](crate::validation_plan::ValidationPlan::compile)
//! — including cyclic `$ref` graphs, which the interpretive walker could
//! not reject (it would overflow the stack) and now cannot reach.
use crate::bast::{
BastArray, BastDefKind, BastDoc, BastEnum, BastField, BastRecord, BastStruct, BastType,
BastUnion,
};
use crate::bast::BastDoc;
use crate::error::AlkTypeError;
use crate::schema::AlkTypeKind;
use serde_json::Value;
const DISCRIMINATOR_KEY: &str = "__discriminator";
pub(crate) const DISCRIMINATOR_KEY: &str = "__discriminator";
/// Validate a materialized `value` against the BAST root type.
///
/// This is the `validate_bytes` validation step: after
/// [`crate::materialize::materialize_packed`]/
/// [`crate::materialize::materialize_aligned`] produces a `Value` tree
/// from the bytes, this walker enforces the value-domain constraints
/// expressed in the BAST document. The materializer already guarantees
/// structural correctness; the validator only checks what the bytes
/// alone can't tell you (integer ranges, `maxLength`,
/// enum index bounds, union variant constraints).
/// This is the convenience one-shot form: it compiles a
/// [`ValidationPlan`](crate::validation_plan::ValidationPlan) from `doc`
/// and validates `value` against it. Use this when you hold a BAST
/// document and a materialized `Value` but no engine. In the engine's
/// hot path (`AlkTypeEngine::validate_bytes`, ADR-010/ADR-012 §3), the
/// plan is compiled once and re-walked per buffer without
/// re-touching the document.
///
/// Returns `Err(AlkTypeError::Validation(...))` on the first violated
/// constraint, or `Err(AlkTypeError::Schema(...))` if the BAST document
/// is malformed (a dangling `$ref`, a missing `values` array, etc.).
pub fn validate_value(doc: &BastDoc<'_>, value: &Value) -> Result<(), AlkTypeError> {
let root_def = doc.root_def();
match root_def.kind() {
BastDefKind::Struct(s) => validate_struct(doc, s, value, ""),
BastDefKind::Union(u) => validate_union(doc, u, value, ""),
BastDefKind::Enum(e) => validate_enum(value, "", e),
}
}
/// Recursively validate `value` against `ty`, resolving `$ref`s lazily.
///
/// `field` is the owning field (for `maxLength`/annotations) — `None` for
/// synthetic contexts (array elements, record values, the root). The
/// validator reads [`BastField::max_length`] only for `String`/`Bytes`
/// arms, so passing a synthetic field (which has `max_length == None`) is
/// correct for those contexts.
fn validate_typeref(
doc: &BastDoc<'_>,
ty: &BastType<'_>,
field: Option<&BastField<'_>>,
value: &Value,
path: &str,
) -> Result<(), AlkTypeError> {
let resolved = doc.resolve_typeref(ty)?;
match &resolved {
BastType::Primitive(AlkTypeKind::Int8) => validate_int(value, path, -128, 127),
BastType::Primitive(AlkTypeKind::Int16) => validate_int(value, path, -32768, 32767),
BastType::Primitive(AlkTypeKind::Int32) => {
validate_int(value, path, -2147483648, 2147483647)
}
BastType::Primitive(AlkTypeKind::Int64) => validate_int64(value, path),
BastType::Primitive(AlkTypeKind::Uint8) => validate_uint(value, path, 255),
BastType::Primitive(AlkTypeKind::Uint16) => validate_uint(value, path, 65535),
BastType::Primitive(AlkTypeKind::Uint32) => validate_uint(value, path, 4294967295),
BastType::Primitive(AlkTypeKind::Uint64) => validate_uint64(value, path),
BastType::Primitive(AlkTypeKind::Float32) => validate_float(value, path),
BastType::Primitive(AlkTypeKind::Float64) => validate_float(value, path),
BastType::Primitive(AlkTypeKind::Boolean) => validate_bool(value, path),
BastType::Primitive(AlkTypeKind::String) => {
check_string(value, path, field.and_then(|f| f.max_length()))
}
BastType::Primitive(AlkTypeKind::Bytes) => {
check_bytes(value, path, field.and_then(|f| f.max_length()))
}
BastType::Enum(e) => validate_enum(value, path, e),
BastType::Struct(s) => validate_struct(doc, s, value, path),
BastType::Union(u) => validate_union(doc, u, value, path),
BastType::Array(a) => validate_array(doc, a, value, path),
BastType::Record(r) => validate_record(doc, r, value, path),
BastType::Ref(_) => Err(AlkTypeError::Schema(format!(
"bast_validation: unresolved $ref at {path} (resolve_typeref should have deref'd it)"
))),
BastType::Primitive(k) => Err(AlkTypeError::Schema(format!(
"bast_validation: unsupported primitive kind {k} at {path}"
))),
}
}
fn validate_int(value: &Value, path: &str, min: i64, max: i64) -> Result<(), AlkTypeError> {
let n = value
.as_i64()
.ok_or_else(|| validation_err(path, "expected an integer"))?;
if n < min || n > max {
return Err(validation_err(
path,
format!("integer {n} out of range [{min}, {max}]"),
));
}
Ok(())
}
fn validate_int64(value: &Value, path: &str) -> Result<(), AlkTypeError> {
if value.as_i64().is_none() {
return Err(validation_err(path, "expected an i64 integer"));
}
Ok(())
}
fn validate_uint(value: &Value, path: &str, max: u64) -> Result<(), AlkTypeError> {
let n = value
.as_u64()
.ok_or_else(|| validation_err(path, "expected a non-negative integer"))?;
if n > max {
return Err(validation_err(path, format!("integer {n} exceeds {max}")));
}
Ok(())
}
fn validate_uint64(value: &Value, path: &str) -> Result<(), AlkTypeError> {
if value.as_u64().is_none() {
return Err(validation_err(path, "expected a u64 integer"));
}
Ok(())
}
fn validate_float(value: &Value, path: &str) -> Result<(), AlkTypeError> {
match value {
Value::Number(n) => {
let f = n
.as_f64()
.ok_or_else(|| validation_err(path, "expected a number"))?;
if !f.is_finite() {
return Err(validation_err(path, "expected a finite number"));
}
Ok(())
}
_ => Err(validation_err(path, "expected a number")),
}
}
fn validate_bool(value: &Value, path: &str) -> Result<(), AlkTypeError> {
match value {
Value::Bool(_) => Ok(()),
_ => Err(validation_err(path, "expected a boolean")),
}
}
fn check_string(value: &Value, path: &str, max_length: Option<usize>) -> Result<(), AlkTypeError> {
let s = value
.as_str()
.ok_or_else(|| validation_err(path, "expected a string"))?;
if let Some(max) = max_length {
if s.len() > max {
return Err(validation_err(
path,
format!("string byte length {} exceeds maxLength {max}", s.len()),
));
}
}
Ok(())
}
fn check_bytes(value: &Value, path: &str, max_length: Option<usize>) -> Result<(), AlkTypeError> {
let arr = value
.as_array()
.ok_or_else(|| validation_err(path, "expected an array of u8 for bytes"))?;
if let Some(max) = max_length {
if arr.len() > max {
return Err(validation_err(
path,
format!("bytes array length {} exceeds maxLength {max}", arr.len()),
));
}
}
for (i, entry) in arr.iter().enumerate() {
let n = entry.as_u64().ok_or_else(|| {
validation_err(
path,
format!("bytes array entry {i} is not a non-negative integer"),
)
})?;
if n > 255 {
return Err(validation_err(
path,
format!("bytes array entry {i} = {n} is not a u8 (0..=255)"),
));
}
}
Ok(())
}
/// Enum validation on the bytes path: the materializer emits a numeric
/// index (`Value::Number`), and the constraint is that the index is
/// within the `values` array bounds (`0..len-1`).
///
/// This is the **fix for the v0.1.0 dead constraint**: the built-in
/// `enum` keyword checked string membership, but the materializer emitted
/// a numeric index that never matched — so out-of-bounds enum indices
/// silently passed. The BAST-native validator checks the index bounds
/// directly.
fn validate_enum(value: &Value, path: &str, enum_def: &BastEnum<'_>) -> Result<(), AlkTypeError> {
let idx = value
.as_u64()
.ok_or_else(|| validation_err(path, "expected a non-negative integer enum index"))?;
let len = enum_def.values().len() as u64;
if idx >= len {
return Err(validation_err(
path,
format!("enum index {idx} out of bounds (values has {len} entries)"),
));
}
Ok(())
}
fn validate_struct(
doc: &BastDoc<'_>,
struct_def: &BastStruct<'_>,
value: &Value,
path: &str,
) -> Result<(), AlkTypeError> {
let obj = value
.as_object()
.ok_or_else(|| validation_err(path, "expected an object"))?;
for field in struct_def.fields() {
let name = field.name();
let field_path = if path.is_empty() {
name.to_string()
} else {
format!("{path}.{name}")
};
let field_value = obj.get(name).ok_or_else(|| {
validation_err(&field_path, format!("missing field {name:?}"))
})?;
validate_typeref(doc, field.ty(), Some(field), field_value, &field_path)?;
}
Ok(())
}
/// Union validation: read `__discriminator`, look up the variant
/// [`BastType`] in the union's `mapping`, and recurse into the variant.
///
/// This recovers OQ-008 per-variant constraint enforcement (e.g.
/// `maxLength` on a `bytes` field inside a variant struct) without
/// custom keywords — the recursion walks the variant's BAST definition
/// and enforces every field constraint it declares.
fn validate_union(
doc: &BastDoc<'_>,
union_def: &BastUnion<'_>,
value: &Value,
path: &str,
) -> Result<(), AlkTypeError> {
let obj = value
.as_object()
.ok_or_else(|| validation_err(path, "expected an object for union"))?;
let disc = obj.get(DISCRIMINATOR_KEY).ok_or_else(|| {
validation_err(path, "union instance is missing the '__discriminator' field")
})?;
let key = match disc {
Value::String(s) => s.clone(),
Value::Number(n) => n.to_string(),
_ => {
return Err(validation_err(
path,
"union '__discriminator' must be a string or number",
));
}
};
let variant_ty = union_def.variant_for(&key).ok_or_else(|| {
validation_err(path, format!("union discriminator value '{key}' not in mapping"))
})?;
validate_typeref(doc, variant_ty, None, value, path)
}
fn validate_array(
doc: &BastDoc<'_>,
array_def: &BastArray<'_>,
value: &Value,
path: &str,
) -> Result<(), AlkTypeError> {
let arr = value
.as_array()
.ok_or_else(|| validation_err(path, "expected an array"))?;
let count = array_def.count();
if arr.len() != count {
return Err(validation_err(
path,
format!("array length {} does not match declared count {count}", arr.len()),
));
}
let element_ty = array_def.element();
for (i, item) in arr.iter().enumerate() {
let item_path = format!("{path}[{i}]");
validate_typeref(doc, element_ty, None, item, &item_path)?;
}
Ok(())
}
fn validate_record(
doc: &BastDoc<'_>,
record_def: &BastRecord<'_>,
value: &Value,
path: &str,
) -> Result<(), AlkTypeError> {
let obj = value
.as_object()
.ok_or_else(|| validation_err(path, "expected an object for record"))?;
let values_ty = record_def.values();
for (k, v) in obj.iter() {
let entry_path = format!("{path}[{k}]");
validate_typeref(doc, values_ty, None, v, &entry_path)?;
}
Ok(())
/// is malformed (a dangling `$ref`, a cyclic `$ref`, a missing `values`
/// array, etc.).
pub fn validate_value(doc: &BastDoc, value: &Value) -> Result<(), AlkTypeError> {
let plan = crate::validation_plan::ValidationPlan::compile(doc)?;
plan.validate(value)
}
/// Construct a `Validation` error from a path + reason string. The
/// payload is a `jsonschema::ValidationError::custom` so the variant
/// type stays uniform with the `validate_json` path (D-BAST-009).
fn validation_err(path: &str, reason: impl Into<String>) -> AlkTypeError {
pub(crate) fn validation_err(path: &str, reason: impl Into<String>) -> AlkTypeError {
let msg = if path.is_empty() {
reason.into()
} else {
@@ -375,9 +81,11 @@ fn validation_err(path: &str, reason: impl Into<String>) -> AlkTypeError {
#[cfg(test)]
mod tests {
use super::*;
use crate::materialize::materialize_packed;
use crate::read_plan::ReadPlan;
use serde_json::json;
fn doc_from<'a>(root: &'a Value, name: &'a str) -> BastDoc<'a> {
fn doc_from(root: &Value, name: &str) -> BastDoc {
BastDoc::new(root, name).expect("bast doc")
}
@@ -392,13 +100,20 @@ mod tests {
}
fn materialize_and_validate(
doc: &BastDoc<'_>,
root: &Value,
doc: &BastDoc,
buffer: &[u8],
) -> Result<(), AlkTypeError> {
let value = crate::materialize::materialize_packed(doc, buffer)?;
let plan = ReadPlan::compile(root, doc.root_name())?;
let value = materialize_packed(&plan, buffer)?;
validate_value(doc, &value)
}
// The tests below are the *parity* suite: they drive validation
// end-to-end (materialize -> validate_value) through the public API
// exactly as the interpretive walker's tests did, so any behavioral
// change in the compiled plan shows up here.
// ----- Integer ranges --------------------------------------------------
#[test]
@@ -409,8 +124,8 @@ mod tests {
] } }
});
let d = doc_from(&root, "S");
assert!(materialize_and_validate(&d, &u32_le(0)).is_ok());
assert!(materialize_and_validate(&d, &u32_le(0xFFFF_FFFF)).is_ok());
assert!(materialize_and_validate(&root, &d, &u32_le(0)).is_ok());
assert!(materialize_and_validate(&root, &d, &u32_le(0xFFFF_FFFF)).is_ok());
}
#[test]
@@ -421,8 +136,8 @@ mod tests {
] } }
});
let d = doc_from(&root, "S");
assert!(materialize_and_validate(&d, &[127u8]).is_ok());
assert!(materialize_and_validate(&d, &[128u8]).is_ok());
assert!(materialize_and_validate(&root, &d, &[127u8]).is_ok());
assert!(materialize_and_validate(&root, &d, &[128u8]).is_ok());
}
#[test]
@@ -478,8 +193,8 @@ mod tests {
] } }
});
let d = doc_from(&root, "S");
assert!(materialize_and_validate(&d, &prefixed_str_le("hi")).is_ok());
let err = materialize_and_validate(&d, &prefixed_str_le("hello")).unwrap_err();
assert!(materialize_and_validate(&root, &d, &prefixed_str_le("hi")).is_ok());
let err = materialize_and_validate(&root, &d, &prefixed_str_le("hello")).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
}
@@ -493,10 +208,10 @@ mod tests {
let d = doc_from(&root, "S");
let mut buf = u32_le(2);
buf.extend_from_slice(&[0xAA, 0xBB]);
assert!(materialize_and_validate(&d, &buf).is_ok());
assert!(materialize_and_validate(&root, &d, &buf).is_ok());
let mut buf = u32_le(3);
buf.extend_from_slice(&[0xAA, 0xBB, 0xCC]);
let err = materialize_and_validate(&d, &buf).unwrap_err();
let err = materialize_and_validate(&root, &d, &buf).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
}
@@ -526,8 +241,8 @@ mod tests {
}
});
let d = doc_from(&root, "S");
assert!(materialize_and_validate(&d, &u32_le(0)).is_ok());
assert!(materialize_and_validate(&d, &u32_le(2)).is_ok());
assert!(materialize_and_validate(&root, &d, &u32_le(0)).is_ok());
assert!(materialize_and_validate(&root, &d, &u32_le(2)).is_ok());
}
#[test]
@@ -541,7 +256,7 @@ mod tests {
}
});
let d = doc_from(&root, "S");
let err = materialize_and_validate(&d, &u32_le(5)).unwrap_err();
let err = materialize_and_validate(&root, &d, &u32_le(5)).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
}
@@ -572,16 +287,16 @@ mod tests {
let mut buf = vec![2u8];
buf.extend_from_slice(&u32_le(2));
buf.extend_from_slice(&[0xAA, 0xBB]);
assert!(materialize_and_validate(&d, &buf).is_ok());
assert!(materialize_and_validate(&root, &d, &buf).is_ok());
let mut buf = vec![2u8];
buf.extend_from_slice(&u32_le(3));
buf.extend_from_slice(&[0xAA, 0xBB, 0xCC]);
let err = materialize_and_validate(&d, &buf).unwrap_err();
let err = materialize_and_validate(&root, &d, &buf).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
let buf = vec![1u8, 7u8];
assert!(materialize_and_validate(&d, &buf).is_ok());
assert!(materialize_and_validate(&root, &d, &buf).is_ok());
}
#[test]
@@ -606,7 +321,7 @@ mod tests {
let mut buf = prefixed_str_le("data");
buf.extend_from_slice(&u32_le(2));
buf.extend_from_slice(&[0xFF, 0xFE]);
let err = materialize_and_validate(&d, &buf).unwrap_err();
let err = materialize_and_validate(&root, &d, &buf).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
}
@@ -671,7 +386,7 @@ mod tests {
buf.extend_from_slice(&2u16.to_le_bytes());
buf.extend_from_slice(&3u16.to_le_bytes());
buf.extend_from_slice(&4u16.to_le_bytes());
assert!(materialize_and_validate(&d, &buf).is_ok());
assert!(materialize_and_validate(&root, &d, &buf).is_ok());
}
#[test]
@@ -715,7 +430,7 @@ mod tests {
] } }
});
let d = doc_from(&root, "S");
let err = materialize_and_validate(&d, &[0u8; 2]).unwrap_err();
let err = materialize_and_validate(&root, &d, &[0u8; 2]).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
}
@@ -744,4 +459,21 @@ mod tests {
assert!(validate_value(&d, &json!({"flag": false})).is_ok());
assert!(validate_value(&d, &json!({"flag": "yes"})).is_err());
}
// ----- Cyclic schema: the new compile-time guard ----------------------
#[test]
fn cyclic_bast_doc_rejected_with_schema_error() {
let root = json!({
"$defs": {
"S": { "kind": "struct", "fields": [
{ "name": "me", "kind": { "$ref": "#/$defs/S" } }
] }
}
});
let d = doc_from(&root, "S");
let err = validate_value(&d, &json!({})).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
assert!(err.to_string().contains("cyclic"), "got {err:?}");
}
}
src/builder.rs
+91
View File
@@ -372,6 +372,8 @@ impl Schema {
/// Schema type, emitted as the standard `maxLength` keyword.
/// In aligned mode with a variable-length type, reserves this many
/// bytes (strategy 2). In packed mode, validation constraint only.
/// Only honored on `string`/`bytes` fields — the BAST parser rejects
/// it on any other kind (review #006 N3).
pub fn max_length(mut self, max: usize) -> Self {
match &mut self.repr {
Repr::Standard(map) => {
@@ -1384,4 +1386,93 @@ mod tests {
);
assert!(engine.is_ok(), "record should compile: {engine:?}");
}
// ----- Standard JSON-Schema conveniences (review #007 L1) -----------
#[test]
fn l1_standard_type_constructors_produce_type_keyword() {
assert_eq!(Schema::object().build(), json!({ "type": "object" }));
assert_eq!(Schema::array().build(), json!({ "type": "array" }));
assert_eq!(Schema::string_().build(), json!({ "type": "string" }));
assert_eq!(Schema::integer().build(), json!({ "type": "integer" }));
assert_eq!(Schema::number().build(), json!({ "type": "number" }));
assert_eq!(Schema::boolean_().build(), json!({ "type": "boolean" }));
assert_eq!(Schema::null().build(), json!({ "type": "null" }));
assert_eq!(Schema::any().build(), json!({}));
}
#[test]
fn l1_field_on_standard_object_builds_properties() {
let s = Schema::object()
.field("id", Schema::integer())
.field("tag", Schema::string_())
.required(&["id"]);
assert_eq!(
s.build(),
json!({
"type": "object",
"properties": {
"id": { "type": "integer" },
"tag": { "type": "string" }
},
"required": ["id"]
})
);
}
#[test]
fn l1_items_and_additional_properties_on_standard_types() {
let arr = Schema::array().items(Schema::integer());
assert_eq!(
arr.build(),
json!({ "type": "array", "items": { "type": "integer" } })
);
let obj = Schema::object()
.additional_properties(Schema::boolean_());
assert_eq!(
obj.build(),
json!({ "type": "object", "additionalProperties": { "type": "boolean" } })
);
}
#[test]
fn l1_constraint_keywords_emit_expected_json_keys() {
// Each convenience setter must land on the exact JSON Schema
// keyword jsonschema interprets — a typo here would silently
// drop the constraint (nothing else checks the translation).
let s = Schema::integer()
.minimum(1.0)
.maximum(100.0);
assert_eq!(
s.build(),
json!({ "type": "integer", "minimum": 1.0, "maximum": 100.0 })
);
let str_schema = Schema::string_().min_length(2).format("uri").title("T").description("D");
assert_eq!(
str_schema.build(),
json!({ "type": "string", "minLength": 2, "format": "uri", "title": "T", "description": "D" })
);
let arr_schema = Schema::array().min_items(1).max_items(10);
assert_eq!(
arr_schema.build(),
json!({ "type": "array", "minItems": 1, "maxItems": 10 })
);
}
#[test]
fn l1_standard_built_schema_compiles_as_json_validator() {
// End of the translation chain: the emitted JSON is accepted by
// build_validator and the constraints actually bite.
let schema = Schema::object()
.field("id", Schema::integer().minimum(0.0).maximum(10.0))
.required(&["id"]);
let validator = crate::validation::build_validator(&schema.build())
.expect("standard schema compiles");
assert!(validator.validate(&json!({ "id": 5 })).is_ok());
assert!(validator.validate(&json!({ "id": 11 })).is_err());
assert!(validator.validate(&json!({})).is_err());
assert!(validator.validate(&json!({ "id": -1 })).is_err());
}
}
src/data_access.rs
+37
View File
@@ -712,6 +712,43 @@ mod tests {
assert!(matches!(err, AlkTypeError::Access { .. }));
}
#[test]
fn m4_write_bytes_indirect_pair_at_nonzero_offset_data_ok() {
// The indirect write's pair offset and data offset are
// independent; pin the pair landing mid-buffer with the data
// elsewhere (review #006 M4 item 3 — the write_bytes_indirect
// guard family is the canonical overflow-guard pattern, review
// #002 M2, and its nonzero-offset paths were uncovered).
let mut buf = vec![0u8; 24];
let written = write_bytes_indirect(&mut buf, 4, 16, b"xyz", "blob", BE).unwrap();
assert_eq!(written, 8);
assert_eq!(&buf[4..8], &16u32.to_be_bytes());
assert_eq!(&buf[8..12], &3u32.to_be_bytes());
assert_eq!(&buf[16..19], b"xyz");
let bytes = read_bytes_indirect(&buf, 4, "blob", BE).unwrap();
assert_eq!(bytes, b"xyz");
}
#[test]
fn m4_write_bytes_indirect_data_length_exceeds_bounds() {
// The data-region bounds check in write_bytes_indirect: the pair
// fits, but the {data_offset, length} target runs past the buffer
// end — the write must refuse without corrupting the pair.
let mut buf = vec![0u8; 20];
let err = write_bytes_indirect(&mut buf, 0, 16, b"hello", "blob", LE).unwrap_err();
match err {
AlkTypeError::Access { field_path, reason } => {
assert_eq!(field_path, "blob");
assert!(reason.contains("bounds"), "reason: {reason}");
}
other => panic!("expected Access, got {other:?}"),
}
// The pair was already written (offset+length at 0..8) — that's
// fine; the error refuses the data copy only.
assert_eq!(&buf[0..4], &16u32.to_le_bytes());
assert_eq!(&buf[4..8], &5u32.to_le_bytes());
}
#[test]
fn read_at_nonzero_offset() {
let mut buf = vec![0u8; 16];
src/engine.rs
+419 -123
View File
@@ -16,18 +16,20 @@
//! §"The AlkTypeEngine struct" and
//! [overview.md](../../docs/architecture/overview.md).
use crate::bast::{BastDefKind, BastDoc, BastStruct, BastType};
use crate::bast_validation;
use crate::bast::{BastDefKind, BastDoc};
use crate::data_access;
use crate::error::AlkTypeError;
use crate::layout_builder::LayoutBuilder;
use crate::materialize;
use crate::offset_map::OffsetMap;
use crate::read_plan::ReadPlan;
use crate::schema::{AlkTypeKind, Endian, VariableEncoding};
use crate::sequential_reader::{FieldValue, SequentialReader};
use crate::validation;
use crate::validation_plan::ValidationPlan;
use serde_json::Value;
use std::fmt;
use std::sync::Arc;
/// The layout mode selected at engine construction time.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -46,12 +48,14 @@ pub enum LayoutMode {
/// APIs that make sense for that mode.
#[derive(Debug)]
enum Layout {
/// Packed sequential layout. The write-side is [`LayoutBuilder`]; the
/// read-side is a fresh [`SequentialReader`] constructed on demand
/// (ADR-007 — the reader has mutable cursor state that the consumer
/// owns, so the engine is a factory, not a holder).
/// Packed sequential layout. The write-side is [`LayoutBuilder`];
/// the read-side compiled form is the [`ReadPlan`] (ADR-011),
/// shared via `Arc` with every [`SequentialReader`] the factory
/// hands out (ADR-007 — the reader owns its cursor state, so the
/// engine is a factory, not a holder).
Packed {
builder: LayoutBuilder,
builder: Box<LayoutBuilder>,
plan: Arc<ReadPlan>,
},
/// Aligned static layout. Field offsets are precomputed in an
/// [`OffsetMap`] for random access.
@@ -71,7 +75,7 @@ enum Layout {
///
/// Validation is split (D-BAST-006, D-BAST-007):
/// - [`AlkTypeEngine::validate_bytes`] uses the BAST-native validator
/// ([`bast_validation`]) — no `jsonschema` involvement, no external
/// ([`crate::bast_validation`]) — no `jsonschema` involvement, no external
/// JSON Schema required.
/// - [`AlkTypeEngine::validate_json`] / [`AlkTypeEngine::is_valid_json`]
/// use a standard `jsonschema::Validator` compiled at construction
@@ -81,9 +85,9 @@ enum Layout {
pub struct AlkTypeEngine {
layout: Layout,
json_validator: Option<jsonschema::Validator>,
validation_plan: Arc<ValidationPlan>,
endian: Endian,
bast_doc: Value,
root_name: String,
doc: BastDoc,
}
impl AlkTypeEngine {
@@ -109,10 +113,10 @@ impl AlkTypeEngine {
/// BAST document — BAST describes bytes, not JSON shape. It may be
/// authored separately or derived from BAST via future codegen.
///
/// The engine retains a clone of the BAST `Value` so that
/// [`AlkTypeEngine::sequential_reader`] and
/// [`AlkTypeEngine::read_field`] can re-parse the typed tree on
/// demand without lifetime entanglement with the caller's `Value`.
/// The engine retains an owned [`BastDoc`] (ADR-012 §2a) so that
/// [`AlkTypeEngine::validate_bytes`] and
/// [`AlkTypeEngine::read_field`] can walk the typed tree without
/// re-parsing the raw document per call.
///
/// # Errors
///
@@ -139,10 +143,19 @@ impl AlkTypeEngine {
}
};
let endian = struct_node.endian();
// The compiled value-domain constraint tree (ADR-012 §3) — built
// once here, walked per buffer by `validate_bytes`. Its compile
// walk also rejects cyclic `$ref` graphs here, before the layout
// builders below (the standalone walkers now guard themselves
// via `walk_guard::check_ref_graph` — review #006 H2 — so this
// gate is engine-compile-time confirmation, not the only line of
// defense).
let validation_plan = Arc::new(ValidationPlan::compile(&doc)?);
let layout = match mode {
LayoutMode::Packed => {
let builder = LayoutBuilder::new(bast_doc, root_name)?;
Layout::Packed { builder }
let builder = Box::new(LayoutBuilder::new(bast_doc, root_name)?);
let plan = Arc::new(ReadPlan::compile(bast_doc, root_name)?);
Layout::Packed { builder, plan }
}
LayoutMode::Aligned => {
let offset_map = OffsetMap::compute(&doc)?;
@@ -156,9 +169,9 @@ impl AlkTypeEngine {
Ok(Self {
layout,
json_validator,
validation_plan,
endian,
bast_doc: bast_doc.clone(),
root_name: root_name.to_string(),
doc,
})
}
@@ -167,6 +180,11 @@ impl AlkTypeEngine {
self.endian
}
/// The root type name this engine was compiled with (D-BAST-001).
pub fn root_name(&self) -> &str {
self.doc.root_name()
}
/// The layout mode this engine was compiled with.
pub fn mode(&self) -> LayoutMode {
match self.layout {
@@ -194,15 +212,17 @@ impl AlkTypeEngine {
}
/// Construct a fresh [`SequentialReader`] for packed-mode reads
/// (ADR-007). Each call returns a new reader with the cursor at
/// position 0. The consumer owns the reader and calls
/// `read_next`/`read_field`/`reset` on it directly.
/// (ADR-007, ADR-011). Each call returns a new reader with the
/// cursor at position 0, sharing the engine's `Arc<ReadPlan>` (a
/// refcount bump — no re-parse, no document clone). The consumer
/// owns the reader and calls `read_next`/`read_field`/`reset` on it
/// directly.
///
/// Returns `None` if compiled in aligned mode.
pub fn sequential_reader(&self) -> Option<SequentialReader> {
match &self.layout {
Layout::Packed { .. } => {
SequentialReader::new(&self.bast_doc, &self.root_name).ok()
Layout::Packed { plan, .. } => {
Some(SequentialReader::new(Arc::clone(plan)))
}
Layout::Aligned { .. } => None,
}
@@ -256,17 +276,22 @@ impl AlkTypeEngine {
/// Validate a binary buffer against the schema by materializing a
/// `serde_json::Value` tree from the bytes (walking the layout engine)
/// and then validating that `Value` against the BAST-native validator
/// — a recursive walker over the BAST type tree that checks the
/// value-domain constraints the materializer does not (integer
/// ranges, `maxLength`, enum index bounds, union
/// variant constraints). Decided in D-BAST-006; see
/// and then validating that `Value` against the compiled
/// [`ValidationPlan`] — the engine's value-domain constraint tree
/// (integer ranges, `maxLength`, enum index bounds, union variant
/// constraints), built once at [`AlkTypeEngine::compile`] from the
/// BAST document (ADR-010; ADR-012 §3). Decided in D-BAST-006; see
/// [bast-format.md §Validation Model](../../docs/architecture/bast-format.md#validation-model).
///
/// The materialize half is the only per-buffer schema-touching step
/// (the bytes must be decoded against the tree); the validation half
/// walks the compiled plan — no `$ref` re-resolution, no schema
/// re-parse per buffer.
///
/// Dispatches on the engine's layout mode: packed mode walks
/// sequentially from offset 0; aligned mode reads at offsets from
/// the `OffsetMap`. Both produce the same `Value` form; the
/// BAST-native validator is mode-agnostic.
/// validation plan is mode-agnostic.
///
/// # Errors
///
@@ -281,32 +306,54 @@ impl AlkTypeEngine {
/// (D-BAST-009 — the variant type stays uniform with the
/// `validate_json` path).
pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError> {
let doc = BastDoc::new(&self.bast_doc, &self.root_name)?;
let value = match &self.layout {
Layout::Packed { .. } => materialize::materialize_packed(&doc, buffer)?,
Layout::Packed { plan, .. } => {
materialize::materialize_packed(plan, buffer)?
}
Layout::Aligned { offset_map } => {
materialize::materialize_aligned(&doc, buffer, offset_map)?
materialize::materialize_aligned(&self.doc, buffer, offset_map)?
}
};
bast_validation::validate_value(&doc, &value)
self.validation_plan.validate(&value)
}
/// Access the compiled [`ValidationPlan`] (ADR-012 §3).
///
/// The plan is the engine's value-domain constraint tree — built once
/// at [`AlkTypeEngine::compile`] from the BAST document, shared via
/// `Arc`, and walked by [`AlkTypeEngine::validate_bytes`] per buffer
/// without touching the BAST document. Exposed for consumers that
/// want to validate their *own* materialized `Value` trees
/// (e.g. one produced by an external reader) against the same
/// constraints, or that want the plan's
/// [`fingerprint`](ValidationPlan::fingerprint) for caching or
/// schema handshakes.
pub fn validation_plan(&self) -> &Arc<ValidationPlan> {
&self.validation_plan
}
/// Read a field from a buffer at its computed offset (aligned mode).
///
/// Looks up the field's byte range in the [`OffsetMap`] and reads the
/// appropriate type using the [`crate::data_access`] functions. Works
/// for fixed-size primitive kinds and length-prefixed `String`/
/// `Bytes` fields.
/// Looks up the field's [`OffsetEntry`](crate::offset_map::OffsetEntry) in the [`OffsetMap`] and reads
/// the appropriate type using the [`crate::data_access`] functions,
/// dispatching on the entry's [`LeafMeta`](crate::offset_map::LeafMeta) (kind, encoding, effective
/// endian — computed at compile time, ADR-012 §2b). Works for
/// fixed-size primitive kinds and length-prefixed `String`/`Bytes`
/// fields.
///
/// Returns an error if compiled in packed mode — use
/// [`AlkTypeEngine::sequential_reader`] for packed mode. Also
/// returns an error for composite kinds (`Struct`, `Union`, `Array`,
/// `Record`) — those are better handled via the layout-specific APIs.
/// [`AlkTypeEngine::sequential_reader`] for packed mode. Composite
/// kinds error as well: a struct path never has an `OffsetMap` entry
/// (only its leaf fields are recorded — read those by their dotted
/// paths), and `Union`/`Array`/`Record` are better handled via the
/// layout-specific APIs.
///
/// # Errors
///
/// - [`AlkTypeError::Access`] if compiled in packed mode.
/// - [`AlkTypeError::Offset`] if `field_path` is not in the offset map.
/// - [`AlkTypeError::Offset`] if `field_path` is not in the offset
/// map (the reachable failure for any composite path, including
/// struct paths — no entry exists for them).
/// - [`AlkTypeError::Access`] for buffer-too-short or invalid data,
/// propagated from [`crate::data_access`].
pub fn read_field<'a>(
@@ -325,19 +372,14 @@ impl AlkTypeEngine {
});
}
};
let range = offset_map
let entry = offset_map
.get(field_path)
.ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "field not found in offset map".to_string(),
})?;
let doc = BastDoc::new(&self.bast_doc, &self.root_name)?;
let leaf = lookup_leaf_field(&doc, field_path).ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "field schema not found in BAST tree or has no primitive kind".to_string(),
})?;
let kind = leaf.kind;
let endian = leaf.endian;
let (kind, encoding, endian) = (entry.meta.kind, entry.meta.encoding, entry.meta.endian);
let range = entry.range;
match kind {
AlkTypeKind::Int8 => {
let v = data_access::read_i8(buffer, range.start, field_path)?;
@@ -388,7 +430,7 @@ impl AlkTypeEngine {
Ok(FieldValue::Enum(v))
}
AlkTypeKind::String => {
let v = match leaf.encoding {
let v = match encoding {
VariableEncoding::OffsetIndirect => {
data_access::read_string_indirect(buffer, range.start, field_path, endian)?
}
@@ -399,7 +441,7 @@ impl AlkTypeEngine {
Ok(FieldValue::String(v))
}
AlkTypeKind::Bytes => {
let v = match leaf.encoding {
let v = match encoding {
VariableEncoding::OffsetIndirect => {
data_access::read_bytes_indirect(buffer, range.start, field_path, endian)?
}
@@ -409,9 +451,12 @@ impl AlkTypeEngine {
};
Ok(FieldValue::Bytes(v))
}
AlkTypeKind::Struct => Ok(FieldValue::Struct {
start: range.start,
end: range.end,
AlkTypeKind::Struct => Err(AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "read_field does not support composite types (no offset-map entry \
exists for a struct path — only its leaf fields are recorded); \
read the leaf fields by their dotted paths instead"
.to_string(),
}),
AlkTypeKind::Union | AlkTypeKind::Array | AlkTypeKind::Record => {
Err(AlkTypeError::Access {
@@ -426,10 +471,12 @@ impl AlkTypeEngine {
/// Write a field to a buffer at its computed offset (aligned mode).
///
/// Looks up the field's byte range in the [`OffsetMap`] and writes the
/// appropriate type using the [`crate::data_access`] functions. Works
/// for fixed-size primitive kinds and length-prefixed `String`/
/// `Bytes` fields.
/// Looks up the field's [`OffsetEntry`](crate::offset_map::OffsetEntry) in the [`OffsetMap`] and writes
/// the appropriate type using the [`crate::data_access`] functions,
/// dispatching on the entry's [`LeafMeta`](crate::offset_map::LeafMeta) (kind, encoding, effective
/// endian — computed at compile time, ADR-012 §2b). Works for
/// fixed-size primitive kinds and length-prefixed `String`/`Bytes`
/// fields.
///
/// Returns an error if compiled in packed mode — use
/// [`AlkTypeEngine::layout_builder`] for packed mode. Also returns
@@ -458,18 +505,14 @@ impl AlkTypeEngine {
});
}
};
let range = offset_map
let entry = offset_map
.get(field_path)
.ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "field not found in offset map".to_string(),
})?;
let doc = BastDoc::new(&self.bast_doc, &self.root_name)?;
let leaf = lookup_leaf_field(&doc, field_path).ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "field schema not found in BAST tree or has no primitive kind".to_string(),
})?;
let endian = leaf.endian;
let (encoding, endian) = (entry.meta.encoding, entry.meta.endian);
let range = entry.range;
match value {
FieldValue::I8(v) => data_access::write_i8(buffer, range.start, *v, field_path),
FieldValue::I16(v) => {
@@ -502,7 +545,7 @@ impl AlkTypeEngine {
data_access::write_enum(buffer, range.start, *v, field_path, endian)
}
FieldValue::String(v) => {
if leaf.encoding == VariableEncoding::OffsetIndirect {
if encoding == VariableEncoding::OffsetIndirect {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
@@ -515,7 +558,7 @@ impl AlkTypeEngine {
Ok(())
}
FieldValue::Bytes(v) => {
if leaf.encoding == VariableEncoding::OffsetIndirect {
if encoding == VariableEncoding::OffsetIndirect {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
@@ -545,60 +588,11 @@ impl fmt::Debug for AlkTypeEngine {
.field("layout", &self.layout)
.field("json_validator", &self.json_validator.as_ref().map(|_| "<jsonschema::Validator>"))
.field("endian", &self.endian)
.field("root_name", &self.root_name)
.field("root_name", &self.root_name())
.finish()
}
}
/// The resolved leaf-field metadata needed by `read_field`/`write_field`:
/// the field's kind, its variable-length encoding, and its effective
/// endianness (field override, else the enclosing struct's default).
struct LeafFieldInfo {
kind: AlkTypeKind,
encoding: VariableEncoding,
endian: Endian,
}
/// Walk the BAST typed tree to find the leaf field for a dotted field
/// path. Returns `None` if any segment is missing or the resolved type is
/// a `$ref` that can't be resolved.
///
/// Splits `field_path` on `.` and descends into the root struct's
/// `fields` at each step, resolving `$ref`s via [`BastDoc::resolve_typeref`].
/// Array element segments (`field[i]`) are not handled here —
/// `read_field`/`write_field` only address leaf fields.
fn lookup_leaf_field(doc: &BastDoc<'_>, field_path: &str) -> Option<LeafFieldInfo> {
let root_def = doc.root_def();
let mut current_struct: BastStruct<'_> = match root_def.kind() {
BastDefKind::Struct(s) => s.clone(),
_ => return None,
};
let mut struct_endian = current_struct.endian();
let segments: Vec<&str> = field_path.split('.').collect();
let last = segments.len();
for (i, segment) in segments.iter().enumerate() {
let field = current_struct.fields().iter().find(|f| f.name() == *segment)?;
let ty = field.ty();
let resolved = doc.resolve_typeref(ty).ok()?;
let field_endian = field.effective_endian(struct_endian);
if i + 1 == last {
return Some(LeafFieldInfo {
kind: resolved.alk_kind(),
encoding: field.encoding(),
endian: field_endian,
});
}
match &resolved {
BastType::Struct(s) => {
current_struct = s.clone();
struct_endian = s.endian();
}
_ => return None,
}
}
None
}
#[cfg(test)]
mod tests {
use super::*;
@@ -751,6 +745,37 @@ mod tests {
assert!(matches!(err, AlkTypeError::Offset { .. }), "got {err:?}");
}
#[test]
fn read_field_on_nested_struct_path_is_offset_error_not_struct_value() {
// M3: the read_field kind dispatch previously had a Struct arm
// returning FieldValue::Struct — unreachable, because
// OffsetMap::compute records only a nested struct's inner leaf
// entries, never an entry for the struct path itself. Lock the
// honest behavior: the struct path is an Offset miss.
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{
"name": "header",
"kind": {
"kind": "struct",
"fields": [ { "name": "magic", "kind": "uint32" } ]
}
}
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let buf = [0u8; 4];
let err = engine.read_field(&buf, "header").unwrap_err();
assert!(matches!(err, AlkTypeError::Offset { .. }), "got {err:?}");
// The leaf inside the nested struct is reachable by dotted path.
assert!(engine.read_field(&buf, "header.magic").is_ok());
}
#[test]
fn read_field_returns_error_for_composite_types() {
let doc = json!({
@@ -985,7 +1010,7 @@ mod tests {
}
#[test]
fn lookup_leaf_field_walks_dotted_path() {
fn offset_map_entries_carry_leaf_meta() {
let doc = json!({
"$defs": {
"S": {
@@ -1005,11 +1030,12 @@ mod tests {
}
}
});
let d = BastDoc::new(&doc, "S").expect("doc");
assert_eq!(lookup_leaf_field(&d, "header.magic").unwrap().kind, AlkTypeKind::Uint32);
assert_eq!(lookup_leaf_field(&d, "header.version").unwrap().kind, AlkTypeKind::Uint8);
assert!(lookup_leaf_field(&d, "header.missing").is_none());
assert!(lookup_leaf_field(&d, "missing").is_none());
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let map = engine.offset_map().expect("aligned map");
assert_eq!(map.get("header.magic").unwrap().meta.kind, AlkTypeKind::Uint32);
assert_eq!(map.get("header.version").unwrap().meta.kind, AlkTypeKind::Uint8);
assert!(map.get("header.missing").is_none());
assert!(map.get("missing").is_none());
}
#[test]
@@ -1266,6 +1292,85 @@ mod tests {
assert!(engine.validate_bytes(&buf).is_ok(), "valid mixed struct should pass");
}
#[test]
fn c1_validate_bytes_packed_decodes_all_twelve_primitives_le() {
// The plan materializer's i16/i32/i64/u64/f64/bool arms had
// zero public-path executions before this battery (review #007
// C1): every packed validate_bytes test fed u8/u32/string
// shapes.
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "little",
"fields": [
{ "name": "i8", "kind": "int8" },
{ "name": "i16", "kind": "int16" },
{ "name": "i32", "kind": "int32" },
{ "name": "i64", "kind": "int64" },
{ "name": "u8", "kind": "uint8" },
{ "name": "u16", "kind": "uint16" },
{ "name": "u32", "kind": "uint32" },
{ "name": "u64", "kind": "uint64" },
{ "name": "f32", "kind": "float32" },
{ "name": "f64", "kind": "float64" },
{ "name": "b", "kind": "bool" }
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
let mut buf = vec![0u8; 1 + 2 + 4 + 8 + 1 + 2 + 4 + 8 + 4 + 8 + 1];
let mut off = 0;
buf[off] = 0x81; off += 1; // i8 = -127
buf[off..off + 2].copy_from_slice(&(-32000i16).to_le_bytes()); off += 2;
buf[off..off + 4].copy_from_slice(&(-2_000_000_007i32).to_le_bytes()); off += 4;
buf[off..off + 8].copy_from_slice(&(-9_000_000_000_000_000_000i64).to_le_bytes()); off += 8;
buf[off] = 0xAB; off += 1; // u8
buf[off..off + 2].copy_from_slice(&0xBEEFu16.to_le_bytes()); off += 2;
buf[off..off + 4].copy_from_slice(&0xDEADBEEFu32.to_le_bytes()); off += 4;
buf[off..off + 8].copy_from_slice(&0x0102030405060708u64.to_le_bytes()); off += 8;
buf[off..off + 4].copy_from_slice(&1.5f32.to_le_bytes()); off += 4;
buf[off..off + 8].copy_from_slice(&2.5f64.to_le_bytes()); off += 8;
buf[off] = 1; off += 1; // bool
assert_eq!(off, buf.len());
assert!(
engine.validate_bytes(&buf).is_ok(),
"all-twelve-primitive battery must validate"
);
// Corrupted-value rejection: bool 0x40 is not 0/1.
let mut bad = buf.clone();
bad[off - 1] = 0x40;
let err = engine.validate_bytes(&bad).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
}
#[test]
fn c1_validate_bytes_packed_decodes_big_endian_subset() {
// The BE leg: packed validate_bytes had only ever decoded BE
// u32s (the chunk-header fixture) before this test.
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "i16", "kind": "int16" },
{ "name": "u64", "kind": "uint64" },
{ "name": "f64", "kind": "float64" }
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
let mut buf = vec![0u8; 18];
buf[0..2].copy_from_slice(&(-32000i16).to_be_bytes());
buf[2..10].copy_from_slice(&0x0102030405060708u64.to_be_bytes());
buf[10..18].copy_from_slice(&2.5f64.to_be_bytes());
assert!(engine.validate_bytes(&buf).is_ok(), "BE battery must validate");
}
#[test]
fn validate_bytes_with_simple_struct_round_trips() {
let doc = json!({
@@ -1287,6 +1392,197 @@ mod tests {
assert!(engine.validate_bytes(&buf).is_ok());
}
#[test]
fn validate_bytes_aligned_record_last_field_round_trips() {
// M2: the aligned record path (OffsetMap LengthPrefixed entry at
// the prefix offset + materialize_typeref_packed dispatch) had
// zero public-path coverage. Record-as-last-field is the only
// safe inline position (ADR-006, M1).
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "little",
"fields": [
{ "name": "id", "kind": "uint32" },
{ "name": "counts", "kind": { "kind": "record", "values": "uint32" } }
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
// Offsets: id @ 0 (4 bytes), counts prefix @ 4 (4 bytes).
// Wire: 4 (id) + 4 (record count) + [4 (key len) + 1 (key) + 4
// (value)] × 2 = 26 bytes.
let mut buf = vec![0u8; 26];
buf[0..4].copy_from_slice(&7u32.to_le_bytes());
let mut off = 4;
buf[off..off + 4].copy_from_slice(&2u32.to_le_bytes());
off += 4;
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
off += 4;
buf[off] = b'b';
off += 1;
buf[off..off + 4].copy_from_slice(&10u32.to_le_bytes());
off += 4;
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
off += 4;
buf[off] = b'a';
off += 1;
buf[off..off + 4].copy_from_slice(&20u32.to_le_bytes());
assert!(engine.validate_bytes(&buf).is_ok());
// Corrupting bytes inside the first entry (key byte + value
// bytes) produces garbage the value-domain check must reject.
let mut corrupt = buf.clone();
corrupt[12] = 0xFF;
corrupt[13] = 0xFF;
corrupt[14] = 0xFF;
corrupt[15] = 0xFF;
assert!(engine.validate_bytes(&corrupt).is_err());
}
#[test]
fn c2_validate_bytes_aligned_inline_string_and_bytes_default_encoding() {
// The aligned validate_bytes tests covered maxLength
// reservations, offset-indirect, records, and unions — but the
// *default* inline length-prefixed encoding (the most common
// real shape) had zero public-path executions
// (materialize_variable_aligned's LengthPrefixed-else branch,
// review #007 C2). ADR-006 allows inline variable fields only
// in the last position, so each shape gets its own schema.
let string_doc = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "little",
"fields": [
{ "name": "id", "kind": "uint32" },
{ "name": "name", "kind": "string" }
]
}
}
});
let engine =
AlkTypeEngine::compile(&string_doc, "S", LayoutMode::Aligned, None).expect("compile");
// Offsets: id @ 0 (4), name prefix @ 4 (4), name data @ 8 (5).
let mut buf = vec![0u8; 13];
buf[0..4].copy_from_slice(&42u32.to_le_bytes());
buf[4..8].copy_from_slice(&5u32.to_le_bytes());
buf[8..13].copy_from_slice(b"hello");
assert!(
engine.validate_bytes(&buf).is_ok(),
"aligned inline string must validate through the default encoding"
);
// Short buffer: name's declared data runs past the buffer end.
let err = engine.validate_bytes(&buf[..10]).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
let bytes_doc = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "little",
"fields": [
{ "name": "blob", "kind": "bytes" }
]
}
}
});
let engine =
AlkTypeEngine::compile(&bytes_doc, "S", LayoutMode::Aligned, None).expect("compile");
let mut bbuf = vec![0u8; 7];
bbuf[0..4].copy_from_slice(&3u32.to_le_bytes());
bbuf[4..7].copy_from_slice(&[0xAA, 0xBB, 0xCC]);
assert!(
engine.validate_bytes(&bbuf).is_ok(),
"aligned inline bytes must validate through the default encoding"
);
}
// ----- ValidationPlan / validate_bytes (ADR-012 §3, phase 7) ----------
#[test]
fn validation_plan_accessor_returns_compiled_plan() {
let doc = uint32_struct_bast();
let engine =
AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
let plan = engine.validation_plan();
assert!(plan.validate(&json!({"id": 42})).is_ok());
assert!(plan.validate(&json!({"id": -1})).is_err());
// Same schema compiled twice produces equal plans (fingerprint
// contract), so the accessor reflects the compile-time build.
let engine2 =
AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
assert_eq!(plan, engine2.validation_plan());
assert_eq!(plan.fingerprint(), engine2.validation_plan().fingerprint());
}
#[test]
fn validate_bytes_enforces_value_domain_via_plan_in_both_modes() {
let doc = json!({
"$defs": { "S": { "kind": "struct", "endian": "little", "fields": [
{ "name": "status", "kind": { "$ref": "#/$defs/Status" } }
] },
"Status": { "kind": "enum", "values": ["Ok", "Err"] } }
});
// Packed: enum index 5 is out of bounds for the 2-value enum.
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
let err = engine.validate_bytes(&5u32.to_le_bytes()).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
// Aligned: same constraint, offset read path (the enum leaf sits
// at offset 0 — no alignment padding for a lone u32-width leaf).
let engine =
AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let mut buf = [0u8; 8];
buf[0..4].copy_from_slice(&5u32.to_le_bytes());
let err = engine.validate_bytes(&buf).unwrap_err();
assert!(matches!(err, AlkTypeError::Validation(_)), "got {err:?}");
// Valid index passes in both modes.
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
assert!(engine.validate_bytes(&1u32.to_le_bytes()).is_ok());
let engine =
AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let mut buf = [0u8; 8];
buf[0..4].copy_from_slice(&1u32.to_le_bytes());
assert!(engine.validate_bytes(&buf).is_ok());
}
#[test]
fn compile_rejects_cyclic_ref_graph_with_schema_error() {
let doc = json!({
"$defs": {
"A": { "kind": "struct", "fields": [
{ "name": "next", "kind": { "$ref": "#/$defs/B" } }
] },
"B": { "kind": "struct", "fields": [
{ "name": "back", "kind": { "$ref": "#/$defs/A" } }
] }
}
});
let err = AlkTypeEngine::compile(&doc, "A", LayoutMode::Packed, None).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
assert!(err.to_string().contains("cyclic"), "got {err:?}");
let err = AlkTypeEngine::compile(&doc, "A", LayoutMode::Aligned, None).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
}
#[test]
fn validation_plan_is_send_sync_shared() {
let doc = uint32_struct_bast();
let engine =
AlkTypeEngine::compile(&doc, "S", LayoutMode::Packed, None).expect("compile");
fn assert_send_sync<T: Send + Sync>(_: &T) {}
assert_send_sync(engine.validation_plan());
// The engine itself must stay Send + Sync now that it holds the
// owned BastDoc (ADR-012 §2a) — alkcall shares engines across
// hub/spoke threads.
fn assert_engine_send_sync<T: Send + Sync>(_: &T) {}
assert_engine_send_sync(&engine);
let shared:std::sync::Arc<_> = engine.validation_plan().clone();
let handle = std::thread::spawn(move || shared.validate(&json!({"id": 1})).is_ok());
assert!(handle.join().expect("join"));
}
// ----- validate_json / is_valid_json tests (step 6, D-BAST-007) -----
fn uint32_struct_bast() -> Value {
src/layout_builder.rs
+418 -36
View File
@@ -44,9 +44,11 @@ use crate::bast::{
BastUnion,
};
use crate::error::AlkTypeError;
use crate::schema::{AlkTypeKind, Endian, DISCRIMINATOR_PATH, U32_SIZE};
use crate::schema::{
AlkTypeKind, Endian, DISCRIMINATOR_PATH, U32_SIZE, MAX_ARRAY_BYTES,
};
use serde_json::Value;
use std::collections::HashMap;
use std::collections::{BTreeMap, HashMap};
const VARIANT_KEY: &str = "__variant";
@@ -71,6 +73,7 @@ pub struct FieldPosition {
#[derive(Debug)]
pub struct PackedLayout {
fields: Vec<(String, FieldPosition)>,
index: BTreeMap<String, usize>,
total_size: usize,
}
@@ -81,10 +84,8 @@ impl PackedLayout {
/// TUnion byte-offset discriminators, the discriminator is recorded
/// under the synthetic path `"<union_path>.__discriminator"`.
pub fn get(&self, field_path: &str) -> Option<&FieldPosition> {
self.fields
.iter()
.find(|(path, _)| path == field_path)
.map(|(_, pos)| pos)
let idx = *self.index.get(field_path)?;
self.fields.get(idx).map(|(_, pos)| pos)
}
/// The total buffer size needed to hold all fields.
@@ -129,8 +130,7 @@ impl PackedLayout {
/// this is correct for protocol wire formats, which pack fields tightly.
#[derive(Debug)]
pub struct LayoutBuilder {
doc_value: Value,
root_name: String,
doc: BastDoc,
endian: Endian,
}
@@ -140,16 +140,20 @@ impl LayoutBuilder {
/// The root type must be a struct. Endianness is read from the
/// root struct's `endian` annotation (defaults to little-endian).
///
/// The builder retains the raw BAST `Value` and re-parses the typed
/// tree on each [`build`](Self::build) call; this is cheap (the
/// typed tree borrows from the source without cloning field data).
/// The builder parses the owned [`BastDoc`] once here (ADR-012 §2a)
/// and every [`build`](Self::build) call reuses the cached tree —
/// no re-parse, no `Value` clone per build.
///
/// # Errors
///
/// Returns [`AlkTypeError::Schema`] if the document is malformed or
/// the root type is not a struct.
/// Returns [`AlkTypeError::Schema`] if the document is malformed, the
/// root type is not a struct, or the reference graph nests deeper
/// than the walk depth cap (128) or contains a `$ref` cycle (review
/// #006 H2 — the build walk's struct/union recursion is unguarded, so
/// cyclic input must be rejected before the walk, not during it).
pub fn new(bast_doc: &Value, root_name: &str) -> Result<Self, AlkTypeError> {
let doc = BastDoc::new(bast_doc, root_name)?;
crate::walk_guard::check_ref_graph(&doc)?;
let root_def = doc.root_def();
let struct_node = match root_def.kind() {
BastDefKind::Struct(s) => s,
@@ -161,11 +165,7 @@ impl LayoutBuilder {
}
};
let endian = struct_node.endian();
Ok(Self {
doc_value: bast_doc.clone(),
root_name: root_name.to_string(),
endian,
})
Ok(Self { doc, endian })
}
/// Build the packed layout given actual data sizes for variable-length
@@ -187,20 +187,25 @@ impl LayoutBuilder {
/// sizes in `var_sizes`, missing discriminator values, or unknown
/// discriminator values.
pub fn build(&self, var_sizes: &HashMap<String, usize>) -> Result<PackedLayout, AlkTypeError> {
let doc = BastDoc::new(&self.doc_value, &self.root_name)?;
let root_def = doc.root_def();
let root_def = self.doc.root_def();
let struct_node = match root_def.kind() {
BastDefKind::Struct(s) => s,
_ => unreachable!("checked in new; doc_value is immutable"),
_ => {
return Err(AlkTypeError::Schema(
"internal: LayoutBuilder root is not a struct (checked in new)".to_string(),
));
}
};
let mut ctx = BuildCtx {
doc: &doc,
doc: &self.doc,
var_sizes,
fields: Vec::new(),
};
let mut offset: usize = 0;
ctx.walk_struct(struct_node, "", &mut offset)?;
let index = Self::build_index(&ctx.fields);
Ok(PackedLayout {
index,
fields: ctx.fields,
total_size: offset,
})
@@ -210,11 +215,23 @@ impl LayoutBuilder {
pub fn endian(&self) -> Endian {
self.endian
}
/// Build the path→index lookup table over the insertion-ordered
/// `fields` vec. First occurrence wins on duplicate paths (matching
/// the linear-scan `find` this index replaced — `BastStruct::parse`
/// does not reject duplicate field names, review #006 L4).
fn build_index(fields: &[(String, FieldPosition)]) -> BTreeMap<String, usize> {
let mut index = BTreeMap::new();
for (i, (path, _)) in fields.iter().enumerate() {
index.entry(path.clone()).or_insert(i);
}
index
}
}
/// Mutable context threaded through the recursive layout computation.
struct BuildCtx<'d> {
doc: &'d BastDoc<'d>,
doc: &'d BastDoc,
var_sizes: &'d HashMap<String, usize>,
fields: Vec<(String, FieldPosition)>,
}
@@ -227,7 +244,7 @@ impl<'d> BuildCtx<'d> {
/// top level).
fn walk_struct(
&mut self,
struct_node: &BastStruct<'d>,
struct_node: &BastStruct,
prefix: &str,
offset: &mut usize,
) -> Result<(), AlkTypeError> {
@@ -246,7 +263,7 @@ impl<'d> BuildCtx<'d> {
/// appending any field paths to `self.fields`.
fn walk_field(
&mut self,
field: &BastField<'d>,
field: &BastField,
field_path: &str,
offset: &mut usize,
) -> Result<(), AlkTypeError> {
@@ -330,7 +347,7 @@ impl<'d> BuildCtx<'d> {
/// supported in v1.
fn walk_array(
&mut self,
array: &BastArray<'d>,
array: &BastArray,
field_path: &str,
offset: &mut usize,
) -> Result<(), AlkTypeError> {
@@ -352,6 +369,21 @@ impl<'d> BuildCtx<'d> {
reason: format!("element kind {elem_kind} has no fixed size"),
})?;
let count = array.count();
let array_bytes = count
.checked_mul(elem_size)
.ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: format!("array size {count} × {elem_size} overflows usize"),
})?;
if array_bytes > MAX_ARRAY_BYTES {
return Err(AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: format!(
"array size {count} × {elem_size} = {array_bytes} bytes exceeds the \
compile-time limit of {MAX_ARRAY_BYTES} bytes"
),
});
}
let start = *offset;
for i in 0..count {
@@ -394,7 +426,7 @@ impl<'d> BuildCtx<'d> {
/// Compute the layout for a `BastUnion` field.
fn walk_union(
&mut self,
union_node: &BastUnion<'d>,
union_node: &BastUnion,
field_path: &str,
offset: &mut usize,
) -> Result<(), AlkTypeError> {
@@ -415,7 +447,7 @@ impl<'d> BuildCtx<'d> {
offset: &mut usize,
disc_type: AlkTypeKind,
disc_off: usize,
union_node: &BastUnion<'d>,
union_node: &BastUnion,
) -> Result<(), AlkTypeError> {
let disc_size = disc_type.type_size().ok_or_else(|| AlkTypeError::Offset {
field_path: field_path.to_string(),
@@ -481,14 +513,21 @@ impl<'d> BuildCtx<'d> {
/// Lay out a TUnion with a field-name discriminator.
///
/// The discriminator is a regular field within the variant struct.
/// Wire convention (ADR-011 addendum, review #006 H3): the union's
/// declared `fields` (the discriminator field + any shared fields)
/// are laid out first at the union's offset, then the selected
/// variant's fields follow — the same walk the reader and
/// materializer perform over the compiled `shared` plan. Variants
/// must not re-declare shared fields (enforced in
/// [`BastUnion::parse`]), so the two walks cover disjoint fields.
///
/// The consumer selects the variant by 0-based index in
/// `var_sizes["<union_path>.__variant"]`.
fn walk_field_discriminator_union(
&mut self,
field_path: &str,
offset: &mut usize,
union_node: &BastUnion<'d>,
union_node: &BastUnion,
) -> Result<(), AlkTypeError> {
let variant_key = format!("{field_path}.{VARIANT_KEY}");
let variant_index =
@@ -524,6 +563,10 @@ impl<'d> BuildCtx<'d> {
});
}
};
for field in union_node.fields() {
let field_field_path = format!("{field_path}.{}", field.name());
self.walk_field(field, &field_field_path, offset)?;
}
self.walk_struct(variant_struct, field_path, offset)?;
Ok(())
}
@@ -561,6 +604,121 @@ mod tests {
pairs.iter().map(|(k, v)| (k.to_string(), *v)).collect()
}
// ----- H2: cyclic schemas rejected at new(), not stack overflow ------
#[test]
fn h2_cyclic_ref_rejected_at_new_not_stack_overflow() {
let root = json!({
"$defs": {
"S": { "kind": "struct", "fields": [
{ "name": "me", "kind": { "$ref": "#/$defs/S" } }
] }
}
});
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("cyclic"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn h2_two_def_cycle_rejected_at_new() {
let root = json!({
"$defs": {
"A": { "kind": "struct", "fields": [
{ "name": "next", "kind": { "$ref": "#/$defs/B" } }
] },
"B": { "kind": "struct", "fields": [
{ "name": "back", "kind": { "$ref": "#/$defs/A" } }
] }
}
});
let err = LayoutBuilder::new(&root, "A").unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
}
#[test]
fn h2_diamond_refs_still_build() {
let root = json!({
"$defs": {
"S": { "kind": "struct", "fields": [
{ "name": "a", "kind": { "$ref": "#/$defs/Point" } },
{ "name": "b", "kind": { "$ref": "#/$defs/Point" } }
] },
"Point": { "kind": "struct", "fields": [
{ "name": "x", "kind": "uint16" }
] }
}
});
let layout = build(&root, "S", &var_sizes(&[]));
assert_eq!(
layout.get("a.x"),
Some(&FieldPosition { offset: 0, size: 2, kind: AlkTypeKind::Uint16 })
);
assert_eq!(
layout.get("b.x"),
Some(&FieldPosition { offset: 2, size: 2, kind: AlkTypeKind::Uint16 })
);
}
// ----- L4: path-indexed random access --------------------------------
#[test]
fn l4_get_is_indexed_random_access_over_large_nested_schema() {
let mut fields = Vec::new();
for i in 0..40 {
fields.push(json!({
"name": format!("blk{i}"),
"kind": {
"kind": "struct",
"fields": [
{ "name": "off", "kind": "uint16" },
{ "name": "data", "kind": "uint64" }
]
}
}));
}
let root = json!({
"$defs": { "S": { "kind": "struct", "fields": fields } }
});
let layout = build(&root, "S", &var_sizes(&[]));
assert_eq!(layout.iter().count(), 80);
let last = layout.get("blk39.data").expect("late dotted-path lookup");
assert_eq!(last.offset, 39 * 10 + 2);
assert_eq!(last.size, 8);
assert_eq!(last.kind, AlkTypeKind::Uint64);
assert_eq!(
layout.get("blk12.off").map(|p| p.offset),
Some(12 * 10)
);
assert_eq!(layout.get("nope"), None);
assert_eq!(layout.get("blk39"), None);
}
#[test]
fn l4_duplicate_field_paths_first_occurrence_wins() {
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "a", "kind": "uint8" },
{ "name": "a", "kind": "uint16" }
]
}
}
});
let layout = build(&root, "S", &var_sizes(&[]));
let first = layout.get("a").expect("duplicate path resolves");
assert_eq!(first.offset, 0);
assert_eq!(first.size, 1);
assert_eq!(first.kind, AlkTypeKind::Uint8);
assert_eq!(layout.iter().count(), 2);
}
#[test]
fn fixed_fields_packed_no_alignment_padding() {
let root = json!({
@@ -898,6 +1056,51 @@ mod tests {
assert_eq!(layout.total_size(), 12);
}
// ----- H1: array caps bound untrusted schemas at build time --------
#[test]
fn h1_array_count_above_cap_rejected_at_parse() {
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "vals", "kind": { "kind": "array", "element": "uint8", "count": 2000000000 } }
]
}
}
});
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("compile-time limit"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn h1_array_bytes_above_cap_rejected_at_build_is_defense_in_depth() {
// The byte cap cannot be reached through the public API with the
// current caps (packed mode has no alignment: max fixed element
// is 8 bytes, count <= 2^16, product <= 2^19 << 2^26) — the
// check exists to hold if the element cap or the fixed-size kind
// set ever grows. The reachable byte-cap path is exercised in
// offset_map (aligned mode, align-driven stride).
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "vals", "kind": { "kind": "array", "element": "uint64", "count": 65536 } }
]
}
}
});
let layout = build(&root, "S", &var_sizes(&[]));
assert_eq!(layout.total_size(), 8 * 65536);
}
#[test]
fn array_fixed_count_after_preceding_field() {
let root = json!({
@@ -1221,6 +1424,10 @@ mod tests {
#[test]
fn union_field_name_discriminator() {
// Shared-then-variant wire convention (ADR-011 addendum): the
// union's `fields` (`type`) are laid out first, then the
// variant's own fields (`handle`). The variant must not
// re-declare `type` (BastUnion::parse rejects that).
let root = json!({
"$defs": {
"S": {
@@ -1246,14 +1453,12 @@ mod tests {
"Read": {
"kind": "struct",
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "handle", "kind": "uint32" }
]
},
"Write": {
"kind": "struct",
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "handle", "kind": "uint32" },
{ "name": "length", "kind": "uint32" }
]
@@ -1308,14 +1513,12 @@ mod tests {
"Read": {
"kind": "struct",
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "handle", "kind": "uint32" }
]
},
"Write": {
"kind": "struct",
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "handle", "kind": "uint32" },
{ "name": "length", "kind": "uint32" }
]
@@ -1358,7 +1561,7 @@ mod tests {
},
"Read": {
"kind": "struct",
"fields": [ { "name": "type", "kind": "uint8" } ]
"fields": [ { "name": "handle", "kind": "uint8" } ]
}
}
});
@@ -1390,7 +1593,7 @@ mod tests {
},
"Read": {
"kind": "struct",
"fields": [ { "name": "type", "kind": "uint8" } ]
"fields": [ { "name": "handle", "kind": "uint8" } ]
}
}
});
@@ -1400,6 +1603,185 @@ mod tests {
assert!(matches!(err, AlkTypeError::Offset { .. }));
}
#[test]
fn union_field_name_variant_redeclaring_disc_is_schema_error() {
// H3 enforcement: a variant that re-declares the discriminator
// field (or any shared field) is rejected when the union
// definition is parsed — via the root (LayoutBuilder::new on a
// union root) or via lazy $ref resolution during build.
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [ { "name": "type", "kind": "uint8" } ],
"mapping": {
"read": { "$ref": "#/$defs/Read" }
}
},
"Read": {
"kind": "struct",
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "handle", "kind": "uint32" }
]
}
}
});
// H2: the graph check in `new()` eagerly parses every reachable
// def, so the union's invalid shape surfaces at `new()` — the
// same Schema error, earlier than the old lazy `build()` hit.
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("re-declares"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn union_root_redeclaring_disc_rejected_at_parse() {
let root = json!({
"$defs": {
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [ { "name": "type", "kind": "uint8" } ],
"mapping": {
"read": { "$ref": "#/$defs/Read" }
}
},
"Read": {
"kind": "struct",
"fields": [ { "name": "type", "kind": "uint8" } ]
}
}
});
let err = BastDoc::new(&root, "Event").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("re-declares"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn union_field_name_duplicate_shared_field_is_schema_error() {
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "type", "kind": "uint32" }
],
"mapping": {
"read": { "$ref": "#/$defs/Read" }
}
},
"Read": {
"kind": "struct",
"fields": [ { "name": "handle", "kind": "uint32" } ]
}
}
});
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("more than once"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn union_field_name_missing_disc_field_in_fields_is_schema_error() {
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "kind" },
"fields": [ { "name": "type", "kind": "uint8" } ],
"mapping": {
"read": { "$ref": "#/$defs/Read" }
}
},
"Read": {
"kind": "struct",
"fields": [ { "name": "handle", "kind": "uint32" } ]
}
}
});
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("no field"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn union_field_name_non_first_disc_field_is_schema_error() {
// H3 item 2: the packed reader reads the disc at the union's
// start offset, so a disc field that is not the first shared
// field would make it dispatch on the wrong bytes. Rejected at
// parse (probe-verified divergence in review #006).
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "tag" },
"fields": [
{ "name": "seq", "kind": "uint32" },
{ "name": "tag", "kind": "uint8" }
],
"mapping": {
"read": { "$ref": "#/$defs/Read" }
}
},
"Read": {
"kind": "struct",
"fields": [ { "name": "handle", "kind": "uint32" } ]
}
}
});
let err = LayoutBuilder::new(&root, "S").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("must be the first"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
fn iter_returns_fields_in_layout_order() {
let root = json!({
src/lib.rs
+26 -13
View File
@@ -9,23 +9,29 @@
//!
//! - **BAST parser** ([`bast`]): Typed tree over a BAST document —
//! `BastDoc`/`BastDef`/`BastStruct`/`BastField`/`BastType`/etc.
//! Borrows from the source `serde_json::Value` without cloning field
//! data.
//! Owns its data (ADR-012 §2a) so consumers (`AlkTypeEngine`,
//! `LayoutBuilder`) can store the parsed tree without lifetime
//! entanglement; the clone happens once at `BastDoc::new`.
//! - **Layout engine** ([`offset_map`], [`layout_builder`],
//! [`sequential_reader`]): Two layout modes — aligned static for
//! mmap-friendly formats, packed sequential for protocol wire formats.
//! All three consume the BAST typed tree.
//! [`sequential_reader`], [`read_plan`]): Two layout modes — aligned
//! static for mmap-friendly formats, packed sequential for protocol
//! wire formats. All consume the BAST typed tree; `read_plan` is the
//! packed read-side compiled form (ADR-011).
//! - **Data access** ([`data_access`]): Typed read/write at computed
//! offsets, zero-copy for fixed-size types.
//! - **TUnion dispatch** ([`tunion`]): Byte-offset and field-name
//! discriminator dispatch over `BastUnion`.
//! - **Validation** ([`validation`], [`bast_validation`]): two
//! validators for two paths. `bast_validation` is the BAST-native
//! validator for `validate_bytes` — a recursive walker over the BAST
//! type tree (D-BAST-006). `validation` builds a standard
//! `jsonschema::Validator` from a consumer-provided JSON Schema for
//! `validate_json` / `is_valid_json` (D-BAST-007) — no custom keywords,
//! no BAST involvement.
//! - **Validation** ([`validation`], [`bast_validation`],
//! [`validation_plan`]): two validators for two paths.
//! `validation_plan` is the compiled `ValidationPlan` — a
//! compile-once-walk-many constraint tree over the BAST document's
//! value-domain constraints, walked by `validate_bytes` per buffer
//! without re-touching the document (ADR-012 §3). `bast_validation`
//! hosts the one-shot `validate_value` wrapper over the plan.
//! `validation` builds a standard `jsonschema::Validator` from a
//! consumer-provided JSON Schema for `validate_json` /
//! `is_valid_json` (D-BAST-007) — no custom keywords, no BAST
//! involvement.
//! - **Builder** ([`builder`]): Fluent Rust API for constructing BAST
//! documents (binary layout) and standard JSON Schemas (JSON
//! validation) at runtime, producing `serde_json::Value` (ADR-009,
@@ -48,10 +54,13 @@ pub mod error;
pub mod layout_builder;
pub mod materialize;
pub mod offset_map;
pub mod read_plan;
pub mod schema;
pub mod sequential_reader;
pub mod tunion;
pub mod validation;
pub mod validation_plan;
pub(crate) mod walk_guard;
pub use bast_meta::BAST_META_SCHEMA;
pub use bast::{
@@ -62,8 +71,12 @@ pub use builder::{Definitions, Discriminator, Schema};
pub use engine::{LayoutMode, AlkTypeEngine};
pub use error::AlkTypeError;
pub use layout_builder::{FieldPosition, LayoutBuilder, PackedLayout};
pub use offset_map::{ByteRange, OffsetMap};
pub use offset_map::{ByteRange, LeafMeta, OffsetEntry, OffsetMap};
pub use read_plan::{
CompositePlan, DiscriminatorPlan, FieldPlan, ReadKind, ReadPlan,
};
pub use schema::{Endian, AlkTypeKind, VariableEncoding};
pub use sequential_reader::{FieldValue, SequentialReader};
pub use tunion::UnionDispatch;
pub use validation::build_validator;
pub use validation_plan::{ValidField, ValidNode, ValidVariant, ValidationPlan};
src/materialize.rs
+1178 -139
View File
File diff suppressed because it is too large. Load diff
src/offset_map.rs
+821 -101
View File
File diff suppressed because it is too large. Load diff
src/read_plan.rs
+1659
View File
File diff suppressed because it is too large. Load diff
src/schema.rs
+33 -2
View File
@@ -16,6 +16,37 @@ use std::fmt;
pub(crate) const U32_SIZE: usize = 4;
pub(crate) const DISCRIMINATOR_PATH: &str = "__discriminator";
/// Maximum declared array element count. Schemas are untrusted input
/// (AGENTS.md §3): every layout walker expands `count` into per-element
/// work (offset-map/layout entries, per-element reads), so an unbounded
/// count is a memory/CPU denial-of-service. The cap applies at parse
/// time, before any walker sees the array; 2^16 elements bounds the
/// per-array compile-time entry cost at ~10 MB.
pub(crate) const MAX_ARRAY_ELEMENTS: usize = 1 << 16;
/// Maximum computed byte size of a single array (`count × element
/// stride`). Bounds the arithmetic product independently of
/// [`MAX_ARRAY_ELEMENTS`] so multi-megabyte strides cannot combine with
/// a legal count into an unbounded layout request.
pub(crate) const MAX_ARRAY_BYTES: usize = 1 << 26;
/// Maximum declared `align` annotation (struct- or field-level).
/// Schemas are untrusted input (AGENTS.md §3): `align_up` rounds the
/// running offset by the full annotation, so an unbounded align lets a
/// one-field schema declare an exabyte-scale layout (`total_size = 2^63`
/// from `align: 2^62` — review #006 N2 probe). Honest layouts never
/// need more than page granularity (4096).
pub(crate) const MAX_ALIGN: usize = 4096;
/// Maximum declared `maxLength` annotation. Schemas are untrusted input
/// (AGENTS.md §3): a string/bytes reservation contributes its full
/// `maxLength` to `total_size`, so an unbounded value lets a one-field
/// schema declare a terabyte-scale layout (`total_size = 2^40` from
/// `maxLength: 1e12` — review #007 F2 probe). The cap matches
/// [`MAX_ARRAY_BYTES`]'s rationale (a single fixed reservation no
/// larger than the largest legal array).
pub(crate) const MAX_LENGTH: usize = 1 << 26;
/// The 18 BAST kinds recognized by the engine.
///
/// Each variant corresponds to a lowercase BAST kind string
@@ -202,14 +233,14 @@ impl fmt::Display for AlkTypeKind {
}
/// Byte endianness for multi-byte integer and float fields.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum Endian {
Little,
Big,
}
/// The encoding strategy for a variable-length type.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub enum VariableEncoding {
/// `[length: u32][data]` — the default. Length prefix at a known offset,
/// variable data follows immediately.
src/sequential_reader.rs
+973 -559
View File
File diff suppressed because it is too large. Load diff
src/tunion.rs
+128 -13
View File
@@ -46,7 +46,7 @@ pub struct UnionDispatch {
/// `mapping`.
pub fn read_byte_discriminator(
buffer: &[u8],
union_node: &BastUnion<'_>,
union_node: &BastUnion,
endian: Endian,
) -> Result<UnionDispatch, AlkTypeError> {
let (offset, disc_type) = match union_node.discriminator() {
@@ -106,18 +106,21 @@ pub fn read_byte_discriminator(
/// - [`AlkTypeError::Schema`] if the union does not have a field-name
/// discriminator, the discriminator field is not declared in
/// `fields`, or the field's kind is not one of `string` / `uint8` /
/// `enum`.
/// `uint16` / `uint32` / `enum` (the same kind set the compiled
/// reader's `plan_discriminator_string_value` accepts — review #006
/// N1 closed the divergence so both public dispatch paths answer
/// identically).
/// - [`AlkTypeError::Access`] if the buffer is too short to contain the
/// discriminator field, or if the read value is not present in the
/// union's `mapping`.
pub fn read_field_discriminator(
buffer: &[u8],
union_node: &BastUnion<'_>,
union_node: &BastUnion,
disc_field_offset: usize,
endian: Endian,
) -> Result<UnionDispatch, AlkTypeError> {
let name = match union_node.discriminator() {
BastDiscriminator::Field { name } => *name,
BastDiscriminator::Field { name } => name.as_str(),
BastDiscriminator::Byte { .. } => {
return Err(AlkTypeError::Schema(
"read_field_discriminator requires a field-name discriminator".to_string(),
@@ -152,6 +155,14 @@ pub fn read_field_discriminator(
let v = read_u8(buffer, disc_field_offset, name)?;
(v.to_string(), 1)
}
AlkTypeKind::Uint16 => {
let v = read_u16(buffer, disc_field_offset, name, endian)?;
(v.to_string(), 2)
}
AlkTypeKind::Uint32 => {
let v = read_u32(buffer, disc_field_offset, name, endian)?;
(v.to_string(), U32_SIZE)
}
AlkTypeKind::Enum => {
let v = read_enum(buffer, disc_field_offset, name, endian)?;
(v.to_string(), U32_SIZE)
@@ -193,9 +204,9 @@ pub fn read_field_discriminator(
/// - [`AlkTypeError::Schema`] if the union has no `mapping` entries or
/// the `key` is not present.
pub fn resolve_variant<'a>(
union_node: &'a BastUnion<'a>,
union_node: &'a BastUnion,
key: &str,
) -> Result<&'a BastType<'a>, AlkTypeError> {
) -> Result<&'a BastType, AlkTypeError> {
union_node
.variant_for(key)
.ok_or_else(|| AlkTypeError::Schema(format!("unknown mapping key: {key}")))
@@ -211,7 +222,7 @@ pub fn resolve_variant<'a>(
///
/// - [`AlkTypeError::Schema`] if the discriminator is a field-name
/// discriminator.
pub fn discriminator_size(union_node: &BastUnion<'_>) -> Result<usize, AlkTypeError> {
pub fn discriminator_size(union_node: &BastUnion) -> Result<usize, AlkTypeError> {
match union_node.discriminator() {
BastDiscriminator::Byte { disc_type, .. } => match disc_type {
AlkTypeKind::Uint8 => Ok(1),
@@ -228,7 +239,7 @@ pub fn discriminator_size(union_node: &BastUnion<'_>) -> Result<usize, AlkTypeEr
}
fn verify_mapping_key(
union_node: &BastUnion<'_>,
union_node: &BastUnion,
key: &str,
field_path: &str,
raw_value: &str,
@@ -252,7 +263,7 @@ mod tests {
const LE: Endian = Endian::Little;
const BE: Endian = Endian::Big;
fn doc_union<'a>(root: &'a serde_json::Value, name: &'a str) -> BastUnion<'a> {
fn doc_union(root: &serde_json::Value, name: &str) -> BastUnion {
let doc = BastDoc::new(root, name).expect("bast doc");
match doc.root_def().kind() {
crate::bast::BastDefKind::Union(u) => u.clone(),
@@ -421,6 +432,78 @@ mod tests {
assert_eq!(d.discriminator_size, 1);
}
#[test]
fn n1_read_field_discriminator_uint16_little_endian() {
// N1: tunion now accepts the same field-disc kinds as the plan
// reader (string/uint8/uint16/uint32/enum) — uint16/uint32 were
// previously rejected with a Schema error, diverging from
// `plan_discriminator_string_value`.
let root = field_union_root("type", "uint16");
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..2].copy_from_slice(&1u16.to_le_bytes());
let d = read_field_discriminator(&buf, &u, 0, LE).expect("read");
assert_eq!(d.key, "1");
assert_eq!(d.variant_offset, 2);
assert_eq!(d.discriminator_size, 2);
}
#[test]
fn n1_read_field_discriminator_uint16_big_endian() {
let root = field_union_root("type", "uint16");
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..2].copy_from_slice(&1u16.to_be_bytes());
let d = read_field_discriminator(&buf, &u, 0, BE).expect("read");
assert_eq!(d.key, "1");
assert_eq!(d.discriminator_size, 2);
}
#[test]
fn n1_read_field_discriminator_uint32_little_endian() {
let root = json!({
"$defs": {
"U": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [ { "name": "type", "kind": "uint32" } ],
"mapping": {
"1024": { "kind": "struct", "fields": [ { "name": "n", "kind": "uint32" } ] }
}
}
}
});
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..4].copy_from_slice(&1024u32.to_le_bytes());
let d = read_field_discriminator(&buf, &u, 0, LE).expect("read");
assert_eq!(d.key, "1024");
assert_eq!(d.variant_offset, 4);
assert_eq!(d.discriminator_size, 4);
}
#[test]
fn n1_read_field_discriminator_uint32_big_endian() {
let root = json!({
"$defs": {
"U": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [ { "name": "type", "kind": "uint32" } ],
"mapping": {
"1024": { "kind": "struct", "fields": [ { "name": "n", "kind": "uint32" } ] }
}
}
}
});
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..4].copy_from_slice(&1024u32.to_be_bytes());
let d = read_field_discriminator(&buf, &u, 0, BE).expect("read");
assert_eq!(d.key, "1024");
assert_eq!(d.discriminator_size, 4);
}
#[test]
fn read_field_discriminator_string_big_endian() {
let root = field_union_root("type", "string");
@@ -453,6 +536,9 @@ mod tests {
#[test]
fn read_field_discriminator_field_not_found_is_schema_error() {
// H3 enforcement: the missing-discriminator-field case is now
// rejected at parse (BastUnion::parse), before any dispatch
// reader can see the union.
let root = json!({
"$defs": {
"U": {
@@ -463,10 +549,13 @@ mod tests {
}
}
});
let u = doc_union(&root, "U");
let buf = [0u8; 4];
let err = read_field_discriminator(&buf, &u, 0, LE).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)));
let err = BastDoc::new(&root, "U").unwrap_err();
match err {
AlkTypeError::Schema(reason) => {
assert!(reason.contains("no field"), "reason: {reason}");
}
other => panic!("expected Schema error, got {other:?}"),
}
}
#[test]
@@ -540,4 +629,30 @@ mod tests {
let err = discriminator_size(&u).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)));
}
#[test]
fn l2_read_field_discriminator_enum_dispatches_on_index() {
// Review #007 L2: the enum arm of read_field_discriminator had
// zero executions — N1's parity tests covered uint16/uint32 but
// skipped enum, the last arm of the documented kind set.
let root = field_union_root("kind", "enum");
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..4].copy_from_slice(&0u32.to_le_bytes());
let d = read_field_discriminator(&buf, &u, 0, LE).expect("read");
assert_eq!(d.key, "0");
assert_eq!(d.variant_offset, 4);
assert_eq!(d.discriminator_size, 4);
}
#[test]
fn l2_read_field_discriminator_enum_big_endian() {
let root = field_union_root("kind", "enum");
let u = doc_union(&root, "U");
let mut buf = vec![0u8; 8];
buf[0..4].copy_from_slice(&1u32.to_be_bytes());
let d = read_field_discriminator(&buf, &u, 0, Endian::Big).expect("read");
assert_eq!(d.key, "1");
assert_eq!(d.variant_offset, 4);
}
}
src/validation_plan.rs
+1133
View File
File diff suppressed because it is too large. Load diff
src/walk_guard.rs
+241
View File
@@ -0,0 +1,241 @@
//! Shared reference-graph guard for the standalone schema walkers.
//!
//! [`check_ref_graph`] is the one-shot pre-walk `AlkTypeEngine::compile`
//! runs before any layout builder, and the one
//! [`crate::offset_map::OffsetMap::compute`],
//! [`crate::layout_builder::LayoutBuilder::new`], and
//! [`crate::materialize::materialize_aligned`] run at their own entry so
//! each is untrusted-input-safe when called without the engine. It
//! rejects reference graphs that nest deeper than [`MAX_GRAPH_DEPTH`] or
//! that contain a `$ref` cycle — the two shapes that would otherwise
//! overflow the walkers' unguarded struct/union recursion (review #006
//! H2; AGENTS.md §3: a cyclic or adversarially deep schema must produce
//! a handleable error, not a stack overflow).
//!
//! The compiled forms keep their own inline guards (their compile walks
//! inline types too, so a graph check alone is not enough for them); the
//! error text intentionally mirrors theirs ("cyclic $ref through…",
//! "compile depth exceeded…") so downstream matching sees one shape.
use crate::bast::{BastDefKind, BastDoc, BastStruct, BastType};
use crate::error::AlkTypeError;
use std::collections::BTreeSet;
/// Maximum `$ref`-graph depth accepted by [`check_ref_graph`]. Matches
/// the plan compilers' `MAX_COMPILE_DEPTH` (128) so every walker rejects
/// the same documents.
pub(crate) const MAX_GRAPH_DEPTH: usize = 128;
pub(crate) fn depth_err(path: &str) -> AlkTypeError {
AlkTypeError::Schema(format!(
"schema walk: compile depth exceeded {MAX_GRAPH_DEPTH} at {path} \
(cyclic $ref or adversarially deep nesting)"
))
}
pub(crate) fn cycle_err(name: &str, path: &str) -> AlkTypeError {
AlkTypeError::Schema(format!(
"schema walk: cyclic $ref through {name:?} at {path}"
))
}
/// Reject cyclic or over-deep `$ref` graphs before a recursive walker
/// sees the document.
///
/// One walk over the reachable definitions: inline structs recurse into
/// their fields; named defs are entered with the path-scoped cycle set
/// (a def currently being expanded) and the depth counter. Diamond
/// references (two fields `$ref`-ing the same def, neither nested inside
/// the other) are allowed — `seen` is removed on exit, so only genuine
/// cycles trip it, the same semantics the plan compilers use.
pub(crate) fn check_ref_graph(doc: &BastDoc) -> Result<(), AlkTypeError> {
let root_def = doc.root_def();
let mut seen = BTreeSet::new();
let path = root_def.name().to_string();
match root_def.kind() {
BastDefKind::Struct(s) => check_struct(doc, s, &path, 0, &mut seen),
BastDefKind::Union(u) => {
check_typeref_list(
doc,
u.mapping().iter().map(|(_, ty)| ty),
&path,
0,
&mut seen,
)?;
check_field_list(doc, u.fields(), &path, 0, &mut seen)
}
BastDefKind::Enum(_) => Ok(()),
}
}
fn check_struct(
doc: &BastDoc,
s: &BastStruct,
path: &str,
depth: usize,
seen: &mut BTreeSet<String>,
) -> Result<(), AlkTypeError> {
if depth > MAX_GRAPH_DEPTH {
return Err(depth_err(path));
}
check_field_list(doc, s.fields(), path, depth, seen)
}
fn check_field_list(
doc: &BastDoc,
fields: &[crate::bast::BastField],
path: &str,
depth: usize,
seen: &mut BTreeSet<String>,
) -> Result<(), AlkTypeError> {
for field in fields {
let field_path = format!("{path}.{}", field.name());
check_typeref(doc, field.ty(), &field_path, depth, seen)?;
}
Ok(())
}
fn check_typeref(
doc: &BastDoc,
ty: &BastType,
path: &str,
depth: usize,
seen: &mut BTreeSet<String>,
) -> Result<(), AlkTypeError> {
if depth > MAX_GRAPH_DEPTH {
return Err(depth_err(path));
}
match ty {
BastType::Ref(r) => {
let name = r.name();
if !seen.insert(name.to_string()) {
return Err(cycle_err(name, path));
}
let def = doc.resolve_ref(r)?;
let def_path = format!("{path} -> {name}");
let out = match def.kind() {
BastDefKind::Struct(s) => {
check_struct(doc, s, &def_path, depth + 1, seen)
}
BastDefKind::Union(u) => {
check_typeref_list(doc, u.mapping().iter().map(|(_, ty)| ty), &def_path, depth + 1, seen)?;
check_field_list(doc, u.fields(), &def_path, depth + 1, seen)
}
BastDefKind::Enum(_) => Ok(()),
};
seen.remove(name);
out
}
BastType::Struct(s) => check_struct(doc, s, path, depth + 1, seen),
BastType::Union(u) => {
check_typeref_list(doc, u.mapping().iter().map(|(_, ty)| ty), path, depth + 1, seen)?;
check_field_list(doc, u.fields(), path, depth + 1, seen)
}
BastType::Array(a) => {
check_typeref(doc, a.element(), path, depth + 1, seen)
}
BastType::Record(r) => check_typeref(doc, r.values(), path, depth + 1, seen),
BastType::Primitive(_) | BastType::Enum(_) => Ok(()),
}
}
fn check_typeref_list<'a, I>(
doc: &BastDoc,
tys: I,
path: &str,
depth: usize,
seen: &mut BTreeSet<String>,
) -> Result<(), AlkTypeError>
where
I: IntoIterator<Item = &'a BastType>,
{
for ty in tys {
check_typeref(doc, ty, path, depth, seen)?;
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn self_cycle_rejected() {
let root = json!({
"$defs": {
"S": { "kind": "struct", "fields": [
{ "name": "me", "kind": { "$ref": "#/$defs/S" } }
] }
}
});
let doc = BastDoc::new(&root, "S").expect("cyclic doc parses");
let err = check_ref_graph(&doc).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
assert!(err.to_string().contains("cyclic"), "got {err:?}");
}
#[test]
fn two_def_cycle_rejected() {
let root = json!({
"$defs": {
"A": { "kind": "struct", "fields": [
{ "name": "next", "kind": { "$ref": "#/$defs/B" } }
] },
"B": { "kind": "struct", "fields": [
{ "name": "back", "kind": { "$ref": "#/$defs/A" } }
] }
}
});
let doc = BastDoc::new(&root, "A").expect("cyclic doc parses");
let err = check_ref_graph(&doc).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
}
#[test]
fn diamond_refs_allowed() {
let root = json!({
"$defs": {
"S": { "kind": "struct", "fields": [
{ "name": "a", "kind": { "$ref": "#/$defs/Point" } },
{ "name": "b", "kind": { "$ref": "#/$defs/Point" } }
] },
"Point": { "kind": "struct", "fields": [
{ "name": "x", "kind": "uint16" }
] }
}
});
let doc = BastDoc::new(&root, "S").expect("doc");
assert!(check_ref_graph(&doc).is_ok());
}
#[test]
fn deep_ref_chain_rejected_not_overflow() {
// 201 named defs chained by refs — depth 201 exceeds the cap of
// 128, but the check itself must complete (bounded stack, clean
// error). Note the chain uses distinct defs, so the cycle set
// never trips; only the depth cap stops it.
let mut defs = serde_json::Map::new();
defs.insert(
"L200".to_string(),
json!({ "kind": "struct", "fields": [ { "name": "v", "kind": "uint8" } ] }),
);
for i in (0..200).rev() {
defs.insert(
format!("L{i}"),
json!({
"kind": "struct",
"fields": [ { "name": "next", "kind": { "$ref": format!("#/$defs/L{}", i + 1) } } ]
}),
);
}
let root = json!({ "$defs": defs });
let doc = BastDoc::new(&root, "L0").expect("deep doc parses (no cycle)");
let err = check_ref_graph(&doc).unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
assert!(
err.to_string().contains("depth exceeded"),
"expected depth error, got {err:?}"
);
}
}
tests/error_paths.rs
+5 -3
View File
@@ -376,7 +376,8 @@ fn sequential_reader_buffer_too_short_returns_access_error() {
}
});
let buffer = [0u8; 2];
let mut reader = SequentialReader::new(&root, "S").unwrap();
let plan = ReadPlan::compile(&root, "S").unwrap();
let mut reader = SequentialReader::new(std::sync::Arc::new(plan));
let err = reader.read_next(&buffer).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
}
@@ -392,7 +393,8 @@ fn sequential_reader_unknown_field_returns_schema_error() {
}
});
let buffer = [0u8; 4];
let mut reader = SequentialReader::new(&root, "S").unwrap();
let plan = ReadPlan::compile(&root, "S").unwrap();
let mut reader = SequentialReader::new(std::sync::Arc::new(plan));
let err = reader.read_field(&buffer, "missing").unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
}
@@ -409,7 +411,7 @@ fn sequential_reader_new_non_struct_returns_schema_error() {
"A": { "kind": "struct", "fields": [] }
}
});
let err = SequentialReader::new(&root, "U").unwrap_err();
let err = ReadPlan::compile(&root, "U").unwrap_err();
assert!(matches!(err, AlkTypeError::Schema(_)), "got {err:?}");
}
tests/poc_roundtrip.rs
+145 -36
View File
@@ -44,29 +44,29 @@ fn fixed_size_round_trip_via_offset_map() -> Result<(), AlkTypeError> {
let mut buffer = vec![0u8; offset_map.total_size()];
let id_range = offset_map.get("id").expect("id range");
data_access::write_u32(&mut buffer, id_range.start, 42, "id", Endian::Little)?;
data_access::write_u32(&mut buffer, id_range.start(), 42, "id", Endian::Little)?;
let score_range = offset_map.get("score").expect("score range");
data_access::write_f32(&mut buffer, score_range.start, 1.5, "score", Endian::Little)?;
data_access::write_f32(&mut buffer, score_range.start(), 1.5, "score", Endian::Little)?;
let flag_range = offset_map.get("flag").expect("flag range");
data_access::write_u8(&mut buffer, flag_range.start, 1, "flag")?;
data_access::write_u8(&mut buffer, flag_range.start(), 1, "flag")?;
let count_range = offset_map.get("count").expect("count range");
data_access::write_u16(
&mut buffer,
count_range.start,
count_range.start(),
1000,
"count",
Endian::Little,
)?;
assert_eq!(
data_access::read_u32(&buffer, id_range.start, "id", Endian::Little)?,
data_access::read_u32(&buffer, id_range.start(), "id", Endian::Little)?,
42
);
let score = data_access::read_f32(&buffer, score_range.start, "score", Endian::Little)?;
let score = data_access::read_f32(&buffer, score_range.start(), "score", Endian::Little)?;
assert!((score - 1.5).abs() < 0.001, "score: {score}");
assert_eq!(data_access::read_u8(&buffer, flag_range.start, "flag")?, 1);
assert_eq!(data_access::read_u8(&buffer, flag_range.start(), "flag")?, 1);
assert_eq!(
data_access::read_u16(&buffer, count_range.start, "count", Endian::Little)?,
data_access::read_u16(&buffer, count_range.start(), "count", Endian::Little)?,
1000
);
Ok(())
@@ -184,29 +184,29 @@ fn nested_struct_round_trip_via_offset_map() -> Result<(), AlkTypeError> {
let header_magic = offset_map.get("header.magic").expect("header.magic");
let payload_prefix = offset_map.get("payload").expect("payload");
assert_eq!(header_version.start, 0);
assert_eq!(header_magic.start, 4);
assert_eq!(payload_prefix.start, 8);
assert_eq!(header_version.start(), 0);
assert_eq!(header_magic.start(), 4);
assert_eq!(payload_prefix.start(), 8);
let data = b"body-data".to_vec();
let mut buffer = vec![0u8; offset_map.total_size() + data.len()];
data_access::write_u32(
&mut buffer,
header_version.start,
header_version.start(),
1,
"header.version",
Endian::Little,
)?;
data_access::write_u32(
&mut buffer,
header_magic.start,
header_magic.start(),
0xCAFEBABE,
"header.magic",
Endian::Little,
)?;
data_access::write_bytes(
&mut buffer,
payload_prefix.start,
payload_prefix.start(),
&data,
"payload",
Endian::Little,
@@ -215,18 +215,18 @@ fn nested_struct_round_trip_via_offset_map() -> Result<(), AlkTypeError> {
assert_eq!(
data_access::read_u32(
&buffer,
header_version.start,
header_version.start(),
"header.version",
Endian::Little
)?,
1
);
assert_eq!(
data_access::read_u32(&buffer, header_magic.start, "header.magic", Endian::Little)?,
data_access::read_u32(&buffer, header_magic.start(), "header.magic", Endian::Little)?,
0xCAFEBABE
);
assert_eq!(
data_access::read_bytes(&buffer, payload_prefix.start, "payload", Endian::Little)?,
data_access::read_bytes(&buffer, payload_prefix.start(), "payload", Endian::Little)?,
&data[..]
);
Ok(())
@@ -257,9 +257,9 @@ fn nested_struct_round_trip_via_engine_aligned() -> Result<(), AlkTypeError> {
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None)?;
let offset_map = engine.offset_map().expect("aligned mode");
assert_eq!(offset_map.get("header.version").unwrap().start, 0);
assert_eq!(offset_map.get("header.flags").unwrap().start, 1);
assert_eq!(offset_map.get("payload_len").unwrap().start, 4);
assert_eq!(offset_map.get("header.version").unwrap().start(), 0);
assert_eq!(offset_map.get("header.flags").unwrap().start(), 1);
assert_eq!(offset_map.get("payload_len").unwrap().start(), 4);
let mut buffer = vec![0u8; offset_map.total_size()];
engine.write_field(&mut buffer, "header.version", &FieldValue::U8(1))?;
@@ -302,23 +302,23 @@ fn big_endian_round_trip_via_offset_map() -> Result<(), AlkTypeError> {
let id_range = offset_map.get("id").expect("id");
let offset_range = offset_map.get("offset").expect("offset");
assert_eq!(id_range.start, 0);
assert_eq!(offset_range.start, 8);
assert_eq!(id_range.start(), 0);
assert_eq!(offset_range.start(), 8);
let value: f64 = std::f64::consts::PI;
let mut buffer = vec![0u8; offset_map.total_size()];
data_access::write_u32(&mut buffer, id_range.start, 0x01020304, "id", endian)?;
data_access::write_f64(&mut buffer, offset_range.start, value, "offset", endian)?;
data_access::write_u32(&mut buffer, id_range.start(), 0x01020304, "id", endian)?;
data_access::write_f64(&mut buffer, offset_range.start(), value, "offset", endian)?;
assert_eq!(&buffer[0..4], &[0x01, 0x02, 0x03, 0x04]);
assert_eq!(&buffer[4..8], &[0x00, 0x00, 0x00, 0x00]);
assert_eq!(&buffer[8..16], value.to_be_bytes());
assert_eq!(
data_access::read_u32(&buffer, id_range.start, "id", endian)?,
data_access::read_u32(&buffer, id_range.start(), "id", endian)?,
0x01020304
);
let read = data_access::read_f64(&buffer, offset_range.start, "offset", endian)?;
let read = data_access::read_f64(&buffer, offset_range.start(), "offset", endian)?;
assert!((read - value).abs() < 1e-12);
Ok(())
}
@@ -342,17 +342,17 @@ fn alignment_padding_round_trip_u8_then_u32() -> Result<(), AlkTypeError> {
let flag_range = offset_map.get("flag").expect("flag");
let id_range = offset_map.get("id").expect("id");
assert_eq!(flag_range.start, 0);
assert_eq!(flag_range.end, 1);
assert_eq!(id_range.start, 4);
assert_eq!(id_range.end, 8);
assert_eq!(flag_range.start(), 0);
assert_eq!(flag_range.end(), 1);
assert_eq!(id_range.start(), 4);
assert_eq!(id_range.end(), 8);
assert_eq!(offset_map.total_size(), 8);
let mut buffer = vec![0u8; offset_map.total_size()];
data_access::write_u8(&mut buffer, flag_range.start, 0xAB, "flag")?;
data_access::write_u8(&mut buffer, flag_range.start(), 0xAB, "flag")?;
data_access::write_u32(
&mut buffer,
id_range.start,
id_range.start(),
0x01020304,
"id",
Endian::Little,
@@ -363,11 +363,11 @@ fn alignment_padding_round_trip_u8_then_u32() -> Result<(), AlkTypeError> {
assert_eq!(&buffer[4..8], 0x01020304u32.to_le_bytes());
assert_eq!(
data_access::read_u8(&buffer, flag_range.start, "flag")?,
data_access::read_u8(&buffer, flag_range.start(), "flag")?,
0xAB
);
assert_eq!(
data_access::read_u32(&buffer, id_range.start, "id", Endian::Little)?,
data_access::read_u32(&buffer, id_range.start(), "id", Endian::Little)?,
0x01020304
);
Ok(())
@@ -459,7 +459,8 @@ fn sequential_reader_round_trip_packed_buffer() -> Result<(), AlkTypeError> {
let after = 1 + 4 + payload.len();
data_access::write_u8(&mut buffer, after, 99, "tail")?;
let mut reader = SequentialReader::new(&root, "S")?;
let plan = ReadPlan::compile(&root, "S")?;
let mut reader = SequentialReader::new(std::sync::Arc::new(plan));
assert_eq!(reader.endian(), Endian::Little);
assert_eq!(reader.position(), 0);
@@ -502,7 +503,8 @@ fn sequential_reader_read_field_walks_preceding_fields() -> Result<(), AlkTypeEr
data_access::write_u32(&mut buffer, 1, 0xDEADBEEF, "b", Endian::Little)?;
data_access::write_u8(&mut buffer, 5, 9, "c")?;
let mut reader = SequentialReader::new(&root, "S")?;
let plan = ReadPlan::compile(&root, "S")?;
let mut reader = SequentialReader::new(std::sync::Arc::new(plan));
let value = reader.read_field(&buffer, "c")?;
assert_eq!(value, FieldValue::U8(9));
assert_eq!(reader.position(), 6);
@@ -601,4 +603,111 @@ fn tunion_byte_offset_discriminator_size_lookup() -> Result<(), AlkTypeError> {
assert_eq!(tunion::discriminator_size(u16_union)?, 2);
assert_eq!(tunion::discriminator_size(u32_union)?, 4);
Ok(())
}
// ---------------------------------------------------------------------------
// L6 (review #006): the single roundtrip test through a field-disc union.
// Write with LayoutBuilder → read with SequentialReader → materialize →
// validate_bytes on the engine — one schema, all three packed-mode
// consumers, so the H3 wire-convention split can never reappear silently.
//
// Fixture shape: the union's `fields` carry the discriminator (`type`,
// uint8) *and* a second shared field (`seq`, uint32); the variant
// (`Read`) declares only its own field (`handle`) — it does NOT
// re-declare the discriminator or any shared field (forbidden since the
// ADR-011 addendum). The disc field is a uint8, so the mapping keys are
// stringified integers ("1" = read). The builder lays out
// shared-then-variant:
// type@0 (1B), seq@1 (4B), handle@5 (4B) — total 9.
// ---------------------------------------------------------------------------
#[test]
fn field_disc_union_roundtrip_build_read_materialize_validate() -> Result<(), AlkTypeError> {
let root = json!({
"$defs": {
"S": {
"kind": "struct",
"endian": "little",
"fields": [
{ "name": "event", "kind": { "$ref": "#/$defs/Event" } },
{ "name": "trailer", "kind": "uint8" }
]
},
"Event": {
"kind": "union",
"discriminator": { "kind": "field", "name": "type" },
"fields": [
{ "name": "type", "kind": "uint8" },
{ "name": "seq", "kind": "uint32" }
],
"mapping": {
"1": { "$ref": "#/$defs/Read" },
"2": { "$ref": "#/$defs/Write" }
}
},
"Read": {
"kind": "struct",
"fields": [ { "name": "handle", "kind": "uint32" } ]
},
"Write": {
"kind": "struct",
"fields": [ { "name": "handle", "kind": "uint32" } ]
}
}
});
// --- Write side: LayoutBuilder (shared-then-variant layout) --------
let builder = LayoutBuilder::new(&root, "S")?;
let layout = builder.build(&var_sizes(&[("event.__variant", 0)]))?;
assert_eq!(layout.total_size(), 9 + 1, "shared(1+4) + variant(4) + trailer(1)");
let handle_pos = layout.get("event.handle").expect("event.handle");
assert_eq!(handle_pos.offset, 5, "variant fields start after shared (type@0, seq@1)");
let disc_pos = layout.get("event.type").expect("event.type");
assert_eq!(disc_pos.offset, 0);
let seq_pos = layout.get("event.seq").expect("event.seq");
assert_eq!(seq_pos.offset, 1);
let mut buffer = vec![0u8; layout.total_size()];
data_access::write_u8(&mut buffer, 0, 1, "event.type")?; // mapping key "1"
data_access::write_u32(&mut buffer, 1, 77, "event.seq", Endian::Little)?;
data_access::write_u32(&mut buffer, 5, 4242, "event.handle", Endian::Little)?;
data_access::write_u8(&mut buffer, 9, 55, "trailer")?;
// --- Engine (packed) + reader + materializer + validator -----------
let engine = AlkTypeEngine::compile(&root, "S", LayoutMode::Packed, None)?;
// validate_bytes (materializer + ValidationPlan) accepts the buffer.
engine.validate_bytes(&buffer)?;
// SequentialReader: the union field reports the disc value and the
// variant start (after the shared walk).
let mut reader = engine.sequential_reader().expect("packed mode has reader");
let (name, value) = reader.read_next(&buffer)?.expect("event");
assert_eq!(name, "event");
let (disc, variant_start) = match &value {
FieldValue::Union { discriminator, variant_start } => (discriminator.clone(), *variant_start),
other => panic!("expected Union, got {other:?}"),
};
assert_eq!(disc, "1", "uint8 disc value 1 stringifies to the mapping key");
assert_eq!(variant_start, 5, "variant starts after the shared walk");
let (name, value) = reader.read_next(&buffer)?.expect("trailer");
assert_eq!(name, "trailer");
assert_eq!(value, FieldValue::U8(55));
// materialize_packed through the engine's plan: shared fields land
// in the object, the variant's handle flattens in.
let plan = engine_sequential_plan(&root, "S")?;
let value = alktype::materialize::materialize_packed(&plan, &buffer)?;
assert_eq!(value["event"]["type"], json!(1));
assert_eq!(value["event"]["seq"], json!(77));
assert_eq!(value["event"]["handle"], json!(4242));
assert_eq!(value["event"]["__discriminator"], json!("1"));
assert_eq!(value["trailer"], json!(55));
Ok(())
}
fn engine_sequential_plan(root: &serde_json::Value, name: &str) -> Result<std::sync::Arc<ReadPlan>, AlkTypeError> {
Ok(std::sync::Arc::new(ReadPlan::compile(root, name)?))
}
tests/tunion_dispatch.rs
+1 -1
View File
@@ -49,7 +49,7 @@ fn sftp_like_byte_union_doc() -> serde_json::Value {
})
}
fn union_of<'a>(root: &'a serde_json::Value, name: &'a str) -> BastUnion<'a> {
fn union_of(root: &serde_json::Value, name: &str) -> BastUnion {
let doc = BastDoc::new(root, name).expect("bast doc");
match doc.root_def().kind() {
BastDefKind::Union(u) => u.clone(),