Port alknet-typedef crate from alknet

Copy the binary struct engine (src/, tests/) verbatim from alknet/crates/alknet-typedef and create a standalone Cargo.toml (workspace-inherited fields inlined). Port the architecture docs (specs, ADRs 095-102, OQs 069-071) from alknet's nested multi-crate layout to a flat single-crate layout, fixing relative link paths. Build, 295 tests, and clippy all pass clean.
2026-08-02 05:59:12 +00:00
parent eac7ad88b3
commit 2c4a4994dc
36 changed files with 13805 additions and 0 deletions
--- a/docs/architecture/layout-engine.md
+++ b/docs/architecture/layout-engine.md
@@ -0,0 +1,360 @@
+---
+status: draft
+last_updated: 2026-07-22
+---
+
+# alknet-typedef — Layout Engine
+
+The layout engine: offset computation, the two layout modes (packed
+sequential vs aligned static), alignment, endianness, and variable-length
+field handling. This is the novel code — the recursive walk of the schema
+JSON that computes byte positions for each field.
+
+## The Two Layout Modes
+
+The POCs surfaced that protocols and mmap-friendly formats need different
+layout strategies. This is the most important architectural finding —
+decided in [ADR-096](decisions/096-two-layout-modes-packed-vs-aligned.md).
+
+### Mode 1: Packed sequential (protocol wire formats)
+
+Fields are packed with no alignment padding. Variable-length fields shift
+all subsequent fields. Used by SFTP, channels, TTY, and most binary
+protocols.
+
+**Components:**
+
+- **`LayoutBuilder`** — constructed via `LayoutBuilder::new(schema)` (requires `TypeDef:Struct` at the top level), then `builder.build(&var_sizes) -> Result<PackedLayout, TypedefError>` where `var_sizes: &HashMap<String, usize>` maps variable-length field paths (and TUnion discriminator/variant keys) to their actual byte sizes. Used at write time when the consumer knows the data sizes upfront. The builder computes positions only; the consumer writes data via the [`data_access`](data-access.md) functions at the computed positions.
+- **`SequentialReader`** — constructed via `SequentialReader::new(schema)`, then driven by `reader.read_next(&buffer) -> Result<Option<(String, FieldValue)>, TypedefError>` until `Ok(None)`, or `reader.read_field(&buffer, path)` to seek a single field (which walks all preceding fields to reach the target). `reader.reset()` rewinds to the start. Used at read time when the consumer is parsing an incoming frame.
+
+**How it works:**
+
+For a struct with fields `[u8, u32, string]` where the string is 10 bytes:
+
+```
+LayoutBuilder::build(var_sizes: {"payload": 10}):
+  field[0] u8:     offset 0, size 1
+  field[1] u32:    offset 1, size 4
+  field[2] string: offset 5, size 4 (length prefix) + 10 (data)
+  total: 19
+
+SequentialReader::read_next (read):
+  read u8 at offset 0
+  read u32 at offset 1
+  read u32 length prefix at offset 5 → data_len
+  read string data at offset 9, length data_len
+  next field at offset 9 + data_len
+```
+
+There is no alignment padding. The `u32` at offset 1 is unaligned — this
+is correct for protocol wire formats, which pack fields tightly.
+
+**Variable-length fields in packed mode:**
+
+The `LayoutBuilder` takes actual data sizes for variable-length fields
+to compute correct positions for subsequent fields. The consumer must
+know the data sizes before writing — this is inherent to packed layouts.
+
+The `SequentialReader` reads each field's length prefix to determine the
+data extent and the position of the next field. The reader walks the
+buffer sequentially; it cannot jump to field N without reading fields
+0..N-1 first.
+
+### Mode 2: Aligned static (mmap-friendly formats)
+
+Fields have fixed positions with natural alignment padding.
+Variable-length fields get a 4-byte length prefix at a known offset; the
+variable data is not included in the static layout. Used by metatensor
+and safetensors.
+
+**Component:**
+
+- **`OffsetMap`** — constructed via `OffsetMap::compute(schema) -> Result<Self, TypedefError>` (requires `TypeDef:Struct` at the top level). Walks the schema once, computes fixed byte positions for each field based on type sizes and alignment. The output is a flat table of `(field_path, byte_range)` pairs (see [Public Types](#public-types)). Used for both read and write at known offsets.
+
+**How it works:**
+
+For a struct with fields `[u8, u32, f32]` and natural alignment:
+
+```
+OffsetMap:
+  field[0] u8:  offset 0, size 1
+  field[1] u32: offset 4, size 4  (3 bytes padding after u8)
+  field[2] f32: offset 8, size 4
+  total: 12 (struct aligned to 4)
+```
+
+The `u32` is aligned to offset 4 (its natural alignment). The consumer
+can read `field[1]` at offset 4 without reading `field[0]` first — random
+access by field path.
+
+**Variable-length fields in aligned mode:**
+
+Variable-length fields get a 4-byte length prefix at a known offset. The
+variable data lives outside the static layout — either immediately after
+the fixed fields (inline length-prefixing) or in a separate data region
+(offset indirection). The `OffsetMap` records the position of the length
+prefix (or the `{offset, length}` pair for offset-indirect fields).
+
+For inline length-prefixing, the variable data follows the fixed fields
+but is not included in the `OffsetMap`'s field ranges. The consumer reads
+the length prefix from the `OffsetMap`'s known offset, then slices the
+data region.
+
+For offset indirection, the field is a struct `{offset: u32, length: u32}`
+at a known position in the `OffsetMap`. The consumer reads the offset and
+length, then slices the separate data region.
+
+### Inline length-prefixing in aligned mode — non-final field restriction
+
+Inline length-prefixed variable fields in aligned mode are only allowed
+as the **last field** in their struct. A non-final inline
+length-prefixed variable field is rejected at `OffsetMap::compute` time
+with a `TypedefError::Offset` — the `OffsetMap` reserves only 4 bytes
+(the length prefix), but `data_access::write_string` writes prefix +
+data inline, which would clobber subsequent fields. Non-final variable
+fields must use `maxLength` (fixed-size reservation) or
+`"encoding": "offset-indirect"`. See
+[ADR-100](decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md).
+
+## Offset Computation Algorithm
+
+The offset computation is a recursive walk of the schema JSON. The
+algorithm is the same for both modes; the difference is whether alignment
+padding is inserted between fields.
+
+### Fixed-size types
+
+For each fixed-size type, the algorithm:
+1. Determines the type's byte size from the `TypeDef:*` kind.
+2. In aligned mode: inserts padding to satisfy the type's alignment
+   (or the field's `align` annotation, or the struct's `align` default).
+3. Records the field's `(start, end)` range.
+4. Advances the current offset by the type's size.
+
+### Composite types
+
+**`TStruct`:** Recurse into the struct's `properties`. The inner fields
+are computed relative to the struct's start offset. The struct's total
+size is the sum of its fields' sizes (plus alignment padding in aligned
+mode). The struct itself may have an `align` annotation that rounds up
+its total size.
+
+**`TUnion`:** TUnion is supported in packed sequential mode only. In
+aligned static mode, `OffsetMap::compute` rejects `TUnion` fields with
+`TypedefError::Offset` — see
+[ADR-102](decisions/102-reject-tunion-in-aligned-mode.md). Unions
+are the protocol dispatch pattern (SFTP type bytes, call protocol event
+types); mmap-friendly formats use structs and arrays, not tagged unions.
+
+In packed sequential mode, the discriminator occupies
+`offset..offset + discriminator_size` bytes. For byte-offset
+discriminators, the variant struct starts at `offset + discriminator_size`.
+For field-name discriminators, the discriminator is just another field —
+its offset is computed like any other field, and the variant struct
+follows at the end of the discriminator field.
+
+Variant sizes depend on the actual sizes of variable-length fields within
+each variant, which aren't known at schema time. The `LayoutBuilder`
+takes the actual variant discriminator value and data sizes at write time,
+computes the size of the selected variant, and uses that for the union's
+total size. The `SequentialReader` reads the discriminator first, looks
+up the variant schema, then reads the variant struct sequentially — it
+doesn't need to know the union's total size upfront.
+
+**`TArray` of fixed-size elements:** Element stride = element size (plus
+alignment padding in aligned mode). Element `i` starts at
+`array_offset + i × stride`. The array's total size is `count × stride`.
+
+**`TArray` of variable-length-element structs:** Deferred for v1
+(OQ-069).
+
+### Variable-length types
+
+The typedef engine supports three strategies for variable-length types
+(see [schema-layer.md](schema-layer.md) §Variable-length types and
+[ADR-097](decisions/097-schema-annotations.md) §3 for the full
+annotation shapes).
+
+**Strategy 1: Inline length-prefixing (default).**
+1. Records the position of the 4-byte length prefix.
+2. In aligned mode: the length prefix is aligned; the variable data is
+   not included in the static layout.
+3. In packed mode: the `LayoutBuilder` takes the actual data size to
+   compute the length prefix value and the position of subsequent fields.
+   The `SequentialReader` reads the length prefix to determine the data
+   extent and the position of the next field.
+
+**Strategy 2: Fixed-size reservation (`maxLength`).**
+1. In aligned static mode: reserves `maxLength` bytes at a fixed offset.
+   Data shorter than `maxLength` is zero-padded. Subsequent fields have
+   known, unchanging offsets — the field is fixed-size from the layout
+   perspective. This is the database `VARCHAR(N)` pattern.
+2. In packed sequential mode: `maxLength` is a validation constraint
+   only. The engine uses strategy 1 (inline length-prefixing) because
+   protocols don't benefit from fixed-size reservation.
+
+**Strategy 3: Offset indirection (`"encoding": "offset-indirect"`).**
+1. The field is a struct `{offset: u32, length: u32}`.
+2. The `OffsetMap` records the position of this struct.
+3. The consumer provides the data region separately. This is the
+   metatensor blob tensor pattern — the index struct lives in one region,
+   the blob data lives in another.
+
+### Nested structs and field paths
+
+Nested structs produce dotted field paths: `header.version`,
+`header.magic`. The offset computation propagates the field path prefix
+during recursion. Both `OffsetMap` and `PackedLayout` store fully-qualified
+paths; the `iter()` method of each yields fields in schema `properties`
+order, with nested struct fields appearing inline under their parent's
+path prefix.
+
+### Endianness
+
+Endianness is per-schema (ADR-097). The offset computation is
+endian-agnostic — it computes byte positions, not byte values. The
+read/write functions apply endianness when converting between bytes and
+typed values. The engine reads the `"endian"` annotation from the schema
+and byte-swaps accordingly. All fixed-size types — including `TEnum`
+(u32 index) — follow the schema's endianness.
+
+## Mode Selection
+
+The consumer selects the mode at engine construction time via the
+`LayoutMode` enum, passed to `TypedefEngine::compile`:
+
+```rust
+pub enum LayoutMode {
+    /// Packed sequential — for protocol wire formats (SFTP, channels, TTY).
+    Packed,
+    /// Aligned static — for mmap-friendly formats (metatensor, safetensors).
+    Aligned,
+}
+```
+
+The choice is determined by the use case, not by the schema:
+
+- **Protocol consumer** (SFTP, binary call frames, TTY negotiation):
+  `LayoutMode::Packed` → uses `LayoutBuilder` for writing and
+  `SequentialReader` for reading.
+- **mmap consumer** (metatensor): `LayoutMode::Aligned` → uses `OffsetMap`
+  for both reading and writing at known offsets.
+
+The same schema can be used in either mode. A schema describing an SFTP
+packet can be consumed by a `SequentialReader` (for parsing incoming
+frames) and a `LayoutBuilder` (for constructing outgoing frames). A schema
+describing a metatensor layout can be consumed by an `OffsetMap` (for
+mmap access).
+
+`TypedefEngine` exposes mode-appropriate accessors: `engine.offset_map()`
+returns `Some(&OffsetMap)` in aligned mode and `None` in packed mode;
+`engine.layout_builder()` returns `Some(&LayoutBuilder)` in packed mode
+and `None` in aligned mode. `engine.sequential_reader()` returns
+`Option<SequentialReader>` (an owned fresh reader, not a reference — the
+reader has mutable cursor state that the consumer owns; see
+[ADR-101](decisions/101-packed-mode-read-factory.md)) in packed
+mode and `None` in aligned mode. See [validation.md](validation.md)
+§"The TypedefEngine struct" for the engine API.
+
+## Public Types
+
+The layout engine produces three public types, one per layout component.
+All are re-exported from the crate root.
+
+### `ByteRange` (aligned mode)
+
+```rust
+pub struct ByteRange {
+    pub start: usize,  // inclusive
+    pub end: usize,    // exclusive
+}
+```
+
+A half-open byte range produced by `OffsetMap::compute` for each field.
+`end - start` is the field's byte size in the static layout (for
+variable-length fields: the length prefix, the `{offset, length}` pair,
+or the `maxLength` reservation — not the variable data). `ByteRange`
+provides `len()` and `is_empty()`.
+
+### `FieldPosition` (packed mode)
+
+```rust
+pub struct FieldPosition {
+    pub offset: usize,
+    pub size: usize,
+    pub kind: TypeDefKind,
+}
+```
+
+A field's computed position in a packed layout, produced by
+`LayoutBuilder::build`. For variable-length fields, `size` is `4` (the
+length prefix); for fixed-size fields, `size` is the type's byte size.
+`kind` records the field's `TypeDef:*` kind so the consumer can dispatch
+to the correct `data_access` read/write function.
+
+### `PackedLayout` (packed mode)
+
+The result of `LayoutBuilder::build`: a map of `field_path → FieldPosition`
+plus the total buffer size needed.
+
+```rust
+impl PackedLayout {
+    pub fn get(&self, field_path: &str) -> Option<&FieldPosition>;
+    pub fn total_size(&self) -> usize;
+    pub fn iter(&self) -> impl Iterator<Item = &(String, FieldPosition)>;
+}
+```
+
+`get` looks up a field by dotted path. For TUnion byte-offset
+discriminators, the discriminator is recorded under the synthetic path
+`"<union_path>.__discriminator"`. `iter` yields fields in layout order
+(schema `properties` order, with nested struct fields appearing inline
+under their parent's path prefix).
+
+### `OffsetMap` (aligned mode)
+
+A flat table of `(field_path, byte_range)` pairs computed from a schema.
+
+```rust
+impl OffsetMap {
+    pub fn compute(schema: &Value) -> Result<Self, TypedefError>;
+    pub fn get(&self, field_path: &str) -> Option<&ByteRange>;
+    pub fn total_size(&self) -> usize;
+    pub fn iter(&self) -> impl Iterator<Item = &(String, ByteRange)>;
+}
+```
+
+`compute` requires a `TypeDef:Struct` at the top level. `total_size`
+includes trailing alignment padding. `iter` yields fields in insertion
+order (schema `properties` order, nested struct fields appearing inline).
+
+## Design Decisions
+
+| Decision | ADR | Summary |
+|----------|-----|---------|
+| Two layout modes | [ADR-096](decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential for protocols; aligned static for mmap formats |
+| Schema annotations | [ADR-097](decisions/097-schema-annotations.md) | Endianness, alignment, encoding annotations that control layout behavior |
+| Non-final inline variable fields | [ADR-100](decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Rejected in aligned mode (would clobber subsequent fields); use `maxLength` or `offset-indirect` |
+| Packed-mode read factory | [ADR-101](decisions/101-packed-mode-read-factory.md) | `engine.sequential_reader()` returns an owned fresh reader, not a reference |
+| TUnion in aligned mode | [ADR-102](decisions/102-reject-tunion-in-aligned-mode.md) | Rejected for v1 (broken semantics; no current consumer needs it) |
+
+## Open Questions
+
+See [open-questions.md](open-questions.md) for full details.
+
+- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs
+  — requires lazy walking logic; blocked on a concrete consumer that
+  needs it.
+
+## References
+
+- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1
+  (aligned OffsetMap) and POC 2 (packed LayoutBuilder/SequentialReader)
+- [ADR-096](decisions/096-two-layout-modes-packed-vs-aligned.md) —
+  the two layout modes decision
+- [ADR-097](decisions/097-schema-annotations.md) — schema
+  annotations
+- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds and their
+  byte sizes
+- [data-access.md](data-access.md) — read/write functions that use the
+  computed offsets