diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 10fad7e..bd8c884 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -296,6 +296,12 @@ adapter location map is now consistent: all HTTP-backed adapters | [crates/channels/channels-adapter.md](crates/channels/channels-adapter.md) | draft | `ChannelsAdapter`, `ChannelManager`, demux/mux contracts (REQ-CH-01..04), two-pump pattern (ADR-078) | | [crates/channels/channel-operations.md](crates/channels/channel-operations.md) | draft | `channel/open`/`close`/`control`/`resources/subscribe`, ACL flow, `direction` semantics, hub relay contract (ADR-079) | | [crates/channels/channel-client.md](crates/channels/channel-client.md) | draft | `ChannelClient` — client side of a channels connection, transport-agnostic `from_connection` primary; dial lives in `AlknetClient` (ADR-089); bidirectionality preserved | +| [crates/typedef/README.md](crates/typedef/README.md) | draft | alknet-typedef crate — binary struct engine; JSON Schema with `TypeDef:*` custom keywords → offset map + read/write + validation | +| [crates/typedef/overview.md](crates/typedef/overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries | +| [crates/typedef/schema-layer.md](crates/typedef/schema-layer.md) | draft | The 16 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations | +| [crates/typedef/layout-engine.md](crates/typedef/layout-engine.md) | draft | Offset computation, two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling | +| [crates/typedef/data-access.md](crates/typedef/data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading | +| [crates/typedef/validation.md](crates/typedef/validation.md) | draft | Custom keyword validators for all 16 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation | ## ADR Table @@ -395,10 +401,14 @@ adapter location map is now consistent: all HTTP-backed adapters | [092](decisions/092-bistream-as-the-handler-leaf.md) | `BiStream` as the Handler Leaf — Unify the Split-Pair `accept_bi` | Accepted (amends ADR-070's `accept_bi` return type; amends ADR-065's `from_stream`/`from_bidi` constructors; amends ADR-074's `ChannelBidiStreamSource::accept_bi` return type; `Connection::from_stream` removed; `from_bidi` is the only public stream constructor) | | [093](decisions/093-channels-pure-channel-multiplexing.md) | alknet-channels — Pure Channel Multiplexing (8-Byte Header, No `stream_type`) | Accepted (amends ADR-071 — 8-byte header; ADR-074 — `into_sub_streams` removed; reverses ADR-077 — TTY always uses its 5-byte format; amends the channels-facing clauses of ADR-072/073/075/076/080/081) | | [094](decisions/094-per-identity-channel-cap.md) | Per-Identity Channel Cap as DoS Defense | Accepted (amends ADR-076 — per-connection `max_channels` reframed as a memory bound; 256 per `PeerId` enforced via `ChannelLifecyclePolicy` in `channels-call`; symmetric; spoke caps hub as direct caller) | +| [095](decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | alknet-typedef — Purpose, Scope, and the jsonschema Engine | Accepted | +| [096](decisions/096-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | Accepted | +| [097](decisions/097-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, and TUnion Discriminators | Accepted | +| [098](decisions/098-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | Accepted | ## Open Questions -Open questions are tracked in [open-questions.md](open-questions.md) — an index of theme-grouped tables (68 OQs across 20 themes) with a cross-theme [Deferred / Blocked](open-questions.md#deferred--blocked) section surfacing the safe-exit deferrals. Each OQ lives in its own file under [`questions/`](questions/) (`NNN-slug.md`, mirroring the ADR convention). +Open questions are tracked in [open-questions.md](open-questions.md) — an index of theme-grouped tables (71 OQs across 21 themes) with a cross-theme [Deferred / Blocked](open-questions.md#deferred--blocked) section surfacing the safe-exit deferrals. Each OQ lives in its own file under [`questions/`](questions/) (`NNN-slug.md`, mirroring the ADR convention). ## Document Lifecycle diff --git a/docs/architecture/crates/typedef/README.md b/docs/architecture/crates/typedef/README.md new file mode 100644 index 0000000..65859a3 --- /dev/null +++ b/docs/architecture/crates/typedef/README.md @@ -0,0 +1,106 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef + +The binary struct engine: a small Rust crate that takes a JSON Schema +with `TypeDef:*` custom keywords and produces an offset map, read/write +functions, and validation — all driven by the schema. The schema is the +format definition; the engine is generic. + +## Documents + +| Document | Status | Description | +|----------|--------|-------------| +| [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries | +| [schema-layer.md](schema-layer.md) | draft | The 17 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations | +| [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling | +| [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading | +| [validation.md](validation.md) | draft | Custom keyword validators for all 16 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation, `TypedefEngine` | + +## Applicable ADRs + +| ADR | Title | Relevance | +|-----|-------|-----------| +| [095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries | +| [096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` | +| [097](../../decisions/097-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations | +| [098](../../decisions/098-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors | + +## Relevant Open Questions + +| OQ | Title | Status | Relevance | +|----|-------|--------|-----------| +| OQ-069 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it | +| OQ-070 | `no_std` + `alloc` support | deferred(scope) | Target `std` for v1; blocked on an embedded use case | +| OQ-071 | Builder API for schema construction | deferred(scope) | Schemas are authored in TypeBox or hand-written JSON for v1; blocked on a concrete need | + +## Key Design Principles + +1. **The schema is the format.** A JSON Schema with `TypeDef:*` custom + keywords is both the validation spec and the layout spec. No separate + format definition, no separate parser, no separate validator. One + schema, three uses: validate, compute offsets, access data. See + [overview.md](overview.md) and [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md). + +2. **jsonschema is the validation engine, not a custom engine.** The + `jsonschema` crate (v0.46.5, Draft 2020-12) handles validation with + custom keyword support. The novel code is the offset computation, not + the validation. This eliminates ~14,000 lines of hand-rolled schema + engines (typebox-rs, alktype). See [schema-layer.md](schema-layer.md) + and [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md). + +3. **Two layout modes for two use cases.** Packed sequential + (`LayoutBuilder`/`SequentialReader`) for protocol wire formats (SFTP, + channels, TTY). Aligned static (`OffsetMap`) for mmap-friendly formats + (metatensor). The consumer selects the mode; the schema is the same. + See [layout-engine.md](layout-engine.md) and + [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md). + +4. **Variable-length types default to inline length-prefixing.** + `[length: u32][data]` is the universal pattern used by channels, SFTP, + TTY, and most binary protocols. Offset indirection (the metatensor + blob tensor pattern) is opt-in via the `encoding` annotation. See + [layout-engine.md](layout-engine.md) and + [ADR-097](../../decisions/097-schema-annotations.md). + +5. **TUnion supports both byte-offset and field-name discriminators.** + Byte-offset for protocol dispatch (SFTP type bytes, call protocol + event types). Field-name for the typedef.ts string pattern. See + [data-access.md](data-access.md) and + [ADR-097](../../decisions/097-schema-annotations.md). + +6. **Endianness is per-schema, default little-endian.** The engine reads + the `"endian"` annotation and byte-swaps accordingly. SFTP consumers + specify `"endian": "big"`. See [layout-engine.md](layout-engine.md) + and [ADR-097](../../decisions/097-schema-annotations.md). + +7. **Validation is opt-in, built once at load time.** The jsonschema + validator is compiled once at schema load time. Access-time validation + is a fast `is_valid()` check. High-throughput paths can skip + validation; security-sensitive paths can validate every frame. See + [validation.md](validation.md) and + [ADR-098](../../decisions/098-error-handling-validation-strategy.md). + +8. **Not a serialization framework.** The typedef engine is not a + general-purpose serde replacement. It operates on raw byte buffers at + computed offsets — no intermediate `Value` tree, no reflection, no + dynamic dispatch per field. For JSON data, use serde. For binary data + with a known schema, use typedef. See [overview.md](overview.md) and + [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md). + +## References + +- `docs/research/alknet-typedef/findings.md` — POC results (26 tests + passing, two layout modes, TUnion dispatch, endianness) +- `docs/research/call-channels-unification/findings.md` §"alknet-typedef: + JSON Schema as the binary struct engine" — the origin of this research + thread +- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox + schema kinds (619 lines) +- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12) +- `/workspace/alknet-typedef-poc/` — the POC code (disposable) +- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef +- `/workspace/@alkimiadev/alktype/` — prior attempt, replaced by typedef diff --git a/docs/architecture/crates/typedef/data-access.md b/docs/architecture/crates/typedef/data-access.md new file mode 100644 index 0000000..3ba6bc7 --- /dev/null +++ b/docs/architecture/crates/typedef/data-access.md @@ -0,0 +1,224 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef — Data Access + +The data access layer: read/write functions, TUnion dispatch, field paths, +zero-copy access for fixed-size types, and length-prefix reading for +variable-length types. This is the consumer-facing API — given a compiled +`TypedefEngine` and a byte buffer, read and write fields at +schema-computed offsets. + +## Read/Write Model + +The typedef engine operates on raw byte buffers (`&[u8]` for reading, +`&mut [u8]` for writing). There is no intermediate `Value` tree, no +reflection, no dynamic dispatch per field. The engine uses the offset map +(or `LayoutBuilder`/`SequentialReader`) to locate fields, then performs +typed access at the computed positions. + +### Fixed-size types + +Fixed-size types (`TFloat32`, `TInt32`, `TUint8`, etc.) are accessed via +zero-copy pointer casts: + +```rust +// Read a u32 at a known offset +fn read_u32(buffer: &[u8], offset: usize, endian: Endian) -> u32 { + let bytes: [u8; 4] = buffer[offset..offset+4].try_into().unwrap(); + match endian { + Endian::Little => u32::from_le_bytes(bytes), + Endian::Big => u32::from_be_bytes(bytes), + } +} + +// Write a u32 at a known offset +fn write_u32(buffer: &mut [u8], offset: usize, value: u32, endian: Endian) { + let bytes = match endian { + Endian::Little => value.to_le_bytes(), + Endian::Big => value.to_be_bytes(), + }; + buffer[offset..offset+4].copy_from_slice(&bytes); +} +``` + +The engine applies endianness at access time based on the schema's +`"endian"` annotation (ADR-097). The offset computation is +endian-agnostic. + +### Variable-length types (inline length-prefixing) + +For variable-length types with inline length-prefixing (the default): + +```rust +// Read a length-prefixed string +fn read_string<'a>(buffer: &'a [u8], offset: usize) -> &'a str { + let len = u32::from_le_bytes(buffer[offset..offset+4].try_into().unwrap()) as usize; + std::str::from_utf8(&buffer[offset+4..offset+4+len]).unwrap() +} + +// Write a length-prefixed string +fn write_string(buffer: &mut [u8], offset: usize, value: &str) { + let data = value.as_bytes(); + buffer[offset..offset+4].copy_from_slice(&(data.len() as u32).to_le_bytes()); + buffer[offset+4..offset+4+data.len()].copy_from_slice(data); +} +``` + +The engine reads the 4-byte length prefix at the field's offset, then +slices the data that follows. For writing, the engine writes the length +prefix + data. + +In packed sequential mode, the `SequentialReader` uses the length prefix +to determine the position of the next field. In aligned static mode, the +`OffsetMap` records the position of the length prefix; the variable data +is accessed separately. + +### Variable-length types (offset indirection) + +For variable-length types with offset indirection (opt-in): + +```rust +// Read an offset-indirect string +fn read_string_indirect(data_region: &[u8], offset: usize) -> &str { + let ptr_offset = u32::from_le_bytes(data_region[offset..offset+4].try_into().unwrap()) as usize; + let ptr_length = u32::from_le_bytes(data_region[offset+4..offset+8].try_into().unwrap()) as usize; + std::str::from_utf8(&data_region[ptr_offset..ptr_offset+ptr_length]).unwrap() +} +``` + +The field is a struct `{offset: u32, length: u32}` at a known position +in the `OffsetMap`. The consumer provides the data region separately; the +engine reads the offset and length, then slices the data region. + +## TUnion Dispatch + +TUnion dispatch reads the discriminator value, looks up the variant +schema, and then reads the variant's fields. The dispatch mechanism +differs by discriminator kind (ADR-097). + +### Byte-offset discriminator + +```rust +fn read_union(buffer: &[u8], schema: &Value) -> Result { + let disc = &schema["discriminator"]; + let offset = disc["offset"].as_u64().unwrap() as usize; + let disc_type = disc["type"].as_str().unwrap(); // e.g., "TypeDef:Uint8" + + // Read the discriminator value + let disc_value: u8 = read_u8(buffer, offset); + let key = disc_value.to_string(); // "5", "6", "101" + + // Look up the variant schema + let mapping = &schema["mapping"]; + let variant_schema = &mapping[&key]; + + // Read the variant struct starting at offset + discriminator_size + let variant_offset = offset + 1; // discriminator_size for Uint8 + read_struct(buffer, variant_offset, variant_schema) +} +``` + +The discriminator is a fixed-size integer at a known byte offset. The +mapping keys are stringified integers. The variant struct starts at +`offset + discriminator_size`. + +This is the SFTP `Packet` enum pattern — byte 0 is the type byte, bytes +1..N are the variant struct. The call protocol's 5 event types +(`call.requested` → 0x01, etc.) use the same pattern. + +### Field-name discriminator + +```rust +fn read_union_field(buffer: &[u8], schema: &Value) -> Result { + let disc = &schema["discriminator"]; + let field_name = disc["name"].as_str().unwrap(); // e.g., "type" + + // Read the discriminator field like any other field + let disc_value = read_field(buffer, field_name, schema)?; + + // Look up the variant schema + let mapping = &schema["mapping"]; + let variant_schema = &mapping[disc_value.as_str().unwrap()]; + + // Read the variant struct + read_struct(buffer, variant_offset, variant_schema) +} +``` + +The discriminator is a named field within the struct. Its offset is +computed like any other field. The mapping keys are string values. + +## Field Paths + +Fields are addressed by dotted paths: `"header.version"`, `"payload.data"`. +The `OffsetMap` stores fully-qualified paths. The read/write functions +accept a field path and look up the byte range: + +```rust +fn read_f32(&self, buffer: &[u8], field_path: &str) -> Result { + let range = self.offset_map.get(field_path) + .ok_or_else(|| TypedefError::Offset { + field_path: field_path.to_string(), + reason: "field not found in offset map".to_string(), + })?; + if buffer.len() < range.end { + return Err(TypedefError::Access { + field_path: field_path.to_string(), + reason: format!("buffer too short: need {} bytes, have {}", range.end, buffer.len()), + }); + } + Ok(read_f32_raw(buffer, range.start, self.endian)) +} +``` + +Nested structs produce nested field paths. The offset computation +propagates the field path prefix during recursion, so the `OffsetMap` +contains entries like `"header.version"` and `"header.magic"`. + +## Zero-Copy Access + +For fixed-size types, the engine provides zero-copy access — the consumer +gets a reference to the bytes in the buffer, not a copy. This is +important for performance-sensitive paths (metatensor tensor access, +high-throughput protocol parsing). + +For variable-length types with inline length-prefixing, the engine +returns a slice of the buffer — the string or byte array data is not +copied. The consumer gets a `&str` or `&[u8]` that borrows from the +input buffer. + +For offset-indirect types, the consumer provides the data region; the +engine returns a slice of that region. + +## Error Handling + +Read/write errors carry the field path for debugging. See +[ADR-098](../../decisions/098-error-handling-validation-strategy.md) and +[validation.md](validation.md) for the full error model. + +## Design Decisions + +| Decision | ADR | Summary | +|----------|-----|---------| +| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Determines whether offsets are fixed (OffsetMap) or sequential (SequentialReader) | +| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness, encoding, and TUnion discriminator shapes that control data access | +| Error handling | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | Field-path-carrying errors for read/write operations | + +## Open Questions + +See [open-questions.md](../../open-questions.md) for full details. + +- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs + — affects the sequential walking logic for array access. + +## References + +- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1 + (read/write round-trip) and POC 2 (SFTP byte-identical round-trip) +- [layout-engine.md](layout-engine.md) — offset computation that produces + the positions this layer reads/writes at +- [validation.md](validation.md) — validation that runs on the same + buffers diff --git a/docs/architecture/crates/typedef/layout-engine.md b/docs/architecture/crates/typedef/layout-engine.md new file mode 100644 index 0000000..7d87091 --- /dev/null +++ b/docs/architecture/crates/typedef/layout-engine.md @@ -0,0 +1,254 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef — Layout Engine + +The layout engine: offset computation, the two layout modes (packed +sequential vs aligned static), alignment, endianness, and variable-length +field handling. This is the novel code — the recursive walk of the schema +JSON that computes byte positions for each field. + +## The Two Layout Modes + +The POCs surfaced that protocols and mmap-friendly formats need different +layout strategies. This is the most important architectural finding — +decided in [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md). + +### Mode 1: Packed sequential (protocol wire formats) + +Fields are packed with no alignment padding. Variable-length fields shift +all subsequent fields. Used by SFTP, channels, TTY, and most binary +protocols. + +**Components:** + +- **`LayoutBuilder`** — takes a schema and actual data sizes for + variable-length fields, computes byte positions for each field in a + packed layout. Used at write time when the consumer knows the data + sizes upfront. +- **`SequentialReader`** — walks a buffer field-by-field according to the + schema, reading length prefixes to determine variable-length data + positions. Used at read time when the consumer is parsing an incoming + frame. + +**How it works:** + +For a struct with fields `[u8, u32, string]`: + +``` +LayoutBuilder (write): + field[0] u8: offset 0, size 1 + field[1] u32: offset 1, size 4 + field[2] string: offset 5, size 4 (length prefix) + data_len + total: 9 + data_len + +SequentialReader (read): + read u8 at offset 0 + read u32 at offset 1 + read u32 length prefix at offset 5 → data_len + read string data at offset 9, length data_len + next field at offset 9 + data_len +``` + +There is no alignment padding. The `u32` at offset 1 is unaligned — this +is correct for protocol wire formats, which pack fields tightly. + +**Variable-length fields in packed mode:** + +The `LayoutBuilder` takes actual data sizes for variable-length fields +to compute correct positions for subsequent fields. The consumer must +know the data sizes before writing — this is inherent to packed layouts. + +The `SequentialReader` reads each field's length prefix to determine the +data extent and the position of the next field. The reader walks the +buffer sequentially; it cannot jump to field N without reading fields +0..N-1 first. + +### Mode 2: Aligned static (mmap-friendly formats) + +Fields have fixed positions with natural alignment padding. +Variable-length fields get a 4-byte length prefix at a known offset; the +variable data is not included in the static layout. Used by metatensor +and safetensors. + +**Component:** + +- **`OffsetMap`** — walks the schema once, computes fixed byte positions + for each field based on type sizes and alignment. The output is a flat + table of `(field_path, byte_range)` pairs. Used for both read and write + at known offsets. + +**How it works:** + +For a struct with fields `[u8, u32, f32]` and natural alignment: + +``` +OffsetMap: + field[0] u8: offset 0, size 1 + field[1] u32: offset 4, size 4 (3 bytes padding after u8) + field[2] f32: offset 8, size 4 + total: 12 (struct aligned to 4) +``` + +The `u32` is aligned to offset 4 (its natural alignment). The consumer +can read `field[1]` at offset 4 without reading `field[0]` first — random +access by field path. + +**Variable-length fields in aligned mode:** + +Variable-length fields get a 4-byte length prefix at a known offset. The +variable data lives outside the static layout — either immediately after +the fixed fields (inline length-prefixing) or in a separate data region +(offset indirection). The `OffsetMap` records the position of the length +prefix (or the `{offset, length}` pair for offset-indirect fields). + +For inline length-prefixing, the variable data follows the fixed fields +but is not included in the `OffsetMap`'s field ranges. The consumer reads +the length prefix from the `OffsetMap`'s known offset, then slices the +data region. + +For offset indirection, the field is a struct `{offset: u32, length: u32}` +at a known position in the `OffsetMap`. The consumer reads the offset and +length, then slices the separate data region. + +## Offset Computation Algorithm + +The offset computation is a recursive walk of the schema JSON. The +algorithm is the same for both modes; the difference is whether alignment +padding is inserted between fields. + +### Fixed-size types + +For each fixed-size type, the algorithm: +1. Determines the type's byte size from the `TypeDef:*` kind. +2. In aligned mode: inserts padding to satisfy the type's alignment + (or the field's `align` annotation, or the struct's `align` default). +3. Records the field's `(start, end)` range. +4. Advances the current offset by the type's size. + +### Composite types + +**`TStruct`:** Recurse into the struct's `properties`. The inner fields +are computed relative to the struct's start offset. The struct's total +size is the sum of its fields' sizes (plus alignment padding in aligned +mode). The struct itself may have an `align` annotation that rounds up +its total size. + +**`TUnion`:** The discriminator occupies `offset..offset + discriminator_size` +bytes. For byte-offset discriminators, the variant struct starts at +`offset + discriminator_size`. For field-name discriminators, the +discriminator is just another field — its offset is computed like any +other field, and the variant struct follows at the end of the +discriminator field. + +In aligned static mode, the union's total size is `discriminator_size + +max(variant_sizes)`, where variant sizes are computed from the schema +(variable-length data lives outside the static layout). + +In packed sequential mode, variant sizes depend on the actual sizes of +variable-length fields within each variant, which aren't known at schema +time. The `LayoutBuilder` takes the actual variant discriminator value +and data sizes at write time, computes the size of the selected variant, +and uses that for the union's total size. The `SequentialReader` reads +the discriminator first, looks up the variant schema, then reads the +variant struct sequentially — it doesn't need to know the union's total +size upfront. + +**`TArray` of fixed-size elements:** Element stride = element size (plus +alignment padding in aligned mode). Element `i` starts at +`array_offset + i × stride`. The array's total size is `count × stride`. + +**`TArray` of variable-length-element structs:** Deferred for v1 +(OQ-069). + +### Variable-length types + +The typedef engine supports three strategies for variable-length types +(see [schema-layer.md](schema-layer.md) §Variable-length types and +[ADR-097](../../decisions/097-schema-annotations.md) §3 for the full +annotation shapes). + +**Strategy 1: Inline length-prefixing (default).** +1. Records the position of the 4-byte length prefix. +2. In aligned mode: the length prefix is aligned; the variable data is + not included in the static layout. +3. In packed mode: the `LayoutBuilder` takes the actual data size to + compute the length prefix value and the position of subsequent fields. + The `SequentialReader` reads the length prefix to determine the data + extent and the position of the next field. + +**Strategy 2: Fixed-size reservation (`maxLength`).** +1. In aligned static mode: reserves `maxLength` bytes at a fixed offset. + Data shorter than `maxLength` is zero-padded. Subsequent fields have + known, unchanging offsets — the field is fixed-size from the layout + perspective. This is the database `VARCHAR(N)` pattern. +2. In packed sequential mode: `maxLength` is a validation constraint + only. The engine uses strategy 1 (inline length-prefixing) because + protocols don't benefit from fixed-size reservation. + +**Strategy 3: Offset indirection (`"encoding": "offset-indirect"`).** +1. The field is a struct `{offset: u32, length: u32}`. +2. The `OffsetMap` records the position of this struct. +3. The consumer provides the data region separately. This is the + metatensor blob tensor pattern — the index struct lives in one region, + the blob data lives in another. + +### Nested structs and field paths + +Nested structs produce dotted field paths: `header.version`, +`header.magic`. The offset computation propagates the field path prefix +during recursion. The `OffsetMap` stores fully-qualified paths. + +### Endianness + +Endianness is per-schema (ADR-097). The offset computation is +endian-agnostic — it computes byte positions, not byte values. The +read/write functions apply endianness when converting between bytes and +typed values. The engine reads the `"endian"` annotation from the schema +and byte-swaps accordingly. + +## Mode Selection + +The consumer selects the mode at engine construction time. The choice is +determined by the use case, not by the schema: + +- **Protocol consumer** (SFTP, binary call frames, TTY negotiation): + uses `LayoutBuilder` for writing and `SequentialReader` for reading. +- **mmap consumer** (metatensor): uses `OffsetMap` for both reading and + writing. + +The same schema can be used in either mode. A schema describing an SFTP +packet can be consumed by a `SequentialReader` (for parsing incoming +frames) and a `LayoutBuilder` (for constructing outgoing frames). A schema +describing a metatensor layout can be consumed by an `OffsetMap` (for +mmap access). + +## Design Decisions + +| Decision | ADR | Summary | +|----------|-----|---------| +| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential for protocols; aligned static for mmap formats | +| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness, alignment, encoding annotations that control layout behavior | + +## Open Questions + +See [open-questions.md](../../open-questions.md) for full details. + +- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs + — requires lazy walking logic; blocked on a concrete consumer that + needs it. + +## References + +- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1 + (aligned OffsetMap) and POC 2 (packed LayoutBuilder/SequentialReader) +- [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) — + the two layout modes decision +- [ADR-097](../../decisions/097-schema-annotations.md) — schema + annotations +- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds and their + byte sizes +- [data-access.md](data-access.md) — read/write functions that use the + computed offsets diff --git a/docs/architecture/crates/typedef/overview.md b/docs/architecture/crates/typedef/overview.md new file mode 100644 index 0000000..bf8f3a1 --- /dev/null +++ b/docs/architecture/crates/typedef/overview.md @@ -0,0 +1,198 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef — Overview + +The binary struct engine: a small Rust crate that takes a JSON Schema +with `TypeDef:*` custom keywords and produces an offset map, read/write +functions, and validation — all driven by the schema. The schema is the +format definition; the engine is generic. + +This document covers the crate's purpose, the "schema is the format" +principle, its dependency edges, consumers, and scope boundaries. +Component details are in the sibling documents. + +## What + +`alknet-typedef` is a library crate that consumes JSON Schemas annotated +with `TypeDef:*` custom keywords (the same kinds defined in TypeBox's +`typedef.ts`) and produces three capabilities: + +1. **An offset map** — walks the schema, computes byte offsets for each + field based on type sizes, field order, and alignment. +2. **Read/write functions** — given a `&[u8]` buffer and a field path, + read the field's bytes at its offset (zero-copy for fixed-size types). + Given a `&mut [u8]` buffer, write a value at its offset. +3. **Validation** — via `jsonschema` custom keywords, validates that a + buffer's bytes match the schema's type constraints. + +The heavy lifting is done by the `jsonschema` crate (validation) and +`serde_json` (schema parsing). The novel code is the offset computation +— a recursive walk of the schema JSON that computes byte positions for +each field. The custom keyword implementations are ~10 lines each. + +The crate is ~1,900 lines (POC verified, 26 tests passing). It replaces +two prior attempts that built their own jsonschema engines — typebox-rs +(~8,400 lines) and alktype (~5,600 lines) — with `jsonschema` + an +offset map + ~50 lines of custom keyword implementations. See +[ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md). + +## Why + +The crate's purpose is to be the binary struct engine for every alknet +component that reads or writes binary data at computed offsets. Instead +of per-protocol serde structs (russh-sftp's 29 packet types), per-handler +wire format code (TTY's 5-byte format parser), or per-format offset +computation (metatensor's tensor access), all of these become instances +of the same engine with different schemas. + +The guiding insight: + +> **The schema is the format.** A JSON Schema with `TypeDef:Float32`, +> `TypeDef:Struct`, `TypeDef:Union` etc. is both the validation spec and +> the layout spec. No separate format definition, no separate parser, no +> separate validator. One schema, three uses: validate, compute offsets, +> access data. + +This is the convergence of three threads identified in the +call-channels-unification research: the `typedef.ts` schema kinds from +TypeBox, the russh-sftp protocol packets, and the metatensor format. The +common pattern: a JSON Schema describes the shape of binary data, and +the binary data is the struct's bytes at computed offsets. + +The crate was bumped up in the timeline when the call-channels-unification +research surfaced that channels, TTY, and the binary call protocol are +all variations on the same wire-format family — `[discriminant][length][payload]`. +The typedef engine makes the "channels is call with a binary data plane" +unification concrete: the binary data plane's wire format is the call +protocol's own schema system, just binary-encoded. The `channel_open` +marker says "use binary framing"; the typedef engine says "here's how to +read/write the binary payload." + +## The "Schema Is the Format" Principle + +A JSON Schema with `TypeDef:*` custom keywords serves three roles +simultaneously: + +| Role | Mechanism | When | +|------|-----------|------| +| **Validation spec** | `jsonschema` custom keywords | Load time (build validator), access time (validate buffer) | +| **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map) | +| **Data access** | Read/write at computed offsets | Access time (read field, write field) | + +No separate format definition, no separate parser, no separate validator. +The schema is the single source of truth for the binary format. Adding a +new field to a protocol is adding a property to the schema JSON — the +engine computes the new offsets automatically. + +This is the same principle as `#[repr(C)]` struct field access, but at +runtime from a portable JSON Schema instead of at compile-time from +language-specific annotations. The schema is the ABI contract. + +## Dependencies + +``` +alknet-typedef +├── jsonschema (v0.46.5, Draft 2020-12) — validation engine, custom keyword support +├── serde_json (with preserve_order) — schema parsing; field order is load-bearing +└── (no tokio, no platform deps) — WASM-clean by construction +``` + +`alknet-typedef` is dependency-light: `jsonschema` + `serde_json` only. +No tokio, no platform deps. Compiles to `wasm32-unknown-unknown` for +browser use. The `jsonschema` crate is already in the workspace at +`/workspace/jsonschema/` but not yet used by any alknet crate — typedef +is the first consumer. + +`serde_json` requires the `preserve_order` feature because field order +is load-bearing for binary layouts. The order of properties in the +schema JSON determines the order of fields in the binary struct. + +## Consumers + +| Consumer | Schema describes | Engine provides | +|----------|-----------------|-----------------| +| russh-sftp | 29 packet structs + Packet union (byte discriminator) | Read/write SFTP frames from bytes | +| metatensor | Model layout (ConvNet struct, tensor refs) | Offset map for mmap'd tensor access | +| binary call frames | `call.requested` / `call.responded` / etc. structs | Read/write binary call frames | +| TTY negotiation | `NegotiateRequest` / `NegotiateResponse` structs | Read/write TTY control frames | +| channels wire | `ChunkHeader { channel_id, length }` | Already trivial (8 bytes, no schema needed) | + +The russh-sftp case is the most instructive and the highest-value POC +target. The `Packet` enum's `TryFrom<&mut Bytes>` impl is a hand-written +dispatch on a type byte followed by serde deserialization. Under typedef, +the dispatch is `TUnion` with a byte-offset discriminator — the schema +says "byte 0 is the discriminator, bytes 1..N are the variant struct." +The engine reads the discriminator, looks up the variant schema, computes +offsets, reads fields. Same result, no per-packet-type code. + +## Scope Boundaries (What This Is Not) + +These boundaries are decided in [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md). + +- **Not metatensor.** typedef is the binary struct *engine*. Metatensor + is a *format* (8-byte header + JSON header + binary data) that uses the + typedef engine for its offset computation and tensor access. +- **Not a Value system.** TypeBox's `Value.Diff`, `Value.Migrate`, + `Value.Convert` — schema evolution — is out of scope for v1. The engine + should not do anything that explicitly blocks adding a Value system + later. +- **Not a code generator.** typebox-rs's `codegen/` module is a separate + concern. The typedef engine consumes schemas; it does not generate them. +- **Not a schema builder.** The typedef engine does not provide a fluent + API for constructing schemas. Schemas are plain JSON — authored in + TypeBox, generated by ujsx components, or hand-written. A builder API + is deferred (OQ-071). +- **Not a serialization framework.** The typedef engine is not a + general-purpose serde replacement. It operates on raw byte buffers at + computed offsets — no intermediate `Value` tree, no reflection, no + dynamic dispatch per field. For JSON data, use serde. For binary data + with a known schema, use typedef. + +## Architecture (component pointers) + +- **[schema-layer.md](schema-layer.md)** — the 17 `TypeDef:*` kinds, + jsonschema custom keyword integration, TypeBox interop, schema + annotations (endianness, alignment, encoding, TUnion discriminators). +- **[layout-engine.md](layout-engine.md)** — offset computation, the two + layout modes (packed sequential vs aligned static), alignment, + endianness, variable-length field handling. +- **[data-access.md](data-access.md)** — read/write functions, TUnion + dispatch, field paths, zero-copy access for fixed-size types, + length-prefix reading for variable-length types. +- **[validation.md](validation.md)** — custom keyword validators for all + 16 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time + validation, `TypedefEngine` as the compiled form of a schema. + +## Design Decisions + +| Decision | ADR | Summary | +|----------|-----|---------| +| Purpose, scope, and the jsonschema engine | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries | +| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential (`LayoutBuilder`/`SequentialReader`) for protocols; aligned static (`OffsetMap`) for mmap formats | +| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name) | +| Error handling and validation | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping | + +## Open Questions + +See [open-questions.md](../../open-questions.md) for full details. + +- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs. +- **OQ-070** (deferred(scope)): `no_std` + `alloc` support. +- **OQ-071** (deferred(scope)): Builder API for schema construction. + +## References + +- `docs/research/alknet-typedef/findings.md` — POC results (26 tests + passing, two layout modes, TUnion dispatch, endianness) +- `docs/research/call-channels-unification/findings.md` §"alknet-typedef: + JSON Schema as the binary struct engine" — the origin of this research + thread +- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox + schema kinds (619 lines) +- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12) +- `/workspace/alknet-typedef-poc/` — the POC code (disposable) +- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef +- `/workspace/@alkimiadev/alktype/` — prior attempt, replaced by typedef diff --git a/docs/architecture/crates/typedef/schema-layer.md b/docs/architecture/crates/typedef/schema-layer.md new file mode 100644 index 0000000..76d2819 --- /dev/null +++ b/docs/architecture/crates/typedef/schema-layer.md @@ -0,0 +1,348 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef — Schema Layer + +The schema layer: the 16 `TypeDef:*` custom type kinds, their mapping to +Rust types and byte sizes, the `jsonschema` custom keyword integration, +TypeBox interop, and the concrete JSON shapes for schema-level annotations. + +## The 17 TypeDef Kinds + +These are the custom schema kinds defined in TypeBox's `typedef.ts` +(`/workspace/@alkdev/typebox/example/typedef/typedef.ts`, 619 lines) and +ported to Rust via `jsonschema` custom keywords. Each kind carries binary +layout semantics — a known byte size (for fixed-size types) or a known +encoding strategy (for variable-length types). + +| Kind | TypeBox key | Rust type | Size | Category | +|------|-------------|-----------|------|----------| +| `TFloat32` | `TypeDef:Float32` | `f32` | 4 | fixed | +| `TFloat64` | `TypeDef:Float64` | `f64` | 8 | fixed | +| `TInt8` | `TypeDef:Int8` | `i8` | 1 | fixed | +| `TInt16` | `TypeDef:Int16` | `i16` | 2 | fixed | +| `TInt32` | `TypeDef:Int32` | `i32` | 4 | fixed | +| `TUint8` | `TypeDef:Uint8` | `u8` | 1 | fixed | +| `TUint16` | `TypeDef:Uint16` | `u16` | 2 | fixed | +| `TUint32` | `TypeDef:Uint32` | `u32` | 4 | fixed | +| `TBoolean` | `TypeDef:Boolean` | `bool` (0x00=false, 0x01=true) | 1 | fixed | +| `TString` | `TypeDef:String` | length-prefixed UTF-8 | variable | variable | +| `TBytes` | `TypeDef:Bytes` | length-prefixed raw bytes | variable | variable | +| `TStruct` | `TypeDef:Struct` | record of fields | sum of field sizes | composite | +| `TUnion` | `TypeDef:Union` | tagged union | discriminator + variant | composite | +| `TArray` | `TypeDef:Array` | repeated element | count × element size | composite | +| `TEnum` | `TypeDef:Enum` | u32 index into enum values | 4 (fixed) | fixed | +| `TRecord` | `TypeDef:Record` | count-prefixed sequence of (key, value) pairs | variable | variable | +| `TTimestamp` | `TypeDef:Timestamp` | length-prefixed RFC 3339 string | variable | variable | + +### Fixed-size types + +`TFloat32`, `TFloat64`, `TInt8`, `TInt16`, `TInt32`, `TUint8`, `TUint16`, +`TUint32`, `TBoolean`, and `TEnum` have known byte sizes. The offset +computation uses these sizes directly. Read/write is zero-copy pointer +cast for these types. + +**`TBoolean` byte representation:** `0x00` = false, `0x01` = true. Other +values are invalid and produce a `TypedefError::Access` on read. + +**`TEnum` binary representation:** A `u32` index into the enum's declared +values, in declaration order. The first declared value is index 0, the +second is index 1, etc. The enum's values are declared via the standard +JSON Schema `"enum"` keyword (e.g., `"enum": ["read", "write", "execute"]`). +The `TypeDef:Enum` custom keyword signals that the type is an enum for +layout purposes; the built-in `enum` keyword provides the value list. +The `u32` index is always little-endian (enum indices are not protocol +data — they are internal to the schema). See [ADR-097](../../decisions/097-schema-annotations.md). + +### Variable-length types + +`TString`, `TBytes`, `TRecord`, and `TTimestamp` have variable byte sizes. +The typedef engine supports three strategies for handling variable-length +types in binary layouts, selected by the `encoding` annotation and the +standard JSON Schema `maxLength` keyword: + +| Strategy | Encoding annotation | Layout behavior | Use case | +|----------|-------------------|-----------------|----------| +| **Inline length-prefixed** | `"length-prefixed"` (default) | `[length: u32][data]`; shifts subsequent fields in packed mode | Protocol wire formats (SFTP, channels, TTY) | +| **Fixed-size reservation** | (none — uses `maxLength`) | `[data: maxLength bytes]`, zero-padded; fixed offset in aligned mode | mmap-friendly formats where max size is known (database `VARCHAR(N)` pattern) | +| **Offset indirection** | `"offset-indirect"` | `{offset: u32, length: u32}` pointing into a separate data region | Blob tensors, metatensor variable-length data (the blob tensor pattern) | + +**Strategy 1: Inline length-prefixing (default).** The field's fixed +portion is a 4-byte length prefix at a computed offset. The variable data +follows immediately after. In packed sequential mode, the length prefix +determines the position of subsequent fields. In aligned static mode, the +length prefix is at a known offset; the variable data is not included in +the static layout. This is the universal pattern used by channels, SFTP, +TTY, and most binary protocols. + +**Strategy 2: Fixed-size reservation.** When a variable-length field +declares `maxLength` (a standard JSON Schema keyword), the engine reserves +`maxLength` bytes at a fixed offset in aligned static mode. Data shorter +than `maxLength` is zero-padded; data longer than `maxLength` is a +validation error. This makes the field fixed-size from the layout +perspective — subsequent fields have known, unchanging offsets. This is +the database `VARCHAR(N)` pattern and the metatensor struct-tensor +pattern for fields with known maximum sizes. + +In packed sequential mode, `maxLength` is a validation constraint only — +the engine still uses inline length-prefixing (strategy 1) because +protocols don't benefit from fixed-size reservation. + +**Strategy 3: Offset indirection.** The field is a struct +`{offset: u32, length: u32}` at a known position. The consumer provides +the data region separately; the engine reads the offset and length, then +slices the data region. This is the metatensor blob tensor pattern — the +index struct lives in one region, the blob data lives in another. Enables +mmap-friendly random access to variable-length data without parsing +length prefixes and without reserving worst-case space. + +**Default strategy selection:** +- In packed sequential mode: always strategy 1 (inline length-prefixing). + `maxLength` is a validation constraint only. +- In aligned static mode: strategy 2 (fixed-size reservation) if + `maxLength` is declared; strategy 3 (offset indirection) if + `"encoding": "offset-indirect"` is declared; strategy 1 (inline + length-prefixing) otherwise. + +**Length prefix endianness:** The 4-byte length prefix (strategies 1 and 3) +respects the schema's `"endian"` annotation (ADR-097). In little-endian +mode, the length is `u32::from_le_bytes`. In big-endian mode, the length +is `u32::from_be_bytes`. This ensures SFTP consumers (big-endian) have +consistent byte order for both field values and length prefixes. + +**`TBytes`:** Raw bytes — no UTF-8 constraint. The payload is `&[u8]`. +Otherwise identical to `TString` in layout (same three strategies). + +**`TRecord`:** A string-keyed map. Binary layout is a count-prefixed +sequence of `(key, value)` pairs: `[count: u32][key_len: u32][key_bytes] +[value_len: u32][value_bytes]...`. The count is the number of entries. +Each key is a length-prefixed UTF-8 string. Each value is the record's +declared value type (specified via the `"values"` property in the schema, +e.g., `"values": { "TypeDef:Float32": true }`). The count prefix respects +the schema's endianness. In aligned static mode with `maxLength`, the +entire record is reserved at `maxLength` bytes (zero-padded). + +**`TTimestamp`:** An RFC 3339 timestamp string (the internet profile of +ISO 8601). Stored as a length-prefixed UTF-8 string (strategy 1) or +fixed-size reservation (strategy 2 with `maxLength`). The engine does not +parse or validate the timestamp format beyond UTF-8 — the jsonschema +validator checks RFC 3339 conformance at the JSON level. + +`TArray` is variable-length when the element type is variable-length or +when the count is not known at schema time. For fixed-size element arrays +with a known count, the size is `element_size × count`. + +**`TArray` count declaration:** The array count is declared via the +standard JSON Schema `"minItems"` and `"maxItems"` keywords. When +`minItems == maxItems`, the array has a fixed count known at schema time. +When they differ or are absent, the count is variable and the array uses +a length-prefixed encoding: `[count: u32][element_0]...[element_N]`. +The count prefix respects the schema's endianness. + +### Composite types + +`TStruct` and `TUnion` are composite — their size is the sum of their +fields' sizes (plus alignment padding in aligned static mode). The offset +computation recurses into their properties. + +## jsonschema Custom Keyword Integration + +The `jsonschema` crate (v0.46.5, Draft 2020-12) supports custom keywords +via the `with_keyword` API. Each `TypeDef:*` kind is registered as a +custom keyword: + +```rust +let validator = jsonschema::options() + .with_keyword("TypeDef:Float32", factory) + .with_keyword("TypeDef:Int32", factory) + .with_keyword("TypeDef:Struct", factory) + // ... all 17 kinds + .build(&schema)?; +``` + +The factory closure receives the parent schema object, the keyword's +value, and the schema path — enabling cross-keyword awareness. The +`TypeDef:Struct` validator, for example, inspects the parent's +`properties` to validate each field against its declared `TypeDef:*` kind. + +Each custom keyword implementation is ~10 lines. The `jsonschema` crate +handles all structural validation (object properties, required fields, +array items, enum values) — the custom keywords only need to validate +the leaf type constraints. See [validation.md](validation.md) for the +validator implementations. + +This is the same pattern as TypeBox's `TypeRegistry.Set` on the JS side. +Same semantics, different language, same JSON Schema wire format. A +TypeBox schema serialized to JSON feeds directly into +`jsonschema::validator_for(&schema)` on the Rust side — zero translation. + +## TypeBox Interop + +TypeBox modules render to standard JSON Schema under `$defs`. A TypeBox +schema like: + +```typescript +const TensorRef = Type.Object({ + dtype: Type.Union([Type.Literal("F32"), Type.Literal("I16")]), + shape: Type.Array(Type.Number()), + data_offsets: Type.Tuple([Type.Number(), Type.Number()]) +}); +``` + +serialized to JSON is a standard JSON Schema with `type: "object"`, +`properties`, and `required`. That JSON feeds directly into the typedef +engine. The `TypeDef:*` custom keywords are added by TypeBox's +`TypeRegistry.Set` — they appear in the serialized JSON as additional +properties on the schema object. + +The typedef engine does not depend on TypeBox or any JS toolchain. It +consumes JSON — whether that JSON was authored in TypeBox, generated by +a ujsx component, or hand-written. The schema is the interface. + +## Schema Annotations + +Schema-level annotations control binary layout behavior. These are +decided in [ADR-097](../../decisions/097-schema-annotations.md). + +### Endianness + +Schema-level annotation with a default of little-endian: + +```json +{ "TypeDef:Struct": true, "endian": "big", "properties": { ... } } +``` + +- `"endian": "little"` (default) — read/write in little-endian byte order. +- `"endian": "big"` — read/write in big-endian byte order. +- Applies to the entire schema and all nested types. + +### Alignment + +Both struct-level and field-level, with field-level overriding: + +```json +{ + "TypeDef:Struct": true, + "align": 256, + "properties": { + "weight": { "TypeDef:Float32": true, "align": 16 } + } +} +``` + +- Struct-level `"align"` sets the default for all fields. +- Field-level `"align"` overrides the struct default. +- Default alignment: 1 for u8/bool, 2 for u16/i16, 4 for u32/i32/f32, + 8 for u64/i64/f64, max field alignment for structs. +- Only meaningful in aligned static mode (ADR-096). Ignored in packed + sequential mode. + +### Variable-length encoding + +The typedef engine supports three strategies for variable-length types +(see §Variable-length types above for full details). The strategy is +selected by the `encoding` annotation and the standard JSON Schema +`maxLength` keyword: + +```json +// Strategy 1: Inline length-prefixing (default, shorthand) +{ "TypeDef:String": true } + +// Strategy 1: Explicit inline length-prefixing +{ "TypeDef:String": { "encoding": "length-prefixed" } } + +// Strategy 2: Fixed-size reservation (uses standard maxLength) +{ "TypeDef:String": true, "maxLength": 256 } + +// Strategy 3: Offset indirection (opt-in) +{ "TypeDef:String": { "encoding": "offset-indirect" } } +``` + +- `"encoding": "length-prefixed"` (default) — 4-byte length prefix at + computed offset, variable data follows immediately. Used by protocol + wire formats. +- `maxLength` (standard JSON Schema keyword) — in aligned static mode, + reserves `maxLength` bytes at a fixed offset (zero-padded). Makes the + field fixed-size from the layout perspective. In packed sequential + mode, `maxLength` is a validation constraint only. +- `"encoding": "offset-indirect"` — field is a struct + `{offset: u32, length: u32}` pointing into a separate data region. + The consumer provides the data region separately. Used by metatensor + blob tensors. +- Applies to all variable-length types: `TypeDef:String`, `TypeDef:Bytes`, + `TypeDef:Array`, `TypeDef:Record`, `TypeDef:Timestamp`. + +### TUnion discriminators + +Two discriminator kinds: byte-offset (protocol dispatch) and field-name +(typedef.ts pattern). + +**Byte-offset discriminator** (SFTP type bytes, call protocol event types): + +```json +{ + "TypeDef:Union": true, + "discriminator": { + "kind": "byte", + "offset": 0, + "type": "TypeDef:Uint8" + }, + "mapping": { + "5": { "$ref": "#/$defs/Read" }, + "6": { "$ref": "#/$defs/Write" }, + "101": { "$ref": "#/$defs/Status" } + } +} +``` + +- `"offset"` — byte position of the discriminator. +- `"type"` — the `TypeDef:*` kind of the discriminator (typically + `TypeDef:Uint8`). +- Mapping keys are stringified integers. The variant struct starts at + `offset + discriminator_size`. + +**Field-name discriminator** (typedef.ts pattern): + +```json +{ + "TypeDef:Union": true, + "discriminator": { + "kind": "field", + "name": "type" + }, + "mapping": { + "read": { "$ref": "#/$defs/Read" }, + "write": { "$ref": "#/$defs/Write" } + } +} +``` + +- `"name"` — the field name holding the discriminator value. +- Mapping keys are string values matching the discriminator field's value. +- The discriminator field is just another field in the struct. + +Mapping values may be either inline schemas or `$ref` pointers. Both work. + +## Design Decisions + +| Decision | ADR | Summary | +|----------|-----|---------| +| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Concrete JSON shapes for endianness, alignment, encoding, and TUnion discriminators | +| Purpose and scope | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine; "schema is the format" principle | + +## Open Questions + +See [open-questions.md](../../open-questions.md) for full details. + +- **OQ-071** (deferred(scope)): Builder API for schema construction. + +## References + +- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox + schema kinds (619 lines) +- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12) +- [ADR-097](../../decisions/097-schema-annotations.md) — schema + annotation shapes +- [validation.md](validation.md) — custom keyword validator implementations diff --git a/docs/architecture/crates/typedef/validation.md b/docs/architecture/crates/typedef/validation.md new file mode 100644 index 0000000..65ac600 --- /dev/null +++ b/docs/architecture/crates/typedef/validation.md @@ -0,0 +1,292 @@ +--- +status: draft +last_updated: 2026-07-20 +--- + +# alknet-typedef — Validation + +The validation layer: custom keyword validators for all 17 `TypeDef:*` +kinds, the `TypedefError` enum, load-time vs access-time validation +strategy, and the `TypedefEngine` as the compiled form of a schema. + +## Validation Strategy + +Validation is delegated to the `jsonschema` crate (v0.46.5, Draft +2020-12). The typedef engine does not implement its own validation — +it registers custom keyword validators for each `TypeDef:*` kind and +lets `jsonschema` handle the structural validation (object properties, +required fields, array items, enum values). + +The strategy is decided in [ADR-098](../../decisions/098-error-handling-validation-strategy.md): + +1. **Load time:** Parse the schema JSON, build the layout engine, build the + jsonschema validator. This is the `TypedefEngine::compile(schema)` constructor. +2. **Access time:** Use the compiled engine for repeated read/write + operations. Validation is opt-in per operation. + +### What validation validates + +The jsonschema validator operates on `serde_json::Value` instances — it +validates JSON representations of data, not raw byte buffers. This is +the correct separation of concerns: + +- **JSON validation** (jsonschema): validates that a JSON document + conforms to the schema. Used for validating hand-written schemas, + TypeBox output, JSON payloads, or the JSON representation of a binary + struct after deserialization. +- **Binary access validation** (data access layer): the read/write + functions perform type-level validation at access time — range checks + for integers, UTF-8 validity for strings, buffer bounds checking. + These return `TypedefError::Access` with field paths. + +The "schema is the format" principle means the same schema describes +both the JSON shape and the binary layout. The jsonschema validator +checks the JSON shape; the data access layer checks the binary layout. +A consumer that wants to validate a binary buffer end-to-end reads the +buffer into a `Value` tree via the data access layer, then validates +that `Value` against the jsonschema validator. This is a two-step +process, not a single `validate(buffer)` call. + +### The `TypedefEngine` struct + +The `TypedefEngine` is the compiled form of a schema. It supports both +layout modes (ADR-096) via an internal enum: + +```rust +pub struct TypedefEngine { + layout: Layout, // packed or aligned (see below) + validator: jsonschema::Validator, // compiled once at load time +} + +enum Layout { + Packed { + builder: LayoutBuilder, + reader: SequentialReader, + }, + Aligned { + offset_map: OffsetMap, + }, +} +``` + +The consumer selects the mode at construction time. The `Layout` enum +ensures the engine always has the correct layout strategy for the +consumer's use case — a protocol consumer gets `Packed`, an mmap +consumer gets `Aligned`. The validator is mode-agnostic (it operates on +`Value`, not raw bytes). + +## Custom Keyword Validators + +Each `TypeDef:*` kind gets a `Keyword` implementation registered via +`jsonschema::options().with_keyword(...)`. The validators check leaf +type constraints; `jsonschema` handles all structural validation. + +### Numeric type validators + +**`TypeDef:Float32` / `TypeDef:Float64`:** +- Value must be a finite number. +- For `Float32`: value must be representable as `f32` (no precision loss + beyond `f32`'s mantissa). + +**`TypeDef:Int8` / `TypeDef:Int16` / `TypeDef:Int32`:** +- Value must be an integer within the type's range. +- Int8: -128..127, Int16: -32768..32767, Int32: -2147483648..2147483647. + +**`TypeDef:Uint8` / `TypeDef:Uint16` / `TypeDef:Uint32`:** +- Value must be a non-negative integer within the type's range. +- Uint8: 0..255, Uint16: 0..65535, Uint32: 0..4294967295. + +### String and binary validators + +**`TypeDef:String`:** +- Value must be a valid UTF-8 string. +- If `maxLength` is specified in the schema, the string's byte length + must not exceed it. + +**`TypeDef:Bytes`:** +- Value must be a string (JSON represents binary data as a string). +- If `maxLength` is specified, the byte length must not exceed it. + +**`TypeDef:Enum`:** +- The `TypeDef:Enum` custom keyword signals that the type is an enum for + *layout* purposes (the engine needs to know it's a fixed-size u32 index, + not a variable-length string). The built-in `enum` keyword provides the + value list and handles value-membership validation. The custom keyword + validator is a no-op beyond the built-in check — it exists solely for + the layout engine to recognize the type. + +**`TypeDef:Timestamp`:** +- Value must be a valid RFC 3339 timestamp string (the internet profile + of ISO 8601, e.g., `"2026-07-20T15:30:00Z"`). + +### Composite type validators + +**`TypeDef:Struct`:** +- Value must be an object. +- Each property must match its declared `TypeDef:*` kind. +- Required fields must be present. +- The `jsonschema` crate's built-in `properties` and `required` keywords + handle the structural checks — the custom keyword only needs to + validate that each field's value matches its `TypeDef:*` kind. + +**`TypeDef:Union`:** +- The discriminator value must be one of the mapping keys. +- The variant struct must match the declared schema for that discriminator + value. + +**`TypeDef:Array`:** +- Value must be an array. +- Each element must match the array's declared element type. +- If `minItems`/`maxItems` is specified, the array length must be within + bounds. + +### Other validators + +**`TypeDef:Boolean`:** +- Value must be `true` or `false`. + +**`TypeDef:Record`:** +- Value must be an object. +- All values must match the record's declared value type (specified via + the `"values"` property in the schema, e.g., + `"values": { "TypeDef:Float32": true }`). + +### Validator implementation pattern + +Each custom keyword implementation is ~10 lines. Example for +`TypeDef:Float32`: + +```rust +struct Float32Validator; + +impl Keyword for Float32Validator { + fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> { + match instance { + Value::Number(n) if n.as_f64().map_or(false, |f| f.is_finite()) => Ok(()), + _ => Err(ValidationError::custom("expected finite f32-compatible number")), + } + } + fn is_valid(&self, instance: &Value) -> bool { + instance.as_f64().map_or(false, |f| f.is_finite()) + } +} +``` + +Registration: + +```rust +let validator = jsonschema::options() + .with_keyword("TypeDef:Float32", |parent, value, path| { + Ok(Box::new(Float32Validator)) + }) + .build(&schema)?; +``` + +The factory closure receives the parent schema object, the keyword's +value, and the schema path. This enables cross-keyword awareness — for +example, a `TypeDef:Struct` validator can inspect the parent's +`properties` to validate each field against its declared `TypeDef:*` kind. + +## TypedefError + +A single `TypedefError` enum covers all error conditions across the +engine's three phases (schema parsing, offset computation, read/write) +plus validation. Decided in [ADR-098](../../decisions/098-error-handling-validation-strategy.md). + +```rust +pub enum TypedefError { + /// Schema parsing errors (invalid JSON, missing keywords, unknown TypeDef kinds). + Schema(String), + /// Offset computation errors (field not found, unsupported type). + Offset { field_path: String, reason: String }, + /// Read/write errors (buffer too short, invalid UTF-8, value out of range). + Access { field_path: String, reason: String }, + /// Validation errors (delegated to jsonschema). + Validation(ValidationError<'static>), +} +``` + +- **`Schema`** — for errors during `TypedefEngine::compile()`. Invalid + JSON, missing required keywords, unknown `TypeDef:*` kinds. +- **`Offset`** — for errors during offset computation. Field not found + in the schema, type not supported for offset computation, recursive + depth exceeded. Carries the field path. +- **`Access`** — for errors during read/write. Buffer too short, invalid + UTF-8 in a string field, value out of range for the target type. + Carries the field path. +- **`Validation`** — wraps `jsonschema`'s `ValidationError`. The + `'static` lifetime is correct — the validator owns its schema reference + and lives for the lifetime of the `TypedefEngine`. + +### Field-path-carrying errors + +Read/write and offset errors include the field path for debugging: + +```rust +Err(TypedefError::Access { + field_path: "header.version".to_string(), + reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(), +}) +``` + +This makes debugging binary format issues tractable — the error tells +you exactly which field failed and why. + +## Validation Timing + +### Load time: `TypedefEngine::compile()` + +The expensive work happens once at schema load time: +1. Parse the schema JSON (`serde_json::from_str` with `preserve_order`). +2. Compute the offset map (or `LayoutBuilder`/`SequentialReader`). +3. Build the jsonschema validator (`jsonschema::options().with_keyword(...).build(&schema)?`). + +The result is a `TypedefEngine` that can be used for repeated operations. + +### Access time: `engine.validate(buffer)` + +Validation is opt-in per operation. The consumer calls +`engine.validate(buffer)` when validation is desired. The jsonschema +validator is already compiled — `is_valid()` is a fast check against +the compiled validator. + +High-throughput paths can skip validation. Security-sensitive paths +(parsing incoming frames from untrusted peers) can validate every frame. +The choice is the consumer's. + +## Relationship to Read/Write + +Validation and data access are independent operations on the same buffer. +The consumer can: + +1. Validate a buffer to ensure it conforms to the schema. +2. Read fields from the buffer at computed offsets. +3. Both — validate first, then read (defense in depth). + +The engine does not couple validation and access. A consumer that trusts +its data source can skip validation and go straight to read/write. A +consumer that parses untrusted input can validate first, then access. + +## Design Decisions + +| Decision | ADR | Summary | +|----------|-----|---------| +| Error handling and validation | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping | +| Purpose and scope | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine | + +## Open Questions + +None specific to validation. The three typedef OQs (OQ-069, OQ-070, +OQ-071) are about layout, platform support, and schema construction — +not validation. + +## References + +- `docs/research/alknet-typedef/findings.md` §"Validation" — the POC's + custom keyword validators for all 17 kinds +- [ADR-098](../../decisions/098-error-handling-validation-strategy.md) — + error handling and validation strategy +- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds that the + validators check +- [data-access.md](data-access.md) — read/write functions that operate + on the same buffers diff --git a/docs/architecture/decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md b/docs/architecture/decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md new file mode 100644 index 0000000..06623e6 --- /dev/null +++ b/docs/architecture/decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md @@ -0,0 +1,168 @@ +# ADR-095: alknet-typedef — Purpose, Scope, and the jsonschema Engine + +## Status +Accepted + +## Context + +Three threads in the codebase converge on the same pattern: a JSON Schema +describes the shape of binary data, and the binary data is the struct's +bytes at computed offsets. + +1. **typedef.ts** (`/workspace/@alkdev/typebox/example/typedef/typedef.ts`, + 619 lines) defines custom TypeBox schema kinds (`TFloat32`, `TStruct`, + `TUnion`, etc.) that carry binary layout semantics. These are registered + via `TypeRegistry.Set` with custom validators. + +2. **russh-sftp** has 29 packet types, each a struct with typed fields + (`Read { id: u32, handle: String, offset: u64, len: u32 }`). The wire + format is `[length: u32][type: u8][payload]` where payload is the + struct's serde bytes. The `Packet` enum dispatches on the type byte — + a tagged union of structs. Under the typedef lens, each packet is a + `TStruct`; the `Packet` enum is a `TUnion` with a byte-offset + discriminator. + +3. **metatensor** needs an offset map for mmap-friendly tensor access — + given a schema describing a model layout (ConvNet struct, tensor refs), + compute byte offsets for each field so the consumer can read tensor + data at known positions without parsing. + +The common pattern: **a JSON Schema with `TypeDef:*` custom keywords +describes the shape of binary data; the binary data is the struct's bytes +at computed offsets.** The schema is the format definition; the engine is +generic. + +Two prior attempts built their own jsonschema engines — the fatal flaw: + +- **typebox-rs** (`/workspace/@alkimiadev/typebox-rs/`, ~8,400 lines): + a full 26-variant `SchemaKind` enum, a custom `Value` type with typed + arrays, and a 912-line hand-written validator. +- **alktype** (`/workspace/@alkimiadev/alktype/`, ~5,600 lines): a + handler-registry pattern that also implements its own validation for + each type. + +The `jsonschema` crate (v0.46.5, Draft 2020-12) is already in the +workspace at `/workspace/jsonschema/`. It handles validation with custom +keyword support — the novel code is the offset computation, not the +validation. + +The call-channels-unification research +(`docs/research/call-channels-unification/findings.md` §"alknet-typedef: +JSON Schema as the binary struct engine") identified the convergence and +bumped typedef up in the timeline. The POC +(`docs/research/alknet-typedef/findings.md`, 26 tests passing) validated +the approach: a ~1,900-line Rust crate that takes a JSON Schema with +`TypeDef:*` custom keywords and produces an offset map, read/write +functions, and validation — all driven by the schema. + +## Decision + +**alknet-typedef is a small Rust crate that takes a JSON Schema with +`TypeDef:*` custom keywords and produces three capabilities:** + +1. **An offset map** — walks the schema, computes byte offsets for each + field based on type sizes, field order, and alignment. +2. **Read/write functions** — given a `&[u8]` buffer and a field path, + read the field's bytes at its offset (zero-copy for fixed-size types). + Given a `&mut [u8]` buffer, write a value at its offset. +3. **Validation** — via `jsonschema` custom keywords, validates that a + buffer's bytes match the schema's type constraints. + +**The heavy lifting is done by the `jsonschema` crate (validation) and +`serde_json` (schema parsing).** The novel code is the offset computation +— a recursive walk of the schema JSON that computes byte positions for +each field. The custom keyword implementations are ~10 lines each. + +**The schema is the format.** A JSON Schema with `TypeDef:Float32`, +`TypeDef:Struct`, `TypeDef:Union` etc. is both the validation spec and +the layout spec. No separate format definition, no separate parser, no +separate validator. One schema, three uses: validate, compute offsets, +access data. + +**The crate depends on `jsonschema` and `serde_json` (with +`preserve_order`).** No tokio, no platform deps. Compiles to +`wasm32-unknown-unknown` for browser use. The `jsonschema` crate's +`with_keyword("TypeDef:Float32", factory)` API is the integration point +for custom type kinds — each `TypeDef:*` kind maps to a custom keyword +validator in Rust. Same semantics as TypeBox's `TypeRegistry.Set`, same +JSON Schema wire format. + +**The crate targets `std` for v1.** The WASM target has `std` available +via `wasm-bindgen`. If embedded use cases emerge, `no_std` + `alloc` can +be added as a feature gate later — the engine's core (offset computation, +read/write) is already allocation-free. See OQ-070. + +## Consequences + +### Positive + +- **Eliminates ~14,000 lines of hand-rolled schema engines.** typebox-rs + and alktype are replaced by `jsonschema` + an offset map + ~50 lines of + custom keyword implementations. The codebase drops from "a port of + TypeBox" to "jsonschema + an offset map." +- **One schema, three uses.** The same JSON Schema validates, computes + offsets, and drives data access. No separate format definition, parser, + or validator per protocol. +- **Schema-driven, not code-driven.** Adding a new SFTP packet type is + adding a variant to the schema JSON, not writing a new Rust struct + + serde impl. The engine is generic; the schema is the configuration. +- **WASM-clean.** `serde_json` + `jsonschema` + byte manipulation. No + tokio, no platform deps. The same typedef schemas work in browser, + Node, Python (via `wasmtime-py`), Go (via `wazero`), and any other + WASM host. +- **TypeBox interop.** TypeBox modules render to standard JSON Schema + under `$defs`. That JSON feeds directly into `jsonschema::validator_for` + on the Rust side. Zero translation. The same schema validates in both + ecosystems. +- **Defense in depth.** Schema validation at the byte level — a malformed + binary payload fails validation before any consumer touches it. The + `jsonschema` crate's compiled validators are fast enough to run on + every incoming frame. + +### Negative + +- **New dependency on `jsonschema`.** The crate is already in the + workspace but not yet used by any alknet crate. This is the first + consumer. +- **`serde_json` with `preserve_order` is required.** Field order is + load-bearing for binary layouts. The `preserve_order` feature adds a + small compile-time cost. +- **Schema authoring is external.** Schemas are authored in TypeBox (JS) + or hand-written JSON. The typedef engine consumes schemas; it does not + generate them. A builder API is deferred (OQ-071). + +## Scope Boundaries (What This Is Not) + +- **Not metatensor.** typedef is the binary struct *engine*. Metatensor + is a *format* (8-byte header + JSON header + binary data) that uses the + typedef engine for its offset computation and tensor access. +- **Not a Value system.** TypeBox's `Value.Diff`, `Value.Migrate`, + `Value.Convert` — schema evolution — is out of scope for v1. The engine + should not do anything that explicitly blocks adding a Value system + later. +- **Not a code generator.** typebox-rs's `codegen/` module is a separate + concern. The typedef engine consumes schemas; it does not generate them. +- **Not a schema builder.** The typedef engine does not provide a fluent + API for constructing schemas. Schemas are plain JSON. +- **Not a serialization framework.** The typedef engine is not a + general-purpose serde replacement. It operates on raw byte buffers at + computed offsets — no intermediate `Value` tree, no reflection, no + dynamic dispatch per field. For JSON data, use serde. For binary data + with a known schema, use typedef. + +## References + +- `docs/research/alknet-typedef/findings.md` — POC results (26 tests + passing, two layout modes, TUnion dispatch, endianness) +- `docs/research/call-channels-unification/findings.md` §"alknet-typedef: + JSON Schema as the binary struct engine" — the origin of this research + thread +- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox + schema kinds (619 lines) +- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12) +- `/workspace/alknet-typedef-poc/` — the POC code (disposable) +- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout + modes decision +- [ADR-097](097-schema-annotations.md) — schema annotation shapes +- [ADR-098](098-error-handling-validation-strategy.md) — error handling + and validation strategy diff --git a/docs/architecture/decisions/096-two-layout-modes-packed-vs-aligned.md b/docs/architecture/decisions/096-two-layout-modes-packed-vs-aligned.md new file mode 100644 index 0000000..7d1c85c --- /dev/null +++ b/docs/architecture/decisions/096-two-layout-modes-packed-vs-aligned.md @@ -0,0 +1,137 @@ +# ADR-096: Two Layout Modes — Packed Sequential vs Aligned Static + +## Status +Accepted + +## Context + +The POCs surfaced that protocols and mmap-friendly formats need different +layout strategies. POC 1 built an aligned `OffsetMap` with natural +alignment padding — correct for mmap-friendly formats (metatensor) but +wrong for protocol wire formats (SFTP, channels, TTY). POC 2 built a +`LayoutBuilder` and `SequentialReader` for packed sequential layouts — +correct for protocol wire formats but wrong for mmap-friendly formats. + +This is the most important architectural finding from the POCs. The +engine must support both modes; a single layout strategy cannot serve +both use cases. + +### Packed sequential layout (protocol wire formats) + +Protocols pack fields sequentially with no alignment padding. +Variable-length fields shift all subsequent fields. Writing requires +knowing actual data sizes upfront; reading walks the buffer sequentially, +reading length prefixes to determine positions. + +This is the layout used by SFTP (all strings and byte arrays are +length-prefixed inline), channels (`[channel_id: u32][size: u32][payload]`), +TTY (`[stream_type: u8][length: u32][payload]`), and most binary protocols. + +### Aligned static layout (mmap-friendly formats) + +Fields have fixed positions with natural alignment padding. +Variable-length fields get a 4-byte length prefix at a known offset; the +variable data is not included in the static layout. This enables +mmap-friendly random access — the consumer can read field N at a known +offset without parsing the fields before it. + +This is the layout used by metatensor (blob tensor pattern: index struct +in one region, blob data in another) and safetensors (header + aligned +tensor data). + +## Decision + +**The typedef engine supports two layout modes, selected by the consumer +at engine construction time:** + +### Mode 1: Packed sequential (`LayoutBuilder` / `SequentialReader`) + +For protocol wire formats. Fields are packed with no alignment padding. +Variable-length fields shift all subsequent fields. + +- **LayoutBuilder** — takes a schema and actual data sizes for + variable-length fields, computes byte positions for each field in a + packed layout. Used at write time when the consumer knows the data + sizes upfront. +- **SequentialReader** — walks a buffer field-by-field according to the + schema, reading length prefixes to determine variable-length data + positions. Used at read time when the consumer is parsing an incoming + frame. + +The `LayoutBuilder` and `SequentialReader` are the primary interface for +protocol consumers (SFTP, binary call frames, TTY negotiation). + +### Mode 2: Aligned static (`OffsetMap`) + +For mmap-friendly formats. Fields have fixed positions with natural +alignment padding. Variable-length fields get a 4-byte length prefix at +a known offset; the variable data is not included in the static layout. + +- **OffsetMap** — walks the schema once, computes fixed byte positions + for each field based on type sizes and alignment. The output is a flat + table of `(field_path, byte_range)` pairs. Used for both read and write + at known offsets. + +The `OffsetMap` is the primary interface for mmap consumers (metatensor). + +### Variable-length handling in each mode + +**Packed sequential mode:** Variable-length fields are inline +length-prefixed by default (`[length: u32][data]`). The `LayoutBuilder` +takes the actual data size to compute the length prefix value and the +position of subsequent fields. The `SequentialReader` reads the length +prefix to determine the data extent and the position of the next field. + +**Aligned static mode:** Variable-length fields get a 4-byte length +prefix at a known offset. The variable data lives outside the static +layout — either immediately after the fixed fields (inline +length-prefixing) or in a separate data region (offset indirection, the +metatensor blob tensor pattern). The `OffsetMap` records the position of +the length prefix (or the `{offset, length}` pair for offset-indirect +fields). + +### Default for variable-length types + +Inline length-prefixing (`[length: u32][data]`) is the default for all +variable-length types in both modes. This is the universal pattern used +by channels, SFTP, TTY, and most binary protocols. Offset indirection is +opt-in via the `encoding` annotation (see ADR-097). + +## Consequences + +### Positive + +- **One engine, two modes.** The same schema can be used in either mode. + A schema describing an SFTP packet can be consumed by a `SequentialReader` + (for parsing incoming frames) and a `LayoutBuilder` (for constructing + outgoing frames). A schema describing a metatensor layout can be + consumed by an `OffsetMap` (for mmap access). +- **Correct for both use cases.** Packed sequential mode produces + byte-identical output to hand-written protocol serialization (validated + by POC 2's russh-sftp round-trip tests). Aligned static mode produces + correct offsets for mmap-friendly access (validated by POC 1's + alignment tests). +- **No mode confusion.** The consumer explicitly selects the mode at + engine construction time. A protocol consumer never accidentally gets + alignment padding; an mmap consumer never accidentally gets + variable-length field shifting. + +### Negative + +- **Two APIs to learn.** Consumers must choose between + `LayoutBuilder`/`SequentialReader` and `OffsetMap`. The choice is + determined by the use case (protocol vs mmap), not by the schema. +- **Variable-length fields in packed mode require size foreknowledge.** + The `LayoutBuilder` needs actual data sizes for variable-length fields + to compute correct positions for subsequent fields. This is inherent + to packed layouts — the consumer must know the data sizes before + writing. + +## References + +- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1 + (aligned OffsetMap) and POC 2 (packed LayoutBuilder/SequentialReader) +- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) — + purpose and scope +- [ADR-097](097-schema-annotations.md) — schema annotations including + the `encoding` field for variable-length types diff --git a/docs/architecture/decisions/097-schema-annotations.md b/docs/architecture/decisions/097-schema-annotations.md new file mode 100644 index 0000000..074d512 --- /dev/null +++ b/docs/architecture/decisions/097-schema-annotations.md @@ -0,0 +1,232 @@ +# ADR-097: Schema Annotations — Endianness, Alignment, Encoding, and TUnion Discriminators + +## Status +Accepted + +## Context + +The typedef engine needs concrete JSON shapes for schema-level +annotations that control binary layout behavior. The POCs validated the +semantics; this ADR pins the shapes. + +Four annotation categories need concrete shapes: + +1. **Endianness** — safetensors is little-endian, SFTP is big-endian. + The engine needs to know which to use. +2. **Alignment** — different backends have different alignment + requirements (wgpu: 256-byte, protocols: natural, mmap: page). +3. **Variable-length encoding** — inline length-prefixing vs offset + indirection for strings, byte arrays, and other variable-length types. +4. **TUnion discriminators** — byte-offset (protocol dispatch) vs + field-name (typedef.ts pattern). + +## Decision + +### 1. Endianness + +**Schema-level annotation with a default of little-endian.** + +```json +{ + "TypeDef:Struct": true, + "endian": "big", + "properties": { ... } +} +``` + +- `"endian": "little"` (default) — read/write in little-endian byte order. +- `"endian": "big"` — read/write in big-endian byte order. +- The annotation applies to the entire schema and all nested types. +- Mixed endianness within one schema is not supported (pathological; no + known protocol requires it). +- The default is little-endian, matching safetensors, wgpu, and most + modern formats. SFTP consumers specify `"endian": "big"`. + +### 2. Alignment + +**Both struct-level and field-level, with field-level overriding +struct-level.** + +```json +{ + "TypeDef:Struct": true, + "align": 256, + "properties": { + "header": { "TypeDef:Struct": true, "properties": { ... } }, + "weight": { "TypeDef:Float32": true, "align": 16 } + } +} +``` + +- Struct-level `"align"` sets the default alignment for all fields in + that struct. The struct's total size is rounded up to this alignment. +- Field-level `"align"` overrides the struct default for that specific + field. +- Default alignment (when no annotation is present): 1 for u8/bool, 2 + for u16/i16, 4 for u32/i32/f32, 8 for u64/i64/f64, max field alignment + for structs. +- Alignment is only meaningful in aligned static mode (ADR-096). In + packed sequential mode, alignment annotations are ignored — fields are + packed with no padding. + +### 3. Variable-length encoding + +**Three strategies for variable-length types, selected by the `encoding` +annotation and the standard JSON Schema `maxLength` keyword.** + +```json +// Strategy 1: Inline length-prefixing (default, shorthand) +{ "TypeDef:String": true } + +// Strategy 1: Explicit inline length-prefixing +{ "TypeDef:String": { "encoding": "length-prefixed" } } + +// Strategy 2: Fixed-size reservation (uses standard maxLength) +{ "TypeDef:String": true, "maxLength": 256 } + +// Strategy 3: Offset indirection (opt-in) +{ "TypeDef:String": { "encoding": "offset-indirect" } } +``` + +**Strategy 1: Inline length-prefixing (default).** The field's fixed +portion is a 4-byte length prefix at a computed offset. The variable data +follows immediately after. In packed sequential mode, the length prefix +determines the position of subsequent fields. In aligned static mode, the +length prefix is at a known offset; the variable data is not included in +the static layout. This is the universal pattern used by channels, SFTP, +TTY, and most binary protocols. + +**Strategy 2: Fixed-size reservation.** When a variable-length field +declares `maxLength` (a standard JSON Schema keyword), the engine reserves +`maxLength` bytes at a fixed offset in aligned static mode. Data shorter +than `maxLength` is zero-padded; data longer than `maxLength` is a +validation error. This makes the field fixed-size from the layout +perspective — subsequent fields have known, unchanging offsets. This is +the database `VARCHAR(N)` pattern and the metatensor struct-tensor +pattern for fields with known maximum sizes. In packed sequential mode, +`maxLength` is a validation constraint only — the engine still uses +inline length-prefixing (strategy 1). + +**Strategy 3: Offset indirection.** The field is a struct +`{offset: u32, length: u32}` that points into a separate data region. +This is the metatensor blob tensor pattern — the index struct lives in +one region, the blob data lives in another. The consumer provides the +data region separately. Enables mmap-friendly random access to +variable-length data without parsing length prefixes and without +reserving worst-case space. + +**Default strategy selection:** +- In packed sequential mode: always strategy 1 (inline length-prefixing). + `maxLength` is a validation constraint only. +- In aligned static mode: strategy 2 (fixed-size reservation) if + `maxLength` is declared; strategy 3 (offset indirection) if + `"encoding": "offset-indirect"` is declared; strategy 1 (inline + length-prefixing) otherwise. + +- `true` is a shorthand for the default (length-prefixed). This keeps + the common case concise and the override explicit. +- The `encoding` annotation and `maxLength` apply to all variable-length + types: `TypeDef:String`, `TypeDef:Bytes`, `TypeDef:Array`, + `TypeDef:Record`, `TypeDef:Timestamp`. + +### 4. TUnion discriminators + +**Two discriminator kinds: byte-offset (protocol dispatch) and +field-name (typedef.ts pattern).** + +#### Kind A: Byte-offset discriminator + +```json +{ + "TypeDef:Union": true, + "discriminator": { + "kind": "byte", + "offset": 0, + "type": "TypeDef:Uint8" + }, + "mapping": { + "1": { "$ref": "#/$defs/Init" }, + "3": { "$ref": "#/$defs/Open" }, + "5": { "$ref": "#/$defs/Read" }, + "6": { "$ref": "#/$defs/Write" }, + "101": { "$ref": "#/$defs/Status" } + } +} +``` + +- The discriminator is a fixed-size integer at a known byte offset. +- `"offset"` is the byte position of the discriminator within the union's + buffer. +- `"type"` is the `TypeDef:*` kind of the discriminator (typically + `TypeDef:Uint8` for protocol type bytes). +- The mapping keys are stringified integers (`"1"`, `"5"`, `"101"`). + The engine parses the key to match the discriminator value. +- The variant struct starts at `offset + discriminator_size`. +- This is the SFTP `Packet` enum pattern and the call protocol's event + type dispatch. + +#### Kind B: Field-name discriminator + +```json +{ + "TypeDef:Union": true, + "discriminator": { + "kind": "field", + "name": "type" + }, + "mapping": { + "read": { "$ref": "#/$defs/Read" }, + "write": { "$ref": "#/$defs/Write" } + } +} +``` + +- The discriminator is a named field within the struct. +- `"name"` is the field name that holds the discriminator value. +- The mapping keys are string values matching the discriminator field's + value. +- The discriminator field is just another field in the struct — its + offset is computed like any other field. +- This is the typedef.ts `TUnion` pattern. + +#### Mapping values + +Mapping values may be either inline schemas or `$ref` pointers. `$ref` +is cleaner for large unions (29 SFTP variants) but requires a `$defs` +section. Inline schemas are simpler for small unions (5 call protocol +event types). Both work. + +## Consequences + +### Positive + +- **Concrete, validated shapes.** All four annotation categories have + concrete JSON shapes that were validated by the POCs. +- **Sensible defaults.** Little-endian, natural alignment, inline + length-prefixing — the common case requires no annotations. +- **Explicit overrides.** Big-endian, custom alignment, offset + indirection — the uncommon case is explicit and self-documenting. +- **TUnion covers both protocol and typedef.ts patterns.** The + byte-offset discriminator handles SFTP type bytes and call protocol + event types. The field-name discriminator handles the typedef.ts string + pattern. No separate union type needed. + +### Negative + +- **Keyword value shape change.** `"TypeDef:String": true` (boolean) and + `"TypeDef:String": { "encoding": "length-prefixed" }` (object) are both + valid. The engine must handle both shapes. This is a minor parsing + concern — the POC already handles it. +- **Alignment annotations are mode-specific.** Alignment is only + meaningful in aligned static mode. In packed sequential mode, alignment + annotations are ignored. This is documented, not enforced — a consumer + that specifies alignment in packed mode gets no error, just no effect. + +## References + +- `docs/research/alknet-typedef/findings.md` §"Open Questions" — the + annotation shape questions this ADR resolves +- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) — + purpose and scope +- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout + modes (alignment only meaningful in aligned static mode) diff --git a/docs/architecture/decisions/098-error-handling-validation-strategy.md b/docs/architecture/decisions/098-error-handling-validation-strategy.md new file mode 100644 index 0000000..d8c3fd8 --- /dev/null +++ b/docs/architecture/decisions/098-error-handling-validation-strategy.md @@ -0,0 +1,157 @@ +# ADR-098: Error Handling and Validation Strategy + +## Status +Accepted + +## Context + +The typedef engine operates in three phases, each with distinct error +conditions: + +1. **Schema parsing** — invalid JSON, missing required keywords, unknown + `TypeDef:*` kinds, malformed annotations. +2. **Offset computation** — field not found, type not supported for + offset computation, recursive schema depth exceeded. +3. **Read/write** — buffer too short, invalid UTF-8, value out of range + for the target type. +4. **Validation** — type constraint violations (range, UTF-8, field + presence, discriminator membership). + +The engine also needs a clear strategy for *when* validation happens: +once at schema load time (build the validator) vs repeatedly at access +time (validate each buffer). + +## Decision + +### Error type: `TypedefError` + +A single `TypedefError` enum with variants for each error category: + +```rust +pub enum TypedefError { + /// Schema parsing errors. + Schema(String), + /// Offset computation errors. + Offset { field_path: String, reason: String }, + /// Read/write errors. + Access { field_path: String, reason: String }, + /// Validation errors (delegated to jsonschema). + Validation(ValidationError<'static>), +} +``` + +- `Schema` — for invalid JSON, missing required keywords, unknown + `TypeDef:*` kinds. The error message describes the problem. +- `Offset` — for field-not-found, unsupported type for offset + computation, etc. Carries the field path for debugging. +- `Access` — for buffer-too-short, invalid UTF-8, value out of range. + Carries the field path for debugging. +- `Validation` — wraps `jsonschema`'s `ValidationError`. The + `jsonschema` crate already provides rich error messages with schema + paths; the typedef engine does not re-wrap or re-interpret them. + +The `Validation` variant uses `ValidationError<'static>` because the +validator is built once at schema load time and lives for the lifetime +of the `TypedefEngine`. The `'static` lifetime is correct — the validator +owns its schema reference. + +### Validation timing: load-time build, access-time check + +The jsonschema validator is built once at schema load time +(`validator_for(&schema)?`) and then called repeatedly +(`validator.is_valid(&instance)`). The typedef engine follows the same +pattern: + +1. **Load time:** Parse the schema JSON, build the offset map (or + `LayoutBuilder`/`SequentialReader`), build the jsonschema validator. + This is the `TypedefEngine::compile(schema: &Value) -> Result` constructor. +2. **Access time:** Use the compiled engine for repeated read/write + operations. Validation is opt-in per operation — the consumer calls + `engine.validate(buffer)` when validation is desired. + +The `TypedefEngine` struct is the compiled form of a schema: + +```rust +pub struct TypedefEngine { + offset_map: OffsetMap, // or LayoutBuilder/SequentialReader + validator: jsonschema::Validator, // compiled once at load time +} +``` + +### Custom keyword validators + +Each `TypeDef:*` kind gets a `Keyword` implementation registered via +`jsonschema::options().with_keyword(...)`. The validators check: + +- **Numeric types** (`TypeDef:Float32`, `TypeDef:Int8`, etc.): range + constraints (Int8: -128..127, Uint8: 0..255, etc.), finiteness for + floats. +- **`TypeDef:String`**: UTF-8 validity. +- **`TypeDef:Struct`**: field presence and types (delegated to + jsonschema's structural validation — the custom keyword only needs to + validate that the struct's fields match their declared `TypeDef:*` + kinds). +- **`TypeDef:Union`**: discriminator value membership in the mapping. +- **`TypeDef:Array`**: element type conformance. +- **`TypeDef:Boolean`**: value is `true` or `false`. +- **`TypeDef:Timestamp`**: ISO 8601 string format. + +The `jsonschema` crate handles all the structural validation (object +properties, required fields, array items, enum values) — the custom +keywords only need to validate the leaf type constraints. Each custom +keyword implementation is ~10 lines. + +### Read/write errors carry field paths + +Read/write errors include the field path for debugging: + +```rust +// Example: reading a u32 from a buffer that's too short +Err(TypedefError::Access { + field_path: "header.version".to_string(), + reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(), +}) +``` + +This makes debugging binary format issues tractable — the error tells +you exactly which field failed and why. + +## Consequences + +### Positive + +- **Single error type.** Consumers handle one `TypedefError` enum, not + multiple error types from different engine phases. +- **Field-path-carrying errors.** Read/write errors include the field + path, making binary format debugging tractable. +- **Validation is opt-in.** The consumer decides when to validate. + High-throughput paths can skip validation; security-sensitive paths + can validate every frame. +- **jsonschema integration is clean.** The `ValidationError` is wrapped + as-is — no re-interpretation, no information loss. +- **Load-time build, access-time use.** The expensive work (schema + parsing, validator compilation, offset computation) happens once at + load time. Access-time operations are cheap (pointer casts, slice + operations, length-prefix reads). + +### Negative + +- **`ValidationError<'static>` lifetime.** The `'static` lifetime on the + `Validation` variant means the error cannot borrow from the buffer + being validated. This is correct (the validator owns its schema + reference) but may surprise readers who expect a shorter lifetime. +- **No error recovery.** The engine does not attempt to recover from + partial reads or writes. A buffer-too-short error on field N means + fields N+1.. are also unreadable. This is inherent to binary formats + — there is no "skip to next field" without a schema-driven parser. + +## References + +- `docs/research/alknet-typedef/findings.md` §"Open Questions" — error + handling strategy question (OQ 8) +- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) — + purpose and scope +- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout + modes +- [ADR-097](097-schema-annotations.md) — schema annotations diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index e053b1c..7f915e3 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -211,6 +211,14 @@ Door type is separate from whether a decision is made. A two-way door is a decis | [OQ-66](questions/066-alknet-register-wire-protocol.md) | `alknet/register` Wire Protocol | deferred(scope) | one | med | | [OQ-67](questions/067-iroh-proxy-support.md) | iroh Proxy Support (Direct-Connection Peer Exposure) | resolved | one | med | +### alknet-typedef + +| OQ | Title | Status | Door | Pri | +|----|-------|--------|------|-----| +| [OQ-069](questions/069-arrays-of-variable-length-element-structs.md) | Arrays of Variable-Length-Element Structs | deferred(scope) | two | low | +| [OQ-070](questions/070-no-std-alloc-support.md) | `no_std` + `alloc` Support | deferred(scope) | two | low | +| [OQ-071](questions/071-builder-api-for-schema-construction.md) | Builder API for Schema Construction | deferred(scope) | two | med | + ## Deferred / Blocked The safe-exit visibility surface. These questions are parked because the @@ -358,3 +366,37 @@ filtering the tables above. [OQ-64](questions/064-client-side-tls-helper.md). - **Full file**: [OQ-64](questions/064-client-side-tls-helper.md) +### OQ-069: Arrays of Variable-Length-Element Structs + +- **Blocked on**: A concrete consumer that needs arrays of structs with + variable-length fields, where the elements are interleaved + (`[fixed_0][str_0][fixed_1][str_1]...`) and the engine must walk + sequentially rather than use a fixed stride. The SFTP `Name` packet + has `Vec` where `File` contains strings, but SFTP serializes + this as a sequence of length-prefixed strings (the serde `SeqAccess` + pattern), not as an array of fixed-stride structs. Arrays of + fixed-size structs are fully supported. +- **Priority**: low +- **Full file**: [OQ-069](questions/069-arrays-of-variable-length-element-structs.md) + +### OQ-070: `no_std` + `alloc` Support + +- **Blocked on**: An embedded use case that requires `no_std` + `alloc` + (e.g., a microcontroller running Rust without `std`). The WASM target + has `std` available via `wasm-bindgen`. The engine's core (offset + computation, read/write) is already allocation-free; the `jsonschema` + dependency is the only `alloc` consumer. +- **Priority**: low +- **Full file**: [OQ-070](questions/070-no-std-alloc-support.md) + +### OQ-071: Builder API for Schema Construction + +- **Blocked on**: A concrete need for programmatic schema construction + in Rust. The current consumers (SFTP, metatensor, binary call frames, + TTY negotiation) all have schemas that can be hand-written or + generated from TypeBox. A builder API would be a fluent Rust API that + produces the same JSON Schema structure — it would sit on top of the + engine, not inside it. +- **Priority**: medium +- **Full file**: [OQ-071](questions/071-builder-api-for-schema-construction.md) + diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 7e93114..a47e52f 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -103,6 +103,7 @@ alknet-vault (standalone — foundational to ACL: key derivation, identity) │ │ ConnectionCredentials + RemoteIdentity per ADR-091) │ ├── alknet-tls TlsServerConfig + TlsClientConfig + FingerprintPinVerifier — shared TLS config across quinn + TCP+TLS + iroh (ADR-082/087; FingerprintPinVerifier per ADR-089 §5) │ ├── alknet-call CallAdapter on alknet/call, CallClient (spawn_dispatch primary; dial in AlknetClient per ADR-089), OperationRegistry, adapters (no TLS/transport deps) +│ ├── alknet-typedef Binary struct engine — JSON Schema with TypeDef:* custom keywords → offset map + read/write + validation (ADR-095–098); depends on jsonschema + serde_json only; WASM-clean │ ├── alknet-channels │ │ ├── alknet-channels-core pure multiplexer (wire format, demux/mux) — ADR-081 │ │ └── alknet-channels-call channel 0 pre-negotiation + lifecycle ops — ADR-081 diff --git a/docs/architecture/questions/069-arrays-of-variable-length-element-structs.md b/docs/architecture/questions/069-arrays-of-variable-length-element-structs.md new file mode 100644 index 0000000..63d0fb5 --- /dev/null +++ b/docs/architecture/questions/069-arrays-of-variable-length-element-structs.md @@ -0,0 +1,20 @@ +# OQ-069: Arrays of variable-length-element structs + +- **Origin**: [crates/typedef/layout-engine.md](crates/typedef/layout-engine.md), + [crates/typedef/data-access.md](crates/typedef/data-access.md); + `docs/research/alknet-typedef/findings.md` §"Problem 3: Nested structs + and arrays of structs" +- **Status**: deferred(scope) +- **Door type**: Two-way (additive — the engine can add lazy walking + logic without changing the existing fixed-stride array support) +- **Priority**: low +- **Impacts**: Blocks any protocol with interleaved variable-length struct arrays (e.g., a protocol where each array element has a string field and elements are packed as `[fixed_0][str_0][fixed_1][str_1]...`). Does NOT block SFTP `Name` packet handling — SFTP serializes this as a sequence of length-prefixed strings (the serde `SeqAccess` pattern), not as an array of fixed-stride structs. Does NOT block any current consumer. +- **Blocked on**: A concrete consumer that needs arrays of structs with + variable-length fields, where the elements are interleaved + (`[fixed_0][str_0][fixed_1][str_1]...`) and the engine must walk + sequentially rather than use a fixed stride. +- **Resolution**: Not yet decidable. The mechanism (lazy sequential + walking of array elements, reading each element's length prefixes to + find the next element's start) is understood but not needed by any + current consumer. Arrays of fixed-size structs are fully supported. +- **Cross-references**: ADR-096, [layout-engine.md](crates/typedef/layout-engine.md) diff --git a/docs/architecture/questions/070-no-std-alloc-support.md b/docs/architecture/questions/070-no-std-alloc-support.md new file mode 100644 index 0000000..1c90f58 --- /dev/null +++ b/docs/architecture/questions/070-no-std-alloc-support.md @@ -0,0 +1,16 @@ +# OQ-070: `no_std` + `alloc` support + +- **Origin**: [crates/typedef/overview.md](crates/typedef/overview.md); + `docs/research/alknet-typedef/findings.md` §"Open Questions" (OQ 6) +- **Status**: deferred(scope) +- **Door type**: Two-way (additive — can be added as a feature gate + without changing the existing `std` API) +- **Priority**: low +- **Impacts**: Blocks embedded/WASM-bare-metal deployment targets (microcontrollers, `no_std` environments). Does NOT block WASM-browser (has `std` via `wasm-bindgen`). Does NOT block any current deployment target. +- **Blocked on**: An embedded use case that requires `no_std` + `alloc` + (e.g., a microcontroller running Rust without `std`). +- **Resolution**: Not yet decidable. Target `std` for v1. If embedded + use cases emerge, `no_std` + `alloc` can be added as a feature gate + later. The engine's core is already allocation-free; the `jsonschema` + dependency is the only `alloc` consumer. +- **Cross-references**: ADR-095 diff --git a/docs/architecture/questions/071-builder-api-for-schema-construction.md b/docs/architecture/questions/071-builder-api-for-schema-construction.md new file mode 100644 index 0000000..c091002 --- /dev/null +++ b/docs/architecture/questions/071-builder-api-for-schema-construction.md @@ -0,0 +1,24 @@ +# OQ-071: Builder API for schema construction + +- **Origin**: [crates/typedef/schema-layer.md](crates/typedef/schema-layer.md), + [crates/typedef/overview.md](crates/typedef/overview.md); + `docs/research/alknet-typedef/findings.md` (the builder API was noted + as the one detail not covered by the POCs) +- **Status**: deferred(scope) +- **Door type**: Two-way (additive — a builder API can be added without + changing the existing JSON-consumption path) +- **Priority**: medium +- **Impacts**: No current consumer. Schemas are authored in TypeBox (JS) + or hand-written JSON for v1. A Rust builder API would enable + programmatic schema construction in Rust without depending on a JS + toolchain, but no current consumer needs this. +- **Blocked on**: A concrete need for programmatic schema construction + in Rust. The current consumers (SFTP, metatensor, binary call frames, + TTY negotiation) all have schemas that can be hand-written or generated + from TypeBox. +- **Resolution**: Not yet decidable. The builder API is important but + not needed for the initial consumers. The engine's JSON-consumption + path is the primary interface for v1. A builder API would be a fluent + Rust API that produces the same JSON Schema structure — it would sit + on top of the engine, not inside it. +- **Cross-references**: ADR-095, [schema-layer.md](crates/typedef/schema-layer.md)