diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 1626800..ce4609f 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -15,12 +15,20 @@ format definition; the engine is generic. | Document | Status | Description | |----------|--------|-------------| | [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries | -| [schema-layer.md](schema-layer.md) | draft | The 19 `AlkType:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations | +| [schema-layer.md](schema-layer.md) | draft | The 19 `AlkType:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations. *(Current v0.1.0 format; will be rewritten for BAST — see [bast-format.md](bast-format.md).)* | +| [bast-format.md](bast-format.md) | draft | **Target schema format.** BAST (Binary Abstract Syntax Tree): meta-schema, TypeRef, examples, the two-validator model. Supersedes the format-spec content of schema-layer.md when the [BAST pivot](../plans/bast-implementation.md) lands. | | [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling | | [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading | -| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `AlkType:*` kinds, `AlkTypeError`, load-time vs access-time validation, `AlkTypeEngine`; `validate_bytes` for binary buffers (ADR-010) | +| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `AlkType:*` kinds, `AlkTypeError`, load-time vs access-time validation, `AlkTypeEngine`; `validate_bytes` for binary buffers (ADR-010). *(Current v0.1.0 validation; will be rewritten for the validator split — see [bast-format.md §Validation Model](bast-format.md#validation-model).)* | | [builder.md](builder.md) | draft | Fluent Rust API for constructing alktype JSON Schemas at runtime, producing `serde_json::Value`; covers AlkType kinds + standard JSON Schema (ADR-009) | +### In-progress work + +| Document | Status | Description | +|----------|--------|-------------| +| [BAST pivot — research record](../research/bast-pivot.md) | draft | Motivation, POC scope and result, decisions D-BAST-001..009, risks for the BAST format pivot | +| [BAST pivot — implementation plan](../plans/bast-implementation.md) | draft | Ordered implementation steps, the public-API semver contract, and the ADR-sync checklist for the BAST pivot | + ## Applicable ADRs | ADR | Title | Relevance | diff --git a/docs/architecture/bast-format.md b/docs/architecture/bast-format.md new file mode 100644 index 0000000..16928a8 --- /dev/null +++ b/docs/architecture/bast-format.md @@ -0,0 +1,675 @@ +--- +status: draft +last_updated: 2026-08-15 +--- + +# alktype — BAST Format + +**BAST** (Binary Abstract Syntax Tree) is alktype's schema format: a +JSON document that describes binary data layouts using a `kind`-based +vocabulary with `$defs`/`$ref` for composition. BAST replaces the +v0.1.0 `AlkType:*` custom-keyword JSON Schema format. + +This document is the **normative format specification**. It is grounded +in the POC on branch `bast-validator-poc` (commit `f371fe4`) and the +decisions D-BAST-001 through D-BAST-009 in +[the pivot research record](../research/bast-pivot.md#decisions). The +implementation plan is +[`docs/plans/bast-implementation.md`](../plans/bast-implementation.md). + +Until the BAST pivot lands in code, [`schema-layer.md`](schema-layer.md) +describes the *current* (custom-keyword) schema layer. This document +describes the *target* (BAST) schema layer. They coexist temporarily; +the implementation plan's final step retires `schema-layer.md`'s +custom-keyword content. + +## Design Principles + +1. **BAST is a JSON Schema instance.** A BAST document is valid JSON + that conforms to the BAST meta-schema (a standard Draft 2020-12 JSON + Schema). Any JSON Schema validator can check whether a BAST document + is well-formed; editors with JSON Schema support provide autocomplete + and inline validation for free. + +2. **`$defs`/`$ref` for composition.** Named type definitions live in a + top-level `$defs` block. `$ref` handles cross-references and union + variant references — the same pattern as JSON Schema's own `$defs` + and TypeBox's `Type.Module`. No custom reference resolution mechanism. + +3. **`kind`-based vocabulary.** Every type has a `kind` field whose + value is a known string (`"uint32"`, `"struct"`, `"union"`, etc.). + This replaces the `AlkType:*` custom-keyword pattern with a flat, + easily-matched string. The 19 `AlkTypeKind` enum variants are + unchanged; `AlkTypeKind::from_str`/`to_str` map between the enum and + the lowercase BAST strings (D-BAST-002). + +4. **Order is explicit.** Struct fields are an ordered array, not an + object with `properties`. Field order is unambiguous — no reliance on + `serde_json`'s `preserve_order` for correctness — and matches the + mental model of binary layouts. + +5. **Annotations are type-level properties.** Endianness, alignment, + encoding, and discriminators are properties of the type definition + or field, not custom keywords on a separate schema object. Their + *semantics* carry forward unchanged from ADR-003; only their + *location* moves. + +## Document Shape + +Every BAST document has the same top-level shape: + +```json +{ "$defs": { "": { ...TypeDef... }, ... } } +``` + +- The `$defs` block is **required** (D-BAST-003). Single-type documents + are a special case with one entry. A bare struct at the top level + would be a different shape with different parsing logic and no home + for additional definitions — rejected. +- The **root type name** is a required parameter to + `AlkTypeEngine::compile(bast_doc, root_name, mode)` (D-BAST-001). It + selects which `$defs` entry is the top-level type. Convention (first + entry) is fragile and depends on JSON key order; a `$root` marker is + redundant with an explicit parameter. + +## The Meta-Schema + +The BAST meta-schema is a standard JSON Schema (Draft 2020-12) that +validates the *structure* of BAST documents (is it well-formed?). It +lives at a stable URL (`https://alk.dev/bast/v1/schema`) and is embedded +in the crate for offline use. A different validator — the BAST-native +validator (see [Validation Model](#validation-model) below) — validates +*binary data* against a BAST document (are the bytes a valid instance?). +These are different validators for different inputs. + +```json +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "https://alk.dev/bast/v1/schema", + "title": "Binary Abstract Syntax Tree (BAST) v1", + "description": "Meta-schema for BAST documents. A BAST document describes the binary layout of structured data.", + "type": "object", + "properties": { + "$defs": { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/TypeDef" } + } + }, + "required": ["$defs"], + "$defs": { + "TypeDef": { + "oneOf": [ + { "$ref": "#/$defs/StructDef" }, + { "$ref": "#/$defs/UnionDef" }, + { "$ref": "#/$defs/EnumDef" } + ] + }, + "StructDef": { + "type": "object", + "properties": { + "kind": { "const": "struct" }, + "endian": { "enum": ["little", "big"] }, + "align": { "type": "integer", "minimum": 1 }, + "fields": { + "type": "array", + "items": { "$ref": "#/$defs/FieldDef" } + } + }, + "required": ["kind", "fields"], + "additionalProperties": false + }, + "FieldDef": { + "type": "object", + "properties": { + "name": { "type": "string", "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$" }, + "kind": { "$ref": "#/$defs/TypeRef" }, + "endian": { "enum": ["little", "big"] }, + "align": { "type": "integer", "minimum": 1 }, + "encoding": { "enum": ["length-prefixed", "offset-indirect"] }, + "maxLength": { "type": "integer", "minimum": 0 } + }, + "required": ["name", "kind"], + "additionalProperties": false + }, + "TypeRef": { + "oneOf": [ + { + "description": "Primitive type", + "type": "string", + "enum": [ + "int8", "int16", "int32", "int64", + "uint8", "uint16", "uint32", "uint64", + "float32", "float64", + "bool", "string", "bytes", "timestamp" + ] + }, + { + "description": "Reference to a named $defs entry", + "type": "object", + "properties": { + "$ref": { "type": "string", "pattern": "^#/\\$defs/[a-zA-Z_][a-zA-Z0-9_]*$" } + }, + "required": ["$ref"], + "additionalProperties": false + }, + { + "description": "Array type (fixed-size only in v1 — count is required)", + "type": "object", + "properties": { + "kind": { "const": "array" }, + "element": { "$ref": "#/$defs/TypeRef" }, + "count": { "type": "integer", "minimum": 0 } + }, + "required": ["kind", "element", "count"], + "additionalProperties": false + }, + { + "description": "Record (string-keyed map) type", + "type": "object", + "properties": { + "kind": { "const": "record" }, + "values": { "$ref": "#/$defs/TypeRef" } + }, + "required": ["kind", "values"], + "additionalProperties": false + } + ] + }, + "UnionDef": { + "type": "object", + "properties": { + "kind": { "const": "union" }, + "endian": { "enum": ["little", "big"] }, + "fields": { + "type": "array", + "items": { "$ref": "#/$defs/FieldDef" } + }, + "discriminator": { + "oneOf": [ + { + "type": "object", + "properties": { + "kind": { "const": "byte" }, + "offset": { "type": "integer", "minimum": 0 }, + "type": { "enum": ["uint8", "uint16", "uint32"] } + }, + "required": ["kind", "offset", "type"], + "additionalProperties": false + }, + { + "type": "object", + "properties": { + "kind": { "const": "field" }, + "name": { "type": "string" } + }, + "required": ["kind", "name"], + "additionalProperties": false + } + ] + }, + "mapping": { + "type": "object", + "additionalProperties": { "$ref": "#/$defs/TypeRef" } + } + }, + "required": ["kind", "discriminator", "mapping"], + "additionalProperties": false + }, + "EnumDef": { + "type": "object", + "properties": { + "kind": { "const": "enum" }, + "values": { + "type": "array", + "items": { "type": "string" }, + "minItems": 1 + } + }, + "required": ["kind", "values"], + "additionalProperties": false + } + } +} +``` + +## Type Definitions + +### Struct + +```json +{ + "kind": "struct", + "endian": "big", + "align": 256, + "fields": [ + { "name": "channel_id", "kind": "uint32" }, + { "name": "length", "kind": "uint32" } + ] +} +``` + +- `kind` (required): `"struct"`. +- `endian` (optional): `"little"` (default) or `"big"`. Sets the default + for all fields; field-level `endian` overrides. +- `align` (optional): struct-level alignment, only meaningful in aligned + static mode (ADR-002/003). +- `fields` (required): ordered array of [FieldDef](#fielddef). Array + position is field order — no reliance on JSON key order. + +### FieldDef + +```json +{ "name": "handle", "kind": "string", "encoding": "offset-indirect", "maxLength": 256 } +``` + +- `name` (required): identifier, `^[a-zA-Z_][a-zA-Z0-9_]*$`. +- `kind` (required): a [TypeRef](#typeref) — primitive string, `$ref` + object, array object, or record object. +- `endian` (optional): overrides the struct/union default for this field. +- `align` (optional): field-level alignment (aligned mode only). +- `encoding` (optional): `"length-prefixed"` (default) or + `"offset-indirect"`. See [Variable-length encoding](#variable-length-encoding). +- `maxLength` (optional): byte-length cap. See + [Variable-length encoding](#variable-length-encoding). + +### TypeRef + +`TypeRef` is the central mechanism for referencing types. Four forms: + +| Form | Example | Meaning | +|------|---------|---------| +| Primitive string | `"uint32"` | A built-in primitive (see [Primitives](#primitives)) | +| `$ref` object | `{ "$ref": "#/$defs/Read" }` | Reference to a named `$defs` entry | +| Array object | `{ "kind": "array", "element": "uint32", "count": 3 }` | Fixed-size array | +| Record object | `{ "kind": "record", "values": "string" }` | String-keyed map | + +The `$ref` form uses standard JSON Pointer syntax **restricted to +`#/$defs/`** — no external references, no fragment-only pointers, +no bare names. The restriction keeps resolution a single hash lookup +and eliminates the `normalize_refs` step the v0.1.0 engine needed for +TypeBox's bare-name refs. + +Arrays and records are inline type constructors, not top-level `$defs` +entries. Complex element types use nested `$ref`: + +```json +{ "kind": "array", "element": { "$ref": "#/$defs/ComplexElement" }, "count": 4 } +``` + +### Primitives + +The 14 primitive `kind` strings map to the unchanged `AlkTypeKind` +variants (D-BAST-002 — lowercase strings, PascalCase enum variants): + +| BAST kind | `AlkTypeKind` | Rust type | Size | Category | +|-----------|---------------|-----------|------|----------| +| `int8` | `Int8` | `i8` | 1 | fixed | +| `int16` | `Int16` | `i16` | 2 | fixed | +| `int32` | `Int32` | `i32` | 4 | fixed | +| `int64` | `Int64` | `i64` | 8 | fixed | +| `uint8` | `Uint8` | `u8` | 1 | fixed | +| `uint16` | `Uint16` | `u16` | 2 | fixed | +| `uint32` | `Uint32` | `u32` | 4 | fixed | +| `uint64` | `Uint64` | `u64` | 8 | fixed | +| `float32` | `Float32` | `f32` | 4 | fixed | +| `float64` | `Float64` | `f64` | 8 | fixed | +| `bool` | `Boolean` | `bool` (`0x00`=false, `0x01`=true) | 1 | fixed | +| `string` | `String` | length-prefixed UTF-8 | variable | variable | +| `bytes` | `Bytes` | length-prefixed raw bytes | variable | variable | +| `timestamp` | `Timestamp` | length-prefixed RFC 3339 string | variable | variable | + +`int64`/`uint64` are alktype additions (not in TypeBox's `typedef.ts`), +required by SFTP `offset: u64` and metatensor `data_offsets`. JSON +precision caveat per ADR-005 applies: integers beyond `2^53` lose +precision in `serde_json::Value::Number`; the binary path is exact. + +### Enum + +```json +{ + "kind": "enum", + "values": ["Ok", "PermissionDenied", "NoSuchFile", "Failure"] +} +``` + +- `kind` (required): `"enum"`. +- `values` (required): non-empty array of strings, in declaration order. +- Binary representation: a `u32` index into `values` (0-based), encoded + per the struct's endianness. This is a deliberate deviation from + TypeBox's string enum in favor of binary efficiency — a `u32` index is + compact, fixed-size, and sufficient for any realistic enum. + +**Bug fix vs v0.1.0:** The v0.1.0 engine has a dead constraint on the +bytes path — the built-in `enum` keyword checks string membership, but +the materializer emits `Value::Number(index)`, which never matches. The +BAST-native validator (see [Validation Model](#validation-model)) checks +the materialized index against `values.len()` bounds, fixing this. + +### Union + +```json +{ + "kind": "union", + "endian": "big", + "discriminator": { + "kind": "byte", + "offset": 0, + "type": "uint8" + }, + "mapping": { + "1": { "$ref": "#/$defs/Init" }, + "3": { "$ref": "#/$defs/Open" }, + "5": { "$ref": "#/$defs/Read" } + } +} +``` + +- `kind` (required): `"union"`. +- `endian` (optional): default endianness for variant fields. +- `discriminator` (required): one of: + - **Byte-offset**: `{ "kind": "byte", "offset": , "type": "uint8"|"uint16"|"uint32" }`. + The discriminator byte is at `offset`; the variant struct starts at + `offset + discriminator_size`. Mapping keys are stringified integers. + - **Field-name**: `{ "kind": "field", "name": "" }`. The + discriminator is a length-prefixed string field; mapping keys are + string values matching the field's value. The union's `fields` array + (optional, only valid with field-name discriminators per D-BAST-005) + provides the field definitions including the discriminator field. +- `mapping` (required): object mapping discriminator values to + [TypeRef](#typeref) entries (typically `$ref` to `$defs` variants). + +Variant `$ref`s are resolved **lazily** by the materializer and +validator — no `inline_union_variant_refs` compile step (removed under +BAST). The validator recurses into the selected variant's BAST +definition on `__discriminator` lookup, recovering the OQ-008 +per-variant constraint enforcement (e.g., `maxLength` on a `bytes` +field inside a variant) without custom keywords. + +### Array + +```json +{ "kind": "array", "element": "float32", "count": 3 } +``` + +- `kind` (required): `"array"`. +- `element` (required): a [TypeRef](#typeref). +- `count` (required in v1): the fixed element count. **Arrays of + variable-length elements without a `count` are not supported in v1** + (D-BAST-004, aligning with OQ-001). The meta-schema enforces this: + `count` is in `required`. Variable-length collections are available + via `record` instead. + +For fixed-size elements with a known count, the array size is +`element_size × count`. For variable-length elements (e.g., +`"element": "string"`) with a known count, each element carries its own +length prefix — the array is count-prefixed in the sense that the count +is known at schema time, but the total byte size is not. + +### Record + +```json +{ "kind": "record", "values": "string" } +``` + +- `kind` (required): `"record"`. +- `values` (required): a [TypeRef](#typeref) for the value type. + +Binary layout: a count-prefixed sequence of `(key, value)` pairs — +`[count: u32][key_len: u32][key_bytes][value]...` repeated `count` times. +Each key is a length-prefixed UTF-8 string. Each value is encoded per +its `values` type. There is no separate `value_len` prefix — the value's +size is determined by its kind (fixed-size kinds have a known size; +variable-length kinds carry their own length prefix). The count and +key-length prefixes respect the struct's endianness. + +## Variable-Length Encoding + +The three strategies from ADR-003 carry forward with the same semantics, +expressed as field-level properties instead of keyword-value objects: + +| Strategy | BAST syntax | Behavior | +|----------|------------|----------| +| Inline length-prefixed (default) | `{ "name": "handle", "kind": "string" }` | `[u32 length][data]` | +| Fixed-size reservation | `{ "name": "name", "kind": "string", "maxLength": 256 }` | Reserve `maxLength` bytes (aligned mode); validation constraint (packed mode) | +| Offset indirection | `{ "name": "blob", "kind": "bytes", "encoding": "offset-indirect" }` | `{offset: u32, length: u32}` pointing to separate data region | + +**Default strategy selection (unchanged from v0.1.0):** +- **Packed sequential mode:** always inline length-prefixing. + `maxLength` is a validation constraint only. +- **Aligned static mode:** fixed-size reservation if `maxLength` is + declared; offset indirection if `"encoding": "offset-indirect"` is + declared; inline length-prefixing otherwise. + +**Length prefix endianness:** The 4-byte length prefix (strategies 1 +and 3) respects the effective endianness (struct default or field +override). In little-endian mode, `u32::from_le_bytes`; in big-endian +mode, `u32::from_be_bytes`. Ensures SFTP consumers (big-endian) have +consistent byte order for field values and length prefixes. + +Applies to all variable-length types: `string`, `bytes`, `timestamp`, +`record`, and arrays of variable-length elements. + +## Endianness + +Struct-level or union-level property with per-field override (same +semantics as ADR-003): + +- Struct/union-level `"endian"` sets the default for all fields. +- Field-level `"endian"` overrides the struct/union default. +- Default is `"little"` when neither is specified. +- The length prefix for variable-length fields respects the effective + endianness. + +```json +{ + "kind": "struct", + "endian": "big", + "fields": [ + { "name": "id", "kind": "uint32" }, + { "name": "handle", "kind": "string" }, + { "name": "crc", "kind": "uint32", "endian": "little" } + ] +} +``` + +## Alignment + +Struct-level or field-level property, only meaningful in aligned static +mode (same as ADR-003): + +```json +{ + "kind": "struct", + "align": 256, + "fields": [ + { "name": "header", "kind": { "$ref": "#/$defs/Header" } }, + { "name": "weight", "kind": "float32", "align": 16 } + ] +} +``` + +- Struct-level `"align"` sets the default for all fields. +- Field-level `"align"` overrides the struct default. +- Default alignment: 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/ + f32/enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length + prefix), 1 for struct/union/array. Unchanged from v0.1.0. +- Ignored in packed sequential mode. + +## Validation Model + +BAST separates two concerns that the v0.1.0 format conflates, and in +doing so reveals that the engine has **two distinct validation paths** +with different inputs and guarantees. This is the validator split, +decided in D-BAST-006, D-BAST-007, and D-BAST-009. See +[`validation.md`](validation.md) for the current (pre-pivot) validation +layer; this section specifies the target model. + +### Two validators, two inputs + +| Path | Input | Validator | Schema source | +|------|-------|-----------|---------------| +| `validate_bytes(&[u8])` | Raw bytes | BAST-native validator | The BAST document (binary layout + value constraints) | +| `validate_json(&Value)` | Parsed JSON `Value` | Standard `jsonschema::Validator` | A consumer-provided standard JSON Schema | + +**`validate_bytes` — bytes in, BAST is the validator.** The materializer +produces a `Value` tree from bytes. By construction, this `Value` is +*structurally correct*: all declared fields are present (the +materializer iterates the field list), types are correct (`read_u32` +produces `Value::Number`), bounds are checked (via `data_access:: +check_bounds`), UTF-8 is valid (via `from_utf8`), the discriminator is +in the mapping, and the boolean byte is 0 or 1. What the materializer +does NOT check — and what the 19 v0.1.0 custom keyword validators check +afterward — are **value-domain constraints expressed in the BAST +document**. The BAST-native validator is a recursive walker over the +BAST type tree that checks exactly these: + +| Constraint | Validator arm | +|------------|---------------| +| Integer range (Int8..Uint64) | `validate_int`/`validate_uint` with `as_i64`/`as_u64` + range check | +| Int64/Uint64 (full range) | `validate_int64`/`validate_uint64` (JSON precision caveat per ADR-005) | +| Float finiteness (Float32/64) | `validate_float` with `as_f64().is_finite()` | +| String `maxLength` (byte length) | `check_string` reads the field-level `maxLength` | +| Bytes `maxLength` (array length) | `check_bytes` accepts `Value::String` and `Value::Array` forms | +| RFC 3339 timestamp shape | `validate_timestamp` — same non-strict check as v0.1.0 | +| Enum index bounds | `validate_enum` checks `idx < values.len()` — **fixes the v0.1.0 dead constraint** | +| Union variant dispatch | `validate_union` reads `__discriminator`, looks up the variant, recurses via `validate_typeref` | +| Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses | +| Array count | `validate_array` checks `arr.len() == count` and recurses per element | +| Record values | `validate_record` recurses into each value's `values` type | +| Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) | + +No external JSON Schema is required for `validate_bytes`. The BAST +document is the complete specification of the binary format — it +describes both the layout (how to read) and the constraints (what +values are valid). This is the "schema is the format" principle from +ADR-001, now fully realized. + +An optional external JSON Schema can be layered on top for constraints +BAST doesn't express (cross-field consistency, regex patterns on string +content). This is additive, not load-bearing. + +**`validate_json` — JSON in, JSON Schema is the validator.** The +consumer provides a JSON `Value` (e.g., an incoming JSON-RPC request). +The BAST document is irrelevant — BAST describes bytes, not JSON shape. +The right validator for a JSON value is a standard +`jsonschema::Validator` built from a standard JSON Schema document the +consumer provides. This is the path alkcall uses for its `OperationSpec` +JSON validation. No custom keywords; BAST is not involved. + +### What is removed + +Under the BAST pivot, the v0.1.0 validation machinery is removed from +the `validate_bytes` path: + +- All 19 `jsonschema::Keyword` implementations (~200 lines of validator + factories) — replaced by the BAST-native validator (~250 lines, a + flat match with no factories, no trait objects, no sub-validator + pre-computation). +- `inline_union_variant_refs()` — union variant refs are resolved lazily + by the validator and materializer. +- `build_validator()`'s custom-keyword path — repurposed or removed (see + the implementation plan's step 6 for the decision on its fate). + +The `jsonschema` crate **remains a direct dependency** for +`validate_json` and for validating BAST documents against the BAST +meta-schema. The only thing removed is the custom keyword integration +path. The `validate_bytes` path no longer touches `jsonschema` — a +small wasm binary-size win in addition to the architecture +simplification. + +### `AlkTypeError::Validation` payload shape + +**Decided (D-BAST-009):** Keep +`Validation(jsonschema::ValidationError<'static>)`. + +The `validate_bytes` path no longer uses `jsonschema`, so its error +payload is constructed via `jsonschema::ValidationError::custom` purely +to keep the variant's type unchanged. The rationale is consumer +ergonomics on the *combined* path: consumers like alkcall use both +`validate_json` (channel 0, JSON-RPC) and `validate_bytes` (binary +channels) and handle `AlkTypeError::Validation` in one place. A single +uniform payload type means one match arm covers both sources. The +alternative (`Validation(String)`) would force `validate_json` to +flatten its structured errors (instance path, schema path, keyword) to +a `String` via `Display` — the more information-rich path loses data to +accommodate the less rich one. That is the wrong direction. + +The `no_std`/minimal-build angle (OQ-002) that the alternative was +meant to enable is moot: `validate_json` requires `jsonschema` +regardless, so a bytes-only `no_std` build already has to give up +`validate_json` as a separate, larger decision. The right place to +revisit is when/if OQ-002 is actually pursued. + +## Relationship to JSON Schema and TypeBox + +### BAST is a JSON Schema dialect + +BAST is a specific JSON Schema instance format — like how JSON Schema +itself is a JSON document conforming to the JSON Schema meta-schema. +BAST documents conform to the BAST meta-schema. The entire JSON Schema +tooling ecosystem works with BAST: + +- **Validation:** `jsonschema::options().build(&bast_meta_schema)?.validate(&bast_doc)` +- **Editors:** VSCode with `$schema` pointing to the BAST meta-schema URL +- **Documentation:** JSON Schema generators produce human-readable docs + from the meta-schema + +### TypeBox interop + +TypeBox's `Type.Module({...})` pattern maps naturally to BAST's `$defs` +structure. A TypeBox module defining binary types can serialize to BAST +JSON. The relationship: + +- TypeBox → BAST JSON → alktype engine (binary layout) +- TypeBox → standard JSON Schema → jsonschema (JSON validation) + +Same TypeBox source, two output formats, two validators. + +### Not a replacement for JSON Schema + +BAST does not replace JSON Schema for JSON data validation. A BAST +document cannot validate a JSON payload — it describes binary data +layouts and value-domain constraints for bytes. For JSON validation, +consumers use standard JSON Schema documents (which may be derived from +BAST via future codegen, or authored separately). The `validate_json` +path accepts a consumer-provided JSON Schema and uses a standard +`jsonschema::Validator` — BAST is not involved. + +This is the split: `validate_bytes` is BAST-native (the BAST document +is both the layout spec and the validation spec for bytes); +`validate_json` is JSON-Schema-native (a standard JSON Schema is the +validation spec for JSON values). One crate, two validators, two input +types. + +## Decisions + +The BAST format is grounded in decisions D-BAST-001 through D-BAST-009, +recorded in [the pivot research record](../research/bast-pivot.md#decisions). +The implementation-relevant summary: + +| Decision | Summary | +|----------|---------| +| [D-BAST-001](../research/bast-pivot.md#d-bast-001-root-type-selection) | Root type name is a required `compile()` parameter — explicit, not convention | +| [D-BAST-002](../research/bast-pivot.md#d-bast-002-primitive-type-string-set) | Lowercase kind strings (`"uint32"`); `AlkTypeKind` variants stay PascalCase | +| [D-BAST-003](../research/bast-pivot.md#d-bast-003-top-level-defs-requirement) | `$defs` is always required; every document has the same top-level shape | +| [D-BAST-004](../research/bast-pivot.md#d-bast-004-arrays-of-variable-length-elements-deferred) | Arrays require `count` in v1; variable-length-element arrays deferred (OQ-001) | +| [D-BAST-005](../research/bast-pivot.md#d-bast-005-field-name-discriminator-unions) | Field-name discriminator unions supported; optional `fields` array on `UnionDef` | +| [D-BAST-006](../research/bast-pivot.md#d-bast-006-validate_bytes-validation-model) | `validate_bytes` uses the BAST-native validator — no external JSON Schema needed | +| [D-BAST-007](../research/bast-pivot.md#d-bast-007-validate_json-validation-model) | `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema | +| [D-BAST-008](../research/bast-pivot.md#d-bast-008-builder-api--two-output-formats) | One builder, two build methods: `struct_()` → BAST, `object()` → standard JSON Schema | +| [D-BAST-009](../research/bast-pivot.md#d-bast-009-alktypeerrorvalidation-payload-shape) | Keep `Validation(jsonschema::ValidationError<'static>)` — uniform payload for both paths | + +## References + +- [Pivot research record](../research/bast-pivot.md) — motivation, POC + scope and result, decisions D-BAST-001..009, risks +- [Implementation plan](../plans/bast-implementation.md) — ordered + steps, public-API semver contract, ADR-sync checklist +- [ADR-003](decisions/003-schema-annotations.md) — annotation semantics + (carry forward unchanged; only location moves) +- [ADR-005](decisions/005-int64-uint64-first-class-kinds.md) — Int64/ + Uint64 as first-class kinds; JSON precision caveat +- [`schema-layer.md`](schema-layer.md) — the current (v0.1.0) schema + layer; superseded by this document when the pivot lands +- [`validation.md`](validation.md) — the current (v0.1.0) validation + layer; rewritten for the validator split when the pivot lands \ No newline at end of file diff --git a/docs/plans/bast-implementation.md b/docs/plans/bast-implementation.md new file mode 100644 index 0000000..dec7cc8 --- /dev/null +++ b/docs/plans/bast-implementation.md @@ -0,0 +1,532 @@ +--- +status: draft +created: 2026-08-15 +--- + +# BAST Pivot — Implementation Plan + +This is the execution plan for the BAST pivot: replacing alktype's +v0.1.0 `AlkType:*` custom-keyword JSON Schema format with the BAST +(Binary Abstract Syntax Tree) format. It is the **entry point** an +implementing agent reads first. + +Companion documents: + +- [`docs/architecture/bast-format.md`](../architecture/bast-format.md) — + the normative BAST format spec (meta-schema, TypeRef, examples, + validation model). Read this for *what* the format is. +- [`docs/research/bast-pivot.md`](../research/bast-pivot.md) — the + research record: motivation, POC scope and result, decisions + D-BAST-001..009, risks. Read this for *why* and *what was proved*. + The POC lives on branch `bast-validator-poc` (commit `f371fe4`) as + `src/bast_poc.rs` — reference scaffolding, deliberately not merged. + +**Working order:** read this plan top-to-bottom. The Semver Contract +section is the scope-creep guardrail — consult it before each step. +Each step links to the specific spec section it implements and the +relevant D-BAST-* decision anchor. Implement steps in order; each step +lists its verification gate. + +## Semver Contract + +The crate is on crates.io at 0.1.0 with zero real consumers, so a +breaking bump is free — but the contract is explicit so the +implementation doesn't drift. Per AGENTS.md, the 0.1.0 public surface +is the items re-exported from `src/lib.rs`. This table is the +authoritative scope-creep guardrail for the pivot. + +| Public item (from `lib.rs` re-exports) | Class | Change | +|---|---|---| +| `AlkTypeKind` (enum + variants + methods) | **Additive** | Unchanged. 19 variants, same methods. New `from_str()`/`to_str()` mapping for lowercase BAST kind strings (`"uint32"` ↔ `AlkTypeKind::Uint32`) — additive methods. | +| `Endian`, `VariableEncoding`, `DiscriminatorKind` | **Unchanged** | — | +| `AlkTypeEngine::compile` | **Breaking** | Signature: `compile(schema: &mut Value, mode)` → `compile(bast_doc: &Value, root_name: &str, mode)`. Adds required `root_name` param (D-BAST-001); drops `&mut` (BAST needs no in-place `normalize_refs`); input is a BAST document, not a custom-keyword JSON Schema. | +| `AlkTypeEngine::validate_json` | **Breaking (behavioral)** | Signature unchanged `(instance: &Value) -> Result<...>`, but the validator it runs is now a standard `jsonschema::Validator` from a consumer-provided JSON Schema, not a custom-keyword validator built from the alktype schema. The *contract* of what schema validates the instance changes. | +| `AlkTypeEngine::validate_bytes` | **Unchanged (contract)** | Same signature. Internally the validation step switches from `jsonschema::Validator` to the BAST-native validator. Error type unchanged (D-BAST-009). | +| `AlkTypeEngine::is_valid_json` | **Breaking (behavioral)** | Same caveat as `validate_json` — validates against the consumer JSON Schema, not the alktype schema. | +| `AlkTypeEngine` accessors (`endian`, `mode`, `offset_map`, `layout_builder`, `sequential_reader`, `read_field`, `write_field`, etc.) | **Unchanged** | Layout-layer accessors are format-agnostic. | +| `LayoutMode`, `OffsetMap`, `ByteRange` | **Unchanged** | — | +| `LayoutBuilder`, `PackedLayout`, `FieldPosition` | **Unchanged** | — | +| `SequentialReader`, `FieldValue` | **Unchanged** | — | +| `UnionDispatch` | **Unchanged** | — | +| `data_access::*` functions | **Unchanged** | — | +| `AlkTypeError` (all 4 variants) | **Unchanged** | D-BAST-009 keeps `Validation(jsonschema::ValidationError<'static>)`. | +| `Schema` builder (`struct_`, `object`, `field`, `build`, all setters) | **Breaking (output format)** | Public method signatures unchanged. `build()` output changes from custom-keyword JSON to BAST JSON (for `struct_`) / standard JSON Schema (for `object`). Callers that introspect the built `Value` break; callers that pass it straight to `compile` are source-compatible once `compile` takes BAST. | +| `Definitions` builder (`new`, `define`, `define_value`, `build`, `merge_into`) | **Breaking (output format)** | Same as `Schema` — signatures unchanged, `build()`/`merge_into()` output shape changes to BAST `$defs`. | +| `Discriminator` builder enum | **Unchanged** | — | +| `build_validator` (from `validation`) | **Breaking (signature or removal)** | Currently `build_validator(schema: &Value) -> Result` builds a custom-keyword validator. Under the pivot it either (a) is removed (consumers call `jsonschema` directly for standard JSON Schema) or (b) is repurposed to build a standard `jsonschema::Validator` from a consumer-provided standard JSON Schema (no custom keywords). Decision belongs to step 6. Either way the current signature's contract breaks. | +| `get_alktype_kind`, `get_alktype_kind_enum`, `get_alktype_kind_loose`, `get_alktype_kind_loose_enum`, `normalize_refs`, `inline_union_variant_refs`, `resolve_ref`, `resolve_ref_or_inline`, `parse_align`, `parse_discriminator`, `parse_encoding`, `parse_endian`, `parse_max_length` | **Breaking (removal or rework)** | All currently re-exported from `lib.rs`. `normalize_refs` and `inline_union_variant_refs` are removed (BAST needs neither). The `get_alktype_kind*` family is removed (replaced by direct `kind` parsing). The `parse_*` and `resolve_*` functions are reworked to read BAST properties instead of keyword-value objects, or removed if subsumed by the BAST parser. **Open: which of these stay public vs become internal.** Current leaning — drop all from `lib.rs` re-exports (they're engine-internal accessors, not consumer API); the BAST parser exposes a new typed surface instead. Confirmed during step 3. | + +**Net breaking surface:** `compile`, `validate_json`/`is_valid_json` +(contract), `Schema::build`/`Definitions::build` (output format), +`build_validator` (signature/removal), and the ~13 `schema::*` helper +re-exports. **Net additive:** BAST parser, BAST-native validator, +`AlkTypeKind::from_str`/`to_str`. **Net unchanged:** the entire layout ++ data-access + materialize + tunion layer, `AlkTypeError`, the +`Discriminator` builder, `AlkTypeKind` variants. + +### Decisions deferred to their implementation steps + +These are small enough to decide when the step is reached, but are +flagged here so they don't become drive-by semver changes: + +1. **`validate_json` JSON Schema source** (step 6): does the consumer + pass the JSON Schema to `compile` (engine carries a second + validator) or to `validate_json` at call time? The former preserves + the current single-call ergonomics; the latter is more flexible. Not + semver-relevant either way if `validate_json`'s signature can absorb + a new param or stay as-is — needs the call-site analysis. +2. **`build_validator` fate** (step 6): removed vs repurposed. If + repurposed, its signature stays but its contract (no custom + keywords) changes — a behavioral break, not a type break. +3. **`schema::*` helper re-exports** (step 3): drop from `lib.rs` + (engine-internal) vs keep public for consumers that walk schemas. + Leaning: drop — they're accessors for the old format, and the BAST + parser exposes a cleaner typed surface. Confirmed during step 3. + +## Steps + +### Step 1 — Add `AlkTypeKind::from_str`/`to_str` for BAST kind strings + +**Goal:** Add the lowercase-string mapping (`"uint32"` ↔ +`AlkTypeKind::Uint32`) that the BAST parser and validator dispatch on. +This is the additive-only, zero-risk foundation — no existing code +changes. + +**Spec reference:** [bast-format.md §Primitives](../architecture/bast-format.md#primitives), +[D-BAST-002](../research/bast-pivot.md#d-bast-002-primitive-type-string-set). + +**Files:** `src/schema.rs` (the `AlkTypeKind` impl block). No `lib.rs` +change needed — the methods are inherent on the already-re-exported +enum. + +**Implementation notes:** +- `to_str(self) -> &'static str` returns the lowercase BAST string. +- `from_str(s: &str) -> Result` returns + `AlkTypeError::Schema` for unknown strings. This is a new inherent + method, distinct from the existing `FromStr` impl that parses the + v0.1.0 `"AlkType:Uint32"` keyword form. Do not remove the existing + `FromStr` yet — step 8 removes the v0.1.0 accessors. +- Cover all 14 primitive kinds plus `struct`, `union`, `array`, + `record`, `enum` (19 total, matching the enum variants). The + lowercase strings are in the [primitives table](../architecture/bast-format.md#primitives); + composite kinds are `"struct"`, `"union"`, `"array"`, `"record"`, + `"enum"`. + +**Verification:** `cargo test --release` (new unit tests for the +mapping, both directions; existing tests unaffected). `cargo clippy +--all-targets -- -D warnings`. + +--- + +### Step 2 — Embed the BAST meta-schema + +**Goal:** Embed the BAST meta-schema as a `serde_json::Value` constant +in the crate, available for validating BAST documents at compile time +and for publishing at `https://alk.dev/bast/v1/schema`. + +**Spec reference:** [bast-format.md §The Meta-Schema](../architecture/bast-format.md#the-meta-schema). + +**Files:** New `src/bast_meta.rs` (or a `const` in `src/schema.rs` — +match existing module conventions). Re-export the meta-schema `Value` +from `lib.rs` if consumers should be able to validate BAST documents +themselves (likely yes — additive, not semver-relevant). + +**Implementation notes:** +- The meta-schema JSON is in [bast-format.md §The Meta-Schema](../architecture/bast-format.md#the-meta-schema). + Copy it verbatim into a `serde_json::json! {...}` macro invocation or + parse it from an embedded string via `serde_json::from_str`. +- No feature flags (AGENTS.md §6). The meta-schema is a compile-time + constant, no I/O. +- WASM-clean: no `include_str!` of an external file is needed if the + `json!` macro is used; either way is wasm-safe. + +**Verification:** `cargo test --release`. `cargo build --target +wasm32-unknown-unknown --release` (meta-schema is a `Value` constant — +wasm-relevant). `cargo clippy --all-targets -- -D warnings`. + +--- + +### Step 3 — BAST document parser + +**Goal:** Implement the BAST document parser that the layout engines +and materializer use instead of the `get_alktype_kind*` custom-keyword +accessors. This is the natural entry point for the pivot — the largest +step, and the one the rest of the steps build on. + +**Spec reference:** [bast-format.md](../architecture/bast-format.md) +(the whole document — the parser implements the format spec). +[D-BAST-001](../research/bast-pivot.md#d-bast-001-root-type-selection), +[D-BAST-003](../research/bast-pivot.md#d-bast-003-top-level-defs-requirement), +[D-BAST-005](../research/bast-pivot.md#d-bast-005-field-name-discriminator-unions). + +**Files:** New `src/bast.rs` (the parser). The existing `src/schema.rs` +stays for now — steps 4–8 migrate callers off it. Update `src/lib.rs` +to add `pub mod bast;` and re-export the parser's public surface. + +**Implementation notes:** +- The parser reads `kind`/`fields`/annotation properties from BAST + nodes. It produces a typed surface (a small `BastNode` enum or + equivalent) that the layout engines, materializer, and validator can + walk without re-parsing the raw JSON at every node. The POC parsed + lazily from raw JSON in both passes to keep the model honest; a typed + tree is a straightforward follow-on optimization (POC observation 5). + Either is acceptable for the production version; the typed tree is + recommended since three consumers (layout, materialize, validate) + walk the same tree. +- `$ref` resolution: `#/$defs/` only — a single hash lookup. No + `normalize_refs` (BAST refs are always full JSON Pointers), no + `inline_union_variant_refs` (union variant refs resolved lazily by + the validator and materializer). See [bast-format.md §TypeRef](../architecture/bast-format.md#typeref). +- Untrusted input: every path that walks a BAST document must return + `Err(AlkTypeError::Schema)` on a malformed document, never `panic!`/ + `unreachable!` (AGENTS.md §3). The POC's + `malformed_document_produces_schema_error_not_panic` test is the + template. +- **Decide deferred decision #3 here:** drop the `schema::*` helper + re-exports from `lib.rs`, or keep them public. Leaning: drop. The + BAST parser exposes a cleaner typed surface; the v0.1.0 accessors + are engine-internal and not consumer API. + +**Verification:** `cargo test --release` (port the POC's parser tests +— the malformed-document test, the type-ref resolution tests). +`cargo clippy --all-targets -- -D warnings`. The layout engines don't +use the parser yet (step 4 wires it in), so the existing suite still +passes on the old path. + +--- + +### Step 4 — Wire `compile()` to accept a BAST document + root name + +**Goal:** Change `AlkTypeEngine::compile` to the new signature and +have it use the BAST parser instead of the custom-keyword accessors. +The layout engines (`offset_map`, `layout_builder`, +`sequential_reader`) consume the BAST parser's typed output instead of +walking raw JSON with `get_alktype_kind*`. + +**Spec reference:** [bast-format.md §Document Shape](../architecture/bast-format.md#document-shape), +[D-BAST-001](../research/bast-pivot.md#d-bast-001-root-type-selection). +Semver contract: `compile` is **Breaking**. + +**Files:** `src/engine.rs` (the `compile` signature and body). The +layout modules (`src/offset_map.rs`, `src/layout_builder.rs`, +`src/sequential_reader.rs`) — their schema-walking code changes from +`get_alktype_kind*` calls to BAST parser calls. `src/lib.rs` if the +parser's public surface needs re-exporting (step 3 may have done this). + +**Implementation notes:** +- New signature: `pub fn compile(bast_doc: &Value, root_name: &str, + mode: LayoutMode) -> Result`. Note `&Value` (not + `&mut Value`) — BAST needs no in-place `normalize_refs`. +- The engine stores the BAST document (or the parsed typed tree) for + `sequential_reader()`'s factory construction and `read_field`'s kind + lookup. The `Layout` enum and mode dispatch are unchanged. +- `parse_endian`, `parse_align`, `parse_encoding`, `parse_discriminator` + are reworked to read BAST properties (struct/field-level) instead of + keyword-value objects. Their *semantics* are unchanged (ADR-003); + only their *input location* moves. Whether they stay as free + functions or become methods on the typed `BastNode` is an + implementation choice — the POC read properties inline. +- The layout engines are format-agnostic beneath the accessors + (checked offset arithmetic, the two modes, union dispatch). This + step is an accessor swap, not a layout-engine rewrite. + +**Verification:** `cargo test --release` (test inputs must be converted +to BAST format — see step 9 for the full test conversion; this step +converts the layout tests as a sanity check). `cargo clippy +--all-targets -- -D warnings`. `cargo build --target +wasm32-unknown-unknown --release` (layout/wasm-relevant). + +--- + +### Step 5 — BAST-native validator (production version) + +**Goal:** Port the POC's BAST-native validator into a production module +and wire it into `validate_bytes` as the validation step, replacing the +`jsonschema::Validator` call on the bytes path. + +**Spec reference:** [bast-format.md §Validation Model](../architecture/bast-format.md#validation-model), +[D-BAST-006](../research/bast-pivot.md#d-bast-006-validate_bytes-validation-model), +[D-BAST-009](../research/bast-pivot.md#d-bast-009-alktypeerrorvalidation-payload-shape). +POC reference: `src/bast_poc.rs` on branch `bast-validator-poc`. + +**Files:** New `src/bast_validation.rs`. `src/engine.rs` +(`validate_bytes` body — swap the `self.validator.validate(&value)` call +for the BAST-native validator). `src/lib.rs` — add `pub mod +bast_validation;` (the validator is engine-internal; whether it's +re-exported is an implementation choice, leaning no). + +**Implementation notes:** +- The POC is the reference. The validator is a single recursive + function (`validate_typeref`) that dispatches on the BAST `kind`. The + constraint table is in [bast-format.md §Validation Model](../architecture/bast-format.md#validation-model). +- Construct `AlkTypeError::Validation` via + `jsonschema::ValidationError::custom` — the variant's payload type is + unchanged (D-BAST-009). The bytes path no longer touches `jsonschema` + for validation, but the error type retains the `jsonschema` type for + uniformity with the `validate_json` path. +- The validator and materializer share the BAST-walking code structure. + If step 3 produced a typed `BastNode` tree, both consume it. If step + 3 parses lazily, the validator parses lazily too (POC approach). +- Enum index bounds: check the materialized index against + `values.len()` — this **fixes the v0.1.0 dead constraint** (the + built-in `enum` keyword checked string membership, but the + materializer emits `Value::Number(index)`, which never matched). Net + improvement. +- Union variant dispatch: read `__discriminator`, look up the variant's + BAST definition, recurse. Recovers OQ-008 per-variant constraint + enforcement without custom keywords. + +**Verification:** `cargo test --release` — the existing `validate_bytes` +tests are the regression target (test *inputs* change to BAST format +in step 9; expected validation outcomes must be identical). The POC's +20 tests are the reference. `cargo clippy --all-targets -- -D warnings`. +`cargo build --target wasm32-unknown-unknown --release`. + +--- + +### Step 6 — `validate_json` against a consumer-provided JSON Schema + +**Goal:** Update `validate_json`/`is_valid_json` to validate against a +standard `jsonschema::Validator` compiled from a consumer-provided JSON +Schema, not a custom-keyword validator built from the alktype schema. + +**Spec reference:** [bast-format.md §Validation Model](../architecture/bast-format.md#validation-model), +[D-BAST-007](../research/bast-pivot.md#d-bast-007-validate_json-validation-model). +Semver contract: `validate_json`/`is_valid_json` are **Breaking +(behavioral)**; `build_validator` is **Breaking (signature or +removal)**. + +**Files:** `src/engine.rs` (`validate_json`/`is_valid_json` bodies, and +the engine's stored validator field if the JSON Schema is supplied at +compile time). `src/validation.rs` (`build_validator` — repurposed or +removed). `src/lib.rs` (the `build_validator` re-export if removed). + +**Implementation notes:** +- **Decide deferred decision #1 here:** does the consumer pass the JSON + Schema to `compile` (engine carries a second validator) or to + `validate_json` at call time? The former preserves single-call + ergonomics; the latter is more flexible. Needs the alkcall call-site + analysis. Not semver-relevant either way if the signature can absorb + the change. +- **Decide deferred decision #2 here:** `build_validator` removed vs + repurposed. If repurposed, its signature stays but its contract + changes (no custom keywords) — a behavioral break. If removed, drop + the `lib.rs` re-export. +- The `jsonschema` crate remains a direct dependency (for `validate_json` + and for validating BAST documents against the meta-schema). Only the + custom keyword integration is removed. +- The engine may carry two validators: the BAST-native validator (for + `validate_bytes`, from step 5) and the standard `jsonschema::Validator` + (for `validate_json`, from this step). Or `validate_json` takes the + JSON Schema at call time and builds a transient validator. The + decision shapes the engine struct's fields. + +**Verification:** `cargo test --release` (new tests for the +consumer-provided JSON Schema path; existing `validate_json` tests +converted — their schemas were custom-keyword, now standard). `cargo +clippy --all-targets -- -D warnings`. + +--- + +### Step 7 — Builder API produces BAST JSON + +**Goal:** Update the builder's `build()` methods to produce BAST JSON +(for `struct_()`) and standard JSON Schema (for `object()`). Public +method signatures are unchanged; only the output `Value` shape changes. + +**Spec reference:** [bast-format.md](../architecture/bast-format.md) +(the output format), [D-BAST-008](../research/bast-pivot.md#d-bast-008-builder-api--two-output-formats). +Semver contract: `Schema::build`/`Definitions::build` are **Breaking +(output format)**. + +**Files:** `src/builder.rs`. `src/lib.rs` if the builder's public +surface changes (it shouldn't — method signatures are unchanged). + +**Implementation notes:** +- `Schema::struct_().field(...).build()` → BAST JSON (a `$defs` entry + with `kind: "struct"`, ordered `fields` array, type-level + annotations). +- `Schema::object().field(...).build()` → standard JSON Schema (no + `AlkType:*` keywords, no BAST `kind` — just `type`/`properties`/ + `required`). +- `Definitions::build()`/`merge_into()` → a BAST `$defs` block. +- The builder already distinguishes AlkType kinds from JSON Schema types + via naming conventions (`string()` vs `string_()`). The construction + API is the same; only the serialization differs. +- The `Discriminator` builder is unchanged (semver contract: + **Unchanged**). + +**Verification:** `cargo test --release` (builder tests assert on the +output `Value` — update the expected shapes). `cargo clippy +--all-targets -- -D warnings`. + +--- + +### Step 8 — Remove v0.1.0 custom-keyword machinery + +**Goal:** Remove the dead code now that all callers use the BAST parser +and BAST-native validator. + +**Spec reference:** [bast-format.md §What is removed](../architecture/bast-format.md#what-is-removed). +Semver contract: the ~13 `schema::*` helper re-exports are **Breaking +(removal or rework)** (decision #3, confirmed in step 3). + +**Files:** `src/schema.rs` (remove `get_alktype_kind*`, +`normalize_refs`, `inline_union_variant_refs`; rework or remove +`parse_*`/`resolve_*`). `src/validation.rs` (remove the 19 +`jsonschema::Keyword` implementations if not already removed in step 5/6). +`src/lib.rs` (drop the removed items from the `pub use` block). + +**Implementation notes:** +- Remove: all 19 `jsonschema::Keyword` implementations (~200 lines), + `normalize_refs()`, `inline_union_variant_refs()`, the + `get_alktype_kind*` family. +- Rework or remove: `parse_align`, `parse_discriminator`, + `parse_encoding`, `parse_endian`, `parse_max_length`, `resolve_ref`, + `resolve_ref_or_inline`. If the BAST parser subsumes them (likely), + remove them. If any remain useful as free functions over the typed + `BastNode`, keep them internal (not re-exported from `lib.rs`). +- The `jsonschema` crate's `with_keyword(...)` registration calls are + removed from `compile`/`build_validator`. The crate itself stays. +- Drop the removed items from `lib.rs`'s `pub use schema::{ ... }` + block. The BAST parser's public surface replaces them. + +**Verification:** `cargo test --release`. `cargo clippy --all-targets +-- -D warnings`. `cargo doc --no-deps` (the public API surface +changed — doc comments must build). `cargo build --target +wasm32-unknown-unknown --release` (removing code shouldn't add +platform deps). + +--- + +### Step 9 — Convert all tests to BAST format + +**Goal:** Update the full test suite to use BAST format for inputs. +Test assertions (expected validation outcomes, expected offsets, +expected materialized values) must be identical — only the input +schema shape changes. + +**Spec reference:** [bast-format.md](../architecture/bast-format.md) +(input format). + +**Files:** `tests/*.rs` (integration tests), `src/*.rs` inline `#[cfg(test)]` +modules (unit tests). + +**Implementation notes:** +- This may be partially done by steps 4–8 (each step converts the tests + it touches as a sanity check). This step is the sweep: every test + using `AlkType:*` keywords converts to BAST `kind`/`fields`. +- The POC's 20 tests are the reference for BAST-shaped test inputs. +- Expected validation outcomes are the regression target. The + enum-index-bounds test is new behavior (the v0.1.0 dead constraint + is now enforced) — that test's expectation *changes* (was: silently + passed; now: `AlkTypeError::Validation`). This is the intended fix, + not a regression. +- Coverage: 310 crate + 86 integration tests (~396 total). All must + pass. + +**Verification:** `cargo test --release` (the full suite — this is the +gate). `cargo clippy --all-targets -- -D warnings`. + +--- + +### Step 10 — Sync architecture docs and ADRs + +**Goal:** Sync the descriptive docs and ADRs to the shipped code. This +is the final step — per AGENTS.md, ADRs are written post-implementation, +grounded in shipped code. + +**Spec reference:** [Semver Contract §ADR impact](#adr-impact-checklist) +below. + +**Files:** `docs/architecture/README.md`, `docs/architecture/overview.md`, +`docs/architecture/schema-layer.md` (rewrite for the BAST parser), +`docs/architecture/validation.md` (rewrite for the validator split), +`docs/architecture/builder.md` (update `build()` output examples), +`src/lib.rs` (module doc comment). New ADRs: ADR-BAST, ADR-VAL-SPLIT. +Amended ADRs: 001 (superseded), 003, 004, 009, 010. + +**Implementation notes:** +- Rewrite `schema-layer.md` to describe the BAST parser (replaces the + custom-keyword accessor walk-through). The current `schema-layer.md` + content is the v0.1.0 reference; `bast-format.md` already contains + the target spec. Either fold `bast-format.md` into `schema-layer.md` + or keep both with `schema-layer.md` pointing at `bast-format.md` for + the format and describing the parser module. +- Rewrite `validation.md` for the validator split (the [bast-format.md + §Validation Model](../architecture/bast-format.md#validation-model) + content moves here, expanded with the production validator's + details). +- Update `builder.md` output examples to BAST JSON. +- Update `src/lib.rs` module doc comment: "Takes a JSON Schema with + `AlkType:*` custom keywords" → "Takes a BAST document". +- Update `docs/architecture/README.md` index — the document table, the + ADR table (new ADRs, superseded ADR-001), the key design principles + (#1, #2, #7, #10 change wording). +- Remove stale TODOs referencing custom-keyword normalization, + `inline_union_variant_refs`, or the rejected bare-name-ref design + (AGENTS.md §"Architecture Context"). +- `docs/research/bast-pivot.md` is the research record — its status + flips from `draft` to `accepted`/`implemented` and it gains a pointer + to the ADRs that superseded its decisions. + +**Verification:** `cargo doc --no-deps` (doc comments build). +Cross-reference check: every link in this plan, `bast-format.md`, and +the new/updated ADRs resolves. `cargo test --release` (no code change, +but the doc sweep shouldn't break anything). + +## ADR Impact Checklist + +Sync these ADRs when step 10 lands. Per AGENTS.md, ADRs are written +post-implementation, grounded in shipped code. + +| ADR | Action | Reason | +|---|---|---| +| [ADR-001](../architecture/decisions/001-alktype-purpose-scope-jsonschema-engine.md) (purpose, scope, "schema is the format") | **Supersede** | The "schema is the format" principle is retained and strengthened (BAST *is* the format), but the concrete format changes from custom-keyword JSON Schema to BAST. A new ADR (ADR-BAST) records the BAST format as the realization of the principle. ADR-001 Status → Superseded by ADR-BAST. | +| [ADR-002](../architecture/decisions/002-two-layout-modes-packed-vs-aligned.md) (two layout modes) | **Unchanged** | Layout modes are format-agnostic. One-line note that the input format changed but the modes didn't. | +| [ADR-003](../architecture/decisions/003-schema-annotations.md) (annotations) | **Amend** | Annotation *semantics* carry forward unchanged; annotation *location* moves from custom-keyword objects to BAST type-level properties. Amend the "where annotations live" sections, keep the semantics. | +| [ADR-004](../architecture/decisions/004-error-handling-validation-strategy.md) (error handling, validation strategy) | **Amend** | Error enum shape unchanged (D-BAST-009). The "validation strategy" section updates: bytes path uses BAST-native validator, JSON path uses standard `jsonschema`. The load-time/access-time split is retained. | +| [ADR-005](../architecture/decisions/005-int64-uint64-first-class-kinds.md) (Int64/Uint64) | **Unchanged** | Kinds carry forward; JSON precision caveat unchanged. | +| [ADR-006](../architecture/decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md) (reject non-final inline in aligned mode) | **Unchanged** | Layout rule, format-agnostic. | +| [ADR-007](../architecture/decisions/007-packed-mode-read-factory.md) (packed-mode read factory) | **Unchanged** | Reader factory semantics are format-agnostic. | +| [ADR-008](../architecture/decisions/008-reject-tunion-in-aligned-mode.md) (reject TUnion in aligned mode) | **Unchanged** | Layout rule, format-agnostic. | +| [ADR-009](../architecture/decisions/009-builder-api.md) (builder API) | **Amend** | Public method surface unchanged; `build()` output format changes (BAST for `struct_`, standard JSON Schema for `object`). Amend the "output format" section; keep the method catalog. | +| [ADR-010](../architecture/decisions/010-generalized-validation-validate-bytes.md) (`validate_bytes`) | **Amend** | The two-step concept (materialize → validate) is retained. The validation step's *implementation* changes from `jsonschema` custom keywords to the BAST-native validator. Amend the "validation step" section; add a pointer to D-BAST-006/D-BAST-009 and ADR-VAL-SPLIT. | + +**New ADRs to write (post-implementation, grounded in shipped code):** +- **ADR-BAST** — the BAST format, meta-schema, and `$defs`/`$ref`/ + `kind` vocabulary. Supersedes ADR-001's format-specific content. +- **ADR-VAL-SPLIT** (or fold into ADR-004's amend) — the two-validator + model: BAST-native for `validate_bytes`, standard `jsonschema` for + `validate_json`. Records D-BAST-006, D-BAST-007, D-BAST-009. + +**Descriptive docs to sync (post-implementation):** +- `docs/architecture/schema-layer.md` — rewrite for the BAST parser + (replaces the custom-keyword accessor walk-through). +- `docs/architecture/validation.md` — rewrite for the validator split. +- `docs/architecture/builder.md` — update the `build()` output examples + to BAST JSON. +- `src/lib.rs` module doc comment — update the "Takes a JSON Schema + with `AlkType:*` custom keywords" preamble to BAST. +- `docs/architecture/README.md` — update the document table, ADR table, + and key design principles for the pivot. +- `docs/architecture/overview.md` — update the "what" and "why" for + BAST (the crate now takes a BAST document, not a custom-keyword JSON + Schema). + +**Stale TODOs to remove:** any TODO referencing custom-keyword +normalization, `inline_union_variant_refs`, or the rejected +bare-name-ref design — align with the ADRs as AGENTS.md §"Architecture +Context" requires. + +## Verification Commands + +Run these before committing each step. All must pass. Per AGENTS.md: + +```bash +cargo test --release # full suite (~396 tests: 310 crate + 86 integration) +cargo clippy --all-targets -- -D warnings +cargo doc --no-deps # if docs changed (step 8, step 10) +cargo build --target wasm32-unknown-unknown --release # if layout/wasm-relevant code changed (step 2, 4, 5, 8) +cargo publish --dry-run --allow-dirty # before a release (post-step 10) +``` \ No newline at end of file diff --git a/docs/research/bast-pivot.md b/docs/research/bast-pivot.md index a8ac0e0..aeb34b7 100644 --- a/docs/research/bast-pivot.md +++ b/docs/research/bast-pivot.md @@ -1,11 +1,10 @@ --- status: draft created: 2026-08-14 +last_updated: 2026-08-15 --- -# BAST Pivot — Binary Abstract Syntax Tree as the Schema Format - -## Summary +# BAST Pivot — Research Record Replace alktype's custom JSON Schema keywords (`AlkType:Uint32`, `AlkType:Struct`, etc.) with a standalone JSON format — BAST (Binary @@ -24,6 +23,16 @@ annotation properties from a purpose-built format. The builder API's public surface stays the same; only the JSON output format changes internally. +> **Document role.** This is the research record: motivation, POC +> scope and result, decisions, risks, references. The normative format +> specification lives in [`docs/architecture/bast-format.md`](../architecture/bast-format.md). +> The execution plan — ordered implementation steps, the public-API +> semver contract, and the ADR-sync checklist — lives in +> [`docs/plans/bast-implementation.md`](../plans/bast-implementation.md). +> Those documents supersede the format-spec, what-changes, and +> migration-path sections that previously lived here; this record +> keeps the *why* and *what was proved*, not the *how to implement*. + ## Motivation ### Current state @@ -88,987 +97,6 @@ v0.1.0 was published but has zero real consumers (only bots/scanners have downloaded it). A breaking change now is free. Waiting until adoption creates migration cost. -## The BAST Format - -### Design principles - -1. **BAST is a JSON Schema instance.** A BAST document is valid JSON - that conforms to the BAST meta-schema. Any standard JSON Schema - validator can validate a BAST document's structure. - -2. **`$defs`/`$ref` for composition.** Named type definitions live in a - top-level `$defs` block. `$ref` handles cross-references and union - variant references. This is the same pattern as TypeBox's - `Type.Module` and JSON Schema's own `$defs` — no custom reference - resolution mechanism needed. - -3. **`kind`-based vocabulary.** Every type has a `kind` field whose - value is a known string (`"uint32"`, `"struct"`, `"union"`, etc.). - This replaces the `AlkType:*` custom keyword pattern with a flat, - easily-matched string. - -4. **Order is explicit.** Struct fields are an ordered array, not an - object with `properties`. This makes field order unambiguous (no - reliance on `serde_json`'s `preserve_order` for correctness) and - matches the mental model of binary layouts. - -5. **Annotations are type-level properties.** Endianness, alignment, - encoding, and discriminators are properties of the type definition, - not custom keywords on a separate schema object. - -### Examples - -#### Channels chunk header (2-field struct, big-endian) - -```json -{ - "$defs": { - "ChunkHeader": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "channel_id", "kind": "uint32" }, - { "name": "length", "kind": "uint32" } - ] - } - } -} -``` - -#### TTY chunk (3-field struct, big-endian) - -```json -{ - "$defs": { - "TtyChunk": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "stream_id", "kind": "uint8" }, - { "name": "length", "kind": "uint32" } - ] - } - } -} -``` - -#### SFTP Read packet (struct with mixed fixed/variable fields) - -```json -{ - "$defs": { - "Read": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "id", "kind": "uint32" }, - { "name": "handle", "kind": "string" }, - { "name": "offset", "kind": "uint64" }, - { "name": "len", "kind": "uint32" } - ] - } - } -} -``` - -#### SFTP Packet union (byte-offset discriminator) - -```json -{ - "$defs": { - "SftpPacket": { - "kind": "union", - "endian": "big", - "discriminator": { - "kind": "byte", - "offset": 0, - "type": "uint8" - }, - "mapping": { - "1": { "$ref": "#/$defs/Init" }, - "3": { "$ref": "#/$defs/Open" }, - "5": { "$ref": "#/$defs/Read" }, - "6": { "$ref": "#/$defs/Write" }, - "101": { "$ref": "#/$defs/Status" } - } - }, - "Read": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "id", "kind": "uint32" }, - { "name": "handle", "kind": "string" }, - { "name": "offset", "kind": "uint64" }, - { "name": "len", "kind": "uint32" } - ] - }, - "Write": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "id", "kind": "uint32" }, - { "name": "handle", "kind": "string" }, - { "name": "offset", "kind": "uint64" }, - { "name": "data", "kind": "bytes" } - ] - }, - "Status": { - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "id", "kind": "uint32" }, - { "name": "status_code", "kind": "uint32" }, - { "name": "error_message", "kind": "string" }, - { "name": "language_tag", "kind": "string" } - ] - } - } -} -``` - -#### Metatensor header (aligned mode, little-endian, custom alignment) - -```json -{ - "$defs": { - "TensorHeader": { - "kind": "struct", - "endian": "little", - "align": 256, - "fields": [ - { "name": "magic", "kind": "uint64" }, - { "name": "json_length", "kind": "uint64" }, - { "name": "data_offset", "kind": "uint64" } - ] - } - } -} -``` - -#### Array of fixed-size elements with known count - -```json -{ - "$defs": { - "Vector3": { - "kind": "struct", - "fields": [ - { "name": "components", "kind": { "kind": "array", "element": "float32", "count": 3 } } - ] - } - } -} -``` - -#### Array of variable-length elements (deferred — see Decisions) - -Arrays of variable-length elements (e.g., `{ "kind": "array", "element": "string" }` -without a `count`) are **not supported in v1**. The engine rejects them today -(OQ-001), and the meta-schema requires `count` on all array types. This is a -known limitation, not a gap — see [D-BAST-004](#d-bast-004-arrays-of-variable-length-elements-deferred). - -#### Record (string-keyed map) - -```json -{ - "$defs": { - "Headers": { - "kind": "struct", - "fields": [ - { "name": "entries", "kind": { "kind": "record", "values": "string" } } - ] - } - } -} -``` - -#### Enum - -```json -{ - "$defs": { - "StatusCode": { - "kind": "enum", - "values": ["Ok", "PermissionDenied", "NoSuchFile", "Failure"] - } - } -} -``` - -### The BAST meta-schema - -A BAST document is valid JSON that conforms to the BAST meta-schema. -The meta-schema is a standard JSON Schema (Draft 2020-12) that -validates the structure of BAST documents. This means: - -- Any JSON Schema validator can check whether a BAST document is - well-formed before the engine compiles it. -- Editors with JSON Schema support (VSCode, JetBrains) provide - autocomplete and inline validation for BAST documents. -- The format is self-describing — a consumer can inspect the meta-schema - to understand the vocabulary without reading Rust source code. - -The meta-schema lives at a stable URL (e.g., -`https://alk.dev/bast/v1/schema`) and is embedded in the crate for -offline use. - -#### Meta-schema sketch - -```json -{ - "$schema": "https://json-schema.org/draft/2020-12/schema", - "$id": "https://alk.dev/bast/v1/schema", - "title": "Binary Abstract Syntax Tree (BAST) v1", - "description": "Meta-schema for BAST documents. A BAST document describes the binary layout of structured data.", - "type": "object", - "properties": { - "$defs": { - "type": "object", - "additionalProperties": { "$ref": "#/$defs/TypeDef" } - } - }, - "required": ["$defs"], - "$defs": { - "TypeDef": { - "oneOf": [ - { "$ref": "#/$defs/StructDef" }, - { "$ref": "#/$defs/UnionDef" }, - { "$ref": "#/$defs/EnumDef" } - ] - }, - "StructDef": { - "type": "object", - "properties": { - "kind": { "const": "struct" }, - "endian": { "enum": ["little", "big"] }, - "align": { "type": "integer", "minimum": 1 }, - "fields": { - "type": "array", - "items": { "$ref": "#/$defs/FieldDef" } - } - }, - "required": ["kind", "fields"], - "additionalProperties": false - }, - "FieldDef": { - "type": "object", - "properties": { - "name": { "type": "string", "pattern": "^[a-zA-Z_][a-zA-Z0-9_]*$" }, - "kind": { "$ref": "#/$defs/TypeRef" }, - "endian": { "enum": ["little", "big"] }, - "align": { "type": "integer", "minimum": 1 }, - "encoding": { "enum": ["length-prefixed", "offset-indirect"] }, - "maxLength": { "type": "integer", "minimum": 0 } - }, - "required": ["name", "kind"], - "additionalProperties": false - }, - "TypeRef": { - "oneOf": [ - { - "description": "Primitive type", - "type": "string", - "enum": [ - "int8", "int16", "int32", "int64", - "uint8", "uint16", "uint32", "uint64", - "float32", "float64", - "bool", "string", "bytes", "timestamp" - ] - }, - { - "description": "Reference to a named $defs entry", - "type": "object", - "properties": { - "$ref": { "type": "string", "pattern": "^#/\\$defs/[a-zA-Z_][a-zA-Z0-9_]*$" } - }, - "required": ["$ref"], - "additionalProperties": false - }, - { - "description": "Array type (fixed-size only in v1 — count is required)", - "type": "object", - "properties": { - "kind": { "const": "array" }, - "element": { "$ref": "#/$defs/TypeRef" }, - "count": { "type": "integer", "minimum": 0 } - }, - "required": ["kind", "element", "count"], - "additionalProperties": false - }, - { - "description": "Record (string-keyed map) type", - "type": "object", - "properties": { - "kind": { "const": "record" }, - "values": { "$ref": "#/$defs/TypeRef" } - }, - "required": ["kind", "values"], - "additionalProperties": false - } - ] - }, - "UnionDef": { - "type": "object", - "properties": { - "kind": { "const": "union" }, - "endian": { "enum": ["little", "big"] }, - "fields": { - "type": "array", - "items": { "$ref": "#/$defs/FieldDef" } - }, - "discriminator": { - "oneOf": [ - { - "type": "object", - "properties": { - "kind": { "const": "byte" }, - "offset": { "type": "integer", "minimum": 0 }, - "type": { "enum": ["uint8", "uint16", "uint32"] } - }, - "required": ["kind", "offset", "type"], - "additionalProperties": false - }, - { - "type": "object", - "properties": { - "kind": { "const": "field" }, - "name": { "type": "string" } - }, - "required": ["kind", "name"], - "additionalProperties": false - } - ] - }, - "mapping": { - "type": "object", - "additionalProperties": { "$ref": "#/$defs/TypeRef" } - } - }, - "required": ["kind", "discriminator", "mapping"], - "additionalProperties": false - }, - "EnumDef": { - "type": "object", - "properties": { - "kind": { "const": "enum" }, - "values": { - "type": "array", - "items": { "type": "string" }, - "minItems": 1 - } - }, - "required": ["kind", "values"], - "additionalProperties": false - } - } -} -``` - -### Type reference resolution - -`TypeRef` is the central mechanism for referencing types. It has four -forms: - -| Form | Example | Meaning | -|------|---------|---------| -| Primitive string | `"uint32"` | A built-in primitive type | -| `$ref` object | `{ "$ref": "#/$defs/Read" }` | Reference to a named definition | -| Array object | `{ "kind": "array", "element": "uint32" }` | Array of elements | -| Record object | `{ "kind": "record", "values": "string" }` | String-keyed map | - -The `$ref` form uses standard JSON Pointer syntax restricted to -`#/$defs/`. This is a subset of JSON Schema's `$ref` — no -external references, no fragment-only pointers, no bare names. The -restriction keeps resolution simple (single hash lookup) and avoids -the normalization step that the current engine needs for TypeBox's -bare-name refs. - -Arrays and records are inline type constructors, not top-level `$defs` -entries. This keeps the common cases concise while allowing complex -element types via nested `$ref`: - -```json -{ "kind": "array", "element": { "$ref": "#/$defs/ComplexElement" } } -``` - -### Variable-length encoding - -The three strategies from ADR-003 carry forward with the same semantics, -expressed as field-level properties instead of keyword-value objects: - -| Strategy | BAST syntax | Behavior | -|----------|------------|----------| -| Inline length-prefixed (default) | `{ "name": "handle", "kind": "string" }` | `[u32 length][data]` | -| Fixed-size reservation | `{ "name": "name", "kind": "string", "maxLength": 256 }` | Reserve `maxLength` bytes (aligned mode); validation constraint (packed mode) | -| Offset indirection | `{ "name": "blob", "kind": "bytes", "encoding": "offset-indirect" }` | `{offset: u32, length: u32}` pointing to separate data region | - -### Endianness - -Endianness is a struct-level or union-level property with per-field -override, same as ADR-003: - -- Struct-level `"endian"` sets the default for all fields. -- Field-level `"endian"` overrides the struct default. -- Default is `"little"` when neither is specified. -- The length prefix for variable-length fields respects the effective - endianness (struct default or field override). - -```json -{ - "kind": "struct", - "endian": "big", - "fields": [ - { "name": "id", "kind": "uint32" }, - { "name": "handle", "kind": "string" }, - { "name": "offset", "kind": "uint64" }, - { "name": "crc", "kind": "uint32", "endian": "little" } - ] -} -``` - -### Alignment - -Alignment is a struct-level or field-level property, only meaningful in -aligned static mode (same as ADR-003): - -```json -{ - "kind": "struct", - "align": 256, - "fields": [ - { "name": "header", "kind": { "$ref": "#/$defs/Header" } }, - { "name": "weight", "kind": "float32", "align": 16 } - ] -} -``` - -## What Changes - -### JSON format - -| Aspect | Current (custom keywords) | BAST | -|--------|--------------------------|------| -| Type declaration | `"AlkType:Uint32": true` on a property | `"kind": "uint32"` in a field definition | -| Struct fields | `"properties": { "x": {...}, "y": {...} }` | `"fields": [{ "name": "x", ... }, { "name": "y", ... }]` | -| Field order | Implicit via `serde_json` `preserve_order` | Explicit via array position | -| Endianness | `"endian": "big"` on the schema object | `"endian": "big"` on the struct/union definition | -| Variable encoding | `"AlkType:String": { "encoding": "offset-indirect" }` | `{ "name": "x", "kind": "string", "encoding": "offset-indirect" }` | -| Discriminator | `"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" }` | `"discriminator": { "kind": "byte", "offset": 0, "type": "uint8" }` | -| `$ref` | Bare names (`"Read"`) normalized to `#/$defs/Read` | Always `#/$defs/Read` — no normalization needed | -| Top-level container | Schema object with `AlkType:Struct` at root | `{ "$defs": { ... } }` with a root type name | -| JSON validation info | `"type": "object"`, `"required"`, etc. on the same object | Separate concern — not in BAST | - -The engine change is an accessor-layer refactor, not a rewrite. The -schema walkers currently read (kind, field list, annotations) out of -JSON nodes via custom-keyword accessors; under BAST they read the same -information from `kind`/`fields`/annotation properties. Everything -beneath the accessors — checked offset arithmetic, union dispatch, -materialization, the two layout modes — is format-agnostic and carries -over unchanged. - -### Engine internals - -| Component | Change | -|-----------|--------| -| `get_alktype_kind()` / `get_alktype_kind_loose()` | Replaced by parsing `"kind"` field directly from BAST nodes | -| `normalize_refs()` | Removed — BAST `$ref` values are always full JSON Pointers | -| `inline_union_variant_refs()` | Removed — union mapping values are resolved lazily via `$ref` | -| `parse_endian()`, `parse_align()`, `parse_encoding()`, `parse_discriminator()` | Adapted to read from BAST field/struct properties instead of keyword-value objects | -| Custom keyword validators (19 `jsonschema::Keyword` impls) | Removed. Replaced by the BAST-native validator (see below). | -| `build_validator()` | Repurposed. Still builds a `jsonschema::Validator`, but from a standard JSON Schema (no custom keywords). Used for JSON payload validation (call's `OperationSpec` schemas, etc.) and for validating BAST documents against the BAST meta-schema. | -| BAST-native validator | **New.** A recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`: integer ranges, float finiteness, `maxLength`, timestamp shape, enum index bounds, union variant dispatch. Replaces the 19 custom keyword validators for the `validate_bytes` path. See [The Validator Split](#the-validator-split). | -| `validate_json()` | Unchanged in signature. Validates a `Value` against a `jsonschema::Validator` compiled from a standard JSON Schema the consumer provides. No longer uses custom keywords. | -| `validate_bytes()` | Unchanged in concept — materialize `Value` from bytes, then validate. The validation step uses the BAST-native validator (not `jsonschema`) to check value-domain constraints from the BAST document. | - -### Public API - -| Item | Change | -|------|--------| -| `AlkTypeKind` enum | Unchanged — same 19 variants, same methods | -| `Endian`, `VariableEncoding`, `DiscriminatorKind` | Unchanged | -| `AlkTypeEngine` | `compile()` takes a BAST document + root type name instead of a custom-keyword JSON Schema. `validate_json()` validates against a consumer-provided JSON Schema. `validate_bytes()` validates against the BAST document via the BAST-native validator. | -| `LayoutMode`, `OffsetMap`, `ByteRange` | Unchanged | -| `LayoutBuilder`, `PackedLayout`, `FieldPosition` | Unchanged | -| `SequentialReader`, `FieldValue` | Unchanged | -| `UnionDispatch` | Unchanged | -| `data_access` functions | Unchanged | -| `AlkTypeError` | Unchanged (Schema/Offset/Access/Validation variants) | -| `Schema` builder | Public methods unchanged. `build()` produces BAST JSON instead of custom-keyword JSON. | -| `Definitions` builder | Public methods unchanged. `build()` produces a BAST `$defs` block. | -| `Discriminator` builder | Unchanged | - -### What is removed - -- All 19 `jsonschema::Keyword` implementations (~200 lines of validator factories) -- `normalize_refs()` — BAST `$ref` values are always full JSON Pointers -- `inline_union_variant_refs()` — union mapping values are resolved lazily -- `get_alktype_kind()` / `get_alktype_kind_loose()` and their `_enum` variants — replaced by direct `kind` field parsing -- Custom keyword registration with `jsonschema` — the engine no longer calls `jsonschema::options().with_keyword(...)`. The `jsonschema` crate **remains a direct dependency** for standard JSON Schema validation (validating JSON payloads like call's `OperationSpec` schemas, and validating BAST documents against the BAST meta-schema). The only thing removed is the custom keyword integration path. - -### What is added - -- BAST meta-schema (embedded in the crate, published at a stable URL) -- BAST document parser — validates a BAST document against the meta-schema, then extracts type definitions -- `AlkTypeEngine::compile()` takes `(bast_document: &Value, root_name: &str, mode: LayoutMode)` — the root name selects which `$defs` entry is the top-level type -- BAST-native validator — a recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`. Replaces the 19 custom keyword validators for the `validate_bytes` path. Enforces: integer ranges, float finiteness, string/bytes `maxLength`, RFC 3339 timestamp shape, enum index bounds, union variant dispatch (recursing into variants). See [The Validator Split](#the-validator-split). -- BAST meta-schema validation at compile time (optional but recommended — the engine can skip it and trust the caller, or validate as a guard) - -## Semver and ADR Impact - -This section is the scope-creep guardrail for the public API during -implementation. The crate is on crates.io at 0.1.0 with zero real -consumers, so a breaking bump is free — but the contract still needs to -be explicit so the implementation doesn't drift. Per AGENTS.md, the -0.1.0 public surface is the items re-exported from `src/lib.rs`. - -### Public API delta - -| Public item (from `lib.rs` re-exports) | Class | Change | -|---|---|---| -| `AlkTypeKind` (enum + variants + methods) | **Additive** | Unchanged. 19 variants, same methods. New `from_str()`/`to_str()` mapping for lowercase BAST kind strings (`"uint32"` ↔ `AlkTypeKind::Uint32`) — additive methods. | -| `Endian`, `VariableEncoding`, `DiscriminatorKind` | **Unchanged** | — | -| `AlkTypeEngine::compile` | **Breaking** | Signature: `compile(schema: &mut Value, mode)` → `compile(bast_doc: &Value, root_name: &str, mode)`. Adds required `root_name` param (D-BAST-001); drops `&mut` (BAST needs no in-place `normalize_refs`); input is a BAST document, not a custom-keyword JSON Schema. | -| `AlkTypeEngine::validate_json` | **Breaking (behavioral)** | Signature unchanged `(instance: &Value) -> Result<...>`, but the validator it runs is now a standard `jsonschema::Validator` from a consumer-provided JSON Schema supplied at `compile`, not a custom-keyword validator built from the alktype schema. The compiled engine must carry a separate JSON-Schema validator (or `validate_json` takes the JSON Schema at call time — to be decided in step 6). Either way the *contract* of what schema validates the instance changes. | -| `AlkTypeEngine::validate_bytes` | **Unchanged (contract)** | Same signature. Internally the validation step switches from `jsonschema::Validator` to the BAST-native validator. Error type unchanged (D-BAST-009). | -| `AlkTypeEngine::is_valid_json` | **Breaking (behavioral)** | Same caveat as `validate_json` — validates against the consumer JSON Schema, not the alktype schema. | -| `AlkTypeEngine` accessors (`endian`, `mode`, `offset_map`, `layout_builder`, `sequential_reader`, `read_field`, `write_field`, etc.) | **Unchanged** | Layout-layer accessors are format-agnostic. | -| `LayoutMode`, `OffsetMap`, `ByteRange` | **Unchanged** | — | -| `LayoutBuilder`, `PackedLayout`, `FieldPosition` | **Unchanged** | — | -| `SequentialReader`, `FieldValue` | **Unchanged** | — | -| `UnionDispatch` | **Unchanged** | — | -| `data_access::*` functions | **Unchanged** | — | -| `AlkTypeError` (all 4 variants) | **Unchanged** | D-BAST-009 keeps `Validation(jsonschema::ValidationError<'static>)`. | -| `Schema` builder (`struct_`, `object`, `field`, `build`, all setters) | **Breaking (output format)** | Public method signatures unchanged. `build()` output changes from custom-keyword JSON to BAST JSON (for `struct_`) / standard JSON Schema (for `object`). Callers that introspect the built `Value` break; callers that pass it straight to `compile` are source-compatible once `compile` takes BAST. | -| `Definitions` builder (`new`, `define`, `define_value`, `build`, `merge_into`) | **Breaking (output format)** | Same as `Schema` — signatures unchanged, `build()`/`merge_into()` output shape changes to BAST `$defs`. | -| `Discriminator` builder enum | **Unchanged** | — | -| `build_validator` (from `validation`) | **Breaking (signature or removal)** | Currently `build_validator(schema: &Value) -> Result` builds a custom-keyword validator. Under the pivot it either (a) is removed (consumers call `jsonschema` directly for standard JSON Schema) or (b) is repurposed to build a standard `jsonschema::Validator` from a consumer-provided standard JSON Schema (no custom keywords). Decision belongs to step 6. Either way the current signature's contract breaks. | -| `get_alktype_kind`, `get_alktype_kind_enum`, `get_alktype_kind_loose`, `get_alktype_kind_loose_enum`, `normalize_refs`, `inline_union_variant_refs`, `resolve_ref`, `resolve_ref_or_inline`, `parse_align`, `parse_discriminator`, `parse_encoding`, `parse_endian`, `parse_max_length` | **Breaking (removal or rework)** | All currently re-exported from `lib.rs`. `normalize_refs` and `inline_union_variant_refs` are removed (BAST needs neither). The `get_alktype_kind*` family is removed (replaced by direct `kind` parsing). The `parse_*` and `resolve_*` functions are reworked to read BAST properties instead of keyword-value objects, or removed if subsumed by the BAST parser. **Open: which of these stay public vs become internal.** Current leaning — drop all from `lib.rs` re-exports (they're engine-internal accessors, not consumer API); the BAST parser exposes a new typed surface instead. | - -**Net breaking surface:** `compile`, `validate_json`/`is_valid_json` -(contract), `Schema::build`/`Definitions::build` (output format), -`build_validator` (signature/removal), and the ~13 `schema::*` helper -re-exports. **Net additive:** BAST parser, BAST-native validator, -`AlkTypeKind::from_str`/`to_str`. **Net unchanged:** the entire layout -+ data-access + materialize + tunion layer, `AlkTypeError`, the -`Discriminator` builder, `AlkTypeKind` variants. - -### ADR impact - -| ADR | Action | Reason | -|---|---|---| -| [ADR-001](../architecture/decisions/001-alktype-purpose-scope-jsonschema-engine.md) (purpose, scope, "schema is the format") | **Supersede** | The "schema is the format" principle is retained and strengthened (BAST *is* the format), but the concrete format changes from custom-keyword JSON Schema to BAST. A new ADR (ADR-BAST or renumbered) records the BAST format as the realization of the principle. ADR-001's Status → Superseded by ADR-BAST. | -| [ADR-002](../architecture/decisions/002-two-layout-modes-packed-vs-aligned.md) (two layout modes) | **Unchanged** | Layout modes are format-agnostic. No content change; maybe a one-line note that the input format changed but the modes didn't. | -| [ADR-003](../architecture/decisions/003-schema-annotations.md) (endianness, alignment, encoding, discriminators) | **Amend** | Annotation *semantics* carry forward unchanged; annotation *location* moves from custom-keyword objects to BAST type-level properties. Amend the "where annotations live" sections, keep the semantics. | -| [ADR-004](../architecture/decisions/004-error-handling-validation-strategy.md) (`AlkTypeError`, load-time build, access-time check) | **Amend** | Error enum shape unchanged (D-BAST-009). The "validation strategy" section updates: bytes path uses BAST-native validator, JSON path uses standard `jsonschema`. The load-time/access-time split is retained. | -| [ADR-005](../architecture/decisions/005-int64-uint64-first-class-kinds.md) (Int64/Uint64, JSON precision caveat) | **Unchanged** | Kinds carry forward; JSON precision caveat is unchanged. | -| [ADR-006](../architecture/decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md) (reject non-final inline length-prefixed in aligned mode) | **Unchanged** | Layout rule, format-agnostic. | -| [ADR-007](../architecture/decisions/007-packed-mode-read-factory.md) (packed-mode read factory, `sequential_reader()` returns owned reader) | **Unchanged** | Reader factory semantics are format-agnostic. | -| [ADR-008](../architecture/decisions/008-reject-tunion-in-aligned-mode.md) (reject TUnion in aligned mode for v1) | **Unchanged** | Layout rule, format-agnostic. | -| [ADR-009](../architecture/decisions/009-builder-api.md) (builder API) | **Amend** | Public method surface unchanged; `build()` output format changes (BAST for `struct_`, standard JSON Schema for `object`). Amend the "output format" section; keep the method catalog. | -| [ADR-010](../architecture/decisions/010-generalized-validation-validate-bytes.md) (`validate_bytes` — materialize then validate) | **Amend** | The two-step concept (materialize → validate) is retained. The validation step's *implementation* changes from `jsonschema` custom keywords to the BAST-native validator. Amend the "validation step" section; add a pointer to D-BAST-006/D-BAST-009 and the Validator Split section. | - -**New ADRs to write (post-implementation, grounded in shipped code):** -- **ADR-BAST** — the BAST format, meta-schema, and `$defs`/`$ref`/ - `kind` vocabulary. Supersedes ADR-001's format-specific content. -- **ADR-VAL-SPLIT** (or fold into ADR-004's amend) — the two-validator - model: BAST-native for `validate_bytes`, standard `jsonschema` for - `validate_json`. Records D-BAST-006, D-BAST-007, D-BAST-009. - -**Descriptive docs to sync (post-implementation):** -- `docs/architecture/schema-layer.md` — rewrite for the BAST parser - (replaces the custom-keyword accessor walk-through). -- `docs/architecture/validation.md` — rewrite for the validator split. -- `docs/architecture/builder.md` — update the `build()` output examples - to BAST JSON. -- `src/lib.rs` module doc comment — update the "Takes a JSON Schema - with `AlkType:*` custom keywords" preamble to BAST. - -**Stale TODOs to remove:** any TODO referencing custom-keyword -normalization, `inline_union_variant_refs`, or the rejected -bare-name-ref design — align with the ADRs as AGENTS.md §"Architecture -Context" requires. - -### Open decisions deferred to implementation steps - -These are small enough to decide when the step is reached, but are -flagged here so they don't become drive-by semver changes: - -1. **`validate_json` JSON Schema source** (step 6): does the consumer - pass the JSON Schema to `compile` (engine carries a second - validator) or to `validate_json` at call time? The former preserves - the current single-call ergonomics; the latter is more flexible. Not - semver-relevant either way if `validate_json`'s signature can absorb - a new param or stay as-is — needs the call-site analysis. -2. **`build_validator` fate** (step 6): removed vs repurposed. If - repurposed, its signature stays but its contract (no custom - keywords) changes — a behavioral break, not a type break. -3. **`schema::*` helper re-exports** (step 3): drop from `lib.rs` - (engine-internal) vs keep public for consumers that walk schemas. - Leaning: drop — they're accessors for the old format, and the BAST - parser exposes a cleaner typed surface. Confirmed during step 3. - -## The Validator Split - -A key architectural clarification: BAST separates two concerns that the -current format conflates, and in doing so reveals that the current -engine has **two distinct validation paths** that custom keywords -paper over with a single mechanism. - -### Current model (conflated) - -``` -┌─────────────────────────────────────────────┐ -│ JSON Schema with AlkType:* custom keywords │ -│ ┌───────────────────────────────┐ │ -│ │ Binary layout info (AlkType:Uint32) │ │ -│ │ JSON validation info (type, enum) │ │ -│ └───────────────────────────────┘ │ -└─────────────────────────────────────────────┘ - │ - ▼ - AlkTypeEngine::compile() - │ - ├──► Layout (offset map / layout builder) - └──► Validator (jsonschema with 19 custom keywords) - │ - ├──► validate_json(&Value) — JSON value validation - └──► validate_bytes(&[u8]) — materialize, then validate -``` - -One document serves two roles. The engine extracts both layout and -validation from the same JSON tree. A single `jsonschema::Validator` -(with custom keywords) serves both `validate_json` and `validate_bytes`. - -### BAST model (three layers, two validators) - -The current engine's two validation paths have different needs, and -BAST makes the split explicit: - -``` - ┌──────────────────────┐ - │ BAST document │ - │ (binary layout + │ - │ value constraints) │ - └──────────┬───────────┘ - │ - ▼ - AlkTypeEngine::compile() - │ - ┌────────────────┼────────────────┐ - ▼ ▼ - ┌─────────────────┐ ┌──────────────────┐ - │ Layout │ │ BAST-native │ - │ (offset map / │ │ validator │ - │ layout builder)│ │ (walks BAST, │ - └────────┬────────┘ │ checks value │ - │ │ constraints) │ - ▼ └────────┬──────────┘ - ┌─────────────────┐ │ - │ Materializer │ │ - │ (bytes → Value, │ │ - │ guarantees │ │ - │ structure) │ │ - └────────┬────────┘ │ - │ │ - ▔──────────────┬─────────────────────┘ - ▼ - validate_bytes(&[u8]) - (materialize, then check - value constraints against - the BAST schema — no external - JSON Schema needed) - - ┌──────────────────────────┐ - │ JSON Schema document │ (separate concern) - │ (JSON validation) │ - │ type: "object" │ - │ properties: { │ - │ id: { type: "integer"}│ - │ } │ - │ required: ["id"] │ - └────────────┬─────────────┘ - │ - ▼ - build_validator(&json_schema) - │ - ▼ - jsonschema::Validator - (standard JSON Schema, - no custom keywords) - │ - ▼ - validate_json(&Value) - (validates consumer-supplied - JSON against a standard - JSON Schema — BAST not - involved) -``` - -### Why two validators, not one - -The two paths have fundamentally different inputs and guarantees: - -**`validate_bytes(&[u8])` — bytes in, BAST is the validator.** -The materializer produces a `Value` tree from bytes. By construction, -this `Value` is *structurally correct*: all declared fields are present -(because the materializer iterates the field list), types are correct -(because `read_u32` produces a `Value::Number`), bounds are checked -(via `data_access::check_bounds`), UTF-8 is valid (via `from_utf8`), -the discriminator is in the mapping, and the boolean byte is 0 or 1. -What the materializer does NOT check — and what the 19 custom keyword -validators currently check afterward — are **value-domain constraints -expressed in the schema**: - -| Constraint | Current enforcer | BAST-native validator | -|------------|------------------|----------------------| -| Integer range (Int8..Uint64) | Custom keyword | BAST `kind` + range check | -| Float finiteness (Float32/64) | Custom keyword | BAST `kind` + `is_finite()` | -| String `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property | -| Bytes `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property | -| RFC 3339 timestamp shape | Custom keyword | BAST `kind: "timestamp"` + shape check | -| Union variant dispatch | `UnionValidator` (OQ-008) | BAST union — recurse into variant | -| Enum value membership | Built-in `enum` keyword (**broken on bytes path** — see below) | BAST `kind: "enum"` + index-in-range | - -All of these are expressible directly from the BAST document — no -external JSON Schema needed. The BAST-native validator is a recursive -walker over the BAST type tree, checking the materialized `Value` -against each type's constraints. Union dispatch is just recursion: -read `__discriminator`, look up the variant's BAST definition, recurse. - -This **recovers the OQ-008 union dispatch behavior** without custom -keywords and without requiring the consumer to hand-author -discriminator dispatch in a separate JSON Schema. The BAST document -already knows the union's variants and their fields. - -**`validate_json(&Value)` — JSON in, JSON Schema is the validator.** -The consumer provides a JSON `Value` (e.g., an incoming JSON-RPC request -on alkcall's channel 0). The BAST document is irrelevant — BAST -describes bytes, not JSON shape. The right validator for a JSON value -is a standard `jsonschema::Validator` built from a standard JSON -Schema document (which the consumer provides, or which is generated -from BAST via future codegen). This is the path alkcall uses today for -its `OperationSpec` JSON validation. - -### Bug fix: enum validation on the bytes path - -The current engine has a **dead constraint**: when `validate_bytes` -materializes an `AlkType:Enum` field, it reads a raw `u32` index and -emits `Value::Number(index)`. The built-in `enum` keyword lists string -members (e.g., `["Ok", "Error"]`). A numeric index never matches a -string-membered `enum`, so enum membership is silently unenforced on -the bytes path. The BAST-native validator fixes this: for `kind: "enum"`, -it checks that the materialized index is within the `values` array -bounds (0..len-1), which is the correct validation for a binary enum -encoded as an index. - -### What this means for consumers - -**alkcall** today uses alktype for two roles: -1. Binary layout (channels chunk header) — `AlkTypeEngine::compile()` - in packed mode -2. JSON validation (call's `OperationSpec` schemas) — `build_validator()` - or `jsonschema` directly - -Under BAST: -1. Binary layout + validation — `AlkTypeEngine::compile(bast_doc, - "ChunkHeader", Packed)` then `engine.validate_bytes(&buf)`. The - engine uses the BAST-native validator — no external JSON Schema - needed for binary data. All value constraints (`maxLength`, ranges, - enum bounds, union dispatch) are enforced from the BAST document. -2. JSON validation — `build_validator(&json_schema)` — builds a - standard `jsonschema::Validator` from a standard JSON Schema - document (no custom keywords). Consumers that don't use binary at - all (e.g., adapters around remote JSON Schema APIs) use this path - exclusively. - -The builder API produces both formats: -- `Schema::struct_().field(...).build()` → BAST JSON (binary layout) -- `Schema::object().field(...).build()` → standard JSON Schema (JSON - validation) - -Both live in the same crate, same wasm target, same builder surface. -This is the point of alktype: one small wasm-compatible codebase that -handles both binary layout and JSON validation for protocol crates like -alkcall, alktty, and future channel implementations. - -### Validation of binary data - -`validate_bytes()` works as follows: -1. **Materialize** a `Value` tree from binary bytes using the BAST - layout. The materializer guarantees structural correctness (bounds, - UTF-8, bool byte, discriminator lookup, all fields present). -2. **Validate** the materialized `Value` against value-domain - constraints from the BAST document, using the BAST-native - validator. This checks integer ranges, float finiteness, `maxLength`, - timestamp shape, enum index bounds, and recurses into union variants. - -No external JSON Schema is required for `validate_bytes`. The BAST -document is the complete specification of the binary format — it -describes both the layout (how to read) and the constraints (what -values are valid). This is the "schema is the format" principle from -ADR-001, now fully realized: the BAST document is the binary format, -and it is also the validation spec. - -An optional external JSON Schema can be layered on top for constraints -BAST doesn't express (e.g., cross-field consistency, regex patterns on -string content). This is additive, not load-bearing — the binary path -works without it. - -## Relationship to JSON Schema and TypeBox - -### BAST is a JSON Schema dialect - -BAST is a specific JSON Schema instance format — like how JSON Schema -itself is a JSON document that conforms to the JSON Schema meta-schema. -BAST documents conform to the BAST meta-schema. The meta-schema is a -standard JSON Schema (Draft 2020-12). - -This means the entire JSON Schema tooling ecosystem works with BAST: -- Validation: `jsonschema::options().build(&bast_meta_schema)?.validate(&bast_doc)` -- Editors: VSCode with `$schema` pointing to the BAST meta-schema URL -- Documentation: JSON Schema generators can produce human-readable docs - from the meta-schema - -Note: the BAST meta-schema validates the *structure* of a BAST document -(is it well-formed?). The BAST-native validator validates *binary data* -against the BAST document (are the bytes a valid instance of this -format?). These are different validators for different inputs. - -### TypeBox interop - -TypeBox's `Type.Module({...})` pattern maps naturally to BAST's -`$defs` structure. A TypeBox module that defines binary types can -serialize to BAST JSON instead of custom-keyword JSON. The codegen -(`ts-to-module.ts`) could target BAST as an output format. - -The relationship is: -- TypeBox → BAST JSON → alktype engine (binary layout) -- TypeBox → standard JSON Schema → jsonschema (JSON validation) - -Same TypeBox source, two output formats, two validators. - -### Not a replacement for JSON Schema - -BAST does not replace JSON Schema for JSON data validation. A BAST -document cannot validate a JSON payload — it describes binary data -layouts and value-domain constraints for bytes. For JSON validation, -consumers use standard JSON Schema documents (which may be derived -from BAST definitions via codegen, or authored separately). The -`validate_json` path on `AlkTypeEngine` accepts a consumer-provided -JSON Schema and uses a standard `jsonschema::Validator` — BAST is not -involved. - -This is the split: `validate_bytes` is BAST-native (the BAST document -is both the layout spec and the validation spec for bytes); -`validate_json` is JSON-Schema-native (a standard JSON Schema is the -validation spec for JSON values). One crate, two validators, two -input types. - -## Codegen (Future) - -BAST enables code generation that the custom-keyword format makes -awkward. A codegen module (feature-gated behind `codegen`) would: - -1. **Input**: A BAST document (or `SchemaRegistry` equivalent) -2. **Walk**: Iterate `$defs` entries, inspect `kind` values -3. **Map**: `"uint32"` → `u32` (Rust), `number` (TypeScript), `int` (Python) -4. **Emit**: Handlebars templates for struct/enum/union definitions - -### Generated artifacts - -| Target | Type definitions | Binary reader/writer | -|--------|-----------------|---------------------| -| Rust | `struct ChunkHeader { channel_id: u32, length: u32 }` | `fn read_header(buf: &[u8]) -> Result` | -| TypeScript | `interface ChunkHeader { channelId: number; length: number }` | `function readHeader(buf: Uint8Array): ChunkHeader` | -| Python | `@dataclass class ChunkHeader: ...` | `def read_header(buf: bytes) -> ChunkHeader` | - -### Relationship to typebox-rs codegen - -The typebox-rs `codegen/` module (in `/workspace/@alkimiadev/typebox-rs`) -is the reference architecture: -- `SchemaRegistry` for named types with `$ref` resolution -- `RustGenerator` / `TypeScriptGenerator` wrapping `Handlebars` templates -- `schema_to_rust_type()` / `schema_to_ts_type()` mapping functions -- Feature-gated behind `codegen = ["handlebars"]` - -alktype's codegen would follow the same pattern but walk BAST `kind` -values instead of `SchemaKind` enum variants. The handlebars-rs -dependency is WASM-compatible. - -### Scope boundary - -Codegen is out of scope for the BAST pivot itself. The pivot changes -the schema format; codegen builds on top of the new format. It is -described here to show that BAST enables it, not to commit to a -specific implementation timeline. - -## ABI Adapter (Future) - -A BAST document describes the binary interface of a protocol — it is -essentially an ABI specification in JSON. This enables: - -- **Version negotiation**: Two peers exchange BAST documents to agree on - a protocol version. The engine can detect mismatches (field added, - type changed, endianness differs) and either reject or adapt. -- **Schema migration**: A consumer with schema v1 can read data written - by schema v2 if the changes are compatible (fields added at the end, - types widened). The engine can compute a migration plan from the diff - of two BAST documents. -- **WASM interop**: A WASM component can export its BAST schema as part - of its WIT interface, enabling host languages to generate - readers/writers for the component's binary protocol without manual - bindings. - -This is a future capability, not part of the pivot. It is mentioned -because BAST makes it possible in a way that custom keywords scattered -through JSON Schema trees do not. - -## Migration Path - -### Phase 1: BAST format and meta-schema (this pivot) - -1. Define the BAST meta-schema (the JSON Schema that validates BAST - documents) -2. ~~**Run the BAST-native validator POC** — implement the validator, - wire it into `validate_bytes`, confirm the existing test suite passes. - This de-risks the validation model before the full pivot.~~ - **Done.** See [POC Result — BAST-native validator](#poc-result--bast-native-validator). - The hypothesis is confirmed; the production refactor (step 5 below) - is now a straightforward port. POC code lives on branch - `bast-validator-poc` (commit `f371fe4`) as a self-contained - `src/bast_poc.rs` reference — deliberately not merged to main (it is - throwaway scaffolding that gets replaced by the production module in - step 5). -3. Implement BAST document parsing in the engine (replace custom keyword - detection with `kind` field parsing) -4. Update `AlkTypeEngine::compile()` to accept a BAST document + root - type name -5. Implement the BAST-native validator (production version, informed - by the POC). Constructs `AlkTypeError::Validation` via - `jsonschema::ValidationError::custom` per [D-BAST-009](#d-bast-009-alktypeerrorvalidation-payload-shape) - — the `Validation` variant's payload type is unchanged. -6. Update `validate_json()` to accept a consumer-provided JSON Schema - and build a standard `jsonschema::Validator` (no custom keywords) -7. Update the builder API to produce BAST JSON (public methods unchanged) -8. Remove custom keyword validators, `normalize_refs()`, - `inline_union_variant_refs()`, and the `get_alktype_kind*` functions -9. Update all tests to use BAST format -10. Update architecture docs (ADRs, schema-layer.md, etc.) - -### Phase 2: Downstream adoption - -1. Port alkcall's chunk header to BAST (replace hand-rolled `wire.rs` - with `AlkTypeEngine` + `SequentialReader`/`LayoutBuilder`) -2. Port alktty's TTY chunk format to BAST -3. Port the SFTP POC schemas to BAST format - -### Phase 3: Codegen (future) - -1. Add `codegen` feature flag with `handlebars` dependency -2. Implement `RustGenerator` and `TypeScriptGenerator` walking BAST - `kind` values -3. External Handlebars templates in `src/codegen/templates/` - ## POC Scope The layout swap needs no POC — it is a backend swap (custom keywords → @@ -1083,12 +111,11 @@ recursive walker over the BAST type tree — fully replace the 19 custom keyword validators on the `validate_bytes` path, including the OQ-008 union variant dispatch, without regression? -**The POC has been run and succeeded.** The remainder of this section -records the original scope and success criteria (kept for the record); -the outcome is in [POC Result — BAST-native validator](#poc-result--bast-native-validator). -The POC code is on branch `bast-validator-poc` (commit `f371fe4`), not -merged to main — it is reference scaffolding superseded by Phase 1 -step 5. +**The POC has been run and succeeded.** The outcome is recorded in +[POC Result](#poc-result--bast-native-validator) below. The POC code is +on branch `bast-validator-poc` (commit `f371fe4`), not merged to main — +it is reference scaffolding superseded by the production module in +[implementation step 5](../plans/bast-implementation.md#step-5--bast-native-validator-production-version). ### POC: BAST-native validator for `validate_bytes` @@ -1144,10 +171,149 @@ checks. straightforward JSON walking; no empirical risk. - Layout computation — unchanged, already proven. +## POC Result — BAST-native validator + +**Status: succeeded.** The POC is on branch `bast-validator-poc` in +`src/bast_poc.rs` (20 tests, all passing; full crate suite — 416 tests — +green; `cargo clippy --all-targets -- -D warnings` clean; +`cargo build --target wasm32-unknown-unknown --release` clean). + +The POC implements the `validate_bytes` validation model from +[D-BAST-006](#d-bast-006-validate_bytes-validation-model) as a +self-contained module that does **not** touch the production schema / +materializer / validator paths. It reuses only `data_access` (read +primitives), `AlkTypeError` (error type), and `Endian`. The BAST +document parser, a packed-mode materializer, and the BAST-native +validator are all implemented from scratch — that is the point: prove +the model works end-to-end before refactoring the production code. + +### What the POC proves + +The hypothesis from the [POC section](#poc-bast-native-validator-for-validate_bytes) +is confirmed: a recursive walker over the BAST type tree fully +replaces the 19 custom keyword validators on the `validate_bytes` path, +including the OQ-008 union variant dispatch, and fixes the +enum-membership dead constraint — all without `jsonschema` custom +keywords and without an external JSON Schema. + +The hard cases that were the actual de-risking targets all pass: + +- **Union byte-offset discriminator + `maxLength` inside a variant + (OQ-008).** `union_byte_disc_max_length_inside_variant_enforced` + materializes a union with two `$ref` variants, dispatches on a + byte-offset `uint8` discriminator, and enforces `maxLength` on a + `bytes` field inside the selected variant. The validator reads + `__discriminator`, looks up the variant's BAST definition, and + recurses — same behavior as the current `UnionValidator`'s + per-variant sub-validators, but with no `jsonschema` involvement. +- **Union field-name discriminator + `maxLength` inside a variant.** + `union_field_disc_max_length_inside_variant_enforced` covers the + typedef.ts-style discriminator (a length-prefixed string field + selects the variant). Same recursion model. +- **Enum index-bounds fix.** `enum_index_out_of_bounds_rejected` + exercises the constraint that is **broken in the current engine** + (the built-in `enum` keyword checks string membership; the + materializer emits `Value::Number(index)`, which never matches — a + dead constraint). The BAST-native validator checks the materialized + index against the `values` array bounds (0..len-1), which is the + correct validation for a binary enum encoded as an index. Net + improvement, not a regression. +- **Nested struct wrapping a union wrapping a struct.** + `nested_struct_with_union_variant` confirms the recursion composes + through multiple type layers. +- **Arrays of fixed-size structs with `count`.** + `array_of_structs_with_count` covers the `Vector3`-style array + (D-BAST-004). +- **Records (count-prefixed string-keyed maps).** + `record_of_uint32` covers the `TRecord` shape. +- **Untrusted schema input.** `malformed_document_produces_schema_error_not_panic` + confirms a malformed BAST document surfaces as `AlkTypeError::Schema`, + not a panic (AGENTS.md §3). +- **Basic cases** (chunk header, int8/uint32 ranges, string/bytes + `maxLength`, timestamp, bool, short buffer) all pass — if a couple of + basic examples work, all of them do, since the validator is a flat + per-kind dispatch with no per-kind special-casing beyond the range + bounds. + +### How the validator works + +The validator is a single recursive function (`validate_typeref`) that +dispatches on the BAST `kind`. Each arm checks the value-domain +constraint for that kind and, for composites, recurses into the child +type definitions. The materializer (also implemented in the POC) +guarantees structural correctness — bounds, UTF-8, bool byte, +discriminator lookup, all fields present — so the validator only +enforces what the materializer cannot. The full constraint table is in +[`bast-format.md` §Validation Model](../architecture/bast-format.md#validation-model). + +### Observations for the production implementation + +1. **No `jsonschema` dependency for `validate_bytes`.** The validator + only needs `serde_json` (for `Value`) and the BAST document. The + `jsonschema` crate is still a direct dependency for `validate_json` + and for validating BAST documents against the BAST meta-schema, but + the `validate_bytes` path no longer touches it. This is a small wasm + binary-size win in addition to the architecture simplification. + +2. **`$ref` resolution is a single hash lookup.** The POC's + `resolve_ref_or_inline` handles only `#/$defs/Name` pointers — the + only form BAST allows. The current engine's `normalize_refs` / + `inline_union_variant_refs` / `resolve_ref_or_inline` machinery for + bare-name refs and inlined union variants is no longer needed: BAST + `$ref`s are always full JSON Pointers, and union variant refs are + resolved lazily by the validator (the materializer already does this + for the read path). The `inline_union_variant_refs` compile step can + be removed entirely. + +3. **The validator is ~250 lines.** The 19 custom keyword validators + (`src/validation.rs`) plus the macro definitions are ~500 lines and + require the `jsonschema::Keyword` trait plumbing (factory closures, + `Box`, sub-validator construction at factory time). The + BAST-native validator is a flat match — no factories, no trait + objects, no sub-validator pre-computation. The recursion is direct. + +4. **The `AlkTypeError::Validation` variant still wraps + `jsonschema::ValidationError<'static>`.** The POC uses + `jsonschema::ValidationError::custom` to construct these so the + error type is unchanged. This is now the decided shape for the + production refactor — see [D-BAST-009](#d-bast-009-alktypeerrorvalidation-payload-shape). + The rationale is consumer ergonomics: a single uniform payload type + means one match arm covers both `validate_json` and `validate_bytes` + errors downstream, and `validate_json`'s structured errors are worth + preserving rather than flattening to a `String`. + +5. **The materializer and validator share the BAST-walking code + structure.** Both walk the same `kind`/`fields`/`mapping` tree. The + production refactor could share a typed BAST tree (a small + `BastNode` enum) between them so the walk is parsed once. The POC + parses lazily from the raw JSON in both passes to keep the model + honest; a typed tree is a straightforward follow-on optimization, not + a risk. + +### Verdict + +The "how do we reproduce the same behavior?" question is answered: +walk the BAST tree the same way the materializer does, checking the +same value-domain constraints the custom keyword validators check +today. The model is a strict simplification — fewer moving parts, no +`jsonschema` integration on the bytes path, no compile-time +`inline_union_variant_refs` step, no factory closures or trait +objects, and the enum dead-constraint is fixed as a side effect. + +The POC does not wire into `AlkTypeEngine::validate_bytes` — that is +the production refactor ([implementation step 5](../plans/bast-implementation.md#step-5--bast-native-validator-production-version)), +which replaces `validation::build_validator` usage on the bytes path +with the BAST-native validator. The POC's job was to de-risk the model +before that refactor; that job is done. + ## Decisions The following were open questions in earlier drafts. Each is now -resolved. They are recorded here as decisions, not re-litigated. +resolved. They are recorded here as decisions, not re-litigated. The +normative format specification that realizes these decisions is in +[`bast-format.md`](../architecture/bast-format.md); the semver +classification of each is in the +[implementation plan's Semver Contract](../plans/bast-implementation.md#semver-contract). ### D-BAST-001: Root type selection @@ -1180,10 +346,10 @@ no home for additional definitions. **Decision:** Arrays of variable-length elements are **not supported in v1**. The meta-schema requires `count` on all array types, making -arrays fixed-size only. The no-count example has been removed. This -matches the engine's current behavior (it rejects arrays of -variable-length elements) and aligns with OQ-001 (deferred, blocked on -a concrete consumer needing interleaved variable-stride arrays). +arrays fixed-size only. This matches the engine's current behavior (it +rejects arrays of variable-length elements) and aligns with OQ-001 +(deferred, blocked on a concrete consumer needing interleaved +variable-stride arrays). Variable-length collections are still available via `record` (a count-prefixed string-keyed map), which the engine supports. If a @@ -1198,8 +364,8 @@ array on `UnionDef`. When `discriminator.kind == "field"`, the `fields` array provides the union's field definitions (including the discriminator field). When `discriminator.kind == "byte"`, `fields` is absent — the union's layout is purely the variant layout. This -preserves a feature the engine already supports. The meta-schema -update is in [the meta-schema sketch above](#meta-schema-sketch). +preserves a feature the engine already supports. The meta-schema is in +[`bast-format.md` §The Meta-Schema](../architecture/bast-format.md#the-meta-schema). ### D-BAST-006: `validate_bytes` validation model @@ -1209,8 +375,8 @@ constraints on the materialized `Value` — no external JSON Schema needed. This recovers the OQ-008 union variant dispatch behavior (the validator recurses into the variant's BAST definition) and fixes the enum-membership dead constraint (the validator checks the materialized -index against the `values` array bounds). See [The Validator -Split](#the-validator-split) and the [POC](#poc-bast-native-validator-for-validate_bytes). +index against the `values` array bounds). See [`bast-format.md` §Validation +Model](../architecture/bast-format.md#validation-model) and the [POC](#poc-bast-native-validator-for-validate_bytes). An optional external JSON Schema can be layered on top for constraints BAST doesn't express (cross-field consistency, regex patterns on string @@ -1298,8 +464,9 @@ bearing on the other validation path. The right place to revisit this is when/if OQ-002 is actually pursued, not preemptively. The POC already used option 1 (via `ValidationError::custom`); the -production refactor (Phase 1 step 5) follows the same construction -pattern. No semver-relevant change to the `Validation` variant. +production refactor ([implementation step 5](../plans/bast-implementation.md#step-5--bast-native-validator-production-version)) +follows the same construction pattern. No semver-relevant change to the +`Validation` variant. ## Risks and Mitigations @@ -1314,10 +481,45 @@ pattern. No semver-relevant change to the `Validation` variant. | Builder API output format change breaks consumers | No real consumers exist yet (v0.1.0 has zero adoption). The builder's public methods are unchanged; only the JSON output format changes. | | Meta-schema maintenance burden | The meta-schema is small and changes rarely. It's embedded in the crate and published at a stable URL. | +## Future Directions + +These are enabled by BAST but out of scope for the pivot itself. They +are mentioned to show that BAST makes them possible, not to commit to a +specific implementation timeline. + +### Codegen + +BAST enables code generation that the custom-keyword format makes +awkward. A codegen module (feature-gated behind `codegen`) would walk +`$defs` entries, inspect `kind` values, and emit Rust/TypeScript/Python +readers and writers via Handlebars templates. The typebox-rs `codegen/` +module (`/workspace/@alkimiadev/typebox-rs`) is the reference +architecture: `SchemaRegistry` for named types, `RustGenerator`/ +`TypeScriptGenerator` wrapping `Handlebars`, `schema_to_rust_type()`/ +`schema_to_ts_type()` mapping functions. alktype's codegen would follow +the same pattern but walk BAST `kind` values. The handlebars-rs +dependency is WASM-compatible. The pivot changes the schema format; +codegen builds on top of the new format. + +### ABI Adapter + +A BAST document describes the binary interface of a protocol — it is +essentially an ABI specification in JSON. This enables version +negotiation (two peers exchange BAST documents to agree on a protocol +version; the engine detects mismatches and either rejects or adapts), +schema migration (a consumer with schema v1 can read data written by +schema v2 if the changes are compatible), and WASM interop (a WASM +component can export its BAST schema as part of its WIT interface). + ## References -- [ADR-001](../architecture/decisions/001-alktype-purpose-scope-jsonschema-engine.md) — current "schema is the format" principle (to be updated) -- [ADR-003](../architecture/decisions/003-schema-annotations.md) — annotation shapes (endianness, alignment, encoding, discriminators — carry forward to BAST) +- [`docs/architecture/bast-format.md`](../architecture/bast-format.md) — + the normative BAST format spec (meta-schema, TypeRef, examples, + validation model) +- [`docs/plans/bast-implementation.md`](../plans/bast-implementation.md) — + the execution plan (ordered steps, semver contract, ADR-sync checklist) +- [ADR-001](../architecture/decisions/001-alktype-purpose-scope-jsonschema-engine.md) — current "schema is the format" principle (to be superseded by ADR-BAST post-implementation) +- [ADR-003](../architecture/decisions/003-schema-annotations.md) — annotation semantics (carry forward to BAST unchanged) - [ADR-009](../architecture/decisions/009-builder-api.md) — builder API (public surface unchanged, output format changes) - [ADR-010](../architecture/decisions/010-generalized-validation-validate-bytes.md) — `validate_bytes` (unchanged in concept) - `/workspace/research/typebox_research/ujsx/jpath.gen.ts` — TypeBox `Type.Module` pattern (the `$defs`/`$ref` model BAST follows) @@ -1325,153 +527,4 @@ pattern. No semver-relevant change to the `Validation` variant. - `/workspace/research/typebox_research/codegen/ts-to-module.ts` — TypeScript-to-TypeBox codegen (reference for future BAST codegen) - `/workspace/@alkimiadev/typebox-rs/src/codegen/` — Rust/TypeScript codegen from schemas (reference architecture) - `/workspace/alknet-typedef-poc/tests/sftp_roundtrip_test.rs` — SFTP POC proving byte-identical output (to be replicated with BAST) -- `/workspace/@alkdev/alkcall/src/channels/wire.rs` — hand-rolled chunk header (target for BAST replacement) - -## POC Result — BAST-native validator - -**Status: succeeded.** The POC is on branch `bast-validator-poc` in -`src/bast_poc.rs` (20 tests, all passing; full crate suite — 416 tests — -green; `cargo clippy --all-targets -- -D warnings` clean; -`cargo build --target wasm32-unknown-unknown --release` clean). - -The POC implements the `validate_bytes` validation model from -[D-BAST-006](#d-bast-006-validate_bytes-validation-model) as a -self-contained module that does **not** touch the production schema / -materializer / validator paths. It reuses only `data_access` (read -primitives), `AlkTypeError` (error type), and `Endian`. The BAST -document parser, a packed-mode materializer, and the BAST-native -validator are all implemented from scratch — that is the point: prove -the model works end-to-end before refactoring the production code. - -### What the POC proves - -The hypothesis from the [POC section](#poc-bast-native-validator-for-validate_bytes) -is confirmed: a recursive walker over the BAST type tree fully -replaces the 19 custom keyword validators on the `validate_bytes` path, -including the OQ-008 union variant dispatch, and fixes the -enum-membership dead constraint — all without `jsonschema` custom -keywords and without an external JSON Schema. - -The hard cases that were the actual de-risking targets all pass: - -- **Union byte-offset discriminator + `maxLength` inside a variant - (OQ-008).** `union_byte_disc_max_length_inside_variant_enforced` - materializes a union with two `$ref` variants, dispatches on a - byte-offset `uint8` discriminator, and enforces `maxLength` on a - `bytes` field inside the selected variant. The validator reads - `__discriminator`, looks up the variant's BAST definition, and - recurses — same behavior as the current `UnionValidator`'s - per-variant sub-validators, but with no `jsonschema` involvement. -- **Union field-name discriminator + `maxLength` inside a variant.** - `union_field_disc_max_length_inside_variant_enforced` covers the - typedef.ts-style discriminator (a length-prefixed string field - selects the variant). Same recursion model. -- **Enum index-bounds fix.** `enum_index_out_of_bounds_rejected` - exercises the constraint that is **broken in the current engine** - (the built-in `enum` keyword checks string membership; the - materializer emits `Value::Number(index)`, which never matches — a - dead constraint). The BAST-native validator checks the materialized - index against the `values` array bounds (0..len-1), which is the - correct validation for a binary enum encoded as an index. Net - improvement, not a regression. -- **Nested struct wrapping a union wrapping a struct.** - `nested_struct_with_union_variant` confirms the recursion composes - through multiple type layers. -- **Arrays of fixed-size structs with `count`.** - `array_of_structs_with_count` covers the `Vector3`-style array - (D-BAST-004). -- **Records (count-prefixed string-keyed maps).** - `record_of_uint32` covers the `TRecord` shape. -- **Untrusted schema input.** `malformed_document_produces_schema_error_not_panic` - confirms a malformed BAST document surfaces as `AlkTypeError::Schema`, - not a panic (AGENTS.md §3). -- **Basic cases** (chunk header, int8/uint32 ranges, string/bytes - `maxLength`, timestamp, bool, short buffer) all pass — if a couple of - basic examples work, all of them do, since the validator is a flat - per-kind dispatch with no per-kind special-casing beyond the range - bounds. - -### How the validator works - -The validator is a single recursive function -(`validate_typeref`) that dispatches on the BAST `kind`. Each arm -checks the value-domain constraint for that kind and, for composites, -recurses into the child type definitions. The materializer (also -implemented in the POC) guarantees structural correctness — bounds, -UTF-8, bool byte, discriminator lookup, all fields present — so the -validator only enforces what the materializer cannot: - -| Constraint | Validator arm | -|------------|---------------| -| Integer range (Int8..Uint64) | `validate_int` / `validate_uint` with `as_i64`/`as_u64` + range check | -| Int64/Uint64 (full range) | `validate_int64` / `validate_uint64` (JSON precision caveat per ADR-005) | -| Float finiteness (Float32/64) | `validate_float` with `as_f64().is_finite()` | -| String `maxLength` (byte length) | `check_string` reads the field-level `maxLength` annotation | -| Bytes `maxLength` (array length) | `check_bytes` accepts both `Value::String` and `Value::Array` forms | -| RFC 3339 timestamp shape | `validate_timestamp` reuses the same non-strict check as the current engine | -| Enum index bounds | `validate_enum` checks `idx < values.len()` — the dead-constraint fix | -| Union variant dispatch | `validate_union` reads `__discriminator`, looks up the variant, recurses via `validate_typeref` | -| Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses | -| Array count | `validate_array` checks `arr.len() == count` and recurses per element | -| Record values | `validate_record` recurses into each value's `values` type | -| Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) | - -### Observations for the production implementation - -1. **No `jsonschema` dependency for `validate_bytes`.** The validator - only needs `serde_json` (for `Value`) and the BAST document. The - `jsonschema` crate is still a direct dependency for `validate_json` - and for validating BAST documents against the BAST meta-schema, but - the `validate_bytes` path no longer touches it. This is a small wasm - binary-size win in addition to the architecture simplification. - -2. **`$ref` resolution is a single hash lookup.** The POC's - `resolve_ref_or_inline` handles only `#/$defs/Name` pointers — the - only form BAST allows. The current engine's `normalize_refs` / - `inline_union_variant_refs` / `resolve_ref_or_inline` machinery for - bare-name refs and inlined union variants is no longer needed: BAST - `$ref`s are always full JSON Pointers, and union variant refs are - resolved lazily by the validator (the materializer already does this - for the read path). The `inline_union_variant_refs` compile step can - be removed entirely. - -3. **The validator is ~250 lines.** The 19 custom keyword validators - (`src/validation.rs`) plus the macro definitions are ~500 lines and - require the `jsonschema::Keyword` trait plumbing (factory closures, - `Box`, sub-validator construction at factory time). The - BAST-native validator is a flat match — no factories, no trait - objects, no sub-validator pre-computation. The recursion is direct. - -4. **The `AlkTypeError::Validation` variant still wraps - `jsonschema::ValidationError<'static>`.** The POC uses - `jsonschema::ValidationError::custom` to construct these so the - error type is unchanged. This is now the decided shape for the - production refactor — see [D-BAST-009](#d-bast-009-alktypeerrorvalidation-payload-shape). - The rationale is consumer ergonomics: a single uniform payload type - means one match arm covers both `validate_json` and `validate_bytes` - errors downstream, and `validate_json`'s structured errors are worth - preserving rather than flattening to a `String`. - -5. **The materializer and validator share the BAST-walking code - structure.** Both walk the same `kind`/`fields`/`mapping` tree. The - production refactor could share a typed BAST tree (a small - `BastNode` enum) between them so the walk is parsed once. The POC - parses lazily from the raw JSON in both passes to keep the model - honest; a typed tree is a straightforward follow-on optimization, not - a risk. - -### Verdict - -The "how do we reproduce the same behavior?" question is answered: -walk the BAST tree the same way the materializer does, checking the -same value-domain constraints the custom keyword validators check -today. The model is a strict simplification — fewer moving parts, no -`jsonschema` integration on the bytes path, no compile-time -`inline_union_variant_refs` step, no factory closures or trait -objects, and the enum dead-constraint is fixed as a side effect. - -The POC does not wire into `AlkTypeEngine::validate_bytes` — that is -the production refactor (Phase 1 step 5 in the Migration Path), which -replaces `validation::build_validator` usage on the bytes path with -the BAST-native validator. The POC's job was to de-risk the model -before that refactor; that job is done. +- `/workspace/@alkdev/alkcall/src/channels/wire.rs` — hand-rolled chunk header (target for BAST replacement) \ No newline at end of file