From 19f8162f1a1c9c13cd3b128b3b94a0973fdca497 Mon Sep 17 00:00:00 2001 From: "glm-5.2" Date: Sat, 15 Aug 2026 09:05:55 +0000 Subject: [PATCH] Refine BAST pivot: validation model, POC scope, resolve open questions - Rewrite Validator Split around BAST-native validator for validate_bytes (walks BAST, checks value-domain constraints, no external JSON Schema needed); validate_json uses standard jsonschema::Validator from consumer-provided JSON Schema - Add targeted POC: BAST-native validator replacing 19 custom keyword validators on the bytes path, verified via existing test suite - Fix meta-schema: require count on arrays (variable-element arrays deferred per OQ-001), add optional fields array to UnionDef for field-name discriminators - Remove lying no-count array example, replace with deferred note - Document dead enum constraint (materialized index never matches string-membered enum); BAST-native validator fixes it via index bounds check - Resolve all 7 OQs + 3 spec gaps as D-BAST-001 through D-BAST-008 - Update Migration Path, Risks table, Engine internals to reflect BAST-native validator - Clarify 'no pocs needed' was an overcorrection: layout swap needs no POC, but the validation model does Verification: docs-only change, no code affected --- docs/research/bast-pivot.md | 572 +++++++++++++++++++++++------------- 1 file changed, 363 insertions(+), 209 deletions(-) diff --git a/docs/research/bast-pivot.md b/docs/research/bast-pivot.md index eb662e8..9538ef6 100644 --- a/docs/research/bast-pivot.md +++ b/docs/research/bast-pivot.md @@ -260,20 +260,12 @@ adoption creates migration cost. } ``` -#### Array of variable-length elements (count-prefixed) +#### Array of variable-length elements (deferred — see Decisions) -```json -{ - "$defs": { - "StringList": { - "kind": "struct", - "fields": [ - { "name": "items", "kind": { "kind": "array", "element": "string" } } - ] - } - } -} -``` +Arrays of variable-length elements (e.g., `{ "kind": "array", "element": "string" }` +without a `count`) are **not supported in v1**. The engine rejects them today +(OQ-001), and the meta-schema requires `count` on all array types. This is a +known limitation, not a gap — see [D-BAST-004](#d-bast-004-arrays-of-variable-length-elements-deferred). #### Record (string-keyed map) @@ -393,14 +385,14 @@ offline use. "additionalProperties": false }, { - "description": "Array type", + "description": "Array type (fixed-size only in v1 — count is required)", "type": "object", "properties": { "kind": { "const": "array" }, "element": { "$ref": "#/$defs/TypeRef" }, "count": { "type": "integer", "minimum": 0 } }, - "required": ["kind", "element"], + "required": ["kind", "element", "count"], "additionalProperties": false }, { @@ -420,6 +412,10 @@ offline use. "properties": { "kind": { "const": "union" }, "endian": { "enum": ["little", "big"] }, + "fields": { + "type": "array", + "items": { "$ref": "#/$defs/FieldDef" } + }, "discriminator": { "oneOf": [ { @@ -578,10 +574,11 @@ over unchanged. | `normalize_refs()` | Removed — BAST `$ref` values are always full JSON Pointers | | `inline_union_variant_refs()` | Removed — union mapping values are resolved lazily via `$ref` | | `parse_endian()`, `parse_align()`, `parse_encoding()`, `parse_discriminator()` | Adapted to read from BAST field/struct properties instead of keyword-value objects | -| Custom keyword validators (19 `jsonschema::Keyword` impls) | Removed. These were the only thing using `jsonschema`'s custom keyword API. | +| Custom keyword validators (19 `jsonschema::Keyword` impls) | Removed. Replaced by the BAST-native validator (see below). | | `build_validator()` | Repurposed. Still builds a `jsonschema::Validator`, but from a standard JSON Schema (no custom keywords). Used for JSON payload validation (call's `OperationSpec` schemas, etc.) and for validating BAST documents against the BAST meta-schema. | -| `validate_json()` | Unchanged in signature. Validates a `Value` against the engine's compiled `jsonschema::Validator` (now built from a standard JSON Schema instead of one with custom keywords). | -| `validate_bytes()` | Unchanged in concept — materialize `Value` from bytes, then validate. The materialization step walks BAST instead of custom-keyword JSON. | +| BAST-native validator | **New.** A recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`: integer ranges, float finiteness, `maxLength`, timestamp shape, enum index bounds, union variant dispatch. Replaces the 19 custom keyword validators for the `validate_bytes` path. See [The Validator Split](#the-validator-split). | +| `validate_json()` | Unchanged in signature. Validates a `Value` against a `jsonschema::Validator` compiled from a standard JSON Schema the consumer provides. No longer uses custom keywords. | +| `validate_bytes()` | Unchanged in concept — materialize `Value` from bytes, then validate. The validation step uses the BAST-native validator (not `jsonschema`) to check value-domain constraints from the BAST document. | ### Public API @@ -589,7 +586,7 @@ over unchanged. |------|--------| | `AlkTypeKind` enum | Unchanged — same 19 variants, same methods | | `Endian`, `VariableEncoding`, `DiscriminatorKind` | Unchanged | -| `AlkTypeEngine` | `compile()` takes a BAST document + root type name instead of a custom-keyword JSON Schema. `validate_json()` may change. `validate_bytes()` unchanged. | +| `AlkTypeEngine` | `compile()` takes a BAST document + root type name instead of a custom-keyword JSON Schema. `validate_json()` validates against a consumer-provided JSON Schema. `validate_bytes()` validates against the BAST document via the BAST-native validator. | | `LayoutMode`, `OffsetMap`, `ByteRange` | Unchanged | | `LayoutBuilder`, `PackedLayout`, `FieldPosition` | Unchanged | | `SequentialReader`, `FieldValue` | Unchanged | @@ -613,105 +610,215 @@ over unchanged. - BAST meta-schema (embedded in the crate, published at a stable URL) - BAST document parser — validates a BAST document against the meta-schema, then extracts type definitions - `AlkTypeEngine::compile()` takes `(bast_document: &Value, root_name: &str, mode: LayoutMode)` — the root name selects which `$defs` entry is the top-level type +- BAST-native validator — a recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`. Replaces the 19 custom keyword validators for the `validate_bytes` path. Enforces: integer ranges, float finiteness, string/bytes `maxLength`, RFC 3339 timestamp shape, enum index bounds, union variant dispatch (recursing into variants). See [The Validator Split](#the-validator-split). - BAST meta-schema validation at compile time (optional but recommended — the engine can skip it and trust the caller, or validate as a guard) ## The Validator Split A key architectural clarification: BAST separates two concerns that the -current format conflates. +current format conflates, and in doing so reveals that the current +engine has **two distinct validation paths** that custom keywords +paper over with a single mechanism. ### Current model (conflated) ``` ┌─────────────────────────────────────────────┐ │ JSON Schema with AlkType:* custom keywords │ -│ ┌───────────────────────────────────────┐ │ +│ ┌───────────────────────────────┐ │ │ │ Binary layout info (AlkType:Uint32) │ │ │ │ JSON validation info (type, enum) │ │ -│ └───────────────────────────────────────┘ │ +│ └───────────────────────────────┘ │ └─────────────────────────────────────────────┘ │ ▼ AlkTypeEngine::compile() │ ├──► Layout (offset map / layout builder) - └──► Validator (jsonschema with custom keywords) + └──► Validator (jsonschema with 19 custom keywords) + │ + ├──► validate_json(&Value) — JSON value validation + └──► validate_bytes(&[u8]) — materialize, then validate ``` One document serves two roles. The engine extracts both layout and -validation from the same JSON tree. +validation from the same JSON tree. A single `jsonschema::Validator` +(with custom keywords) serves both `validate_json` and `validate_bytes`. -### BAST model (separated) +### BAST model (three layers, two validators) + +The current engine's two validation paths have different needs, and +BAST makes the split explicit: ``` -┌──────────────────────┐ ┌──────────────────────────┐ -│ BAST document │ │ JSON Schema document │ -│ (binary layout) │ │ (JSON validation) │ -│ │ │ │ -│ kind: "struct" │ │ type: "object" │ -│ fields: [ │ │ properties: { │ -│ { name, kind } │ │ id: { type: "integer"} │ -│ ] │ │ } │ -│ endian: "big" │ │ required: ["id"] │ -└──────────┬───────────┘ └────────────┬─────────────┘ - │ │ - ▼ ▼ - AlkTypeEngine::compile() build_validator(&json_schema) - │ │ - ▼ ▼ - Layout (offset map / jsonschema::Validator - layout builder) (standard JSON Schema) - │ │ - ▼ ▼ - validate_bytes(&[u8]) validate_json(&Value) + ┌──────────────────────┐ + │ BAST document │ + │ (binary layout + │ + │ value constraints) │ + └──────────┬───────────┘ + │ + ▼ + AlkTypeEngine::compile() + │ + ┌────────────────┼────────────────┐ + ▼ ▼ + ┌─────────────────┐ ┌──────────────────┐ + │ Layout │ │ BAST-native │ + │ (offset map / │ │ validator │ + │ layout builder)│ │ (walks BAST, │ + └────────┬────────┘ │ checks value │ + │ │ constraints) │ + ▼ └────────┬──────────┘ + ┌─────────────────┐ │ + │ Materializer │ │ + │ (bytes → Value, │ │ + │ guarantees │ │ + │ structure) │ │ + └────────┬────────┘ │ + │ │ + ▔──────────────┬─────────────────────┘ + ▼ + validate_bytes(&[u8]) + (materialize, then check + value constraints against + the BAST schema — no external + JSON Schema needed) + + ┌──────────────────────────┐ + │ JSON Schema document │ (separate concern) + │ (JSON validation) │ + │ type: "object" │ + │ properties: { │ + │ id: { type: "integer"}│ + │ } │ + │ required: ["id"] │ + └────────────┬─────────────┘ + │ + ▼ + build_validator(&json_schema) + │ + ▼ + jsonschema::Validator + (standard JSON Schema, + no custom keywords) + │ + ▼ + validate_json(&Value) + (validates consumer-supplied + JSON against a standard + JSON Schema — BAST not + involved) ``` -Two documents, two validators, two concerns. The BAST document -describes binary layout. The JSON Schema document describes JSON data -shape. They can be linked (a BAST struct can reference a JSON Schema by -`$id` for validation) but they are separate documents. +### Why two validators, not one -Both paths use `jsonschema` under the hood — the BAST path uses it to -validate BAST documents against the BAST meta-schema at compile time; -the JSON path uses it to validate JSON payloads against standard JSON -Schema documents. The only thing removed is the custom keyword -registration path (`jsonschema::options().with_keyword(...)`). +The two paths have fundamentally different inputs and guarantees: + +**`validate_bytes(&[u8])` — bytes in, BAST is the validator.** +The materializer produces a `Value` tree from bytes. By construction, +this `Value` is *structurally correct*: all declared fields are present +(because the materializer iterates the field list), types are correct +(because `read_u32` produces a `Value::Number`), bounds are checked +(via `data_access::check_bounds`), UTF-8 is valid (via `from_utf8`), +the discriminator is in the mapping, and the boolean byte is 0 or 1. +What the materializer does NOT check — and what the 19 custom keyword +validators currently check afterward — are **value-domain constraints +expressed in the schema**: + +| Constraint | Current enforcer | BAST-native validator | +|------------|------------------|----------------------| +| Integer range (Int8..Uint64) | Custom keyword | BAST `kind` + range check | +| Float finiteness (Float32/64) | Custom keyword | BAST `kind` + `is_finite()` | +| String `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property | +| Bytes `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property | +| RFC 3339 timestamp shape | Custom keyword | BAST `kind: "timestamp"` + shape check | +| Union variant dispatch | `UnionValidator` (OQ-008) | BAST union — recurse into variant | +| Enum value membership | Built-in `enum` keyword (**broken on bytes path** — see below) | BAST `kind: "enum"` + index-in-range | + +All of these are expressible directly from the BAST document — no +external JSON Schema needed. The BAST-native validator is a recursive +walker over the BAST type tree, checking the materialized `Value` +against each type's constraints. Union dispatch is just recursion: +read `__discriminator`, look up the variant's BAST definition, recurse. + +This **recovers the OQ-008 union dispatch behavior** without custom +keywords and without requiring the consumer to hand-author +discriminator dispatch in a separate JSON Schema. The BAST document +already knows the union's variants and their fields. + +**`validate_json(&Value)` — JSON in, JSON Schema is the validator.** +The consumer provides a JSON `Value` (e.g., an incoming JSON-RPC request +on alkcall's channel 0). The BAST document is irrelevant — BAST +describes bytes, not JSON shape. The right validator for a JSON value +is a standard `jsonschema::Validator` built from a standard JSON +Schema document (which the consumer provides, or which is generated +from BAST via future codegen). This is the path alkcall uses today for +its `OperationSpec` JSON validation. + +### Bug fix: enum validation on the bytes path + +The current engine has a **dead constraint**: when `validate_bytes` +materializes an `AlkType:Enum` field, it reads a raw `u32` index and +emits `Value::Number(index)`. The built-in `enum` keyword lists string +members (e.g., `["Ok", "Error"]`). A numeric index never matches a +string-membered `enum`, so enum membership is silently unenforced on +the bytes path. The BAST-native validator fixes this: for `kind: "enum"`, +it checks that the materialized index is within the `values` array +bounds (0..len-1), which is the correct validation for a binary enum +encoded as an index. ### What this means for consumers **alkcall** today uses alktype for two roles: -1. Binary layout (channels chunk header) — `AlkTypeEngine::compile()` in - packed mode +1. Binary layout (channels chunk header) — `AlkTypeEngine::compile()` + in packed mode 2. JSON validation (call's `OperationSpec` schemas) — `build_validator()` or `jsonschema` directly Under BAST: -1. Binary layout — `AlkTypeEngine::compile(bast_doc, "ChunkHeader", - Packed)` — same flow, different input format -2. JSON validation — `build_validator(&json_schema)` — same API, - now builds a standard `jsonschema::Validator` (no custom keywords). - Consumers that don't use binary at all (e.g., adapters around remote - JSON Schema APIs) use this path exclusively. +1. Binary layout + validation — `AlkTypeEngine::compile(bast_doc, + "ChunkHeader", Packed)` then `engine.validate_bytes(&buf)`. The + engine uses the BAST-native validator — no external JSON Schema + needed for binary data. All value constraints (`maxLength`, ranges, + enum bounds, union dispatch) are enforced from the BAST document. +2. JSON validation — `build_validator(&json_schema)` — builds a + standard `jsonschema::Validator` from a standard JSON Schema + document (no custom keywords). Consumers that don't use binary at + all (e.g., adapters around remote JSON Schema APIs) use this path + exclusively. -The builder API can produce both formats: -- `Schema::struct_().field(...).build()` → BAST JSON -- `Schema::object().field(...).build()` → standard JSON Schema +The builder API produces both formats: +- `Schema::struct_().field(...).build()` → BAST JSON (binary layout) +- `Schema::object().field(...).build()` → standard JSON Schema (JSON + validation) + +Both live in the same crate, same wasm target, same builder surface. +This is the point of alktype: one small wasm-compatible codebase that +handles both binary layout and JSON validation for protocol crates like +alkcall, alktty, and future channel implementations. ### Validation of binary data -`validate_bytes()` still works: materialize a `Value` tree from binary -bytes using the BAST layout, then validate that `Value` against a JSON -Schema. The JSON Schema can be: -- Derived from the BAST definition (a codegen step, future) -- Provided separately by the consumer -- A standard JSON Schema that the consumer already has for JSON - validation of the same logical type +`validate_bytes()` works as follows: +1. **Materialize** a `Value` tree from binary bytes using the BAST + layout. The materializer guarantees structural correctness (bounds, + UTF-8, bool byte, discriminator lookup, all fields present). +2. **Validate** the materialized `Value` against value-domain + constraints from the BAST document, using the BAST-native + validator. This checks integer ranges, float finiteness, `maxLength`, + timestamp shape, enum index bounds, and recurses into union variants. -The materialization step walks the BAST layout (unchanged from current -behavior — it walks the schema to compute offsets and read fields). The -validation step uses a standard `jsonschema::Validator` (no custom -keywords needed — the `Value` tree is already typed by the -materialization). +No external JSON Schema is required for `validate_bytes`. The BAST +document is the complete specification of the binary format — it +describes both the layout (how to read) and the constraints (what +values are valid). This is the "schema is the format" principle from +ADR-001, now fully realized: the BAST document is the binary format, +and it is also the validation spec. + +An optional external JSON Schema can be layered on top for constraints +BAST doesn't express (e.g., cross-field consistency, regex patterns on +string content). This is additive, not load-bearing — the binary path +works without it. ## Relationship to JSON Schema and TypeBox @@ -728,6 +835,11 @@ This means the entire JSON Schema tooling ecosystem works with BAST: - Documentation: JSON Schema generators can produce human-readable docs from the meta-schema +Note: the BAST meta-schema validates the *structure* of a BAST document +(is it well-formed?). The BAST-native validator validates *binary data* +against the BAST document (are the bytes a valid instance of this +format?). These are different validators for different inputs. + ### TypeBox interop TypeBox's `Type.Module({...})` pattern maps naturally to BAST's @@ -744,10 +856,19 @@ Same TypeBox source, two output formats, two validators. ### Not a replacement for JSON Schema BAST does not replace JSON Schema for JSON data validation. A BAST -document cannot validate a JSON payload. It describes binary data -layouts. For JSON validation, consumers use standard JSON Schema -documents (which may be derived from BAST definitions via codegen, or -authored separately). +document cannot validate a JSON payload — it describes binary data +layouts and value-domain constraints for bytes. For JSON validation, +consumers use standard JSON Schema documents (which may be derived +from BAST definitions via codegen, or authored separately). The +`validate_json` path on `AlkTypeEngine` accepts a consumer-provided +JSON Schema and uses a standard `jsonschema::Validator` — BAST is not +involved. + +This is the split: `validate_bytes` is BAST-native (the BAST document +is both the layout spec and the validation spec for bytes); +`validate_json` is JSON-Schema-native (a standard JSON Schema is the +validation spec for JSON values). One crate, two validators, two +input types. ## Codegen (Future) @@ -814,15 +935,22 @@ through JSON Schema trees do not. 1. Define the BAST meta-schema (the JSON Schema that validates BAST documents) -2. Implement BAST document parsing in the engine (replace custom keyword +2. **Run the BAST-native validator POC** — implement the validator, + wire it into `validate_bytes`, confirm the existing test suite passes. + This de-risks the validation model before the full pivot. +3. Implement BAST document parsing in the engine (replace custom keyword detection with `kind` field parsing) -3. Update `AlkTypeEngine::compile()` to accept a BAST document + root +4. Update `AlkTypeEngine::compile()` to accept a BAST document + root type name -4. Update the builder API to produce BAST JSON (public methods unchanged) -5. Remove custom keyword validators, `normalize_refs()`, +5. Implement the BAST-native validator (production version, informed + by the POC) +6. Update `validate_json()` to accept a consumer-provided JSON Schema + and build a standard `jsonschema::Validator` (no custom keywords) +7. Update the builder API to produce BAST JSON (public methods unchanged) +8. Remove custom keyword validators, `normalize_refs()`, `inline_union_variant_refs()`, and the `get_alktype_kind*` functions -6. Update all tests to use BAST format -7. Update architecture docs (ADRs, schema-layer.md, etc.) +9. Update all tests to use BAST format +10. Update architecture docs (ADRs, schema-layer.md, etc.) ### Phase 2: Downstream adoption @@ -838,156 +966,179 @@ through JSON Schema trees do not. `kind` values 3. External Handlebars templates in `src/codegen/templates/` -## Spec Gaps +## POC Scope -No POCs are needed for this pivot. It is a backend swap (custom -keywords → `kind`-based format) on top of a proven layout engine, not -new protocol invention. The layout engine's byte-identity is already -proven (alknet-typedef-poc, alktype-builder-poc) and the layout code is -unchanged, so there is nothing empirical left to de-risk. What remains -are three spec gaps where the draft promises more than the engine -delivers, or expresses less than the engine supports. Each is a -decision, not an unknown. +The layout swap needs no POC — it is a backend swap (custom keywords → +`kind`-based format) on top of a proven layout engine. The layout +engine's byte-identity is already proven (alknet-typedef-poc, +alktype-builder-poc) and the layout code is unchanged, so there is +nothing empirical to de-risk there. -### Gap 1: `validate_bytes` semantics after keyword validator removal +One targeted POC **is** needed to de-risk the validation model. The +risk is specific and falsifiable: can a BAST-native validator — a +recursive walker over the BAST type tree — fully replace the 19 custom +keyword validators on the `validate_bytes` path, including the OQ-008 +union variant dispatch, without regression? -The current engine's `UnionValidator` dispatches to variant schemas at -validation time (OQ-008), so `validate_bytes` on a union checks variant -field constraints (e.g. `maxLength` on a `Bytes` field inside a -variant). Removing the 19 custom keyword validators deletes this -dispatch. The spec must state what `validate_bytes` validates: +### POC: BAST-native validator for `validate_bytes` -- Structural readability only (bounds, discriminator lookup, UTF-8) — - constraint checks move to a separately-provided JSON Schema -- Full constraint validation via a JSON Schema the consumer provides - (OQ-BAST-006's leaning) — but variant constraints then require the - consumer to hand-author `__discriminator` dispatch, a regression - from today's behavior +**Hypothesis:** A recursive walker over the BAST type tree can enforce +all value-domain constraints that the 19 custom keyword validators +currently enforce, recovering the OQ-008 union variant dispatch +behavior, and fixing the enum-membership dead constraint on the bytes +path — all without `jsonschema` custom keywords and without requiring +the consumer to provide an external JSON Schema. -### Gap 2: Arrays of variable-length elements +**Scope:** +1. Implement the BAST-native validator as a new module + (`src/bast_validation.rs` or similar) +2. The validator walks a materialized `Value` tree against the BAST + type definitions, checking: + - Integer ranges (Int8..Uint64) + - Float finiteness (Float32/64) + - String `maxLength` (UTF-8 byte length) + - Bytes `maxLength` (byte length) + - RFC 3339 timestamp shape (non-strict, matching current behavior) + - Enum index bounds (0..values.len()-1 — **fixes the dead constraint**) + - Union variant dispatch (read `__discriminator`, look up variant + BAST definition, recurse) + - Boolean validity (materializer already checks, but the validator + should confirm) +3. Wire it into `validate_bytes()` as the validation step (replacing + the `jsonschema::Validator` call) +4. Run the **existing test suite** — the tests encode all current + expected validation behavior. If they pass, the POC succeeds. -The example in §"The BAST Format" shows -`{ "kind": "array", "element": "string" }` (count-prefixed). The engine -explicitly rejects arrays of variable-length elements ("TArray of -variable-length element kind ... is not supported"). Either the example -is out of scope for v1 (remove it and note the restriction in the -meta-schema), or variable-element arrays are a new feature with their -own layout semantics. +**Success criteria:** +- All existing `validate_bytes` tests pass without modification to their + assertions (test *inputs* will change to BAST format, but the + expected validation outcomes must be identical) +- The union variant dispatch tests (OQ-008) pass — `maxLength` on a + `Bytes` field inside a union variant is enforced +- The enum index-bounds validation works (new behavior — currently + broken, so this is a fix, not a regression) -### Gap 3: Field-name discriminator unions +**Failure path:** If the POC reveals that the BAST-native validator +cannot cleanly express some constraint (e.g., a constraint that relies +on JSON Schema's structural keywords in a way that's hard to +reimplement), the fallback is the "structural-only + external JSON +Schema" model from the original Gap 1 — but this is unlikely given that +the materializer already guarantees structure, leaving only value-domain +checks. -The meta-schema's `UnionDef` has no `fields` array, but the engine's -field-name discriminator reads the discriminator field from the union's -`properties` object. As sketched, BAST cannot express a feature the -engine already supports. The meta-schema needs a field-name- -discriminator shape (e.g. a `fields` array on `UnionDef`), or -field-name discriminators are dropped for v1. +**Out of scope for this POC:** +- `validate_json` — this path uses a standard `jsonschema::Validator` + from a consumer-provided JSON Schema, not the BAST-native validator. + No POC needed; it's a standard `jsonschema` usage. +- BAST document parsing / meta-schema validation — the parser is + straightforward JSON walking; no empirical risk. +- Layout computation — unchanged, already proven. -## Open Questions +## Decisions -### OQ-BAST-001: Root type selection +The following were open questions in earlier drafts. Each is now +resolved. They are recorded here as decisions, not re-litigated. -How does the engine know which `$defs` entry is the root type? Options: -- Explicit: `AlkTypeEngine::compile(bast_doc, "ChunkHeader", Packed)` -- Convention: the first entry in `$defs` (fragile — depends on JSON - key order) -- Marker: a `"$root": "ChunkHeader"` property on the BAST document +### D-BAST-001: Root type selection -**Leaning**: Explicit. The root type name is a required parameter to -`compile()`. This is unambiguous and matches how consumers think about -it ("compile the ChunkHeader schema"). +**Decision:** Explicit. The root type name is a required parameter to +`compile()`: `AlkTypeEngine::compile(bast_doc, "ChunkHeader", Packed)`. +This is unambiguous and matches how consumers think about it ("compile +the ChunkHeader schema"). Convention (first entry in `$defs`) is fragile +and depends on JSON key order; a `$root` marker is redundant with an +explicit parameter. -### OQ-BAST-002: JSON validation schema linkage +### D-BAST-002: Primitive type string set -How does a BAST struct reference a JSON Schema for `validate_bytes`? -Options: -- Separate parameter: `engine.validate_bytes(buf, Some(&json_schema))` -- Embedded reference: `{ "kind": "struct", "validation": { "$ref": "https://..." } }` -- Convention: same `$id` base, different fragment +**Decision:** Lowercase (`"uint32"`, `"int8"`, `"float64"`, `"bool"`, +`"string"`, `"bytes"`, `"timestamp"`). Matches JSON Schema's own +convention (`"string"`, `"integer"`, `"boolean"`), is easier to type, +and is the convention in the TypeBox research examples. The +`AlkTypeKind` enum variants remain PascalCase in Rust — the mapping is +a simple `from_str()` impl. -**Leaning**: Separate parameter for v1. The JSON Schema is a separate -document; the engine doesn't need to know about it at compile time. -`validate_bytes()` accepts an optional `&jsonschema::Validator` that -the consumer provides. This keeps BAST focused on binary layout. +### D-BAST-003: Top-level `$defs` requirement -### OQ-BAST-003: Primitive type string set +**Decision:** Always `$defs`. Every BAST document has the same +top-level shape: `{ "$defs": { ... } }`. Single-type documents are a +special case with one entry. The `$defs` block is the namespace; the +root type name (D-BAST-001) selects the entry point. A bare struct at +the top level would be a special case with different parsing logic and +no home for additional definitions. -The current proposal uses lowercase strings for primitives: -`"uint32"`, `"int8"`, `"float64"`, `"bool"`, `"string"`, `"bytes"`, -`"timestamp"`. Alternatives: -- PascalCase: `"Uint32"`, `"Int8"` (matches `AlkTypeKind` variant names) -- UPPER_CASE: `"UINT32"`, `"INT8"` -- Prefixed: `"bast:uint32"` (namespaced, but verbose) +### D-BAST-004: Arrays of variable-length elements (deferred) -**Leaning**: Lowercase. Matches JSON Schema's own convention -(`"string"`, `"integer"`, `"boolean"`), is easier to type, and is the -convention in the TypeBox research examples. The `AlkTypeKind` enum -variants remain PascalCase in Rust — the mapping is a simple -`from_str()` impl. +**Decision:** Arrays of variable-length elements are **not supported +in v1**. The meta-schema requires `count` on all array types, making +arrays fixed-size only. The no-count example has been removed. This +matches the engine's current behavior (it rejects arrays of +variable-length elements) and aligns with OQ-001 (deferred, blocked on +a concrete consumer needing interleaved variable-stride arrays). -### OQ-BAST-004: Array count — fixed vs variable +Variable-length collections are still available via `record` (a +count-prefixed string-keyed map), which the engine supports. If a +consumer needs a variable-length array of fixed-size elements, they +can use a record with integer-stringified keys as a workaround, or +wait for OQ-001 to be addressed. -When is an array fixed-size (no count prefix) vs variable-size (count -prefix in binary)? Options: -- Explicit `count` field: `{ "kind": "array", "element": "uint32", "count": 3 }` → fixed -- Absent `count`: `{ "kind": "array", "element": "uint32" }` → variable (count-prefixed) -- Separate `minItems`/`maxItems` like current JSON Schema convention +### D-BAST-005: Field-name discriminator unions -**Leaning**: Explicit `count` for fixed, absent for variable. This is -clearer than `minItems == maxItems` and matches the BAST principle of -explicit layout information. +**Decision:** Supported. The meta-schema includes an optional `fields` +array on `UnionDef`. When `discriminator.kind == "field"`, the +`fields` array provides the union's field definitions (including the +discriminator field). When `discriminator.kind == "byte"`, `fields` is +absent — the union's layout is purely the variant layout. This +preserves a feature the engine already supports. The meta-schema +update is in [the meta-schema sketch above](#meta-schema-sketch). -### OQ-BAST-005: Top-level `$defs` requirement +### D-BAST-006: `validate_bytes` validation model -Should a BAST document always have a top-level `$defs` block, or can -a single struct be the root? Options: -- Always `$defs`: `{ "$defs": { "ChunkHeader": { "kind": "struct", ... } } }` -- Bare struct: `{ "kind": "struct", "fields": [...] }` (no `$defs`) +**Decision:** BAST-native validator. The `validate_bytes` path uses a +recursive walker over the BAST type tree to check value-domain +constraints on the materialized `Value` — no external JSON Schema +needed. This recovers the OQ-008 union variant dispatch behavior (the +validator recurses into the variant's BAST definition) and fixes the +enum-membership dead constraint (the validator checks the materialized +index against the `values` array bounds). See [The Validator +Split](#the-validator-split) and the [POC](#poc-bast-native-validator-for-validate_bytes). -**Leaning**: Always `$defs`. Consistency — every BAST document has the -same top-level shape. Single-type documents are a special case of the -general form. The `$defs` block is the namespace; the root type name -selects the entry point. +An optional external JSON Schema can be layered on top for constraints +BAST doesn't express (cross-field consistency, regex patterns on string +content). This is additive, not load-bearing. -### OQ-BAST-006: `validate_json` on `AlkTypeEngine` +### D-BAST-007: `validate_json` validation model -With custom keyword validators removed, what does -`AlkTypeEngine::validate_json()` do? Options: -- Remove it — the engine is for binary layout; JSON validation is a - separate concern -- Keep it with a separately-provided JSON Schema — the engine holds a - `jsonschema::Validator` compiled from a JSON Schema the consumer - provides at compile time -- Keep it as a convenience that validates the materialized `Value` - against the BAST structure itself (type checks only, no range - constraints) +**Decision:** Standard JSON Schema validator. `validate_json` on +`AlkTypeEngine` validates a consumer-provided JSON `Value` against a +`jsonschema::Validator` compiled from a standard JSON Schema document +the consumer provides at compile time. No custom keywords. The BAST +document is not involved in this path — BAST describes bytes, not JSON +shape. This preserves the `validate_json` / `validate_bytes` symmetry +from ADR-010, but the two paths now use different validators (standard +`jsonschema` for JSON, BAST-native for bytes), reflecting their +different inputs and guarantees. -**Leaning**: Keep it with a separately-provided JSON Schema. The engine -already compiles a validator at load time (ADR-004). The validator just -comes from a standard JSON Schema instead of custom keywords. This -preserves the `validate_json` / `validate_bytes` symmetry from ADR-010. +The JSON Schema may be authored separately or derived from BAST via +future codegen. For alkcall's channel 0 (JSON-RPC), the JSON Schema is +the `OperationSpec` schema, authored independently of any BAST +document. -### OQ-BAST-007: Builder API — two output formats +### D-BAST-008: Builder API — two output formats -The builder API currently produces one JSON format (custom keywords). -Under BAST, it needs to produce two: -1. BAST JSON (for `Schema::struct_()`, `Schema::uint32()`, etc.) -2. Standard JSON Schema (for `Schema::object()`, `Schema::string_()`, etc.) +**Decision:** One builder, two build methods. The construction API is +the same (field names, types, annotations); only the output format +differs. `Schema::struct_().field(...).build()` → BAST JSON (binary +layout). `Schema::object().field(...).build()` → standard JSON Schema +(JSON validation). The builder already distinguishes AlkType kinds from +JSON Schema types via naming conventions (`string()` vs `string_()`). -Should these be two separate builder types, or one builder with a mode -flag? Options: -- Two builders: `BastBuilder` and `JsonSchemaBuilder` (or `Schema::bast` - and `Schema::json`) -- One builder with mode: `Schema::new(Mode::Bast)` / `Schema::new(Mode::JsonSchema)` -- One builder, two build methods: `schema.build_bast()` / - `schema.build_json_schema()` - -**Leaning**: One builder, two build methods. The construction API is the -same (field names, types, annotations); only the output format differs. -`Schema::struct_().field(...).build()` → BAST. -`Schema::object().field(...).build()` → standard JSON Schema. The -builder already distinguishes AlkType kinds from JSON Schema types via -naming conventions (`string()` vs `string_()`). +Both output formats live in the same crate. This is the point of +alktype: one small wasm-compatible codebase that handles both binary +layout and JSON validation for protocol crates. alkcall uses both — +channel 0 is JSON (standard JSON Schema), binary channels use BAST. +Future crates (alktty, tunnels, sftp, git) will use BAST for their +binary formats. The codegen feature (future) will generate +readers/writers from BAST documents for these crates. ## Risks and Mitigations @@ -995,9 +1146,12 @@ naming conventions (`string()` vs `string_()`). |------|-----------| | BAST format doesn't cover all 19 type kinds | The format is designed to cover all 19. The meta-schema is the spec — if a kind can't be expressed, the meta-schema is wrong. | | `$ref` resolution complexity moves from engine to schema authoring | BAST `$ref` values are always full JSON Pointers (`#/$defs/Name`). No normalization, no bare names. Resolution is a single hash lookup. | -| Losing `jsonschema`'s structural validation (required fields, etc.) | BAST is for binary layout, not JSON validation. Structural constraints belong in the JSON Schema document, not the BAST document. | +| BAST-native validator misses a constraint the custom keywords enforced | The [POC](#poc-bast-native-validator-for-validate_bytes) runs the existing test suite, which encodes all current expected validation behavior. If a constraint is missed, a test fails before the pivot lands. | +| Losing OQ-008 union variant dispatch | The BAST-native validator recurses into the variant's BAST definition on `__discriminator` lookup — same behavior, no custom keywords. Covered by the POC. | +| Enum membership broken on bytes path | Already broken today (dead constraint). The BAST-native validator fixes it by checking the materialized index against the `values` array bounds. Net improvement. | +| `validate_json` loses custom keyword validation | `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema. Consumers that relied on custom keywords for JSON validation need to provide equivalent standard JSON Schema keywords. No real consumers exist yet. | | Builder API output format change breaks consumers | No real consumers exist yet (v0.1.0 has zero adoption). The builder's public methods are unchanged; only the JSON output format changes. | -| Meta-schema maintenance burden | The meta-schema is small (~150 lines) and changes rarely. It's embedded in the crate and published at a stable URL. | +| Meta-schema maintenance burden | The meta-schema is small and changes rarely. It's embedded in the crate and published at a stable URL. | ## References