alktype/docs/research/bast-pivot.md

status, created, last_updated

status	created	last_updated
draft	2026-08-14	2026-08-15

BAST Pivot — Research Record

Replace alktype's custom JSON Schema keywords (AlkType:Uint32, AlkType:Struct, etc.) with a standalone JSON format — BAST (Binary Abstract Syntax Tree) — that describes binary data layouts using a kind-based vocabulary with $defs/$ref for composition. BAST is itself a valid JSON Schema instance (it has a meta-schema), making it self-validating, editor-friendly, and trivially consumable from any language with a JSON parser.

The engine's core logic (layout computation, data access, union dispatch, two layout modes) is unchanged. Only the schema-walking accessor layer changes: instead of detecting AlkType:* keywords scattered through a JSON Schema tree, the walkers read kind/fields/ annotation properties from a purpose-built format.

The builder API's public surface stays the same; only the JSON output format changes internally.

Document role. This is the research record: motivation, POC scope and result, decisions, risks, references. The normative format specification lives in docs/architecture/bast-format.md. The execution plan — ordered implementation steps, the public-API semver contract, and the ADR-sync checklist — lives in docs/plans/bast-implementation.md. Those documents supersede the format-spec, what-changes, and migration-path sections that previously lived here; this record keeps the why and what was proved, not the how to implement.

Motivation

Current state

alktype v0.1.0 embeds binary layout information inside standard JSON Schema documents via custom keywords:

{
  "AlkType:Struct": true,
  "type": "object",
  "properties": {
    "channel_id": { "AlkType:Uint32": true, "type": "integer" },
    "length":     { "AlkType:Uint32": true, "type": "integer" }
  },
  "endian": "big"
}

This works for the Rust engine — it walks the tree, detects keywords, computes offsets. But it creates friction for everything outside Rust:

1. Cross-language consumption. A Python, Go, or TypeScript consumer that wants to parse an alktype schema must re-implement custom keyword detection. The format is not self-describing — you need to know that AlkType:Uint32 means "4-byte unsigned integer" and that it can appear as either true or { "encoding": "..." }.

2. Code generation. Generating Rust/TypeScript/Python readers and writers from a schema requires walking an arbitrary JSON Schema tree looking for custom keywords. A kind-based format with known keys makes this a straightforward structural walk.

3. Tooling. Editors, linters, and schema validators don't understand AlkType:* keywords. A BAST document with a published meta-schema gets autocomplete, validation, and documentation in any JSON Schema- aware editor for free.

4. Two concerns in one document. The current format conflates binary layout (what the engine needs) with JSON validation (what jsonschema needs). A type: "object" with properties and required is a JSON validation concern; AlkType:Uint32 is a binary layout concern. They live in the same JSON object but serve different masters.

The downstream pain is real

The alkcall agent's review identified that the channels 8-byte chunk header is hand-rolled with manual bit shifts — alktype's binary layout capability is unused because the custom-keyword format is awkward to integrate for a simple 2-field struct. alktty plans to hand-roll its 5-byte TTY chunk format for the same reason. SFTP's 29 packet types were proven byte-identical with alktype in the POC, but the production path requires defining 29 schemas in the custom-keyword format.

All three cases are the same pattern: a small binary struct that needs a schema-driven reader/writer. BAST makes this trivial — a 10-line JSON file replaces hand-rolled bit shifts.

Timing

v0.1.0 was published but has zero real consumers (only bots/scanners have downloaded it). A breaking change now is free. Waiting until adoption creates migration cost.

POC Scope

The layout swap needs no POC — it is a backend swap (custom keywords → kind-based format) on top of a proven layout engine. The layout engine's byte-identity is already proven (alknet-typedef-poc, alktype-builder-poc) and the layout code is unchanged, so there is nothing empirical to de-risk there.

One targeted POC was needed to de-risk the validation model. The risk was specific and falsifiable: can a BAST-native validator — a recursive walker over the BAST type tree — fully replace the 19 custom keyword validators on the validate_bytes path, including the OQ-008 union variant dispatch, without regression?

The POC has been run and succeeded. The outcome is recorded in POC Result below. The POC code is on branch bast-validator-poc (commit f371fe4), not merged to main — it is reference scaffolding superseded by the production module in implementation step 5.

POC: BAST-native validator for validate_bytes

Hypothesis: A recursive walker over the BAST type tree can enforce all value-domain constraints that the 19 custom keyword validators currently enforce, recovering the OQ-008 union variant dispatch behavior, and fixing the enum-membership dead constraint on the bytes path — all without jsonschema custom keywords and without requiring the consumer to provide an external JSON Schema.

Scope:

1. Implement the BAST-native validator as a new module (src/bast_validation.rs or similar)
2. The validator walks a materialized Value tree against the BAST type definitions, checking:
   • Integer ranges (Int8..Uint64)
   • Float finiteness (Float32/64)
   • String maxLength (UTF-8 byte length)
   • Bytes maxLength (byte length)
   • RFC 3339 timestamp shape (non-strict, matching current behavior)
   • Enum index bounds (0..values.len()-1 — fixes the dead constraint)
   • Union variant dispatch (read __discriminator, look up variant BAST definition, recurse)
   • Boolean validity (materializer already checks, but the validator should confirm)
3. Wire it into validate_bytes() as the validation step (replacing the jsonschema::Validator call)
4. Run the existing test suite — the tests encode all current expected validation behavior. If they pass, the POC succeeds.

Success criteria:

• All existing validate_bytes tests pass without modification to their assertions (test inputs will change to BAST format, but the expected validation outcomes must be identical)
• The union variant dispatch tests (OQ-008) pass — maxLength on a Bytes field inside a union variant is enforced
• The enum index-bounds validation works (new behavior — currently broken, so this is a fix, not a regression)

Failure path: If the POC reveals that the BAST-native validator cannot cleanly express some constraint (e.g., a constraint that relies on JSON Schema's structural keywords in a way that's hard to reimplement), the fallback is the "structural-only + external JSON Schema" model from the original Gap 1 — but this is unlikely given that the materializer already guarantees structure, leaving only value-domain checks.

Out of scope for this POC:

• validate_json — this path uses a standard jsonschema::Validator from a consumer-provided JSON Schema, not the BAST-native validator. No POC needed; it's a standard jsonschema usage.
• BAST document parsing / meta-schema validation — the parser is straightforward JSON walking; no empirical risk.
• Layout computation — unchanged, already proven.

POC Result — BAST-native validator

Status: succeeded. The POC is on branch bast-validator-poc in src/bast_poc.rs (20 tests, all passing; full crate suite — 416 tests — green; cargo clippy --all-targets -- -D warnings clean; cargo build --target wasm32-unknown-unknown --release clean).

The POC implements the validate_bytes validation model from D-BAST-006 as a self-contained module that does not touch the production schema / materializer / validator paths. It reuses only data_access (read primitives), AlkTypeError (error type), and Endian. The BAST document parser, a packed-mode materializer, and the BAST-native validator are all implemented from scratch — that is the point: prove the model works end-to-end before refactoring the production code.

What the POC proves

The hypothesis from the POC section is confirmed: a recursive walker over the BAST type tree fully replaces the 19 custom keyword validators on the validate_bytes path, including the OQ-008 union variant dispatch, and fixes the enum-membership dead constraint — all without jsonschema custom keywords and without an external JSON Schema.

The hard cases that were the actual de-risking targets all pass:

• Union byte-offset discriminator + maxLength inside a variant (OQ-008). union_byte_disc_max_length_inside_variant_enforced materializes a union with two $ref variants, dispatches on a byte-offset uint8 discriminator, and enforces maxLength on a bytes field inside the selected variant. The validator reads __discriminator, looks up the variant's BAST definition, and recurses — same behavior as the current UnionValidator's per-variant sub-validators, but with no jsonschema involvement.
• Union field-name discriminator + maxLength inside a variant. union_field_disc_max_length_inside_variant_enforced covers the typedef.ts-style discriminator (a length-prefixed string field selects the variant). Same recursion model.
• Enum index-bounds fix. enum_index_out_of_bounds_rejected exercises the constraint that is broken in the current engine (the built-in enum keyword checks string membership; the materializer emits Value::Number(index), which never matches — a dead constraint). The BAST-native validator checks the materialized index against the values array bounds (0..len-1), which is the correct validation for a binary enum encoded as an index. Net improvement, not a regression.
• Nested struct wrapping a union wrapping a struct. nested_struct_with_union_variant confirms the recursion composes through multiple type layers.
• Arrays of fixed-size structs with count. array_of_structs_with_count covers the Vector3-style array (D-BAST-004).
• Records (count-prefixed string-keyed maps). record_of_uint32 covers the TRecord shape.
• Untrusted schema input. malformed_document_produces_schema_error_not_panic confirms a malformed BAST document surfaces as AlkTypeError::Schema, not a panic (AGENTS.md §3).
• Basic cases (chunk header, int8/uint32 ranges, string/bytes maxLength, timestamp, bool, short buffer) all pass — if a couple of basic examples work, all of them do, since the validator is a flat per-kind dispatch with no per-kind special-casing beyond the range bounds.

How the validator works

The validator is a single recursive function (validate_typeref) that dispatches on the BAST kind. Each arm checks the value-domain constraint for that kind and, for composites, recurses into the child type definitions. The materializer (also implemented in the POC) guarantees structural correctness — bounds, UTF-8, bool byte, discriminator lookup, all fields present — so the validator only enforces what the materializer cannot. The full constraint table is in bast-format.md §Validation Model.

Observations for the production implementation

1. No jsonschema dependency for validate_bytes. The validator only needs serde_json (for Value) and the BAST document. The jsonschema crate is still a direct dependency for validate_json and for validating BAST documents against the BAST meta-schema, but the validate_bytes path no longer touches it. This is a small wasm binary-size win in addition to the architecture simplification.

2. $ref resolution is a single hash lookup. The POC's resolve_ref_or_inline handles only #/$defs/Name pointers — the only form BAST allows. The current engine's normalize_refs / inline_union_variant_refs / resolve_ref_or_inline machinery for bare-name refs and inlined union variants is no longer needed: BAST $refs are always full JSON Pointers, and union variant refs are resolved lazily by the validator (the materializer already does this for the read path). The inline_union_variant_refs compile step can be removed entirely.

3. The validator is ~250 lines. The 19 custom keyword validators (src/validation.rs) plus the macro definitions are ~500 lines and require the jsonschema::Keyword trait plumbing (factory closures, Box<dyn Keyword>, sub-validator construction at factory time). The BAST-native validator is a flat match — no factories, no trait objects, no sub-validator pre-computation. The recursion is direct.

4. The AlkTypeError::Validation variant still wraps jsonschema::ValidationError<'static>. The POC uses jsonschema::ValidationError::custom to construct these so the error type is unchanged. This is now the decided shape for the production refactor — see D-BAST-009. The rationale is consumer ergonomics: a single uniform payload type means one match arm covers both validate_json and validate_bytes errors downstream, and validate_json's structured errors are worth preserving rather than flattening to a String.

5. The materializer and validator share the BAST-walking code structure. Both walk the same kind/fields/mapping tree. The production refactor could share a typed BAST tree (a small BastNode enum) between them so the walk is parsed once. The POC parses lazily from the raw JSON in both passes to keep the model honest; a typed tree is a straightforward follow-on optimization, not a risk.

Verdict

The "how do we reproduce the same behavior?" question is answered: walk the BAST tree the same way the materializer does, checking the same value-domain constraints the custom keyword validators check today. The model is a strict simplification — fewer moving parts, no jsonschema integration on the bytes path, no compile-time inline_union_variant_refs step, no factory closures or trait objects, and the enum dead-constraint is fixed as a side effect.

The POC does not wire into AlkTypeEngine::validate_bytes — that is the production refactor (implementation step 5), which replaces validation::build_validator usage on the bytes path with the BAST-native validator. The POC's job was to de-risk the model before that refactor; that job is done.

Decisions

The following were open questions in earlier drafts. Each is now resolved. They are recorded here as decisions, not re-litigated. The normative format specification that realizes these decisions is in bast-format.md; the semver classification of each is in the implementation plan's Semver Contract.

D-BAST-001: Root type selection

Decision: Explicit. The root type name is a required parameter to compile(): AlkTypeEngine::compile(bast_doc, "ChunkHeader", Packed). This is unambiguous and matches how consumers think about it ("compile the ChunkHeader schema"). Convention (first entry in $defs) is fragile and depends on JSON key order; a $root marker is redundant with an explicit parameter.

D-BAST-002: Primitive type string set

Decision: Lowercase ("uint32", "int8", "float64", "bool", "string", "bytes", "timestamp"). Matches JSON Schema's own convention ("string", "integer", "boolean"), is easier to type, and is the convention in the TypeBox research examples. The AlkTypeKind enum variants remain PascalCase in Rust — the mapping is a simple from_str() impl.

D-BAST-003: Top-level $defs requirement

Decision: Always $defs. Every BAST document has the same top-level shape: { "$defs": { ... } }. Single-type documents are a special case with one entry. The $defs block is the namespace; the root type name (D-BAST-001) selects the entry point. A bare struct at the top level would be a special case with different parsing logic and no home for additional definitions.

D-BAST-004: Arrays of variable-length elements (deferred)

Decision: Arrays of variable-length elements are not supported in v1. The meta-schema requires count on all array types, making arrays fixed-size only. This matches the engine's current behavior (it rejects arrays of variable-length elements) and aligns with OQ-001 (deferred, blocked on a concrete consumer needing interleaved variable-stride arrays).

Variable-length collections are still available via record (a count-prefixed string-keyed map), which the engine supports. If a consumer needs a variable-length array of fixed-size elements, they can use a record with integer-stringified keys as a workaround, or wait for OQ-001 to be addressed.

D-BAST-005: Field-name discriminator unions

Decision: Supported. The meta-schema includes an optional fields array on UnionDef. When discriminator.kind == "field", the fields array provides the union's field definitions (including the discriminator field). When discriminator.kind == "byte", fields is absent — the union's layout is purely the variant layout. This preserves a feature the engine already supports. The meta-schema is in bast-format.md §The Meta-Schema.

D-BAST-006: validate_bytes validation model

Decision: BAST-native validator. The validate_bytes path uses a recursive walker over the BAST type tree to check value-domain constraints on the materialized Value — no external JSON Schema needed. This recovers the OQ-008 union variant dispatch behavior (the validator recurses into the variant's BAST definition) and fixes the enum-membership dead constraint (the validator checks the materialized index against the values array bounds). See bast-format.md §Validation Model and the POC.

An optional external JSON Schema can be layered on top for constraints BAST doesn't express (cross-field consistency, regex patterns on string content). This is additive, not load-bearing.

D-BAST-007: validate_json validation model

Decision: Standard JSON Schema validator. validate_json on AlkTypeEngine validates a consumer-provided JSON Value against a jsonschema::Validator compiled from a standard JSON Schema document the consumer provides at compile time. No custom keywords. The BAST document is not involved in this path — BAST describes bytes, not JSON shape. This preserves the validate_json / validate_bytes symmetry from ADR-010, but the two paths now use different validators (standard jsonschema for JSON, BAST-native for bytes), reflecting their different inputs and guarantees.

The JSON Schema may be authored separately or derived from BAST via future codegen. For alkcall's channel 0 (JSON-RPC), the JSON Schema is the OperationSpec schema, authored independently of any BAST document.

D-BAST-008: Builder API — two output formats

Decision: One builder, two build methods. The construction API is the same (field names, types, annotations); only the output format differs. Schema::struct_().field(...).build() → BAST JSON (binary layout). Schema::object().field(...).build() → standard JSON Schema (JSON validation). The builder already distinguishes AlkType kinds from JSON Schema types via naming conventions (string() vs string_()).

Both output formats live in the same crate. This is the point of alktype: one small wasm-compatible codebase that handles both binary layout and JSON validation for protocol crates. alkcall uses both — channel 0 is JSON (standard JSON Schema), binary channels use BAST. Future crates (alktty, tunnels, sftp, git) will use BAST for their binary formats. The codegen feature (future) will generate readers/writers from BAST documents for these crates.

D-BAST-009: AlkTypeError::Validation payload shape

Status: decided. Keep Validation(jsonschema::ValidationError<'static>).

AlkTypeError::Validation currently wraps jsonschema::ValidationError<'static>. Under the BAST pivot the validate_bytes path no longer uses jsonschema at all (confirmed by the POC — observation 1), so the error payload on that path is constructed via jsonschema::ValidationError::custom purely to keep the variant's type unchanged. The two options were:

1. Keep Validation(jsonschema::ValidationError<'static>). Simplest — ValidationError::custom is public and 'static, so the bytes path can construct it without a real jsonschema validator. Cost: the error type retains its jsonschema dependency even though the bytes path no longer drives it. validate_json still uses jsonschema, so the dependency isn't removable either way — but the error type carries jsonschema only for one of its two callers.
2. Introduce Validation(String) (or a small structured payload). Drops the jsonschema type from the public error enum. This is a semver-relevant public-API change (the Validation variant's payload type changes), so per AGENTS.md it requires an explicit decision, not a drive-by. Benefit: the error type is jsonschema-free, which matters if a future no_std/minimal build wants to drop jsonschema from the bytes-only path (relates to OQ-002).

Rationale for option 1: The deciding factor is consumer ergonomics on the combined path. Consumers like alkcall use both validate_json (channel 0, JSON-RPC) and validate_bytes (binary channels) and handle AlkTypeError::Validation in one place. A single uniform payload type means one match arm covers both sources — no Validation(jsonschema_err) vs Validation(string) branching downstream. Option 2 would force validate_json to flatten its structured errors (instance path, schema path, keyword) to a String via Display just to match a bytes-path shape — the more information-rich path loses data to accommodate the less rich one. That is the wrong direction.

The no_std/minimal-build angle (OQ-002) that option 2 was meant to enable is moot in practice: validate_json requires jsonschema regardless, so a bytes-only no_std build already has to give up validate_json as a separate, larger decision. Dropping the type from one error variant does not unlock that build — the dependency is load- bearing on the other validation path. The right place to revisit this is when/if OQ-002 is actually pursued, not preemptively.

The POC already used option 1 (via ValidationError::custom); the production refactor (implementation step 5) follows the same construction pattern. No semver-relevant change to the Validation variant.

Risks and Mitigations

Risk	Mitigation
BAST format doesn't cover all 19 type kinds	The format is designed to cover all 19. The meta-schema is the spec — if a kind can't be expressed, the meta-schema is wrong.
$ref resolution complexity moves from engine to schema authoring	BAST $ref values are always full JSON Pointers (#/$defs/Name). No normalization, no bare names. Resolution is a single hash lookup.
BAST-native validator misses a constraint the custom keywords enforced	The POC runs the existing test suite, which encodes all current expected validation behavior. If a constraint is missed, a test fails before the pivot lands.
Losing OQ-008 union variant dispatch	The BAST-native validator recurses into the variant's BAST definition on __discriminator lookup — same behavior, no custom keywords. Covered by the POC.
Enum membership broken on bytes path	Already broken today (dead constraint). The BAST-native validator fixes it by checking the materialized index against the values array bounds. Net improvement.
validate_json loses custom keyword validation	validate_json uses a standard jsonschema::Validator from a consumer-provided JSON Schema. Consumers that relied on custom keywords for JSON validation need to provide equivalent standard JSON Schema keywords. No real consumers exist yet.
Builder API output format change breaks consumers	No real consumers exist yet (v0.1.0 has zero adoption). The builder's public methods are unchanged; only the JSON output format changes.
Meta-schema maintenance burden	The meta-schema is small and changes rarely. It's embedded in the crate and published at a stable URL.

Future Directions

These are enabled by BAST but out of scope for the pivot itself. They are mentioned to show that BAST makes them possible, not to commit to a specific implementation timeline.

Codegen

BAST enables code generation that the custom-keyword format makes awkward. A codegen module (feature-gated behind codegen) would walk $defs entries, inspect kind values, and emit Rust/TypeScript/Python readers and writers via Handlebars templates. The typebox-rs codegen/ module (/workspace/@alkimiadev/typebox-rs) is the reference architecture: SchemaRegistry for named types, RustGenerator/ TypeScriptGenerator wrapping Handlebars, schema_to_rust_type()/ schema_to_ts_type() mapping functions. alktype's codegen would follow the same pattern but walk BAST kind values. The handlebars-rs dependency is WASM-compatible. The pivot changes the schema format; codegen builds on top of the new format.

ABI Adapter

A BAST document describes the binary interface of a protocol — it is essentially an ABI specification in JSON. This enables version negotiation (two peers exchange BAST documents to agree on a protocol version; the engine detects mismatches and either rejects or adapts), schema migration (a consumer with schema v1 can read data written by schema v2 if the changes are compatible), and WASM interop (a WASM component can export its BAST schema as part of its WIT interface).

References

• docs/architecture/bast-format.md — the normative BAST format spec (meta-schema, TypeRef, examples, validation model)
• docs/plans/bast-implementation.md — the execution plan (ordered steps, semver contract, ADR-sync checklist)
• ADR-001 — current "schema is the format" principle (to be superseded by ADR-BAST post-implementation)
• ADR-003 — annotation semantics (carry forward to BAST unchanged)
• ADR-009 — builder API (public surface unchanged, output format changes)
• ADR-010 — validate_bytes (unchanged in concept)
• /workspace/research/typebox_research/ujsx/jpath.gen.ts — TypeBox Type.Module pattern (the $defs/$ref model BAST follows)
• /workspace/research/typebox_research/ujsx/mdast.gen.ts — TypeBox cross-module references and composite types
• /workspace/research/typebox_research/codegen/ts-to-module.ts — TypeScript-to-TypeBox codegen (reference for future BAST codegen)
• /workspace/@alkimiadev/typebox-rs/src/codegen/ — Rust/TypeScript codegen from schemas (reference architecture)
• /workspace/alknet-typedef-poc/tests/sftp_roundtrip_test.rs — SFTP POC proving byte-identical output (to be replicated with BAST)
• /workspace/@alkdev/alkcall/src/channels/wire.rs — hand-rolled chunk header (target for BAST replacement)