alktype/docs/architecture/README.md

status, last_updated

status	last_updated
accepted	2026-08-15

alktype

The binary struct engine: a small Rust crate that takes a BAST (Binary Abstract Syntax Tree) document and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

Documents

Document	Status	Description
overview.md	accepted	Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries
bast-format.md	accepted	Normative BAST format specification. Meta-schema, TypeRef, TypeDef shapes (Struct/Union/Enum/FieldDef), examples, validation model. The format the engine consumes.
schema-layer.md	accepted	The BAST parser (src/bast.rs) — the typed tree (BastDoc/BastDef/BastType/…) every engine module walks, the 18 BAST kinds, the AlkTypeKind enum, and the foundational annotation types.
layout-engine.md	draft	Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling
data-access.md	draft	Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading
validation.md	accepted	The two-validator model (BAST-native for validate_bytes, standard jsonschema for validate_json), AlkTypeError, load-time vs access-time validation, AlkTypeEngine as the compiled form of a BAST document (ADR-010, ADR-VAL-SPLIT).
builder.md	accepted	Fluent Rust API for constructing BAST documents (struct_()) and standard JSON Schemas (object()) at runtime, producing serde_json::Value (ADR-009, D-BAST-008).

In-progress work

Document	Status	Description
BAST pivot — research record	accepted	Motivation, POC scope and result, decisions D-BAST-001..009, risks for the BAST format pivot. Implemented in steps 1–10.
BAST pivot — implementation plan	accepted	Ordered implementation steps, the public-API semver contract, and the ADR-sync checklist for the BAST pivot. Steps 1–10 complete.

Applicable ADRs

ADR	Title	Relevance
001	Purpose, Scope, and the jsonschema Engine	What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries. Format-specific content superseded by ADR-BAST; purpose/scope retained.
BAST	BAST (Binary Abstract Syntax Tree) as the Schema Format	The BAST format, meta-schema, $defs/$ref/kind vocabulary. Supersedes ADR-001's format-specific content; records D-BAST-001..009.
VAL-SPLIT	Two-Validator Model — BAST-Native for Bytes, Standard jsonschema for JSON	validate_bytes uses the BAST-native validator; validate_json uses a standard jsonschema::Validator from a consumer-provided JSON Schema. Records D-BAST-006/007/009.
002	Two Layout Modes — Packed Sequential vs Aligned Static	The most important architectural finding; when to use each mode; LayoutBuilder/SequentialReader vs OffsetMap (format-agnostic — input format changed, modes didn't)
003	Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators	Annotation semantics (carry forward unchanged); annotation location moved to BAST type-level properties under the pivot
004	Error Handling and Validation Strategy	AlkTypeError enum (shape unchanged, D-BAST-009); load-time build, access-time check; field-path-carrying errors. Validation-strategy section refined by ADR-VAL-SPLIT.
005	Int64/Uint64 as First-Class Kinds	64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat
006	Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode	Prevents silent data corruption (inline variable data clobbering subsequent fields)
007	Packed-Mode Read API — Engine as SequentialReader Factory	engine.sequential_reader() returns an owned reader, not a reference
008	Reject TUnion in Aligned Mode for v1	Unions are the protocol pattern; aligned-mode union semantics were broken
009	Builder API for Schema Construction	Fluent Rust API producing serde_json::Value; covers BAST kinds + standard JSON Schema; resolves OQ-003. Output format amended to BAST / standard JSON Schema by ADR-BAST.
010	Generalized Validation — validate_bytes on AlkTypeEngine	Single-call binary-buffer validation; materialize Value from bytes, then validate. Validation step amended to the BAST-native validator by ADR-VAL-SPLIT.
011	Compiled Read Plan for Packed Mode	ReadPlan — the packed read-side compiled form, symmetric to OffsetMap (aligned) and PackedLayout (packed write). Closes review #004's 400x read-path gap; retires ADR-007's "re-parse on demand" framing. Accepted.
012	Plan Fingerprinting and Closing the Deferred M1 Sites in 0.3.0	ReadPlan/OffsetMap Hash + Eq + fingerprint(); owned BastDoc (lifetime removal); OffsetMap carries LeafMeta to close the aligned-side M1 sites. Bundles with ADR-011 into one 0.3.0 breaking release. Proposed.

Relevant Open Questions

OQ	Title	Status	Relevance
OQ-001	Arrays of variable-length-element structs	deferred(scope)	Requires lazy walking logic; blocked on a concrete consumer that needs it. BAST arrays require count in v1 (D-BAST-004), aligning with this deferral.
OQ-002	no_std + alloc support	deferred(scope)	Target std for v1; blocked on an embedded use case
OQ-003	Builder API for schema construction	resolved (ADR-009)	Shipped in v0.1.0; alkcall is the concrete consumer; see builder.md
OQ-004	Discriminator::Field name — &str or String	resolved	String, for ownership simplicity
OQ-005	Union materialization shape — byte-offset vs field-name consistency	resolved	Both kinds return { "__discriminator": <value>, ...variant-fields }
OQ-006	Builder spec Example 3 — wrap Union in a Struct	resolved	builder.md Example 3 wraps the union in a Schema::struct_().field("payload", ...)
OQ-007	Bytes materialization — lossy UTF-8 conversion	resolved	Array of u8: materializer produces Value::Array of Value::Number; BAST-native validator accepts both Value::String and Value::Array
OQ-008	UnionValidator variant dispatch	resolved	BAST-native validator recurses into the selected variant's BAST definition on __discriminator lookup — no custom keywords, no inline_union_variant_refs

Key Design Principles

1. The schema is the format. A BAST document is both the layout spec and the validation spec for bytes. No separate format definition, no separate parser, no separate validator. One schema, three uses: validate, compute offsets, access data. See overview.md, ADR-001, and ADR-BAST.

2. BAST is a JSON Schema dialect, not a custom format. A BAST document is valid JSON conforming to the BAST meta-schema (a standard Draft 2020-12 JSON Schema). Any JSON Schema validator can check whether a BAST document is well-formed; editors with JSON Schema support provide autocomplete for free. See bast-format.md and ADR-BAST.

3. Two layout modes for two use cases. Packed sequential (LayoutBuilder/SequentialReader) for protocol wire formats (SFTP, channels, TTY). Aligned static (OffsetMap) for mmap-friendly formats (metatensor). The consumer selects the mode; the BAST document is the same. See layout-engine.md and ADR-002.

4. Variable-length types default to inline length-prefixing. [length: u32][data] is the universal pattern used by channels, SFTP, TTY, and most binary protocols. Offset indirection (the metatensor blob tensor pattern) is opt-in via the encoding annotation. See layout-engine.md and ADR-003.

5. TUnion supports both byte-offset and field-name discriminators. Byte-offset for protocol dispatch (SFTP type bytes, call protocol event types). Field-name for the typedef.ts string pattern. See data-access.md and ADR-003.

6. Endianness is per-schema, default little-endian. The engine reads the struct-level "endian" annotation and byte-swaps accordingly. SFTP consumers specify "endian": "big". See layout-engine.md and ADR-003.

7. Two validators for two input types. validate_bytes(&[u8]) uses the BAST-native validator (a recursive walker over the BAST type tree — no jsonschema involvement). validate_json(&Value) uses a standard jsonschema::Validator from a consumer-provided JSON Schema (BAST is not involved — BAST describes bytes, not JSON shape). One AlkTypeError::Validation variant covers both (D-BAST-009). See validation.md and ADR-VAL-SPLIT.

8. Not a serialization framework. The alktype engine is not a general-purpose serde replacement. It operates on raw byte buffers at computed offsets — no reflection, no dynamic dispatch per field. For JSON data, use serde. For binary data with a known BAST document, use alktype. See overview.md and ADR-001.

9. Schemas can be built at runtime from Rust. A fluent builder API produces serde_json::Value for both BAST documents (struct_()) and standard JSON Schemas (object()), covering alkcall's two roles (binary layout + JSON payloads) from one module. The builder is additive — consumers with static BAST documents continue to load JSON. See builder.md and ADR-009.

10. Two validation entry points, one engine. validate_json(&Value) for already-parsed JSON (call's payloads); validate_bytes(&[u8]) for binary buffers (channels' chunk header). Different validators, one AlkTypeError::Validation variant. See validation.md, ADR-010, and ADR-VAL-SPLIT.

References

• @alkdev/alknet: docs/research/alknet-typedef/findings.md — POC results (26 tests passing, two layout modes, TUnion dispatch, endianness)
• @alkdev/alknet: docs/research/call-channels-unification/findings.md §"alknet-typedef: JSON Schema as the binary struct engine" — the origin of this research thread
• @alkdev/alknet: typebox/example/typedef/typedef.ts — the TypeBox schema kinds (619 lines)
• @alkdev/alknet: jsonschema/ — the jsonschema crate (v0.46.5, Draft 2020-12)
• @alkdev/alknet: alknet-typedef-poc/ — the POC code (disposable)
• @alkdev/alknet: typebox-rs/ — prior attempt, replaced by alktype
• @alkdev/alknet: alktype-prototype/ — prior attempt (the @alkdev/alktype prototype, a handler-registry pattern; not to be confused with this crate, which reuses the name but is backed by the jsonschema crate)

Note

: The research findings, POC code, and prior-attempt paths above refer to the parent @alkdev/alknet workspace where this crate originated. They are preserved here as historical context for the architectural decisions; the artifacts themselves are not part of this standalone repo.