alktype/docs/architecture/overview.md

status, last_updated

status	last_updated
accepted	2026-08-15

alktype — Overview

The binary struct engine: a small Rust crate that takes a BAST (Binary Abstract Syntax Tree) document and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

This document covers the crate's purpose, the "schema is the format" principle, its dependency edges, consumers, and scope boundaries. Component details are in the sibling documents.

What

alktype is a library crate that consumes BAST documents and produces three capabilities:

1. An offset map — walks the BAST typed tree, computes byte offsets for each field based on type sizes, field order, and alignment.
2. Read/write functions — given a &[u8] buffer and a field path, read the field's bytes at its offset (zero-copy for fixed-size types). Given a &mut [u8] buffer, write a value at its offset.
3. Validation — two validators for two input types:
   • validate_bytes(&[u8]) uses the BAST-native validator (a recursive walker over the BAST type tree) to check the value-domain constraints the materializer doesn't (integer ranges, maxLength, enum index bounds, union variant constraints).
   • validate_json(&Value) uses a standard jsonschema::Validator compiled from a consumer-provided JSON Schema (BAST is not involved — BAST describes bytes, not JSON shape).

BAST is a JSON document that describes binary data layouts using a kind-based vocabulary with $defs/$ref for composition. BAST is itself a valid JSON Schema instance (it has a meta-schema), making it self-validating, editor-friendly, and trivially consumable from any language with a JSON parser. See ADR-BAST and bast-format.md.

The heavy lifting is done by the jsonschema crate (the validate_json path and BAST document meta-schema validation) and serde_json (BAST document parsing). The novel code is the offset computation — a recursive walk of the BAST typed tree that computes byte positions for each field — and the BAST-native validator — a flat recursive match over the same tree. See ADR-001 (purpose/scope) and ADR-BAST (format).

The crate replaces two prior attempts that built their own jsonschema engines — typebox-rs (~8,400 lines) and the @alkdev/alktype prototype (~5,600 lines) — with a BAST parser + an offset map + a BAST-native validator + the jsonschema crate for the JSON-validation path. See ADR-001.

Why

The crate's purpose is to be a binary struct engine for components that read or write binary data at computed offsets. Instead of per-protocol serde structs (russh-sftp's 29 packet types), per-handler wire format code (TTY's 5-byte format parser), or per-format offset computation (metatensor's tensor access), all of these become instances of the same engine with different BAST documents.

The guiding insight:

The schema is the format. A BAST document is both the layout spec and the validation spec for bytes. No separate format definition, no separate parser, no separate validator. One schema, three uses: validate, compute offsets, access data.

This is the convergence of three threads identified in the call-channels-unification research: the typedef.ts schema kinds from TypeBox, the russh-sftp protocol packets, and the metatensor format. The common pattern: a schema describes the shape of binary data, and the binary data is the struct's bytes at computed offsets.

The crate was bumped up in the timeline when the call-channels-unification research surfaced that channels, TTY, and the binary call protocol are all variations on the same wire-format family — [discriminant][length][payload]. The alktype engine makes the "channels is call with a binary data plane" unification concrete: the binary data plane's wire format is the call protocol's own schema system, just binary-encoded. The channel_open marker says "use binary framing"; the alktype engine says "here's how to read/write the binary payload."

The "Schema Is the Format" Principle

A BAST document serves three roles simultaneously:

Role	Mechanism	When
Validation spec (bytes)	BAST-native validator (recursive walker over the BAST type tree)	Load time (parse typed tree), access time (validate_bytes)
Validation spec (JSON)	Standard jsonschema::Validator from a consumer-provided JSON Schema	Load time (build validator), access time (validate_json)
Layout spec	Offset computation from type sizes + field order	Load time (build offset map)
Data access	Read/write at computed offsets	Access time (read field, write field)

No separate format definition, no separate parser, no separate validator. The BAST document is the single source of truth for the binary format. Adding a new field to a protocol is adding an entry to the BAST fields array — the engine computes the new offsets automatically.

This is the same principle as #[repr(C)] struct field access, but at runtime from a portable JSON document instead of at compile-time from language-specific annotations. The BAST document is the ABI contract.

Dependencies

alktype
├── jsonschema (v0.46, Draft 2020-12, default-features=false) — validate_json path + BAST meta-schema validation
├── serde_json (with preserve_order)                          — BAST document parsing; mapping iteration order is load-bearing
└── (no tokio, no platform deps)                              — WASM-clean by construction

alktype is dependency-light: jsonschema + serde_json only. No tokio, no platform deps. Compiles to wasm32-unknown-unknown for browser use. The validate_bytes path does not touch jsonschema for validation (it uses jsonschema::ValidationError::custom only for the error payload type, D-BAST-009) — a small wasm binary-size win.

serde_json's preserve_order feature remains a dependency. Under BAST, struct field order is explicit (the fields array), so layout correctness no longer depends on it; but mapping iteration order and the Definitions $defs block order are still load-bearing for the parser's lazy resolution and the builder's output.

Consumers

Consumer	Schema describes	Engine provides
alkcall (v0.1.0 first consumer)	channels' ChunkHeader { channel_id: u32 BE, length: u32 BE } + call's OperationSpec.input_schema / output_schema / error_schemas	validate_bytes for the 8-byte chunk header; validate_json for call's JSON payloads; builder API for both
russh-sftp	29 packet structs + Packet union (byte discriminator)	Read/write SFTP frames from bytes
metatensor	Model layout (ConvNet struct, tensor refs)	Offset map for mmap'd tensor access
binary call frames	call.requested / call.responded / etc. structs	Read/write binary call frames
TTY negotiation	NegotiateRequest / NegotiateResponse structs	Read/write TTY control frames
channels wire	ChunkHeader { channel_id, length }	8-byte chunk header (now in scope for alkcall; was "trivial" pre-v0.1.0)

alkcall — the merged alknet-call (call protocol) + alknet-channels (channel multiplexing) extraction from @alkdev/alknet — is the consumer that bumped the builder API (OQ-003) and generalized validation (ADR-010) into v0.1.0. It uses alktype for two distinct schema roles (binary layout + JSON payloads) from one library. See builder.md and validation.md §"validate_bytes".

The russh-sftp case is the most instructive and the highest-value POC target. The Packet enum's TryFrom<&mut Bytes> impl is a hand-written dispatch on a type byte followed by serde deserialization. Under alktype, the dispatch is a BAST union with a byte-offset discriminator — the schema says "byte 0 is the discriminator, bytes 1..N are the variant struct." The engine reads the discriminator, looks up the variant schema, computes offsets, reads fields. Same result, no per-packet-type code.

Scope Boundaries (What This Is Not)

These boundaries are decided in ADR-001 and ADR-BAST.

• Not metatensor. alktype is the binary struct engine. Metatensor is a format (8-byte header + JSON header + binary data) that uses the alktype engine for its offset computation and tensor access.
• Not a Value system. TypeBox's Value.Diff, Value.Migrate, Value.Convert — schema evolution — is out of scope for v1. The engine should not do anything that explicitly blocks adding a Value system later.
• Not a code generator. typebox-rs's codegen/ module is a separate concern. The alktype engine consumes BAST documents; it does not generate them.
• Schema builder is in scope as of v0.1.0. A fluent Rust API for constructing BAST documents and standard JSON Schemas at runtime, producing serde_json::Value, is shipped in v0.1.0 (ADR-009, resolves OQ-003). The builder covers BAST kinds (struct_()) and standard JSON Schema (object()); see builder.md. BAST documents may still be authored in TypeBox, generated by ujsx components, or hand-written — the builder is an additional construction path, not a replacement.
• Not a serialization framework. The alktype engine is not a general-purpose serde replacement. It operates on raw byte buffers at computed offsets — no intermediate Value tree (except for the validate_bytes materialization step), no reflection, no dynamic dispatch per field. For JSON data, use serde. For binary data with a known schema, use alktype.
• Not a JSON-payload validator. BAST describes bytes, not JSON shape. validate_json validates a JSON Value against a consumer-provided standard JSON Schema, not against the BAST document. See ADR-VAL-SPLIT.

Architecture (component pointers)

• schema-layer.md — the BAST parser (the typed surface every engine module walks), the 18 BAST kinds, the AlkTypeKind enum, and the foundational annotation types.
• bast-format.md — the normative BAST format specification (meta-schema, TypeRef, examples, validation model).
• layout-engine.md — offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length field handling.
• data-access.md — read/write functions, TUnion dispatch, field paths, zero-copy access for fixed-size types, length-prefix reading for variable-length types.
• validation.md — the two-validator model (BAST-native for validate_bytes, standard jsonschema for validate_json), AlkTypeError, load-time vs access-time validation, AlkTypeEngine as the compiled form of a BAST document.
• builder.md — fluent Rust API for constructing BAST documents and standard JSON Schemas at runtime, producing serde_json::Value. Covers BAST kinds and standard JSON Schema (ADR-009, D-BAST-008).

Design Decisions

Decision	ADR	Summary
Purpose, scope, and the jsonschema engine	ADR-001	What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries (format-specific content superseded by ADR-BAST)
BAST format	ADR-BAST	BAST as the schema format; meta-schema, $defs/$ref/kind vocabulary; supersedes ADR-001's format-specific content
Two layout modes	ADR-002	Packed sequential (LayoutBuilder/SequentialReader) for protocols; aligned static (OffsetMap) for mmap formats
Schema annotations	ADR-003	Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name) — semantics carry forward; location moved to BAST type-level properties
Error handling and validation	ADR-004	AlkTypeError enum; load-time build, access-time check; field-path-carrying errors; jsonschema ValidationError wrapping (validation-strategy section refined by ADR-VAL-SPLIT)
Two-validator model	ADR-VAL-SPLIT	BAST-native validator for validate_bytes; standard jsonschema::Validator for validate_json; D-BAST-006/007/009
Int64/Uint64 kinds	ADR-005	64-bit integers as first-class kinds (SFTP offsets, metatensor data_offsets)
Non-final inline variable fields	ADR-006	Rejected in aligned mode (would clobber subsequent fields)
Packed-mode read factory	ADR-007	engine.sequential_reader() returns an owned fresh reader
TUnion in aligned mode	ADR-008	Rejected for v1 (broken semantics; no current consumer needs it)
Builder API	ADR-009	Fluent Rust API producing serde_json::Value; covers BAST kinds + standard JSON Schema; resolves OQ-003 (output format amended to BAST / standard JSON Schema by ADR-BAST)
Generalized validation — validate_bytes	ADR-010	Single-call binary-buffer validation on AlkTypeEngine; materialize Value from bytes, then validate (validation step amended to the BAST-native validator by ADR-VAL-SPLIT)

Open Questions

See open-questions.md for full details.

• OQ-001 (deferred(scope)): Arrays of variable-length-element structs — BAST arrays require count in v1 (D-BAST-004), aligning with this deferral.
• OQ-002 (deferred(scope)): no_std + alloc support.
• OQ-003 (resolved by ADR-009): Builder API for schema construction. Shipped in v0.1.0; see builder.md.

References

• @alkdev/alknet: docs/research/alknet-typedef/findings.md — POC results (26 tests passing, two layout modes, TUnion dispatch, endianness)
• @alkdev/alknet: docs/research/call-channels-unification/findings.md §"alknet-typedef: JSON Schema as the binary struct engine" — the origin of this research thread
• @alkdev/alknet: typebox/example/typedef/typedef.ts — the TypeBox schema kinds (619 lines)
• @alkdev/alknet: jsonschema/ — the jsonschema crate (v0.46.5, Draft 2020-12)
• @alkdev/alknet: alknet-typedef-poc/ — the POC code (disposable)
• @alkdev/alknet: typebox-rs/ — prior attempt, replaced by alktype
• @alkdev/alknet: alktype-prototype/ — prior attempt (the @alkdev/alktype prototype; not to be confused with this crate, which reuses the name but is backed by the jsonschema crate)
• BAST pivot research record — motivation, POC scope and result, decisions D-BAST-001..009, risks
• BAST pivot implementation plan — ordered implementation steps, semver contract, ADR-sync checklist

Note

: The research findings, POC code, and prior-attempt paths above refer to the parent @alkdev/alknet workspace where this crate originated. They are preserved here as historical context for the architectural decisions; the artifacts themselves are not part of this standalone repo.