alktype/docs/architecture

status, last_updated

status	last_updated
draft	2026-08-11

alktype

The binary struct engine: a small Rust crate that takes a JSON Schema with AlkType:* custom keywords and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

Documents

Document	Status	Description
overview.md	draft	Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries
schema-layer.md	draft	The 19 AlkType:* kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations. (Current v0.1.0 format; will be rewritten for BAST — see bast-format.md.)
bast-format.md	draft	Target schema format. BAST (Binary Abstract Syntax Tree): meta-schema, TypeRef, examples, the two-validator model. Supersedes the format-spec content of schema-layer.md when the BAST pivot lands.
layout-engine.md	draft	Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling
data-access.md	draft	Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading
validation.md	draft	Custom keyword validators for all 19 AlkType:* kinds, AlkTypeError, load-time vs access-time validation, AlkTypeEngine; validate_bytes for binary buffers (ADR-010). (Current v0.1.0 validation; will be rewritten for the validator split — see bast-format.md §Validation Model.)
builder.md	draft	Fluent Rust API for constructing alktype JSON Schemas at runtime, producing serde_json::Value; covers AlkType kinds + standard JSON Schema (ADR-009)

In-progress work

Document	Status	Description
BAST pivot — research record	draft	Motivation, POC scope and result, decisions D-BAST-001..009, risks for the BAST format pivot
BAST pivot — implementation plan	draft	Ordered implementation steps, the public-API semver contract, and the ADR-sync checklist for the BAST pivot

Applicable ADRs

ADR	Title	Relevance
001	Purpose, Scope, and the jsonschema Engine	What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries
002	Two Layout Modes — Packed Sequential vs Aligned Static	The most important architectural finding; when to use each mode; LayoutBuilder/SequentialReader vs OffsetMap
003	Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators	Concrete JSON shapes for all schema-level annotations
004	Error Handling and Validation Strategy	AlkTypeError enum; load-time build, access-time check; field-path-carrying errors
005	Int64/Uint64 as First-Class Kinds	64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat
006	Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode	Prevents silent data corruption (inline variable data clobbering subsequent fields)
007	Packed-Mode Read API — Engine as SequentialReader Factory	engine.sequential_reader() returns an owned reader, not a reference
008	Reject TUnion in Aligned Mode for v1	Unions are the protocol pattern; aligned-mode union semantics were broken
009	Builder API for Schema Construction	Fluent Rust API producing serde_json::Value; covers AlkType kinds + standard JSON Schema; resolves OQ-003
010	Generalized Validation — validate_bytes on AlkTypeEngine	Single-call binary-buffer validation; materialize Value from bytes, then validate; two methods on one struct, not a trait

Relevant Open Questions

OQ	Title	Status	Relevance
OQ-001	Arrays of variable-length-element structs	deferred(scope)	Requires lazy walking logic; blocked on a concrete consumer that needs it
OQ-002	no_std + alloc support	deferred(scope)	Target std for v1; blocked on an embedded use case
OQ-003	Builder API for schema construction	resolved (ADR-009)	Resolved in v0.1.0; alkcall is the concrete consumer; see builder.md
OQ-004	Discriminator::Field name — &str or String	open	Builder API ownership question; resolve before the SFTP Packet POC's field-name discriminator path
OQ-005	Union materialization shape — byte-offset vs field-name consistency	open	Blocks the SFTP Packet validate_bytes POC (next round)
OQ-006	Builder spec Example 3 — wrap Union in a Struct	open	Documentation fix in builder.md; the engine requires AlkType:Struct at the top level
OQ-007	Bytes materialization — lossy UTF-8 conversion	open	Blocks the SFTP use case for validate_bytes (binary handle/data fields)

Key Design Principles

1. The schema is the format. A JSON Schema with AlkType:* custom keywords is both the validation spec and the layout spec. No separate format definition, no separate parser, no separate validator. One schema, three uses: validate, compute offsets, access data. See overview.md and ADR-001.

2. jsonschema is the validation engine, not a custom engine. The jsonschema crate (v0.46.5, Draft 2020-12) handles validation with custom keyword support. The novel code is the offset computation, not the validation. This eliminates ~14,000 lines of hand-rolled schema engines (typebox-rs, the @alkdev/alktype prototype). See schema-layer.md and ADR-001.

3. Two layout modes for two use cases. Packed sequential (LayoutBuilder/SequentialReader) for protocol wire formats (SFTP, channels, TTY). Aligned static (OffsetMap) for mmap-friendly formats (metatensor). The consumer selects the mode; the schema is the same. See layout-engine.md and ADR-002.

4. Variable-length types default to inline length-prefixing. [length: u32][data] is the universal pattern used by channels, SFTP, TTY, and most binary protocols. Offset indirection (the metatensor blob tensor pattern) is opt-in via the encoding annotation. See layout-engine.md and ADR-003.

5. TUnion supports both byte-offset and field-name discriminators. Byte-offset for protocol dispatch (SFTP type bytes, call protocol event types). Field-name for the typedef.ts string pattern. See data-access.md and ADR-003.

6. Endianness is per-schema, default little-endian. The engine reads the "endian" annotation and byte-swaps accordingly. SFTP consumers specify "endian": "big". See layout-engine.md and ADR-003.

7. Validation is opt-in, built once at load time. The jsonschema validator is compiled once at schema load time. Access-time validation is a fast is_valid() check. High-throughput paths can skip validation; security-sensitive paths can validate every frame. See validation.md and ADR-004.

8. Not a serialization framework. The alktype engine is not a general-purpose serde replacement. It operates on raw byte buffers at computed offsets — no intermediate Value tree, no reflection, no dynamic dispatch per field. For JSON data, use serde. For binary data with a known schema, use alktype. See overview.md and ADR-001.

9. Schemas can be built at runtime from Rust (v0.1.0). A fluent builder API produces serde_json::Value for both AlkType-kind schemas and standard JSON Schema, covering alkcall's two roles (binary layout + JSON payloads) from one module. The builder is additive — consumers with static schemas continue to load JSON. See builder.md and ADR-009.

10. Two validation entry points, one engine (v0.1.0). validate_json(&Value) for already-parsed JSON (call's payloads); validate_bytes(&[u8]) for binary buffers (channels' chunk header). Same underlying jsonschema validator; the bytes path materializes a Value tree via the layout engine, then validates. See validation.md and ADR-010.

References

• @alkdev/alknet: docs/research/alknet-typedef/findings.md — POC results (26 tests passing, two layout modes, TUnion dispatch, endianness)
• @alkdev/alknet: docs/research/call-channels-unification/findings.md §"alknet-typedef: JSON Schema as the binary struct engine" — the origin of this research thread
• @alkdev/alknet: typebox/example/typedef/typedef.ts — the TypeBox schema kinds (619 lines)
• @alkdev/alknet: jsonschema/ — the jsonschema crate (v0.46.5, Draft 2020-12)
• @alkdev/alknet: alknet-typedef-poc/ — the POC code (disposable)
• @alkdev/alknet: typebox-rs/ — prior attempt, replaced by alktype
• @alkdev/alknet: alktype-prototype/ — prior attempt (the @alkdev/alktype prototype, a handler-registry pattern; not to be confused with this crate, which reuses the name but is backed by the jsonschema crate)

Note

: The research findings, POC code, and prior-attempt paths above refer to the parent @alkdev/alknet workspace where this crate originated. They are preserved here as historical context for the architectural decisions; the artifacts themselves are not part of this standalone repo.