alktype

The binary struct engine: a small Rust crate that takes a BAST (Binary Abstract Syntax Tree) document and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

alktype is a standalone crate with two dependencies: jsonschema (for JSON validation and BAST meta-schema validation) and serde_json (for BAST document parsing). No tokio, no platform deps, no unsafe. Compiles to wasm32-unknown-unknown.

What it is

BAST is a JSON document that describes binary data layouts using a kind-based vocabulary with $defs/$ref for composition. BAST is itself a valid JSON Schema instance (it has a meta-schema), making it self-validating, editor-friendly, and trivially consumable from any language with a JSON parser. See docs/architecture/bast-format.md for the normative format spec.

A BAST document serves three roles simultaneously:

Role	Mechanism	When
Validation spec (bytes)	Compiled ValidationPlan walk over the materialized Value (ADR-012)	Access time (validate_bytes)
Validation spec (JSON)	Standard jsonschema::Validator from a consumer-provided JSON Schema	Load time (build validator), access time (validate_json)
Layout spec	Offset computation from type sizes + field order	Load time (build offset map / packed layout)
Data access	Read/write at computed offsets	Access time (read field, write field)
Wire access (packed)	Compiled ReadPlan (ADR-011) — compile-once, no per-read schema walk	Access time (SequentialReader)

No separate format definition, no separate parser, no separate validator. The BAST document is the single source of truth for the binary format. Adding a new field to a protocol is adding an entry to the BAST fields array — the engine computes the new offsets automatically.

This is the same principle as #[repr(C)] struct field access, but at runtime from a portable JSON document instead of at compile time from language-specific annotations. The BAST document is the ABI contract.

Usage

Build the BAST document with the fluent Rust builder (ADR-009), compile it once into an [AlkTypeEngine], then read/write fields at computed offsets:

use alktype::{AlkTypeEngine, Definitions, Endian, LayoutMode, Schema, FieldValue};

// Channels' 8-byte chunk header: big-endian, packed mode.
let doc = Definitions::new().build_doc("ChunkHeader", Schema::struct_()
    .endian(Endian::Big)
    .field("channel_id", Schema::uint32())
    .field("length",     Schema::uint32()));

// `json_schema: None` — no JSON-validation path needed for a binary-only schema.
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;

// Write a frame into a buffer. For fixed-size structs, the byte
// positions are a direct read off the layout — channel_id at 0,
// length at 4. (For variable-length fields, use LayoutBuilder to
// compute positions from known data sizes.)
let mut buf = vec![0u8; 8];
alktype::data_access::write_u32(&mut buf, 0, 42, "channel_id", Endian::Big)?;
alktype::data_access::write_u32(&mut buf, 4,  7, "length",     Endian::Big)?;

// Validate the bytes against the BAST document in one call.
engine.validate_bytes(&buf)?; // materializes a Value, then runs the BAST-native validator

// Read the frame back sequentially (packed mode is sequential by
// construction — variable-length fields shift subsequent fields).
let mut reader = engine.sequential_reader().expect("packed mode");
let (name, value) = reader.read_next(&buf)?.expect("first field");
assert_eq!(name, "channel_id");
assert_eq!(value, FieldValue::U32(42));
# Ok::<(), alktype::AlkTypeError>(())

BAST documents may also be authored as plain serde_json::json!{...} literals and passed directly to AlkTypeEngine::compile — the builder is a construction convenience, not a requirement.

use alktype::{AlkTypeEngine, LayoutMode};
use serde_json::json;

let doc = json!({
    "$defs": {
        "ChunkHeader": {
            "kind": "struct",
            "endian": "big",
            "fields": [
                { "name": "channel_id", "kind": "uint32" },
                { "name": "length",     "kind": "uint32" }
            ]
        }
    }
});
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;
# Ok::<(), alktype::AlkTypeError>(())

The 18 BAST kinds

kind	Rust type	Size	Notes
int8	i8	1
int16	i16	2	endian-sensitive
int32	i32	4	endian-sensitive
int64	i64	8	endian-sensitive; JSON precision caveat (ADR-005)
uint8	u8	1
uint16	u16	2	endian-sensitive
uint32	u32	4	endian-sensitive; also the enum/string/bytes length-prefix width
uint64	u64	8	endian-sensitive; JSON precision caveat (ADR-005)
float32	f32	4	endian-sensitive; NaN/inf rejected by validator
float64	f64	8	endian-sensitive; NaN/inf rejected by validator
bool	bool	1	0x00=false, 0x01=true
enum	u32 index	4	index into the values array; bounds-checked by the BAST-native validator
string	length-prefixed UTF-8	4 + N	[length: u32][bytes] by default
bytes	length-prefixed raw bytes	4 + N	[length: u32][bytes] by default
struct	record of fields	composite	nested; field paths are dotted ("header.version")
union	tagged union	composite	byte-offset or field-name discriminator
array	repeated element	composite	fixed-size elements with stride; count required in v1 (D-BAST-004)
record	string-keyed map	composite	[count: u32][key, value]...

The 18 kinds map to the AlkTypeKind Rust enum. AlkTypeKind::to_bast_str/ from_bast_str convert between the enum and the lowercase BAST strings (D-BAST-002). Only struct, union, and enum can appear as named $defs entries; primitives, arrays, and records appear as field/element/ value types via TypeRef.

Two layout modes

The consumer selects the layout mode at engine construction time via AlkTypeEngine::compile(bast_doc, root_name, mode, json_schema). The same BAST document can be compiled in either mode. Decided in ADR-002.

Mode	Use case	Read API	Write API
Packed (LayoutMode::Packed)	Protocol wire formats (SFTP, channels, TTY) — fields packed with no alignment padding; variable-length fields shift subsequent fields	[SequentialReader] (walks fields in order)	[LayoutBuilder] (computes positions from known data sizes)
Aligned (LayoutMode::Aligned)	mmap-friendly formats (metatensor, safetensors) — fixed positions with natural alignment padding; variable-length data lives outside the static layout	[OffsetMap] (random access by field path)	OffsetMap (write at known offsets)

Variable-length handling

• Packed mode: [length: u32][data] inline by default. The LayoutBuilder takes actual data sizes to compute positions; the SequentialReader reads the length prefix to find the data extent.
• Aligned mode: a 4-byte length prefix sits at a known offset; the variable data is not part of the static layout. Offset indirection (the metatensor blob pattern: {offset, length} pointing into a separate data region) is opt-in via the field-level encoding annotation. Fixed-size reservation via maxLength is also supported.

Union discriminators

kind: "union" supports two discriminator kinds (ADR-003):

• Byte-offset — a fixed-size integer (uint8/uint16/uint32) at a known byte offset. The SFTP Packet pattern: byte 0 is the type byte, bytes 1..N are the variant struct. Mapping keys are stringified integers. With all-canonical numeric keys the compiled reader dispatches on the raw integer (no per-read stringification).
• Field-name — a named field within the union. The TypeBox typedef.ts pattern. Mapping keys are string values matching the discriminator field's value. The fields array declares the discriminator field (D-BAST-005), which must be its first entry; the variant must not re-declare it or any shared field. The builder lays out the declared fields first, then the variant's own fields (ADR-011 addendum) — builder, reader, materializer, and validator all agree on that convention.

Variant $refs are resolved lazily — no compile-time inlining step.

Endianness

Per-schema, default little-endian. Set "endian": "big" on the root struct (or via Schema::endian(Endian::Big)) and the engine byte-swaps every multi-byte read/write accordingly. Field-level endian overrides the struct default. SFTP consumers specify big-endian; channels' chunk header is big-endian.

Validation

Two entry points on [AlkTypeEngine], two validators for two input types (ADR-VAL-SPLIT):

• validate_bytes(&[u8]) — for raw byte buffers (channels' chunk header, SFTP packets). Materializes a Value tree from the bytes via the layout engine, then runs the compiled ValidationPlan (0.2.0 used an interpretive BAST walker; 0.3.0 compiles the value-domain constraints — integer ranges, maxLength, enum index bounds, union variant dispatch — once at compile time). No jsonschema involvement; the BAST document is the complete validation spec for bytes (D-BAST-006).
• validate_json(&Value) / is_valid_json(&Value) — for already-parsed JSON (call's OperationSpec.input_schema payloads). Validates against a standard jsonschema::Validator compiled at AlkTypeEngine::compile time from a consumer-provided JSON Schema (the json_schema: Option<&Value> parameter). BAST is not involved — BAST describes bytes, not JSON shape (D-BAST-007).

Both paths return AlkTypeError::Validation(jsonschema::ValidationError<'static>) — one uniform payload, one match arm (D-BAST-009).

Validation is opt-in per operation. High-throughput paths can skip it; security-sensitive paths can validate every frame. The BAST-native validator also fixes a v0.1.0 dead constraint: enum index bounds are now checked (the materializer emits a numeric index; the validator checks it against values.len()).

BAST document shape

Every BAST document has the same top-level shape:

{ "$defs": { "<TypeName>": { ...TypeDef... }, ... } }

• The $defs block is required (D-BAST-003).
• The root type name is a required parameter to AlkTypeEngine::compile(bast_doc, root_name, mode, ...) (D-BAST-001) — it selects which $defs entry is the top-level type.
• $ref is restricted to #/$defs/<name> — one hash lookup, no normalization pass.

The BAST meta-schema is embedded in the crate as BAST_META_SCHEMA (re-exported from the crate root) and published at https://alk.dev/bast/v1/schema. Consumers can validate a BAST document's structure with any JSON Schema validator. See docs/architecture/bast-format.md for the full spec.

Crate independence

alktype does not depend on any application or networking crate. It defines its own types (AlkTypeError, AlkTypeEngine, FieldValue, etc.) and is usable in contexts where networking doesn't exist — CLI tools, test harnesses, schema-building utilities, and WASM targets.

Schemas as untrusted input

The crate treats BAST documents as untrusted input. A malformed document returns AlkTypeError::Schema from any engine path — never a panic. This matters for hub/spoke topologies where the remote peer provides the schema (e.g. alkcall accepting an OperationSpec from an arbitrary internet peer). Every unreachable!() site in production code was converted to Err ahead of v0.1.0 (review #002, L2); the BAST parser preserves this invariant — overflow-safe arithmetic (checked_add, usize::try_from) on all offset/count casts.

0.3.0 adds compile-time bounds for adversarial schemas: array counts ≤ 2^16 elements, computed array sizes ≤ 2^26 bytes, align ≤ 4096, maxLength ≤ 2^26, and a shared reference-graph guard that rejects cyclic $refs and >128-deep nesting in every public schema walker. Adversarial buffers fail with Access errors at read time — the materializers never preallocate from declared counts. Reviews #006, #007, and #008 document the audit trail (docs/reviews/).

Documentation

Architecture documentation lives under docs/architecture/:

• Overview — purpose, "schema is the format" principle, dependencies, consumers, scope boundaries
• BAST format — normative format spec: meta-schema, TypeRef, TypeDef shapes, validation model
• Schema layer — the BAST parser (BastDoc/BastDef/BastType typed tree), the 18 kinds, the AlkTypeKind enum
• Layout engine — offset computation, the two layout modes, alignment, endianness
• Data access — read/write functions, TUnion dispatch, field paths, zero-copy access
• Validation — the two-validator model, AlkTypeError, load-time vs access-time validation
• Builder — fluent Rust API for constructing BAST documents and standard JSON Schemas at runtime
• Architecture decisions (ADRs) — purpose/scope (ADR-001), BAST format (ADR-BAST), two-validator model (ADR-VAL-SPLIT), two layout modes (ADR-002), schema annotations (ADR-003), error handling (ADR-004), int64/uint64 kinds (ADR-005), non-final inline variable fields (ADR-006), packed-mode read factory (ADR-007), TUnion in aligned mode (ADR-008), builder API (ADR-009), validate_bytes (ADR-010), compiled read plan (ADR-011), plan fingerprinting + ValidationPlan (ADR-012)

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.