alktype/README.md

alktype

The binary struct engine: a small Rust crate that takes a JSON Schema with AlkType:* custom keywords and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

alktype is a standalone crate with two dependencies: jsonschema (for validation) and serde_json (for schema parsing). No tokio, no platform deps, no unsafe. Compiles to wasm32-unknown-unknown.

What it is

A JSON Schema annotated with AlkType:* custom keywords serves three roles simultaneously:

Role	Mechanism	When
Validation spec	jsonschema custom keywords	Load time (build validator), access time (validate buffer)
Layout spec	Offset computation from type sizes + field order	Load time (build offset map / packed layout)
Data access	Read/write at computed offsets	Access time (read field, write field)

No separate format definition, no separate parser, no separate validator. The schema is the single source of truth for the binary format. Adding a new field to a protocol is adding a property to the schema JSON — the engine computes the new offsets automatically.

This is the same principle as #[repr(C)] struct field access, but at runtime from a portable JSON Schema instead of at compile time from language-specific annotations. The schema is the ABI contract.

Usage

Build the schema with the fluent Rust builder (ADR-009), compile it once into an [AlkTypeEngine], then read/write fields at computed offsets:

use alktype::{AlkTypeEngine, Endian, LayoutMode, Schema, FieldValue};

// Channels' 8-byte chunk header: big-endian, packed mode.
let mut schema = Schema::struct_()
    .endian(Endian::Big)
    .field("channel_id", Schema::uint32())
    .field("length",     Schema::uint32())
    .build();

let engine = AlkTypeEngine::compile(&mut schema, LayoutMode::Packed)?;

// Write a frame into a buffer. For fixed-size structs, the byte
// positions are a direct read off the layout — channel_id at 0,
// length at 4. (For variable-length fields, use LayoutBuilder to
// compute positions from known data sizes.)
let mut buf = vec![0u8; 8];
alktype::data_access::write_u32(&mut buf, 0, 42, "channel_id", Endian::Big)?;
alktype::data_access::write_u32(&mut buf, 4,  7, "length",     Endian::Big)?;

// Validate the bytes against the schema in one call.
engine.validate_bytes(&buf)?; // materializes a Value, then validates

// Read the frame back sequentially (packed mode is sequential by
// construction — variable-length fields shift subsequent fields).
let mut reader = engine.sequential_reader().expect("packed mode");
let (name, value) = reader.read_next(&buf)?.expect("first field");
assert_eq!(name, "channel_id");
assert_eq!(value, FieldValue::U32(42));
# Ok::<(), alktype::AlkTypeError>(())

Schemas may also be authored as plain serde_json::json!{...} literals and passed directly to AlkTypeEngine::compile — the builder is a construction convenience, not a requirement.

The 19 AlkType:* kinds

Kind	Rust type	Size	Notes
AlkType:Int8	i8	1
AlkType:Int16	i16	2	endian-sensitive
AlkType:Int32	i32	4	endian-sensitive
AlkType:Int64	i64	8	endian-sensitive; JSON precision caveat (ADR-005)
AlkType:Uint8	u8	1
AlkType:Uint16	u16	2	endian-sensitive
AlkType:Uint32	u32	4	endian-sensitive; also the enum/string/bytes length-prefix width
AlkType:Uint64	u64	8	endian-sensitive; JSON precision caveat (ADR-005)
AlkType:Float32	f32	4	endian-sensitive; NaN/inf rejected by validator
AlkType:Float64	f64	8	endian-sensitive; NaN/inf rejected by validator
AlkType:Boolean	bool	1
AlkType:Enum	u32 index	4	index into the schema's "enum" array
AlkType:String	length-prefixed UTF-8	4 + N	[length: u32][bytes] by default
AlkType:Bytes	length-prefixed raw bytes	4 + N	[length: u32][bytes] by default
AlkType:Timestamp	length-prefixed RFC 3339	4 + N	non-strict string check (see inline docs)
AlkType:Struct	record of fields	composite	nested; field paths are dotted ("header.version")
AlkType:Union	tagged union	composite	byte-offset or field-name discriminator
AlkType:Array	repeated element	composite	fixed-size elements with stride, or variable count
AlkType:Record	string-keyed map	composite	[count: u32][key, value]...

The engine recognizes a kind when the schema object has a key starting with AlkType: whose value is true (the boolean shorthand) or an annotation object (e.g. { "AlkType:String": { "encoding": "offset-indirect" } }).

Two layout modes

The consumer selects the layout mode at engine construction time via AlkTypeEngine::compile(schema, mode). The same schema can be compiled in either mode. Decided in ADR-002.

Mode	Use case	Read API	Write API
Packed (LayoutMode::Packed)	Protocol wire formats (SFTP, channels, TTY) — fields packed with no alignment padding; variable-length fields shift subsequent fields	[SequentialReader] (walks fields in order)	[LayoutBuilder] (computes positions from known data sizes)
Aligned (LayoutMode::Aligned)	mmap-friendly formats (metatensor, safetensors) — fixed positions with natural alignment padding; variable-length data lives outside the static layout	[OffsetMap] (random access by field path)	OffsetMap (write at known offsets)

Variable-length handling

• Packed mode: [length: u32][data] inline by default. The LayoutBuilder takes actual data sizes to compute positions; the SequentialReader reads the length prefix to find the data extent.
• Aligned mode: a 4-byte length prefix sits at a known offset; the variable data is not part of the static layout. Offset indirection (the metatensor blob pattern: {offset, length} pointing into a separate data region) is opt-in via the encoding annotation.

TUnion discriminators

AlkType:Union supports two discriminator kinds (ADR-003):

• Byte-offset — a fixed-size integer at a known byte offset. The SFTP Packet pattern: byte 0 is the type byte, bytes 1..N are the variant struct. Mapping keys are stringified integers.
• Field-name — a named field within the struct. The TypeBox typedef.ts pattern. Mapping keys are string values matching the discriminator field's value.

Endianness

Per-schema, default little-endian. Set "endian": "big" on the top-level schema (or via Schema::endian(Endian::Big)) and the engine byte-swaps every multi-byte read/write accordingly. SFTP consumers specify big-endian; channels' chunk header is big-endian.

Validation

Two entry points on [AlkTypeEngine], one underlying jsonschema validator (ADR-010):

• validate_json(&Value) / is_valid_json(&Value) — for already-parsed JSON (call's payload schemas).
• validate_bytes(&[u8]) — materializes a Value tree from the bytes via the layout engine, then validates that Value. Single-call binary buffer validation.

The validator is compiled once at load time; access-time validation is a fast is_valid() check. High-throughput paths can skip validation; security-sensitive paths can validate every frame.

Crate independence

alktype does not depend on any application or networking crate. It defines its own types (AlkTypeError, AlkTypeEngine, FieldValue, etc.) and is usable in contexts where networking doesn't exist — CLI tools, test harnesses, schema-building utilities, and WASM targets. The upcoming alkcall crate (the alknet-call + alknet-channels unification) depends on alktype for both binary layout and JSON payload schemas; alktype knows nothing about alkcall.

Schemas as untrusted input

The crate treats schemas as untrusted input. A malformed schema returns AlkTypeError::Schema / AlkTypeError::Offset from any engine path — never a panic. This matters for hub/spoke topologies where the remote peer provides the schema (e.g. alkcall accepting an OperationSpec from an arbitrary internet peer). All unreachable!() sites in production code were converted to Err ahead of v0.1.0 (review #002, L2).

Documentation

Architecture documentation lives under docs/architecture/:

• Overview — purpose, "schema is the format" principle, dependencies, consumers, scope boundaries
• Schema layer — the 19 kinds, jsonschema custom keyword integration, schema annotations
• Layout engine — offset computation, the two layout modes, alignment, endianness
• Data access — read/write functions, TUnion dispatch, field paths, zero-copy access
• Validation — custom keyword validators, AlkTypeError, load-time vs access-time validation
• Builder — fluent Rust API for constructing alktype JSON Schemas at runtime
• Architecture decisions (ADRs) — purpose/scope, two layout modes, schema annotations, error handling, int64/uint64 kinds, packed-mode read factory, TUnion in aligned mode, builder API, validate_bytes

License

Licensed under either of

• Apache License, Version 2.0 (LICENSE-APACHE or http://www.apache.org/licenses/LICENSE-2.0)
• MIT license (LICENSE-MIT or http://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.