alktype/docs/architecture/overview.md

status, last_updated

status	last_updated
draft	2026-07-22

alktype — Overview

The binary struct engine: a small Rust crate that takes a JSON Schema with TypeDef:* custom keywords and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic.

This document covers the crate's purpose, the "schema is the format" principle, its dependency edges, consumers, and scope boundaries. Component details are in the sibling documents.

What

alktype is a library crate that consumes JSON Schemas annotated with TypeDef:* custom keywords (the same kinds defined in TypeBox's typedef.ts, plus TypeDef:Bytes, TypeDef:Int64, and TypeDef:Uint64 as alktype additions) and produces three capabilities:

1. An offset map — walks the schema, computes byte offsets for each field based on type sizes, field order, and alignment.
2. Read/write functions — given a &[u8] buffer and a field path, read the field's bytes at its offset (zero-copy for fixed-size types). Given a &mut [u8] buffer, write a value at its offset.
3. Validation — via jsonschema custom keywords, validates that a buffer's bytes match the schema's type constraints.

The heavy lifting is done by the jsonschema crate (validation) and serde_json (schema parsing). The novel code is the offset computation — a recursive walk of the schema JSON that computes byte positions for each field. The custom keyword implementations are small (a few lines each, generated from shared macros — see validation.md).

The crate replaces two prior attempts that built their own jsonschema engines — typebox-rs (~8,400 lines) and the @alkimiadev/alktype prototype (~5,600 lines) — with jsonschema + an offset map + small custom keyword implementations. See ADR-001.

Why

The crate's purpose is to be a binary struct engine for components that read or write binary data at computed offsets. Instead of per-protocol serde structs (russh-sftp's 29 packet types), per-handler wire format code (TTY's 5-byte format parser), or per-format offset computation (metatensor's tensor access), all of these become instances of the same engine with different schemas.

The guiding insight:

The schema is the format. A JSON Schema with TypeDef:Float32, TypeDef:Struct, TypeDef:Union etc. is both the validation spec and the layout spec. No separate format definition, no separate parser, no separate validator. One schema, three uses: validate, compute offsets, access data.

This is the convergence of three threads identified in the call-channels-unification research: the typedef.ts schema kinds from TypeBox, the russh-sftp protocol packets, and the metatensor format. The common pattern: a JSON Schema describes the shape of binary data, and the binary data is the struct's bytes at computed offsets.

The crate was bumped up in the timeline when the call-channels-unification research surfaced that channels, TTY, and the binary call protocol are all variations on the same wire-format family — [discriminant][length][payload]. The typedef engine makes the "channels is call with a binary data plane" unification concrete: the binary data plane's wire format is the call protocol's own schema system, just binary-encoded. The channel_open marker says "use binary framing"; the typedef engine says "here's how to read/write the binary payload."

The "Schema Is the Format" Principle

A JSON Schema with TypeDef:* custom keywords serves three roles simultaneously:

Role	Mechanism	When
Validation spec	jsonschema custom keywords	Load time (build validator), access time (validate buffer)
Layout spec	Offset computation from type sizes + field order	Load time (build offset map)
Data access	Read/write at computed offsets	Access time (read field, write field)

No separate format definition, no separate parser, no separate validator. The schema is the single source of truth for the binary format. Adding a new field to a protocol is adding a property to the schema JSON — the engine computes the new offsets automatically.

This is the same principle as #[repr(C)] struct field access, but at runtime from a portable JSON Schema instead of at compile-time from language-specific annotations. The schema is the ABI contract.

Dependencies

alktype
├── jsonschema (v0.46.5, Draft 2020-12) — validation engine, custom keyword support
├── serde_json (with preserve_order)     — schema parsing; field order is load-bearing
└── (no tokio, no platform deps)         — WASM-clean by construction

alktype is dependency-light: jsonschema + serde_json only. No tokio, no platform deps. Compiles to wasm32-unknown-unknown for browser use. The jsonschema crate is already in the workspace at /workspace/jsonschema/ — alktype is its first consumer.

serde_json requires the preserve_order feature because field order is load-bearing for binary layouts. The order of properties in the schema JSON determines the order of fields in the binary struct.

Consumers

Consumer	Schema describes	Engine provides
russh-sftp	29 packet structs + Packet union (byte discriminator)	Read/write SFTP frames from bytes
metatensor	Model layout (ConvNet struct, tensor refs)	Offset map for mmap'd tensor access
binary call frames	call.requested / call.responded / etc. structs	Read/write binary call frames
TTY negotiation	NegotiateRequest / NegotiateResponse structs	Read/write TTY control frames
channels wire	ChunkHeader { channel_id, length }	Already trivial (8 bytes, no schema needed)

The russh-sftp case is the most instructive and the highest-value POC target. The Packet enum's TryFrom<&mut Bytes> impl is a hand-written dispatch on a type byte followed by serde deserialization. Under typedef, the dispatch is TUnion with a byte-offset discriminator — the schema says "byte 0 is the discriminator, bytes 1..N are the variant struct." The engine reads the discriminator, looks up the variant schema, computes offsets, reads fields. Same result, no per-packet-type code.

Scope Boundaries (What This Is Not)

These boundaries are decided in ADR-001.

• Not metatensor. typedef is the binary struct engine. Metatensor is a format (8-byte header + JSON header + binary data) that uses the typedef engine for its offset computation and tensor access.
• Not a Value system. TypeBox's Value.Diff, Value.Migrate, Value.Convert — schema evolution — is out of scope for v1. The engine should not do anything that explicitly blocks adding a Value system later.
• Not a code generator. typebox-rs's codegen/ module is a separate concern. The typedef engine consumes schemas; it does not generate them.
• Not a schema builder. The typedef engine does not provide a fluent API for constructing schemas. Schemas are plain JSON — authored in TypeBox, generated by ujsx components, or hand-written. A builder API is deferred (OQ-003).
• Not a serialization framework. The typedef engine is not a general-purpose serde replacement. It operates on raw byte buffers at computed offsets — no intermediate Value tree, no reflection, no dynamic dispatch per field. For JSON data, use serde. For binary data with a known schema, use typedef.

Architecture (component pointers)

• schema-layer.md — the 19 TypeDef:* kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations (endianness, alignment, encoding, TUnion discriminators).
• layout-engine.md — offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length field handling.
• data-access.md — read/write functions, TUnion dispatch, field paths, zero-copy access for fixed-size types, length-prefix reading for variable-length types.
• validation.md — custom keyword validators for all 19 TypeDef:* kinds, TypedefError, load-time vs access-time validation, TypedefEngine as the compiled form of a schema.

Design Decisions

Decision	ADR	Summary
Purpose, scope, and the jsonschema engine	ADR-001	What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries
Two layout modes	ADR-002	Packed sequential (LayoutBuilder/SequentialReader) for protocols; aligned static (OffsetMap) for mmap formats
Schema annotations	ADR-003	Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name)
Error handling and validation	ADR-004	TypedefError enum; load-time build, access-time check; field-path-carrying errors; jsonschema ValidationError wrapping
Int64/Uint64 kinds	ADR-005	64-bit integers as first-class kinds (SFTP offsets, metatensor data_offsets)
Non-final inline variable fields	ADR-006	Rejected in aligned mode (would clobber subsequent fields)
Packed-mode read factory	ADR-007	engine.sequential_reader() returns an owned fresh reader
TUnion in aligned mode	ADR-008	Rejected for v1 (broken semantics; no current consumer needs it)

Open Questions

See open-questions.md for full details.

• OQ-001 (deferred(scope)): Arrays of variable-length-element structs.
• OQ-002 (deferred(scope)): no_std + alloc support.
• OQ-003 (deferred(scope)): Builder API for schema construction.

References

• docs/research/alknet-typedef/findings.md — POC results (26 tests passing, two layout modes, TUnion dispatch, endianness)
• docs/research/call-channels-unification/findings.md §"alknet-typedef: JSON Schema as the binary struct engine" — the origin of this research thread
• /workspace/@alkdev/typebox/example/typedef/typedef.ts — the TypeBox schema kinds (619 lines)
• /workspace/jsonschema/ — the jsonschema crate (v0.46.5, Draft 2020-12)
• /workspace/alknet-typedef-poc/ — the POC code (disposable)
• /workspace/@alkimiadev/typebox-rs/ — prior attempt, replaced by typedef
• /workspace/@alkimiadev/alktype/ — prior attempt (the @alkimiadev/alktype prototype; not to be confused with this crate, which reuses the name but is backed by the jsonschema crate)