alktype/docs/architecture/schema-layer.md

status, last_updated

status	last_updated
accepted	2026-08-15

alktype — Schema Layer

The schema layer: the BAST (Binary Abstract Syntax Tree) format and the typed parser that the layout engines, materializer, and BAST-native validator walk. BAST replaces the v0.1.0 AlkType:* custom-keyword JSON Schema format decided in ADR-001; the pivot is recorded in ADR-BAST and grounded in D-BAST-001..009.

The normative format specification is bast-format.md (meta-schema, TypeRef, examples, validation model). This document describes the implementation — the typed parser in src/bast.rs and the foundational AlkTypeKind enum in src/schema.rs — and points at the format spec for shape details.

The 18 BAST Kinds

BAST uses lowercase kind strings ("uint32", "struct", "union", etc.). The engine represents them as the AlkTypeKind Rust enum — one variant per kind — providing compile-time exhaustiveness checking and integer-discriminant dispatch (a jump table) instead of string comparison at every field access.

BAST kind	AlkTypeKind	Rust type	Size	Category
int8	Int8	i8	1	fixed
int16	Int16	i16	2	fixed
int32	Int32	i32	4	fixed
int64	Int64	i64	8	fixed
uint8	Uint8	u8	1	fixed
uint16	Uint16	u16	2	fixed
uint32	Uint32	u32	4	fixed
uint64	Uint64	u64	8	fixed
float32	Float32	f32	4	fixed
float64	Float64	f64	8	fixed
bool	Boolean	bool (0x00=false, 0x01=true)	1	fixed
string	String	length-prefixed UTF-8	variable	variable
bytes	Bytes	length-prefixed raw bytes	variable	variable
struct	Struct	record of fields	sum of field sizes	composite
union	Union	tagged union	discriminator + variant	composite
array	Array	repeated element	count × element size	composite
enum	Enum	u32 index into enum values	4 (fixed)	fixed
record	Record	count-prefixed (key, value) pairs	variable	variable

int64/uint64 are alktype additions — TypeBox's typedef.ts tops out at 32-bit integers. Required by SFTP Read/Write offset: u64 and metatensor data_offsets. See ADR-005.

The AlkTypeKind enum

src/schema.rs defines the enum:

pub enum AlkTypeKind {
    Int8, Int16, Int32, Int64,
    Uint8, Uint16, Uint32, Uint64,
    Float32, Float64,
    Boolean, Enum,
    String, Bytes,
    Struct, Union, Array, Record,
}

The enum carries the kind's binary-layout metadata as inherent methods:

Method	Returns	Notes
to_bast_str(self)	&'static str	The lowercase BAST kind string ("uint32")
from_bast_str(s)	Result<AlkTypeKind, AlkTypeError>	Parses a lowercase BAST kind string; AlkTypeError::Schema for unknowns
type_size(self)	Option<usize>	Some(N) for fixed-size kinds; None for variable/composite
natural_alignment(self)	usize	1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array
is_fixed_size(self)	bool	True for the 12 fixed-size primitive kinds
is_composite(self)	bool	True for Struct, Union, Array, Record
is_variable_length(self)	bool	True for String, Bytes, Record
needs_endian(self)	bool	True for kinds whose read/write takes an Endian parameter

AlkTypeKind implements Display, backed by to_bast_str so the layout engines, materializer, validator, and parser surface the canonical BAST name in error messages. from_bast_str is the inverse and is the dispatch point the BAST parser uses to map a kind string to the enum variant (D-BAST-002).

Foundational annotation types

src/schema.rs also defines the two annotation enums (semantics unchanged from ADR-003; only their location in the document moved — see ADR-BAST and bast-format.md §Variable-Length Encoding):

pub enum Endian { Little, Big }
pub enum VariableEncoding { LengthPrefixed, OffsetIndirect }

The Discriminator builder enum lives in src/builder.rs (the builder's domain); the BAST parser's typed discriminator view is BastDiscriminator.

The BAST Parser (bast module)

src/bast.rs is the typed surface over a BAST document. Three consumers walk the same tree — the layout engines (offset_map, layout_builder, sequential_reader), the materialize layer, and the bast_validation validator — so a typed view pays for itself: each walks matched arms over BastType instead of re-parsing raw JSON at every node. Borrowing (not cloning) the source serde_json::Value keeps the parse allocation-free beyond the small typed nodes themselves.

Document shape

Every BAST document has the same top-level shape:

{ "$defs": { "<TypeName>": { ...TypeDef... }, ... } }

• The $defs block is required (D-BAST-003). Single-type documents are a special case with one entry.
• The root type name is a required parameter to AlkTypeEngine::compile(bast_doc, root_name, mode, ...) (D-BAST-001). It selects which $defs entry is the top-level type; convention (first entry) is fragile and depends on JSON key order, so an explicit parameter is used instead.

See bast-format.md for the normative TypeDef shapes (Struct, Union, Enum, FieldDef, TypeRef) and the meta-schema.

Typed tree

The parser produces a borrowed typed tree:

Type	Role
BastDoc<'a>	The parsed document: the root Value, the chosen root name, and the parsed root BastDef. Entry point via BastDoc::new(root, root_name).
BastDef<'a>	A named $defs entry — { name, kind: BastDefKind, source }. Only struct/union/enum can live at the top level.
BastDefKind<'a>	Struct(BastStruct) / Union(BastUnion) / Enum(BastEnum).
BastStruct<'a>	{ endian, align, fields: Vec<BastField>, source }. Field order is the fields array order (BAST design principle #4 — no reliance on serde_json's preserve_order).
BastField<'a>	{ name, ty: BastType, endian, align, encoding, max_length, source }. Annotations are field-level properties (ADR-003 semantics, BAST location).
BastUnion<'a>	{ endian, discriminator, fields, mapping: Vec<(key, BastType)>, source }. Variant $refs resolve lazily — no compile-time inlining.
BastDiscriminator<'a>	Byte { offset, disc_type } / Field { name }. The typed view of the discriminator object.
BastEnum<'a>	{ values: Vec<&'a str>, source }. Non-empty (enforced).
BastType<'a>	A TypeRef — Primitive(AlkTypeKind) / Ref(BastRef) / Array(BastArray) / Record(BastRecord) / Struct(...) / Union(...) / Enum(...). The central mechanism for typing fields, array elements, record values, and union variants.
BastRef<'a>	A $ref restricted to #/$defs/<name>. Carries just the name.
BastArray<'a>	{ element: Box<BastType>, count, source }. count is required in v1 (D-BAST-004).
BastRecord<'a>	{ values: Box<BastType>, source }.

All of these are re-exported from the crate root (pub use bast::{...} in src/lib.rs).

$ref resolution

BAST $refs are always full JSON Pointers restricted to #/$defs/<name> — no external references, no fragment-only pointers, no bare names (rejected by the parser). The restriction keeps resolution a single hash lookup and eliminates the v0.1.0 normalize_refs pass that rewrote TypeBox's bare-name refs.

BastDoc exposes three resolution helpers:

Method	Purpose
lookup_def(name) -> Result<&'a Value, AlkTypeError>	The single hash lookup into $defs.
resolve_ref(r: &BastRef) -> Result<BastDef, AlkTypeError>	Resolve a BastRef to its BastDef.
resolve_typeref(ty: &BastType) -> Result<BastType, AlkTypeError>	Deref one $ref level, or return the inline type unchanged. The composite-walkers call this.
resolve_typeref_as_def(ty, path) -> Result<BastDef, AlkTypeError>	Resolve a BastType to a BastDef, wrapping inline composites in a synthetic def. Convenient for the validator/materializer.

Variant $refs (union mapping entries) are resolved lazily by the materializer and validator via these helpers — no inline_union_variant_refs compile step (removed under BAST). The parser only records the BastRef target name.

Untrusted input

Every path that walks a BAST document returns Err(AlkTypeError::Schema) on a malformed document, never panic!/unreachable!/unwrap (AGENTS.md §3 — the downstream alkcall consumer accepts schemas from arbitrary internet peers in its hub/spoke topology). Overflow-safe arithmetic (checked_add, usize::try_from) is used for any offset/count cast (AGENTS.md §4).

A malformed document (missing $defs, missing kind, unknown kind string, dangling $ref, empty mapping, non-struct/union/enum at the top level, a field-name union without a fields array, etc.) surfaces as AlkTypeError::Schema with a path-annotated message.

What the parser does not do

• No meta-schema validation. BastDoc::new parses structurally (every reachable def parses to a typed BastDef) but does not run the BAST meta-schema. Consumers that want full structural validation can run the meta-schema via jsonschema directly (BAST_META_SCHEMA is re-exported from the crate root). The parser's structural checks catch the cases that matter for layout/materialize/validate; the meta-schema is the authoritative well-formedness check.
• No eager full-document parse. Only the root definition and the definitions it (transitively) references are parsed eagerly; orphan $defs entries are not checked. Lazy $ref resolution reaches the rest at access time.
• No annotation interpretation. The parser records endian/ align/encoding/maxLength on BastField/BastStruct; the layout engines and validator interpret them (ADR-003 semantics).

Schema Annotations

Annotation semantics carry forward unchanged from ADR-003; only their location moved from v0.1.0's custom-keyword objects to BAST type-level properties. The concrete BAST shapes are in bast-format.md:

• Endianness — struct/union-level endian with field-level override.
• Alignment — struct/field-level align (aligned mode only).
• Variable-length encoding — field-level encoding and maxLength.
• Union discriminators — discriminator object on the union def (byte or field).

The maxLength keyword is not a BAST invention — it is the standard JSON Schema maxLength, repurposed as a byte-length cap. In aligned mode it reserves a fixed-size slot; in packed mode it is a validation constraint only. It applies to string/bytes fields only: the parser rejects it on any other kind (review #006 N3 — elsewhere it was silently unenforced), and in aligned mode a record reservation was silently corrupt (review #006 M5). See bast-format.md §Variable-Length Encoding and ADR-003.

What Was Removed

The v0.1.0 custom-keyword accessor layer was removed in step 8 of the BAST pivot. The schema module retains only the foundational types (AlkTypeKind, Endian, VariableEncoding, shared constants); the BAST parser is the typed surface every engine module walks. Removed:

• get_alktype_kind / get_alktype_kind_enum / get_alktype_kind_loose / get_alktype_kind_loose_enum — superseded by the parser's kind-string dispatch.
• normalize_refs / inline_union_variant_refs (+ recursive helpers) — BAST refs are always #/$defs/<name>; one hash lookup, variant refs resolve lazily.
• parse_encoding / parse_align / parse_max_length / parse_endian — bast.rs has its own BAST-property-form copies (internal to the parser).
• parse_discriminator + DiscriminatorKind — replaced by bast::BastDiscriminator; the builder has its own Discriminator enum.
• resolve_ref / resolve_ref_or_inline — replaced by BastDoc::lookup_def / resolve_typeref.
• FromStr impl, as_str, Endian::from_schema, ALKTYPE_PREFIX, BYTE_DISCRIMINATOR_TYPES, and their unit tests.

See ADR-BAST §"What is removed" and bast-format.md §What is removed.

Design Decisions

Decision	ADR	Summary
BAST format, meta-schema, $defs/$ref/kind vocabulary	ADR-BAST	Supersedes ADR-001's format-specific content; records D-BAST-001..009
Schema annotations	ADR-003	Annotation semantics (carry forward unchanged; only location moves)
Int64/Uint64 kinds	ADR-005	64-bit integers as first-class kinds (required by SFTP offsets and metatensor data_offsets)
Purpose and scope	ADR-001	Why jsonschema not a custom engine; "schema is the format" principle (format-specific content superseded by ADR-BAST)

Open Questions

See open-questions.md for full details.

• OQ-001 (deferred(scope)): Arrays of variable-length-element structs — BAST arrays require count in v1 (D-BAST-004), aligning with this deferral.

References

• bast-format.md — the normative BAST format specification (meta-schema, TypeRef, examples, validation model)
• ADR-BAST — the BAST format decision
• ADR-003 — annotation semantics
• BAST pivot research record — motivation, POC scope and result, decisions D-BAST-001..009
• validation.md — the BAST-native validator and the validate_json JSON-Schema path
• src/bast.rs — the parser implementation
• src/schema.rs — the AlkTypeKind enum and foundational annotation types