alktype/docs/architecture/schema-layer.md

status, last_updated

status	last_updated
draft	2026-07-22

alknet-typedef — Schema Layer

The schema layer: the 19 TypeDef:* custom type kinds, their mapping to Rust types and byte sizes, the jsonschema custom keyword integration, TypeBox interop, and the concrete JSON shapes for schema-level annotations.

The 19 TypeDef Kinds

These are the custom schema kinds defined in TypeBox's typedef.ts (/workspace/@alkdev/typebox/example/typedef/typedef.ts, 619 lines) and ported to Rust via jsonschema custom keywords. Each kind carries binary layout semantics — a known byte size (for fixed-size types) or a known encoding strategy (for variable-length types).

Kind	TypeBox key	Rust type	Size	Category
TFloat32	TypeDef:Float32	f32	4	fixed
TFloat64	TypeDef:Float64	f64	8	fixed
TInt8	TypeDef:Int8	i8	1	fixed
TInt16	TypeDef:Int16	i16	2	fixed
TInt32	TypeDef:Int32	i32	4	fixed
TInt64	TypeDef:Int64	i64	8	fixed
TUint8	TypeDef:Uint8	u8	1	fixed
TUint16	TypeDef:Uint16	u16	2	fixed
TUint32	TypeDef:Uint32	u32	4	fixed
TUint64	TypeDef:Uint64	u64	8	fixed
TBoolean	TypeDef:Boolean	bool (0x00=false, 0x01=true)	1	fixed
TString	TypeDef:String	length-prefixed UTF-8	variable	variable
TBytes	TypeDef:Bytes	length-prefixed raw bytes	variable	variable
TStruct	TypeDef:Struct	record of fields	sum of field sizes	composite
TUnion	TypeDef:Union	tagged union	discriminator + variant	composite
TArray	TypeDef:Array	repeated element	count × element size	composite
TEnum	TypeDef:Enum	u32 index into enum values	4 (fixed)	fixed
TRecord	TypeDef:Record	count-prefixed sequence of (key, value) pairs	variable	variable
TTimestamp	TypeDef:Timestamp	length-prefixed RFC 3339 string	variable	variable

TypeDef:Int64 and TypeDef:Uint64 are alknet-typedef additions — TypeBox's typedef.ts tops out at 32-bit integers. They are required by the primary POC targets: SFTP Read/Write packets have offset: u64, and metatensor data_offsets are u64. See ADR-099.

The TypeDefKind enum

The engine represents the 19 kinds as a Rust enum — TypeDefKind — with one variant per kind (TypeDefKind::Float32, TypeDefKind::Struct, etc.). The enum provides compile-time exhaustiveness checking and integer discriminant dispatch (a jump table) instead of string comparison at every field access. It is pub and re-exported from the crate root.

pub enum TypeDefKind {
    Int8, Int16, Int32, Int64,
    Uint8, Uint16, Uint32, Uint64,
    Float32, Float64,
    Boolean, Enum,
    String, Bytes, Timestamp,
    Struct, Union, Array, Record,
}

The enum carries the kind's binary-layout metadata as inherent methods:

Method	Returns	Notes
as_str(self)	&'static str	The JSON Schema keyword, e.g. "TypeDef:Uint8"
type_size(self)	Option<usize>	Some(N) for fixed-size kinds; None for variable/composite
natural_alignment(self)	usize	1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array
is_fixed_size(self)	bool	True for the 12 fixed-size primitive kinds
is_composite(self)	bool	True for Struct, Union, Array, Record
is_variable_length(self)	bool	True for String, Bytes, Timestamp, Record
needs_endian(self)	bool	True for kinds whose read/write takes an Endian parameter

TypeDefKind implements Display (renders the keyword string) and FromStr (parses the keyword string back into the variant, returning TypedefError::Schema for unknown kinds). The layout engines and the validator dispatch on the enum, not on strings.

Fixed-size types

TFloat32, TFloat64, TInt8, TInt16, TInt32, TUint8, TUint16, TUint32, TBoolean, and TEnum have known byte sizes. The offset computation uses these sizes directly. Read/write is zero-copy pointer cast for these types.

TBoolean byte representation: 0x00 = false, 0x01 = true. Other values are invalid and produce a TypedefError::Access on read.

TEnum binary representation: A u32 index into the enum's declared values, in declaration order. The first declared value is index 0, the second is index 1, etc. The enum's values are declared via the standard JSON Schema "enum" keyword (e.g., "enum": ["read", "write", "execute"]). The TypeDef:Enum custom keyword signals that the type is an enum for layout purposes; the built-in enum keyword provides the value list.

Design note: TypeBox's TEnum is a string enum (variable-length). The typedef engine uses a u32 index instead — a deliberate deviation from TypeBox fidelity in favor of binary efficiency. Most enums have a small number of variants (e.g., the call protocol's 5 event types); a u32 index is compact, fixed-size, and sufficient for any realistic enum. The JSON representation (for validation) remains a string; the binary representation is the u32 index. The u32 index follows the schema's endianness annotation (ADR-097), like all other fixed-size types. In little-endian mode the index is u32::from_le_bytes; in big-endian mode it is u32::from_be_bytes.

Variable-length types

TString, TBytes, TRecord, and TTimestamp have variable byte sizes. The typedef engine supports three strategies for handling variable-length types in binary layouts, selected by the encoding annotation and the standard JSON Schema maxLength keyword:

Strategy	Encoding annotation	Layout behavior	Use case
Inline length-prefixed	"length-prefixed" (default)	[length: u32][data]; shifts subsequent fields in packed mode	Protocol wire formats (SFTP, channels, TTY)
Fixed-size reservation	(none — uses maxLength)	[data: maxLength bytes], zero-padded; fixed offset in aligned mode	mmap-friendly formats where max size is known (database VARCHAR(N) pattern)
Offset indirection	"offset-indirect"	{offset: u32, length: u32} pointing into a separate data region	Blob tensors, metatensor variable-length data (the blob tensor pattern)

Strategy 1: Inline length-prefixing (default). The field's fixed portion is a 4-byte length prefix at a computed offset. The variable data follows immediately after. In packed sequential mode, the length prefix determines the position of subsequent fields. In aligned static mode, the length prefix is at a known offset; the variable data is not included in the static layout. This is the universal pattern used by channels, SFTP, TTY, and most binary protocols.

Strategy 2: Fixed-size reservation. When a variable-length field declares maxLength (a standard JSON Schema keyword), the engine reserves maxLength bytes at a fixed offset in aligned static mode. Data shorter than maxLength is zero-padded; data longer than maxLength is a validation error. This makes the field fixed-size from the layout perspective — subsequent fields have known, unchanging offsets. This is the database VARCHAR(N) pattern and the metatensor struct-tensor pattern for fields with known maximum sizes.

In packed sequential mode, maxLength is a validation constraint only — the engine still uses inline length-prefixing (strategy 1) because protocols don't benefit from fixed-size reservation.

Strategy 3: Offset indirection. The field is a struct {offset: u32, length: u32} at a known position. The consumer provides the data region separately; the engine reads the offset and length, then slices the data region. This is the metatensor blob tensor pattern — the index struct lives in one region, the blob data lives in another. Enables mmap-friendly random access to variable-length data without parsing length prefixes and without reserving worst-case space.

Default strategy selection:

• In packed sequential mode: always strategy 1 (inline length-prefixing). maxLength is a validation constraint only.
• In aligned static mode: strategy 2 (fixed-size reservation) if maxLength is declared; strategy 3 (offset indirection) if "encoding": "offset-indirect" is declared; strategy 1 (inline length-prefixing) otherwise.

Length prefix endianness: The 4-byte length prefix (strategies 1 and 3) respects the schema's "endian" annotation (ADR-097). In little-endian mode, the length is u32::from_le_bytes. In big-endian mode, the length is u32::from_be_bytes. This ensures SFTP consumers (big-endian) have consistent byte order for both field values and length prefixes.

TBytes: Raw bytes — no UTF-8 constraint. The payload is &[u8]. Otherwise identical to TString in layout (same three strategies).

Design note: TypeDef:Bytes is an alknet-typedef addition — it does not exist in TypeBox's typedef.ts (which defines 16 kinds). It is included because raw byte arrays are a common binary protocol primitive (SFTP data payloads, channels payloads, tensor data) and are semantically distinct from UTF-8 strings. In the binary representation, TBytes is raw bytes with no encoding (not base64, not hex). In the JSON representation (for validation), TBytes is a string (JSON has no native byte type).

TRecord: A string-keyed map. The value type is declared via the schema's "values" property (e.g., "values": { "TypeDef:Float32": true }). Binary layout is a count-prefixed sequence of (key, value) pairs: [count: u32][key_len: u32][key_bytes][value]... repeated count times. The count is the number of entries. Each key is a length-prefixed UTF-8 string. Each value is encoded according to its declared TypeDef:* kind — a Record<Uint32> value is 4 raw bytes; a Record<String> value is itself a length-prefixed string; a Record<Struct> value is the struct's fields laid out inline. There is no separate value_len prefix — the value's size is determined by its kind (fixed-size kinds have a known size; variable-length kinds carry their own length prefix). The count and key-length prefixes respect the schema's endianness. In aligned static mode with maxLength, the entire record is reserved at maxLength bytes (zero-padded).

TTimestamp: An RFC 3339 timestamp string (the internet profile of ISO 8601). Stored as a length-prefixed UTF-8 string (strategy 1) or fixed-size reservation (strategy 2 with maxLength). The data-access layer treats timestamps as opaque length-prefixed strings — it does not parse or validate the timestamp format. The jsonschema custom keyword validator checks RFC 3339 conformance at the JSON level (see validation.md).

TArray is variable-length when the element type is variable-length or when the count is not known at schema time. For fixed-size element arrays with a known count, the size is element_size × count.

TArray count declaration: The array count is declared via the standard JSON Schema "minItems" and "maxItems" keywords. When minItems == maxItems, the array has a fixed count known at schema time. When they differ or are absent, the count is variable and the array uses a length-prefixed encoding: [count: u32][element_0]...[element_N]. The count prefix respects the schema's endianness.

Composite types

TStruct and TUnion are composite — their size is the sum of their fields' sizes (plus alignment padding in aligned static mode). The offset computation recurses into their properties.

Schema-Layer Public API

The schema module exposes the foundational types and functions every other module depends on. These are re-exported from the crate root.

get_typedef_kind vs get_typedef_kind_loose

The engine recognizes a TypeDef:* kind on a schema node two ways, because the keyword value may be either a boolean (true) or an annotation object ({ "encoding": "..." }):

Function	Recognizes	Returns
get_typedef_kind(node) -> Option<&str>	Boolean form only ({ "TypeDef:String": true })	The keyword string, e.g. "TypeDef:String"
get_typedef_kind_loose(node) -> Option<&str>	Boolean form and object form	The keyword string
get_typedef_kind_enum(node) -> Option<TypeDefKind>	Boolean form only	The parsed enum variant
get_typedef_kind_loose_enum(node) -> Option<TypeDefKind>	Boolean form and object form	The parsed enum variant

The boolean-form-only functions are used by the validator factories (which reject the object form as a schema error) and the top-level kind-check in OffsetMap::compute / LayoutBuilder::new / SequentialReader::new (which require TypeDef:Struct at the root). The "loose" variants are used by the layout engines during field traversal, so that a variable- length field with an encoding annotation ({ "TypeDef:String": { "encoding": "offset-indirect" } }) is still recognized as a String.

Annotation parsers

Each schema-level annotation has a dedicated parser that reads it from a serde_json::Value node and returns a sensible default when absent:

Function	Annotation	Default
parse_endian(node) -> Endian	"endian"	Endian::Little
parse_align(node) -> Option<usize>	"align"	None
parse_max_length(node) -> Option<usize>	"maxLength"	None
parse_encoding(keyword_value) -> VariableEncoding	"encoding" (within the keyword's value object)	VariableEncoding::LengthPrefixed
parse_discriminator(node) -> Result<DiscriminatorKind, TypedefError>	"discriminator"	(required — returns TypedefError::Schema if absent)

Public enums

pub enum Endian { Little, Big }
pub enum VariableEncoding { LengthPrefixed, OffsetIndirect }
pub enum DiscriminatorKind {
    Byte { offset: usize, disc_type: TypeDefKind },
    Field { name: String },
}

DiscriminatorKind::Byte carries the byte position (offset) and the discriminator's TypeDef:* kind (disc_type, restricted to Uint8/ Uint16/Uint32). DiscriminatorKind::Field carries the discriminator field's name. See data-access.md §"TUnion Dispatch" for how these drive dispatch.

$ref resolution and normalization

Function	Purpose
normalize_refs(schema: &mut Value)	Walks the schema; rewrites every "$ref" whose value is a bare name (no # prefix) to "#/$defs/<name>". Idempotent. Runs once at TypedefEngine::compile time.
resolve_ref(root, ref_path) -> Option<&Value>	Resolves a JSON Pointer $ref (e.g. "#/$defs/Read") against the root schema.
resolve_ref_or_inline(node, root) -> Option<&Value>	If node has a "$ref", resolves it against root; otherwise returns node itself (it's an inline schema).

normalize_refs bridges TypeBox's bare-name ref output and jsonschema's JSON Pointer requirement. The layout engines call resolve_ref_or_inline on every $ref-bearing node they encounter during traversal.

jsonschema Custom Keyword Integration

The jsonschema crate (v0.46.5, Draft 2020-12) supports custom keywords via the with_keyword API. Each TypeDef:* kind is registered as a custom keyword:

let validator = jsonschema::options()
    .with_keyword("TypeDef:Float32", factory)
    .with_keyword("TypeDef:Int32", factory)
    .with_keyword("TypeDef:Struct", factory)
    // ... all 17 kinds
    .build(&schema)?;

The factory closure receives the parent schema object, the keyword's value, and the schema path — enabling cross-keyword awareness. The TypeDef:Struct validator, for example, inspects the parent's properties to validate each field against its declared TypeDef:* kind.

Each custom keyword implementation is ~10 lines. The jsonschema crate handles all structural validation (object properties, required fields, array items, enum values) — the custom keywords only need to validate the leaf type constraints. See validation.md for the validator implementations.

This is the same pattern as TypeBox's TypeRegistry.Set on the JS side. Same semantics, different language, same JSON Schema wire format. A TypeBox schema serialized to JSON feeds into the typedef engine after a single pre-processing step: normalizing $ref values (see below).

TypeBox Interop

TypeBox modules render to standard JSON Schema under $defs. A TypeBox schema like:

const TensorRef = Type.Object({
  dtype: Type.Union([Type.Literal("F32"), Type.Literal("I16")]),
  shape: Type.Array(Type.Number()),
  data_offsets: Type.Tuple([Type.Number(), Type.Number()])
});

serialized to JSON is a standard JSON Schema with type: "object", properties, and required. That JSON feeds into the typedef engine after $ref normalization. The TypeDef:* custom keywords are added by TypeBox's TypeRegistry.Set — they appear in the serialized JSON as additional properties on the schema object.

$ref normalization

TypeBox generates bare-name $ref values (e.g., "$ref": "Read"), referencing sibling definitions within the same $defs block. The jsonschema crate requires full JSON Pointer paths (e.g., "$ref": "#/$defs/Read"). The typedef engine normalizes TypeBox-style refs at schema load time via normalize_refs — a ~20-line recursive walk that rewrites every bare-name "$ref" to "#/$defs/<name>". The normalization is idempotent — full JSON Pointer refs pass through unchanged. It runs once at TypedefEngine::compile time, before the schema is passed to jsonschema or the offset computation.

Verification: The jsonschema crate (v0.46.5) rejects bare-name refs with Resource 'Read' is not present in a registry. Full JSON Pointer refs (#/$defs/Read) resolve correctly. The normalization step bridges the gap between TypeBox's output and jsonschema's input.

The typedef engine does not depend on TypeBox or any JS toolchain. It consumes JSON — whether that JSON was authored in TypeBox, generated by a ujsx component, or hand-written. The schema is the interface.

Schema Annotations

Schema-level annotations control binary layout behavior. These are decided in ADR-097.

Endianness

Schema-level annotation with a default of little-endian:

{ "TypeDef:Struct": true, "endian": "big", "properties": { ... } }

• "endian": "little" (default) — read/write in little-endian byte order.
• "endian": "big" — read/write in big-endian byte order.
• Applies to the entire schema and all nested types.

Alignment

Both struct-level and field-level, with field-level overriding:

{
  "TypeDef:Struct": true,
  "align": 256,
  "properties": {
    "weight": { "TypeDef:Float32": true, "align": 16 }
  }
}

• Struct-level "align" sets the default for all fields.
• Field-level "align" overrides the struct default.
• Default alignment: 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/ enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array.
• Only meaningful in aligned static mode (ADR-096). Ignored in packed sequential mode.

Variable-length encoding

The typedef engine supports three strategies for variable-length types (see §Variable-length types above for full details). The strategy is selected by the encoding annotation and the standard JSON Schema maxLength keyword:

// Strategy 1: Inline length-prefixing (default, shorthand)
{ "TypeDef:String": true }

// Strategy 1: Explicit inline length-prefixing
{ "TypeDef:String": { "encoding": "length-prefixed" } }

// Strategy 2: Fixed-size reservation (uses standard maxLength)
{ "TypeDef:String": true, "maxLength": 256 }

// Strategy 3: Offset indirection (opt-in)
{ "TypeDef:String": { "encoding": "offset-indirect" } }

• "encoding": "length-prefixed" (default) — 4-byte length prefix at computed offset, variable data follows immediately. Used by protocol wire formats.
• maxLength (standard JSON Schema keyword) — in aligned static mode, reserves maxLength bytes at a fixed offset (zero-padded). Makes the field fixed-size from the layout perspective. In packed sequential mode, maxLength is a validation constraint only.
• "encoding": "offset-indirect" — field is a struct {offset: u32, length: u32} pointing into a separate data region. The consumer provides the data region separately. Used by metatensor blob tensors.
• Applies to all variable-length types: TypeDef:String, TypeDef:Bytes, TypeDef:Array, TypeDef:Record, TypeDef:Timestamp.

TUnion discriminators

Two discriminator kinds: byte-offset (protocol dispatch) and field-name (typedef.ts pattern).

Byte-offset discriminator (SFTP type bytes, call protocol event types):

{
  "TypeDef:Union": true,
  "discriminator": {
    "kind": "byte",
    "offset": 0,
    "type": "TypeDef:Uint8"
  },
  "mapping": {
    "5": { "$ref": "#/$defs/Read" },
    "6": { "$ref": "#/$defs/Write" },
    "101": { "$ref": "#/$defs/Status" }
  }
}

• "offset" — byte position of the discriminator.
• "type" — the TypeDef:* kind of the discriminator (typically TypeDef:Uint8).
• Mapping keys are stringified integers. The variant struct starts at offset + discriminator_size.

Field-name discriminator (typedef.ts pattern):

{
  "TypeDef:Union": true,
  "discriminator": {
    "kind": "field",
    "name": "type"
  },
  "mapping": {
    "read": { "$ref": "#/$defs/Read" },
    "write": { "$ref": "#/$defs/Write" }
  }
}

• "name" — the field name holding the discriminator value.
• Mapping keys are string values matching the discriminator field's value.
• The discriminator field is just another field in the struct.

Mapping values may be either inline schemas or $ref pointers. Both work.

Design Decisions

Decision	ADR	Summary
Schema annotations	ADR-097	Concrete JSON shapes for endianness, alignment, encoding, and TUnion discriminators
Int64/Uint64 kinds	ADR-099	64-bit integers as first-class kinds (required by SFTP offsets and metatensor data_offsets)
Purpose and scope	ADR-095	Why jsonschema not a custom engine; "schema is the format" principle

Open Questions

See open-questions.md for full details.

• OQ-071 (deferred(scope)): Builder API for schema construction.

References

• /workspace/@alkdev/typebox/example/typedef/typedef.ts — the TypeBox schema kinds (619 lines)
• /workspace/jsonschema/ — the jsonschema crate (v0.46.5, Draft 2020-12)
• ADR-097 — schema annotation shapes
• validation.md — custom keyword validator implementations