alkdev/alktype
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rebrand TypeDef to AlkType in code, keyword strings, and docs

Browse Source 
Rename all 19 JSON Schema custom keyword strings from "TypeDef:*"
to "AlkType:*" (e.g., "TypeDef:Struct" -> "AlkType:Struct")
across source, tests, and docs. This is a breaking change to the
schema format itself — existing schemas using the old keywords
must be updated.

Rename the Rust identifiers:
- TypedefEngine -> AlkTypeEngine
- TypedefError -> AlkTypeError
- TypeDefKind -> AlkTypeKind
- TYPEDEF_PREFIX -> ALKTYPE_PREFIX
- get_typedef_kind{,_loose,_loose_enum,_enum} ->
  get_alktype_kind{,_loose,_loose_enum,_enum}

Update error message strings ("unknown TypeDef kind" ->
"unknown AlkType kind"), 11 test function names containing
typedef_kind/to_typedef_error, and doc-comment prose ("TypeDef
kind" -> "AlkType kind", "typedef engine" -> "alktype
engine", "typedef schema" -> "alktype schema"). Fix the broken
docs/architecture/crates/typedef/ path references in source doc
comments to point at docs/architecture/ directly. Rebrand the
typedef:annotation test fixture and the "not-a-typedef" test
string to their alktype equivalents.

Update ~20 generic "typedef" prose references in the architecture
docs ("typedef is the binary struct engine", "use typedef",
"typedef limitation", "replaced by typedef", etc.) to alktype.
Rename TypedefEngine in the ADR-007 code example to AlkTypeEngine.

Preserve as provenance per the prior prose-rebrand decision:
typedef.ts references (external TypeBox source file),
docs/research/alknet-typedef/findings.md research citations,
/workspace/alknet-typedef-poc/ POC path, and the
"alknet-typedef:" research section headers in findings.

Build, 295 tests, and clippy all pass clean.
This commit is contained in:
glm-5.2
2026-08-02 07:05:53 +00:00
parent 1cfb3638d1
commit 5e268e8f47
31 changed files with 1851 additions and 1851 deletions
Download Patch File Download Diff File Expand all files Collapse all files

16
docs/architecture/README.md
View File

@@ -6,7 +6,7 @@ last_updated: 2026-07-22
# alktype
The binary struct engine: a small Rust crate that takes a JSON Schema
with `TypeDef:*` custom keywords and produces an offset map, read/write
with `AlkType:*` custom keywords and produces an offset map, read/write
functions, and validation — all driven by the schema. The schema is the
format definition; the engine is generic.
@@ -15,10 +15,10 @@ format definition; the engine is generic.
| Document | Status | Description |
|----------|--------|-------------|
| [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
| [schema-layer.md](schema-layer.md) | draft | The 19 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [schema-layer.md](schema-layer.md) | draft | The 19 `AlkType:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
| [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation, `TypedefEngine` |
| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `AlkType:*` kinds, `AlkTypeError`, load-time vs access-time validation, `AlkTypeEngine` |
## Applicable ADRs
@@ -27,7 +27,7 @@ format definition; the engine is generic.
| [001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
| [002](decisions/002-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` |
| [003](decisions/003-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations |
| [004](decisions/004-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors |
| [004](decisions/004-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors |
| [005](decisions/005-int64-uint64-first-class-kinds.md) | Int64/Uint64 as First-Class Kinds | 64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat |
| [006](decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode | Prevents silent data corruption (inline variable data clobbering subsequent fields) |
| [007](decisions/007-packed-mode-read-factory.md) | Packed-Mode Read API — Engine as SequentialReader Factory | `engine.sequential_reader()` returns an owned reader, not a reference |
@@ -43,7 +43,7 @@ format definition; the engine is generic.
## Key Design Principles
1. **The schema is the format.** A JSON Schema with `TypeDef:*` custom
1. **The schema is the format.** A JSON Schema with `AlkType:*` custom
keywords is both the validation spec and the layout spec. No separate
format definition, no separate parser, no separate validator. One
schema, three uses: validate, compute offsets, access data. See
@@ -88,11 +88,11 @@ format definition; the engine is generic.
[validation.md](validation.md) and
[ADR-004](decisions/004-error-handling-validation-strategy.md).
8. **Not a serialization framework.** The typedef engine is not a
8. **Not a serialization framework.** The alktype engine is not a
general-purpose serde replacement. It operates on raw byte buffers at
computed offsets — no intermediate `Value` tree, no reflection, no
dynamic dispatch per field. For JSON data, use serde. For binary data
with a known schema, use typedef. See [overview.md](overview.md) and
with a known schema, use alktype. See [overview.md](overview.md) and
[ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md).
## References
@@ -106,5 +106,5 @@ format definition; the engine is generic.
schema kinds (619 lines)
- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12)
- `/workspace/alknet-typedef-poc/` — the POC code (disposable)
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by alktype
- `/workspace/@alkimiadev/alktype/` — prior attempt (the @alkimiadev/alktype prototype, a handler-registry pattern; not to be confused with this crate, which reuses the name but is backed by the `jsonschema` crate)

66
docs/architecture/data-access.md
View File

@@ -8,7 +8,7 @@ last_updated: 2026-07-22
The data access layer: read/write functions, TUnion dispatch, field paths,
zero-copy access for fixed-size types, and length-prefix reading for
variable-length types. This is the consumer-facing API — given a compiled
`TypedefEngine` and a byte buffer, read and write fields at
`AlkTypeEngine` and a byte buffer, read and write fields at
schema-computed offsets.
This document covers two layers:
@@ -16,11 +16,11 @@ This document covers two layers:
- **Primitive read/write functions** in the `data_access` module —
typed reads/writes at a caller-provided offset. These are the building
blocks used by the layout types (`OffsetMap`, `LayoutBuilder`,
`SequentialReader`) and the `TypedefEngine`. Each operates on a raw
byte buffer at a known offset and returns a `TypedefError::Access`
`SequentialReader`) and the `AlkTypeEngine`. Each operates on a raw
byte buffer at a known offset and returns a `AlkTypeError::Access`
carrying the field path on bounds or encoding failures.
- **The `FieldValue` enum and the higher-level APIs** —
`TypedefEngine::read_field`/`write_field` (aligned mode) and
`AlkTypeEngine::read_field`/`write_field` (aligned mode) and
`SequentialReader::read_next`/`read_field` (packed mode) — which look
up a field's offset via the layout and dispatch to the primitive
functions, returning a unified `FieldValue<'a>`.
@@ -55,7 +55,7 @@ types, signalling the consumer must walk each element sequentially.
## Read/Write Model
The typedef engine operates on raw byte buffers (`&[u8]` for reading,
The alktype engine operates on raw byte buffers (`&[u8]` for reading,
`&mut [u8]` for writing). There is no intermediate `Value` tree, no
reflection, no dynamic dispatch per field. The engine uses the offset map
(or `LayoutBuilder`/`SequentialReader`) to locate fields, then performs
@@ -63,21 +63,21 @@ typed access at the computed positions.
### Higher-level read/write
The `TypedefEngine` and `SequentialReader` provide the primary
The `AlkTypeEngine` and `SequentialReader` provide the primary
consumer-facing read/write APIs. They look up a field's offset via the
layout and dispatch to the primitive `data_access` functions, returning
`FieldValue` (read) or accepting `&FieldValue` (write).
```rust
impl TypedefEngine {
impl AlkTypeEngine {
// Aligned mode: looks up the field's ByteRange in the OffsetMap,
// dispatches to the right data_access function by TypeDefKind.
// Returns TypedefError::Access if compiled in packed mode
// dispatches to the right data_access function by AlkTypeKind.
// Returns AlkTypeError::Access if compiled in packed mode
// (use sequential_reader() for packed mode).
pub fn read_field<'a>(&self, buffer: &'a [u8], field_path: &str)
-> Result<FieldValue<'a>, TypedefError>;
-> Result<FieldValue<'a>, AlkTypeError>;
pub fn write_field(&self, buffer: &mut [u8], field_path: &str,
value: &FieldValue<'_>) -> Result<(), TypedefError>;
value: &FieldValue<'_>) -> Result<(), AlkTypeError>;
// Packed mode: returns an owned fresh SequentialReader (ADR-007).
// Each call returns a new reader with the cursor at position 0.
@@ -90,16 +90,16 @@ impl SequentialReader {
// prefixes to find each field's position. read_field walks all
// preceding fields to reach the target.
pub fn read_next<'a>(&mut self, buffer: &'a [u8])
-> Result<Option<(String, FieldValue<'a>)>, TypedefError>;
-> Result<Option<(String, FieldValue<'a>)>, AlkTypeError>;
pub fn read_field<'a>(&mut self, buffer: &'a [u8], field_path: &str)
-> Result<FieldValue<'a>, TypedefError>;
-> Result<FieldValue<'a>, AlkTypeError>;
pub fn reset(&mut self);
pub fn position(&self) -> usize;
pub fn endian(&self) -> Endian;
}
```
`read_field`/`write_field` on `TypedefEngine` work for the fixed-size
`read_field`/`write_field` on `AlkTypeEngine` work for the fixed-size
primitive kinds and the length-prefixed `String`/`Bytes`/`Timestamp`
fields. Composite kinds (`Struct`, `Union`, `Array`, `Record`) return a
`FieldValue` carrying a layout descriptor (byte range, variant start,
@@ -115,7 +115,7 @@ which the builder consumes at `build` time.
The `data_access` module exposes typed read/write functions for each
primitive kind. Each takes `field_path: &str` for error attribution
(produces a `TypedefError::Access` carrying the path on bounds or
(produces a `AlkTypeError::Access` carrying the path on bounds or
encoding failures) and, for multi-byte types, an `Endian` parameter.
### Fixed-size types
@@ -126,7 +126,7 @@ accessed via zero-copy reads of N bytes at the offset:
```rust
// Read a u32 at a known offset, applying endianness. Bounds-checked.
fn read_u32(buffer: &[u8], offset: usize, field_path: &str, endian: Endian)
-> Result<u32, TypedefError> {
-> Result<u32, AlkTypeError> {
let bytes: [u8; 4] = read_array(buffer, offset, field_path)?;
Ok(match endian {
Endian::Little => u32::from_le_bytes(bytes),
@@ -136,7 +136,7 @@ fn read_u32(buffer: &[u8], offset: usize, field_path: &str, endian: Endian)
// Write a u32 at a known offset, applying endianness. Bounds-checked.
fn write_u32(buffer: &mut [u8], offset: usize, value: u32,
field_path: &str, endian: Endian) -> Result<(), TypedefError> {
field_path: &str, endian: Endian) -> Result<(), AlkTypeError> {
let bytes = match endian {
Endian::Little => value.to_le_bytes(),
Endian::Big => value.to_be_bytes(),
@@ -148,7 +148,7 @@ fn write_u32(buffer: &mut [u8], offset: usize, value: u32,
The engine applies endianness at access time based on the schema's
`"endian"` annotation (ADR-003). The offset computation is
endian-agnostic. The `read_array`/`write_array` helpers perform the
bounds check and produce `TypedefError::Access` with the field path on
bounds check and produce `AlkTypeError::Access` with the field path on
failure.
### TEnum access
@@ -158,7 +158,7 @@ to the `u32` primitives, applying the schema's endianness:
```rust
pub fn read_enum(buffer: &[u8], offset: usize, field_path: &str, endian: Endian)
-> Result<u32, TypedefError> {
-> Result<u32, AlkTypeError> {
read_u32(buffer, offset, field_path, endian)
}
```
@@ -179,11 +179,11 @@ attribution and `endian` for the length prefix:
```rust
// Read a length-prefixed string, borrowing from the buffer.
fn read_string<'a>(buffer: &'a [u8], offset: usize,
field_path: &str, endian: Endian) -> Result<&'a str, TypedefError>;
field_path: &str, endian: Endian) -> Result<&'a str, AlkTypeError>;
// Write a length-prefixed string. Returns total bytes written (4 + data.len()).
fn write_string(buffer: &mut [u8], offset: usize, value: &str,
field_path: &str, endian: Endian) -> Result<usize, TypedefError>;
field_path: &str, endian: Endian) -> Result<usize, AlkTypeError>;
// read_bytes / write_bytes have the same shape — raw bytes, no UTF-8 check.
```
@@ -210,10 +210,10 @@ bytes live in a separate `data_region`:
```rust
fn read_string_indirect<'a>(buffer: &'a [u8], offset: usize,
data_region: &'a [u8], field_path: &str,
endian: Endian) -> Result<&'a str, TypedefError>;
endian: Endian) -> Result<&'a str, AlkTypeError>;
fn read_bytes_indirect<'a>(buffer: &'a [u8], offset: usize,
data_region: &'a [u8], field_path: &str,
endian: Endian) -> Result<&'a [u8], TypedefError>;
endian: Endian) -> Result<&'a [u8], AlkTypeError>;
```
The field is a struct `{offset: u32, length: u32}` at a known position
@@ -247,14 +247,14 @@ fresh `SequentialReader` scoped to the variant).
```rust
/// Read the discriminator value from a byte-offset TUnion. The discriminator
/// is a fixed-size integer (TypeDef:Uint8/Uint16/Uint32) at a known byte
/// is a fixed-size integer (AlkType:Uint8/Uint16/Uint32) at a known byte
/// offset. Returns the mapping key (stringified integer) and the variant
/// struct offset.
pub fn read_byte_discriminator(
buffer: &[u8],
union_schema: &Value,
endian: Endian,
) -> Result<UnionDispatch, TypedefError>;
) -> Result<UnionDispatch, AlkTypeError>;
```
This is the SFTP `Packet` enum pattern — byte 0 is the type byte, bytes
@@ -268,14 +268,14 @@ starts at `offset + discriminator_size`.
/// Read the discriminator value from a field-name TUnion. The
/// discriminator is a named field within the struct — the consumer
/// provides the field's computed offset (from the OffsetMap or
/// LayoutBuilder). Supports TypeDef:String, Uint8, and Enum discriminator
/// LayoutBuilder). Supports AlkType:String, Uint8, and Enum discriminator
/// fields.
pub fn read_field_discriminator(
buffer: &[u8],
union_schema: &Value,
disc_field_offset: usize,
endian: Endian,
) -> Result<UnionDispatch, TypedefError>;
) -> Result<UnionDispatch, AlkTypeError>;
```
The discriminator is a named field within the struct. Its offset is
@@ -291,12 +291,12 @@ the variant's fields starting at the end of the discriminator field.
/// are returned directly. $ref pointers of the form "#/$defs/<name>"
/// are resolved against the union schema's own $defs block.
pub fn resolve_variant<'a>(union_schema: &'a Value, key: &str)
-> Result<&'a Value, TypedefError>;
-> Result<&'a Value, AlkTypeError>;
/// Get the discriminator's byte size (1/2/4 for Uint8/16/32) for a
/// byte-offset TUnion. Field-name discriminators have no fixed size
/// and produce a TypedefError::Schema.
pub fn discriminator_size(union_schema: &Value) -> Result<usize, TypedefError>;
/// and produce a AlkTypeError::Schema.
pub fn discriminator_size(union_schema: &Value) -> Result<usize, AlkTypeError>;
```
### TUnion in the layout engines
@@ -318,13 +318,13 @@ variant before recursing.
Fields are addressed by dotted paths: `"header.version"`, `"payload.data"`.
Both `OffsetMap` and `PackedLayout` store fully-qualified paths (nested
struct fields appear under their parent's path prefix). The higher-level
APIs (`TypedefEngine::read_field`/`write_field`, `SequentialReader::read_field`)
APIs (`AlkTypeEngine::read_field`/`write_field`, `SequentialReader::read_field`)
accept a field path, look up the byte range/position in the layout, and
dispatch to the primitive `data_access` function for the field's kind.
For aligned-mode access, `TypedefEngine::read_field(&buffer, "header.version")`
For aligned-mode access, `AlkTypeEngine::read_field(&buffer, "header.version")`
returns `FieldValue` — it looks up the `ByteRange` in the `OffsetMap`, finds
the field's `TypeDef:*` kind in the schema, and calls the matching
the field's `AlkType:*` kind in the schema, and calls the matching
`data_access::read_*` function. `write_field` is the mirror. Composite
kinds (`Struct`, `Union`, `Array`, `Record`) return a `FieldValue`
carrying a layout descriptor; the consumer recurses with a fresh reader

34
docs/architecture/decisions/001-alktype-purpose-scope-jsonschema-engine.md
View File

@@ -18,7 +18,7 @@ bytes at computed offsets.
(`Read { id: u32, handle: String, offset: u64, len: u32 }`). The wire
format is `[length: u32][type: u8][payload]` where payload is the
struct's serde bytes. The `Packet` enum dispatches on the type byte —
a tagged union of structs. Under the typedef lens, each packet is a
a tagged union of structs. Under the alktype lens, each packet is a
`TStruct`; the `Packet` enum is a `TUnion` with a byte-offset
discriminator.
@@ -27,7 +27,7 @@ bytes at computed offsets.
compute byte offsets for each field so the consumer can read tensor
data at known positions without parsing.
The common pattern: **a JSON Schema with `TypeDef:*` custom keywords
The common pattern: **a JSON Schema with `AlkType:*` custom keywords
describes the shape of binary data; the binary data is the struct's bytes
at computed offsets.** The schema is the format definition; the engine is
generic.
@@ -51,16 +51,16 @@ validation.
The call-channels-unification research
(`docs/research/call-channels-unification/findings.md` §"alknet-typedef:
JSON Schema as the binary struct engine") identified the convergence and
bumped typedef up in the timeline. The POC
bumped alktype up in the timeline. The POC
(`docs/research/alknet-typedef/findings.md`, 26 tests passing) validated
the approach: a ~1,900-line Rust crate that takes a JSON Schema with
`TypeDef:*` custom keywords and produces an offset map, read/write
`AlkType:*` custom keywords and produces an offset map, read/write
functions, and validation — all driven by the schema.
## Decision
**alktype is a small Rust crate that takes a JSON Schema with
`TypeDef:*` custom keywords and produces three capabilities:**
`AlkType:*` custom keywords and produces three capabilities:**
1. **An offset map** — walks the schema, computes byte offsets for each
field based on type sizes, field order, and alignment.
@@ -79,8 +79,8 @@ functions, and validation — all driven by the schema.
— a recursive walk of the schema JSON that computes byte positions for
each field. The custom keyword implementations are ~10 lines each.
**The schema is the format.** A JSON Schema with `TypeDef:Float32`,
`TypeDef:Struct`, `TypeDef:Union` etc. is both the validation spec and
**The schema is the format.** A JSON Schema with `AlkType:Float32`,
`AlkType:Struct`, `AlkType:Union` etc. is both the validation spec and
the layout spec. No separate format definition, no separate parser, no
separate validator. One schema, three uses: validate, compute offsets,
access data.
@@ -88,8 +88,8 @@ access data.
**The crate depends on `jsonschema` and `serde_json` (with
`preserve_order`).** No tokio, no platform deps. Compiles to
`wasm32-unknown-unknown` for browser use. The `jsonschema` crate's
`with_keyword("TypeDef:Float32", factory)` API is the integration point
for custom type kinds — each `TypeDef:*` kind maps to a custom keyword
`with_keyword("AlkType:Float32", factory)` API is the integration point
for custom type kinds — each `AlkType:*` kind maps to a custom keyword
validator in Rust. Same semantics as TypeBox's `TypeRegistry.Set`, same
JSON Schema wire format.
@@ -113,7 +113,7 @@ be added as a feature gate later — the engine's core (offset computation,
adding a variant to the schema JSON, not writing a new Rust struct +
serde impl. The engine is generic; the schema is the configuration.
- **WASM-clean.** `serde_json` + `jsonschema` + byte manipulation. No
tokio, no platform deps. The same typedef schemas work in browser,
tokio, no platform deps. The same alktype schemas work in browser,
Node, Python (via `wasmtime-py`), Go (via `wazero`), and any other
WASM host.
- **TypeBox interop.** TypeBox modules render to standard JSON Schema
@@ -135,27 +135,27 @@ be added as a feature gate later — the engine's core (offset computation,
load-bearing for binary layouts. The `preserve_order` feature adds a
small compile-time cost.
- **Schema authoring is external.** Schemas are authored in TypeBox (JS)
or hand-written JSON. The typedef engine consumes schemas; it does not
or hand-written JSON. The alktype engine consumes schemas; it does not
generate them. A builder API is deferred (OQ-003).
## Scope Boundaries (What This Is Not)
- **Not metatensor.** typedef is the binary struct *engine*. Metatensor
- **Not metatensor.** alktype is the binary struct *engine*. Metatensor
is a *format* (8-byte header + JSON header + binary data) that uses the
typedef engine for its offset computation and tensor access.
alktype engine for its offset computation and tensor access.
- **Not a Value system.** TypeBox's `Value.Diff`, `Value.Migrate`,
`Value.Convert` — schema evolution — is out of scope for v1. The engine
should not do anything that explicitly blocks adding a Value system
later.
- **Not a code generator.** typebox-rs's `codegen/` module is a separate
concern. The typedef engine consumes schemas; it does not generate them.
- **Not a schema builder.** The typedef engine does not provide a fluent
concern. The alktype engine consumes schemas; it does not generate them.
- **Not a schema builder.** The alktype engine does not provide a fluent
API for constructing schemas. Schemas are plain JSON.
- **Not a serialization framework.** The typedef engine is not a
- **Not a serialization framework.** The alktype engine is not a
general-purpose serde replacement. It operates on raw byte buffers at
computed offsets — no intermediate `Value` tree, no reflection, no
dynamic dispatch per field. For JSON data, use serde. For binary data
with a known schema, use typedef.
with a known schema, use alktype.
## References

2
docs/architecture/decisions/002-two-layout-modes-packed-vs-aligned.md
View File

@@ -41,7 +41,7 @@ tensor data).
## Decision
**The typedef engine supports two layout modes, selected by the consumer
**The alktype engine supports two layout modes, selected by the consumer
at engine construction time:**
### Mode 1: Packed sequential (`LayoutBuilder` / `SequentialReader`)

46
docs/architecture/decisions/003-schema-annotations.md
View File

@@ -5,7 +5,7 @@ Accepted
## Context
The typedef engine needs concrete JSON shapes for schema-level
The alktype engine needs concrete JSON shapes for schema-level
annotations that control binary layout behavior. The POCs validated the
semantics; this ADR pins the shapes.
@@ -28,7 +28,7 @@ Four annotation categories need concrete shapes:
```json
{
"TypeDef:Struct": true,
"AlkType:Struct": true,
"endian": "big",
"properties": { ... }
}
@@ -49,11 +49,11 @@ struct-level.**
```json
{
"TypeDef:Struct": true,
"AlkType:Struct": true,
"align": 256,
"properties": {
"header": { "TypeDef:Struct": true, "properties": { ... } },
"weight": { "TypeDef:Float32": true, "align": 16 }
"header": { "AlkType:Struct": true, "properties": { ... } },
"weight": { "AlkType:Float32": true, "align": 16 }
}
}
```
@@ -76,16 +76,16 @@ annotation and the standard JSON Schema `maxLength` keyword.**
```json
// Strategy 1: Inline length-prefixing (default, shorthand)
{ "TypeDef:String": true }
{ "AlkType:String": true }
// Strategy 1: Explicit inline length-prefixing
{ "TypeDef:String": { "encoding": "length-prefixed" } }
{ "AlkType:String": { "encoding": "length-prefixed" } }
// Strategy 2: Fixed-size reservation (uses standard maxLength)
{ "TypeDef:String": true, "maxLength": 256 }
{ "AlkType:String": true, "maxLength": 256 }
// Strategy 3: Offset indirection (opt-in)
{ "TypeDef:String": { "encoding": "offset-indirect" } }
{ "AlkType:String": { "encoding": "offset-indirect" } }
```
**Strategy 1: Inline length-prefixing (default).** The field's fixed
@@ -126,27 +126,27 @@ reserving worst-case space.
- `true` is a shorthand for the default (length-prefixed). This keeps
the common case concise and the override explicit.
- The `encoding` annotation and `maxLength` apply to all variable-length
types: `TypeDef:String`, `TypeDef:Bytes`, `TypeDef:Array`,
`TypeDef:Record`, `TypeDef:Timestamp`.
types: `AlkType:String`, `AlkType:Bytes`, `AlkType:Array`,
`AlkType:Record`, `AlkType:Timestamp`.
### 3a. TRecord value type
`TypeDef:Record` is a string-keyed map. The value type is declared via
`AlkType:Record` is a string-keyed map. The value type is declared via
the `"values"` property in the schema:
```json
{
"TypeDef:Record": true,
"values": { "TypeDef:Float32": true }
"AlkType:Record": true,
"values": { "AlkType:Float32": true }
}
```
- `"values"` is a schema object declaring the `TypeDef:*` kind of all
- `"values"` is a schema object declaring the `AlkType:*` kind of all
values in the record. All values share the same type.
- The binary layout is a count-prefixed sequence of `(key, value)` pairs:
`[count: u32][key_len: u32][key_bytes][value]...` repeated `count`
times. Each key is a length-prefixed UTF-8 string. Each value is
encoded according to its declared `TypeDef:*` kind — a `Record<Uint32>`
encoded according to its declared `AlkType:*` kind — a `Record<Uint32>`
value is 4 raw bytes; a `Record<String>` value is itself a
length-prefixed string; a `Record<Struct>` value is the struct's
fields laid out inline. There is **no separate `value_len` prefix** —
@@ -165,11 +165,11 @@ field-name (typedef.ts pattern).**
```json
{
"TypeDef:Union": true,
"AlkType:Union": true,
"discriminator": {
"kind": "byte",
"offset": 0,
"type": "TypeDef:Uint8"
"type": "AlkType:Uint8"
},
"mapping": {
"1": { "$ref": "#/$defs/Init" },
@@ -184,8 +184,8 @@ field-name (typedef.ts pattern).**
- The discriminator is a fixed-size integer at a known byte offset.
- `"offset"` is the byte position of the discriminator within the union's
buffer.
- `"type"` is the `TypeDef:*` kind of the discriminator (typically
`TypeDef:Uint8` for protocol type bytes).
- `"type"` is the `AlkType:*` kind of the discriminator (typically
`AlkType:Uint8` for protocol type bytes).
- The mapping keys are stringified integers (`"1"`, `"5"`, `"101"`).
The engine parses the key to match the discriminator value.
- The variant struct starts at `offset + discriminator_size`.
@@ -196,7 +196,7 @@ field-name (typedef.ts pattern).**
```json
{
"TypeDef:Union": true,
"AlkType:Union": true,
"discriminator": {
"kind": "field",
"name": "type"
@@ -240,8 +240,8 @@ event types). Both work.
### Negative
- **Keyword value shape change.** `"TypeDef:String": true` (boolean) and
`"TypeDef:String": { "encoding": "length-prefixed" }` (object) are both
- **Keyword value shape change.** `"AlkType:String": true` (boolean) and
`"AlkType:String": { "encoding": "length-prefixed" }` (object) are both
valid. The engine must handle both shapes. This is a minor parsing
concern — the POC already handles it.
- **Alignment annotations are mode-specific.** Alignment is only

48
docs/architecture/decisions/004-error-handling-validation-strategy.md
View File

@@ -5,11 +5,11 @@ Accepted
## Context
The typedef engine operates in three phases, each with distinct error
The alktype engine operates in three phases, each with distinct error
conditions:
1. **Schema parsing** — invalid JSON, missing required keywords, unknown
`TypeDef:*` kinds, malformed annotations.
`AlkType:*` kinds, malformed annotations.
2. **Offset computation** — field not found, type not supported for
offset computation, recursive schema depth exceeded.
3. **Read/write** — buffer too short, invalid UTF-8, value out of range
@@ -23,12 +23,12 @@ time (validate each buffer).
## Decision
### Error type: `TypedefError`
### Error type: `AlkTypeError`
A single `TypedefError` enum with variants for each error category:
A single `AlkTypeError` enum with variants for each error category:
```rust
pub enum TypedefError {
pub enum AlkTypeError {
/// Schema parsing errors.
Schema(String),
/// Offset computation errors.
@@ -41,39 +41,39 @@ pub enum TypedefError {
```
- `Schema` — for invalid JSON, missing required keywords, unknown
`TypeDef:*` kinds. The error message describes the problem.
`AlkType:*` kinds. The error message describes the problem.
- `Offset` — for field-not-found, unsupported type for offset
computation, etc. Carries the field path for debugging.
- `Access` — for buffer-too-short, invalid UTF-8, value out of range.
Carries the field path for debugging.
- `Validation` — wraps `jsonschema`'s `ValidationError`. The
`jsonschema` crate already provides rich error messages with schema
paths; the typedef engine does not re-wrap or re-interpret them.
paths; the alktype engine does not re-wrap or re-interpret them.
The `Validation` variant uses `ValidationError<'static>` because the
validator is built once at schema load time and lives for the lifetime
of the `TypedefEngine`. The `'static` lifetime is correct — the validator
of the `AlkTypeEngine`. The `'static` lifetime is correct — the validator
owns its schema reference.
### Validation timing: load-time build, access-time check
The jsonschema validator is built once at schema load time
(`validator_for(&schema)?`) and then called repeatedly
(`validator.is_valid(&instance)`). The typedef engine follows the same
(`validator.is_valid(&instance)`). The alktype engine follows the same
pattern:
1. **Load time:** Parse the schema JSON, build the offset map (or
`LayoutBuilder`/`SequentialReader`), build the jsonschema validator.
This is the `TypedefEngine::compile(schema: &Value) -> Result<Self,
TypedefError>` constructor.
This is the `AlkTypeEngine::compile(schema: &Value) -> Result<Self,
AlkTypeError>` constructor.
2. **Access time:** Use the compiled engine for repeated read/write
operations. Validation is opt-in per operation — the consumer calls
`engine.validate(buffer)` when validation is desired.
The `TypedefEngine` struct is the compiled form of a schema:
The `AlkTypeEngine` struct is the compiled form of a schema:
```rust
pub struct TypedefEngine {
pub struct AlkTypeEngine {
offset_map: OffsetMap, // or LayoutBuilder/SequentialReader
validator: jsonschema::Validator, // compiled once at load time
}
@@ -81,21 +81,21 @@ pub struct TypedefEngine {
### Custom keyword validators
Each `TypeDef:*` kind gets a `Keyword` implementation registered via
Each `AlkType:*` kind gets a `Keyword` implementation registered via
`jsonschema::options().with_keyword(...)`. The validators check:
- **Numeric types** (`TypeDef:Float32`, `TypeDef:Int8`, etc.): range
- **Numeric types** (`AlkType:Float32`, `AlkType:Int8`, etc.): range
constraints (Int8: -128..127, Uint8: 0..255, etc.), finiteness for
floats.
- **`TypeDef:String`**: UTF-8 validity.
- **`TypeDef:Struct`**: field presence and types (delegated to
- **`AlkType:String`**: UTF-8 validity.
- **`AlkType:Struct`**: field presence and types (delegated to
jsonschema's structural validation — the custom keyword only needs to
validate that the struct's fields match their declared `TypeDef:*`
validate that the struct's fields match their declared `AlkType:*`
kinds).
- **`TypeDef:Union`**: discriminator value membership in the mapping.
- **`TypeDef:Array`**: element type conformance.
- **`TypeDef:Boolean`**: value is `true` or `false`.
- **`TypeDef:Timestamp`**: RFC 3339 string format (the internet profile of ISO 8601).
- **`AlkType:Union`**: discriminator value membership in the mapping.
- **`AlkType:Array`**: element type conformance.
- **`AlkType:Boolean`**: value is `true` or `false`.
- **`AlkType:Timestamp`**: RFC 3339 string format (the internet profile of ISO 8601).
The `jsonschema` crate handles all the structural validation (object
properties, required fields, array items, enum values) — the custom
@@ -108,7 +108,7 @@ Read/write errors include the field path for debugging:
```rust
// Example: reading a u32 from a buffer that's too short
Err(TypedefError::Access {
Err(AlkTypeError::Access {
field_path: "header.version".to_string(),
reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(),
})
@@ -121,7 +121,7 @@ you exactly which field failed and why.
### Positive
- **Single error type.** Consumers handle one `TypedefError` enum, not
- **Single error type.** Consumers handle one `AlkTypeError` enum, not
multiple error types from different engine phases.
- **Field-path-carrying errors.** Read/write errors include the field
path, making binary format debugging tractable.

36
docs/architecture/decisions/005-int64-uint64-first-class-kinds.md
View File

@@ -5,17 +5,17 @@ Accepted
## Context
The typedef engine's kind set (ADR-001, ADR-003) tops out at 32-bit
The alktype engine's kind set (ADR-001, ADR-003) tops out at 32-bit
integers. The POC included `u64` read/write primitives, and the
call-channels-unification research's own SFTP schema example uses
`"TypeDef:Uint64"` for the `offset` field (`Read`/`Write` packets have
`"AlkType:Uint64"` for the `offset` field (`Read`/`Write` packets have
`offset: u64`). Metatensor/safetensors `data_offsets` are also `u64`.
A `TypeDef:Uint64` variant was added to the `TypeDefKind` enum during
A `AlkType:Uint64` variant was added to the `AlkTypeKind` enum during
implementation (the task decomposition correctly identified the gap),
but without an ADR the addition was half-finished: `type_size()`
returned `None`, the layout engines couldn't compute offsets for it, and
the validator didn't register a `TypeDef:Uint64` keyword. The variant
the validator didn't register a `AlkType:Uint64` keyword. The variant
was then removed (commit `14d9cf2`) on the grounds that it was
unintended and a latent panic — but the underlying gap is real: SFTP and
metatensor, the two primary POC targets, both require 64-bit integers.
@@ -28,11 +28,11 @@ is 8 raw bytes, and `from_le_bytes`/`from_be_bytes` work correctly for
the full `u64`/`i64` range. The validation concern is handled by
accepting integer-form JSON values (the `jsonschema` crate's
`as_i64`/`as_u64` methods handle the common range; values past 2^53 are
a JSON representation limitation, not a typedef limitation).
a JSON representation limitation, not a alktype limitation).
## Decision
**Add `TypeDef:Int64` and `TypeDef:Uint64` as first-class kinds.**
**Add `AlkType:Int64` and `AlkType:Uint64` as first-class kinds.**
Both are fixed-size (8 bytes), with natural alignment 8. They follow
the schema's endianness annotation like all other fixed-size types.
@@ -43,20 +43,20 @@ Read/write is via `data_access::read_i64`/`write_i64`/`read_u64`/
| Kind | TypeBox key | Rust type | Size | Alignment |
|------|-------------|-----------|------|-----------|
| `TInt64` | `TypeDef:Int64` | `i64` | 8 | 8 |
| `TUint64` | `TypeDef:Uint64` | `u64` | 8 | 8 |
| `TInt64` | `AlkType:Int64` | `i64` | 8 | 8 |
| `TUint64` | `AlkType:Uint64` | `u64` | 8 | 8 |
### Validation
The custom keyword validators check:
- `TypeDef:Int64`: value must be an integer in `i64::MIN..=i64::MAX`
- `AlkType:Int64`: value must be an integer in `i64::MIN..=i64::MAX`
(`-9223372036854775808` to `9223372036854775807`).
- `TypeDef:Uint64`: value must be a non-negative integer in
- `AlkType:Uint64`: value must be a non-negative integer in
`0..=u64::MAX` (`0` to `18446744073709551615`).
The `jsonschema` crate's `as_i64`/`as_u64` handle the common range.
JSON numbers past 2^53 lose precision in the JSON representation —
this is a JSON limitation, not a typedef limitation. The binary
this is a JSON limitation, not a alktype limitation. The binary
representation (8 raw bytes) is always exact. A consumer that needs
to validate the full 64-bit range from JSON should provide the value
as a JSON integer (which `serde_json` preserves for values up to
@@ -66,13 +66,13 @@ enabled, or when the value fits in `i64`/`u64` without the feature).
### `FieldValue` additions
`FieldValue::I64(i64)` and `FieldValue::U64(u64)` are added to the
unified return type. The `SequentialReader`, `TypedefEngine::read_field`,
and `TypedefEngine::write_field` dispatch on the new kinds.
unified return type. The `SequentialReader`, `AlkTypeEngine::read_field`,
and `AlkTypeEngine::write_field` dispatch on the new kinds.
### Kind count
The engine now has **19** first-class kinds (17 + Int64 + Uint64).
`TypeDefKind::is_fixed_size()` returns `true` for both new kinds.
`AlkTypeKind::is_fixed_size()` returns `true` for both new kinds.
`type_size()` returns `Some(8)`. `natural_alignment()` returns `8`.
`needs_endian()` returns `true`.
@@ -82,8 +82,8 @@ The engine now has **19** first-class kinds (17 + Int64 + Uint64).
- **Unblocks the two primary POC targets.** SFTP `Read`/`Write` packets
(`offset: u64`) and metatensor `data_offsets` (`u64`) are now
expressible in typedef schemas.
- **Completes the half-finished addition.** The `TypeDefKind` enum,
expressible in alktype schemas.
- **Completes the half-finished addition.** The `AlkTypeKind` enum,
`data_access` primitives, and `FieldValue` variants for 64-bit
integers now have matching layout, validator, and engine support.
- **No new design surface.** Int64/Uint64 are fixed-size types that
@@ -94,7 +94,7 @@ The engine now has **19** first-class kinds (17 + Int64 + Uint64).
- **JSON precision caveat.** Values past 2^53 lose precision in the
JSON representation (not in the binary representation). This is a
JSON limitation, not a typedef limitation, but it means the
JSON limitation, not a alktype limitation, but it means the
validation layer cannot perfectly round-trip the full 64-bit range
through JSON `Number` without `arbitrary_precision`. In practice,
SFTP offsets and tensor data offsets are well within 2^53.
@@ -105,7 +105,7 @@ The engine now has **19** first-class kinds (17 + Int64 + Uint64).
## References
- `docs/research/call-channels-unification/findings.md` §"russh-sftp" —
the SFTP schema with `"offset": { "TypeDef:Uint64": true }`
the SFTP schema with `"offset": { "AlkType:Uint64": true }`
- `docs/research/alknet-typedef/findings.md` §"POC 1" — the POC included
u64 read/write
- [ADR-001](001-alktype-purpose-scope-jsonschema-engine.md) —

8
docs/architecture/decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md
View File

@@ -17,7 +17,7 @@ indirection)."
The implementation has a bug: `OffsetMap::compute` reserves only 4 bytes
for an inline length-prefixed variable field (the length prefix), but
`TypedefEngine::write_field` for a `String`/`Bytes` field calls
`AlkTypeEngine::write_field` for a `String`/`Bytes` field calls
`data_access::write_string` at `range.start`, which writes
`[4-byte length][data]` inline — clobbering every subsequent field. The
`read_field` path has the mirror behavior (reads inline), so the engine
@@ -45,12 +45,12 @@ length-prefixing.
**Reject non-final inline length-prefixed variable fields in aligned
static mode at `OffsetMap::compute` time.**
A variable-length field (`TypeDef:String`, `TypeDef:Bytes`,
`TypeDef:Timestamp`, `TypeDef:Record`) in aligned static mode that uses
A variable-length field (`AlkType:String`, `AlkType:Bytes`,
`AlkType:Timestamp`, `AlkType:Record`) in aligned static mode that uses
the default inline length-prefixing strategy (no `maxLength`, no
`offset-indirect`) must be the last field in its struct. If a non-final
inline length-prefixed variable field is encountered,
`OffsetMap::compute` returns `TypedefError::Offset` with a message
`OffsetMap::compute` returns `AlkTypeError::Offset` with a message
explaining that non-final variable fields in aligned mode require
`maxLength` (fixed-size reservation) or `"encoding": "offset-indirect"`
(offset indirection).

4
docs/architecture/decisions/007-packed-mode-read-factory.md
View File

@@ -5,7 +5,7 @@ Accepted
## Context
`TypedefEngine` stores a `SequentialReader` inside its `Layout::Packed`
`AlkTypeEngine` stores a `SequentialReader` inside its `Layout::Packed`
variant. The engine exposes it via
`engine.sequential_reader() -> Option<&SequentialReader>`.
@@ -43,7 +43,7 @@ Three options were considered:
an owned fresh reader, reconstructed from the stored schema.
```rust
impl TypedefEngine {
impl AlkTypeEngine {
/// Construct a fresh SequentialReader for packed-mode reads.
/// Returns None if compiled in aligned mode.
pub fn sequential_reader(&self) -> Option<SequentialReader>;

6
docs/architecture/decisions/008-reject-tunion-in-aligned-mode.md
View File

@@ -12,7 +12,7 @@ aligned mode has three implementation problems:
1. **No variant field offsets.** Only the `__discriminator` byte range
is recorded in the `OffsetMap`. Variant field offsets are not
available anywhere in aligned mode — the consumer must recompute
them by hand. This makes `TypedefEngine::read_field` on a union
them by hand. This makes `AlkTypeEngine::read_field` on a union
variant field impossible.
2. **`find_discriminator_field` takes the first variant's offset.** For
@@ -49,8 +49,8 @@ worse than rejecting it clearly.
**Reject `TUnion` in aligned static mode for v1.**
`OffsetMap::compute` returns `TypedefError::Offset` when it encounters
a `TypeDef:Union` field, with a message explaining that unions are not
`OffsetMap::compute` returns `AlkTypeError::Offset` when it encounters
a `AlkType:Union` field, with a message explaining that unions are not
supported in aligned mode and the consumer should use packed mode (or
restructure as a struct with an explicit discriminator field).

30
docs/architecture/layout-engine.md
View File

@@ -24,8 +24,8 @@ protocols.
**Components:**
- **`LayoutBuilder`** — constructed via `LayoutBuilder::new(schema)` (requires `TypeDef:Struct` at the top level), then `builder.build(&var_sizes) -> Result<PackedLayout, TypedefError>` where `var_sizes: &HashMap<String, usize>` maps variable-length field paths (and TUnion discriminator/variant keys) to their actual byte sizes. Used at write time when the consumer knows the data sizes upfront. The builder computes positions only; the consumer writes data via the [`data_access`](data-access.md) functions at the computed positions.
- **`SequentialReader`** — constructed via `SequentialReader::new(schema)`, then driven by `reader.read_next(&buffer) -> Result<Option<(String, FieldValue)>, TypedefError>` until `Ok(None)`, or `reader.read_field(&buffer, path)` to seek a single field (which walks all preceding fields to reach the target). `reader.reset()` rewinds to the start. Used at read time when the consumer is parsing an incoming frame.
- **`LayoutBuilder`** — constructed via `LayoutBuilder::new(schema)` (requires `AlkType:Struct` at the top level), then `builder.build(&var_sizes) -> Result<PackedLayout, AlkTypeError>` where `var_sizes: &HashMap<String, usize>` maps variable-length field paths (and TUnion discriminator/variant keys) to their actual byte sizes. Used at write time when the consumer knows the data sizes upfront. The builder computes positions only; the consumer writes data via the [`data_access`](data-access.md) functions at the computed positions.
- **`SequentialReader`** — constructed via `SequentialReader::new(schema)`, then driven by `reader.read_next(&buffer) -> Result<Option<(String, FieldValue)>, AlkTypeError>` until `Ok(None)`, or `reader.read_field(&buffer, path)` to seek a single field (which walks all preceding fields to reach the target). `reader.reset()` rewinds to the start. Used at read time when the consumer is parsing an incoming frame.
**How it works:**
@@ -69,7 +69,7 @@ and safetensors.
**Component:**
- **`OffsetMap`** — constructed via `OffsetMap::compute(schema) -> Result<Self, TypedefError>` (requires `TypeDef:Struct` at the top level). Walks the schema once, computes fixed byte positions for each field based on type sizes and alignment. The output is a flat table of `(field_path, byte_range)` pairs (see [Public Types](#public-types)). Used for both read and write at known offsets.
- **`OffsetMap`** — constructed via `OffsetMap::compute(schema) -> Result<Self, AlkTypeError>` (requires `AlkType:Struct` at the top level). Walks the schema once, computes fixed byte positions for each field based on type sizes and alignment. The output is a flat table of `(field_path, byte_range)` pairs (see [Public Types](#public-types)). Used for both read and write at known offsets.
**How it works:**
@@ -109,7 +109,7 @@ length, then slices the separate data region.
Inline length-prefixed variable fields in aligned mode are only allowed
as the **last field** in their struct. A non-final inline
length-prefixed variable field is rejected at `OffsetMap::compute` time
with a `TypedefError::Offset` — the `OffsetMap` reserves only 4 bytes
with a `AlkTypeError::Offset` — the `OffsetMap` reserves only 4 bytes
(the length prefix), but `data_access::write_string` writes prefix +
data inline, which would clobber subsequent fields. Non-final variable
fields must use `maxLength` (fixed-size reservation) or
@@ -125,7 +125,7 @@ padding is inserted between fields.
### Fixed-size types
For each fixed-size type, the algorithm:
1. Determines the type's byte size from the `TypeDef:*` kind.
1. Determines the type's byte size from the `AlkType:*` kind.
2. In aligned mode: inserts padding to satisfy the type's alignment
(or the field's `align` annotation, or the struct's `align` default).
3. Records the field's `(start, end)` range.
@@ -141,7 +141,7 @@ its total size.
**`TUnion`:** TUnion is supported in packed sequential mode only. In
aligned static mode, `OffsetMap::compute` rejects `TUnion` fields with
`TypedefError::Offset` — see
`AlkTypeError::Offset` — see
[ADR-008](decisions/008-reject-tunion-in-aligned-mode.md). Unions
are the protocol dispatch pattern (SFTP type bytes, call protocol event
types); mmap-friendly formats use structs and arrays, not tagged unions.
@@ -170,7 +170,7 @@ alignment padding in aligned mode). Element `i` starts at
### Variable-length types
The typedef engine supports three strategies for variable-length types
The alktype engine supports three strategies for variable-length types
(see [schema-layer.md](schema-layer.md) §Variable-length types and
[ADR-003](decisions/003-schema-annotations.md) §3 for the full
annotation shapes).
@@ -221,7 +221,7 @@ and byte-swaps accordingly. All fixed-size types — including `TEnum`
## Mode Selection
The consumer selects the mode at engine construction time via the
`LayoutMode` enum, passed to `TypedefEngine::compile`:
`LayoutMode` enum, passed to `AlkTypeEngine::compile`:
```rust
pub enum LayoutMode {
@@ -246,7 +246,7 @@ frames) and a `LayoutBuilder` (for constructing outgoing frames). A schema
describing a metatensor layout can be consumed by an `OffsetMap` (for
mmap access).
`TypedefEngine` exposes mode-appropriate accessors: `engine.offset_map()`
`AlkTypeEngine` exposes mode-appropriate accessors: `engine.offset_map()`
returns `Some(&OffsetMap)` in aligned mode and `None` in packed mode;
`engine.layout_builder()` returns `Some(&LayoutBuilder)` in packed mode
and `None` in aligned mode. `engine.sequential_reader()` returns
@@ -254,7 +254,7 @@ and `None` in aligned mode. `engine.sequential_reader()` returns
reader has mutable cursor state that the consumer owns; see
[ADR-007](decisions/007-packed-mode-read-factory.md)) in packed
mode and `None` in aligned mode. See [validation.md](validation.md)
§"The TypedefEngine struct" for the engine API.
§"The AlkTypeEngine struct" for the engine API.
## Public Types
@@ -282,14 +282,14 @@ provides `len()` and `is_empty()`.
pub struct FieldPosition {
pub offset: usize,
pub size: usize,
pub kind: TypeDefKind,
pub kind: AlkTypeKind,
}
```
A field's computed position in a packed layout, produced by
`LayoutBuilder::build`. For variable-length fields, `size` is `4` (the
length prefix); for fixed-size fields, `size` is the type's byte size.
`kind` records the field's `TypeDef:*` kind so the consumer can dispatch
`kind` records the field's `AlkType:*` kind so the consumer can dispatch
to the correct `data_access` read/write function.
### `PackedLayout` (packed mode)
@@ -317,14 +317,14 @@ A flat table of `(field_path, byte_range)` pairs computed from a schema.
```rust
impl OffsetMap {
pub fn compute(schema: &Value) -> Result<Self, TypedefError>;
pub fn compute(schema: &Value) -> Result<Self, AlkTypeError>;
pub fn get(&self, field_path: &str) -> Option<&ByteRange>;
pub fn total_size(&self) -> usize;
pub fn iter(&self) -> impl Iterator<Item = &(String, ByteRange)>;
}
```
`compute` requires a `TypeDef:Struct` at the top level. `total_size`
`compute` requires a `AlkType:Struct` at the top level. `total_size`
includes trailing alignment padding. `iter` yields fields in insertion
order (schema `properties` order, nested struct fields appearing inline).
@@ -354,7 +354,7 @@ See [open-questions.md](open-questions.md) for full details.
the two layout modes decision
- [ADR-003](decisions/003-schema-annotations.md) — schema
annotations
- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds and their
- [schema-layer.md](schema-layer.md) — the 17 AlkType kinds and their
byte sizes
- [data-access.md](data-access.md) — read/write functions that use the
computed offsets

40
docs/architecture/overview.md
View File

@@ -6,7 +6,7 @@ last_updated: 2026-07-22
# alktype — Overview
The binary struct engine: a small Rust crate that takes a JSON Schema
with `TypeDef:*` custom keywords and produces an offset map, read/write
with `AlkType:*` custom keywords and produces an offset map, read/write
functions, and validation — all driven by the schema. The schema is the
format definition; the engine is generic.
@@ -17,8 +17,8 @@ Component details are in the sibling documents.
## What
`alktype` is a library crate that consumes JSON Schemas annotated
with `TypeDef:*` custom keywords (the same kinds defined in TypeBox's
`typedef.ts`, plus `TypeDef:Bytes`, `TypeDef:Int64`, and `TypeDef:Uint64`
with `AlkType:*` custom keywords (the same kinds defined in TypeBox's
`typedef.ts`, plus `AlkType:Bytes`, `AlkType:Int64`, and `AlkType:Uint64`
as alktype additions) and produces three capabilities:
1. **An offset map** — walks the schema, computes byte offsets for each
@@ -52,8 +52,8 @@ engine with different schemas.
The guiding insight:
> **The schema is the format.** A JSON Schema with `TypeDef:Float32`,
> `TypeDef:Struct`, `TypeDef:Union` etc. is both the validation spec and
> **The schema is the format.** A JSON Schema with `AlkType:Float32`,
> `AlkType:Struct`, `AlkType:Union` etc. is both the validation spec and
> the layout spec. No separate format definition, no separate parser, no
> separate validator. One schema, three uses: validate, compute offsets,
> access data.
@@ -67,15 +67,15 @@ the binary data is the struct's bytes at computed offsets.
The crate was bumped up in the timeline when the call-channels-unification
research surfaced that channels, TTY, and the binary call protocol are
all variations on the same wire-format family — `[discriminant][length][payload]`.
The typedef engine makes the "channels is call with a binary data plane"
The alktype engine makes the "channels is call with a binary data plane"
unification concrete: the binary data plane's wire format is the call
protocol's own schema system, just binary-encoded. The `channel_open`
marker says "use binary framing"; the typedef engine says "here's how to
marker says "use binary framing"; the alktype engine says "here's how to
read/write the binary payload."
## The "Schema Is the Format" Principle
A JSON Schema with `TypeDef:*` custom keywords serves three roles
A JSON Schema with `AlkType:*` custom keywords serves three roles
simultaneously:
| Role | Mechanism | When |
@@ -123,7 +123,7 @@ schema JSON determines the order of fields in the binary struct.
The russh-sftp case is the most instructive and the highest-value POC
target. The `Packet` enum's `TryFrom<&mut Bytes>` impl is a hand-written
dispatch on a type byte followed by serde deserialization. Under typedef,
dispatch on a type byte followed by serde deserialization. Under alktype,
the dispatch is `TUnion` with a byte-offset discriminator — the schema
says "byte 0 is the discriminator, bytes 1..N are the variant struct."
The engine reads the discriminator, looks up the variant schema, computes
@@ -133,28 +133,28 @@ offsets, reads fields. Same result, no per-packet-type code.
These boundaries are decided in [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md).
- **Not metatensor.** typedef is the binary struct *engine*. Metatensor
- **Not metatensor.** alktype is the binary struct *engine*. Metatensor
is a *format* (8-byte header + JSON header + binary data) that uses the
typedef engine for its offset computation and tensor access.
alktype engine for its offset computation and tensor access.
- **Not a Value system.** TypeBox's `Value.Diff`, `Value.Migrate`,
`Value.Convert` — schema evolution — is out of scope for v1. The engine
should not do anything that explicitly blocks adding a Value system
later.
- **Not a code generator.** typebox-rs's `codegen/` module is a separate
concern. The typedef engine consumes schemas; it does not generate them.
- **Not a schema builder.** The typedef engine does not provide a fluent
concern. The alktype engine consumes schemas; it does not generate them.
- **Not a schema builder.** The alktype engine does not provide a fluent
API for constructing schemas. Schemas are plain JSON — authored in
TypeBox, generated by ujsx components, or hand-written. A builder API
is deferred (OQ-003).
- **Not a serialization framework.** The typedef engine is not a
- **Not a serialization framework.** The alktype engine is not a
general-purpose serde replacement. It operates on raw byte buffers at
computed offsets — no intermediate `Value` tree, no reflection, no
dynamic dispatch per field. For JSON data, use serde. For binary data
with a known schema, use typedef.
with a known schema, use alktype.
## Architecture (component pointers)
- **[schema-layer.md](schema-layer.md)** — the 19 `TypeDef:*` kinds,
- **[schema-layer.md](schema-layer.md)** — the 19 `AlkType:*` kinds,
jsonschema custom keyword integration, TypeBox interop, schema
annotations (endianness, alignment, encoding, TUnion discriminators).
- **[layout-engine.md](layout-engine.md)** — offset computation, the two
@@ -164,8 +164,8 @@ These boundaries are decided in [ADR-001](decisions/001-alktype-purpose-scope-js
dispatch, field paths, zero-copy access for fixed-size types,
length-prefix reading for variable-length types.
- **[validation.md](validation.md)** — custom keyword validators for all
19 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time
validation, `TypedefEngine` as the compiled form of a schema.
19 `AlkType:*` kinds, `AlkTypeError`, load-time vs access-time
validation, `AlkTypeEngine` as the compiled form of a schema.
## Design Decisions
@@ -174,7 +174,7 @@ These boundaries are decided in [ADR-001](decisions/001-alktype-purpose-scope-js
| Purpose, scope, and the jsonschema engine | [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
| Two layout modes | [ADR-002](decisions/002-two-layout-modes-packed-vs-aligned.md) | Packed sequential (`LayoutBuilder`/`SequentialReader`) for protocols; aligned static (`OffsetMap`) for mmap formats |
| Schema annotations | [ADR-003](decisions/003-schema-annotations.md) | Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name) |
| Error handling and validation | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Error handling and validation | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Int64/Uint64 kinds | [ADR-005](decisions/005-int64-uint64-first-class-kinds.md) | 64-bit integers as first-class kinds (SFTP offsets, metatensor data_offsets) |
| Non-final inline variable fields | [ADR-006](decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Rejected in aligned mode (would clobber subsequent fields) |
| Packed-mode read factory | [ADR-007](decisions/007-packed-mode-read-factory.md) | `engine.sequential_reader()` returns an owned fresh reader |
@@ -199,5 +199,5 @@ See [open-questions.md](open-questions.md) for full details.
schema kinds (619 lines)
- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12)
- `/workspace/alknet-typedef-poc/` — the POC code (disposable)
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by alktype
- `/workspace/@alkimiadev/alktype/` — prior attempt (the @alkimiadev/alktype prototype; not to be confused with this crate, which reuses the name but is backed by the `jsonschema` crate)

2
docs/architecture/questions/003-builder-api-for-schema-construction.md
View File

@@ -9,7 +9,7 @@
changing the existing JSON-consumption path)
- **Priority**: medium
- **Impacts**: Blocks programmatic schema construction in Rust without a
JS toolchain. Any consumer that wants to build typedef schemas at
JS toolchain. Any consumer that wants to build alktype schemas at
runtime from Rust code (rather than loading pre-authored JSON) must
construct the JSON manually or depend on TypeBox. Does NOT block any
current consumer — all v1 consumers (SFTP, metatensor, binary call

150
docs/architecture/schema-layer.md
View File

@@ -5,12 +5,12 @@ last_updated: 2026-07-22
# alktype — Schema Layer
The schema layer: the 19 `TypeDef:*` custom type kinds, their mapping to
The schema layer: the 19 `AlkType:*` custom type kinds, their mapping to
Rust types and byte sizes, the `jsonschema` custom keyword integration,
TypeBox interop, and the concrete JSON shapes for schema-level
annotations.
## The 19 TypeDef Kinds
## The 19 AlkType Kinds
These are the custom schema kinds defined in TypeBox's `typedef.ts`
(`/workspace/@alkdev/typebox/example/typedef/typedef.ts`, 619 lines) and
@@ -20,42 +20,42 @@ encoding strategy (for variable-length types).
| Kind | TypeBox key | Rust type | Size | Category |
|------|-------------|-----------|------|----------|
| `TFloat32` | `TypeDef:Float32` | `f32` | 4 | fixed |
| `TFloat64` | `TypeDef:Float64` | `f64` | 8 | fixed |
| `TInt8` | `TypeDef:Int8` | `i8` | 1 | fixed |
| `TInt16` | `TypeDef:Int16` | `i16` | 2 | fixed |
| `TInt32` | `TypeDef:Int32` | `i32` | 4 | fixed |
| `TInt64` | `TypeDef:Int64` | `i64` | 8 | fixed |
| `TUint8` | `TypeDef:Uint8` | `u8` | 1 | fixed |
| `TUint16` | `TypeDef:Uint16` | `u16` | 2 | fixed |
| `TUint32` | `TypeDef:Uint32` | `u32` | 4 | fixed |
| `TUint64` | `TypeDef:Uint64` | `u64` | 8 | fixed |
| `TBoolean` | `TypeDef:Boolean` | `bool` (0x00=false, 0x01=true) | 1 | fixed |
| `TString` | `TypeDef:String` | length-prefixed UTF-8 | variable | variable |
| `TBytes` | `TypeDef:Bytes` | length-prefixed raw bytes | variable | variable |
| `TStruct` | `TypeDef:Struct` | record of fields | sum of field sizes | composite |
| `TUnion` | `TypeDef:Union` | tagged union | discriminator + variant | composite |
| `TArray` | `TypeDef:Array` | repeated element | count × element size | composite |
| `TEnum` | `TypeDef:Enum` | u32 index into enum values | 4 (fixed) | fixed |
| `TRecord` | `TypeDef:Record` | count-prefixed sequence of (key, value) pairs | variable | variable |
| `TTimestamp` | `TypeDef:Timestamp` | length-prefixed RFC 3339 string | variable | variable |
| `TFloat32` | `AlkType:Float32` | `f32` | 4 | fixed |
| `TFloat64` | `AlkType:Float64` | `f64` | 8 | fixed |
| `TInt8` | `AlkType:Int8` | `i8` | 1 | fixed |
| `TInt16` | `AlkType:Int16` | `i16` | 2 | fixed |
| `TInt32` | `AlkType:Int32` | `i32` | 4 | fixed |
| `TInt64` | `AlkType:Int64` | `i64` | 8 | fixed |
| `TUint8` | `AlkType:Uint8` | `u8` | 1 | fixed |
| `TUint16` | `AlkType:Uint16` | `u16` | 2 | fixed |
| `TUint32` | `AlkType:Uint32` | `u32` | 4 | fixed |
| `TUint64` | `AlkType:Uint64` | `u64` | 8 | fixed |
| `TBoolean` | `AlkType:Boolean` | `bool` (0x00=false, 0x01=true) | 1 | fixed |
| `TString` | `AlkType:String` | length-prefixed UTF-8 | variable | variable |
| `TBytes` | `AlkType:Bytes` | length-prefixed raw bytes | variable | variable |
| `TStruct` | `AlkType:Struct` | record of fields | sum of field sizes | composite |
| `TUnion` | `AlkType:Union` | tagged union | discriminator + variant | composite |
| `TArray` | `AlkType:Array` | repeated element | count × element size | composite |
| `TEnum` | `AlkType:Enum` | u32 index into enum values | 4 (fixed) | fixed |
| `TRecord` | `AlkType:Record` | count-prefixed sequence of (key, value) pairs | variable | variable |
| `TTimestamp` | `AlkType:Timestamp` | length-prefixed RFC 3339 string | variable | variable |
`TypeDef:Int64` and `TypeDef:Uint64` are alktype additions —
`AlkType:Int64` and `AlkType:Uint64` are alktype additions —
TypeBox's `typedef.ts` tops out at 32-bit integers. They are required by
the primary POC targets: SFTP `Read`/`Write` packets have `offset: u64`,
and metatensor `data_offsets` are `u64`. See
[ADR-005](decisions/005-int64-uint64-first-class-kinds.md).
### The `TypeDefKind` enum
### The `AlkTypeKind` enum
The engine represents the 19 kinds as a Rust enum — `TypeDefKind` — with
one variant per kind (`TypeDefKind::Float32`, `TypeDefKind::Struct`, etc.).
The engine represents the 19 kinds as a Rust enum — `AlkTypeKind` — with
one variant per kind (`AlkTypeKind::Float32`, `AlkTypeKind::Struct`, etc.).
The enum provides compile-time exhaustiveness checking and integer
discriminant dispatch (a jump table) instead of string comparison at
every field access. It is `pub` and re-exported from the crate root.
```rust
pub enum TypeDefKind {
pub enum AlkTypeKind {
Int8, Int16, Int32, Int64,
Uint8, Uint16, Uint32, Uint64,
Float32, Float64,
@@ -69,7 +69,7 @@ The enum carries the kind's binary-layout metadata as inherent methods:
| Method | Returns | Notes |
|--------|---------|-------|
| `as_str(self)` | `&'static str` | The JSON Schema keyword, e.g. `"TypeDef:Uint8"` |
| `as_str(self)` | `&'static str` | The JSON Schema keyword, e.g. `"AlkType:Uint8"` |
| `type_size(self)` | `Option<usize>` | `Some(N)` for fixed-size kinds; `None` for variable/composite |
| `natural_alignment(self)` | `usize` | 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array |
| `is_fixed_size(self)` | `bool` | True for the 12 fixed-size primitive kinds |
@@ -77,9 +77,9 @@ The enum carries the kind's binary-layout metadata as inherent methods:
| `is_variable_length(self)` | `bool` | True for String, Bytes, Timestamp, Record |
| `needs_endian(self)` | `bool` | True for kinds whose read/write takes an `Endian` parameter |
`TypeDefKind` implements `Display` (renders the keyword string) and
`AlkTypeKind` implements `Display` (renders the keyword string) and
`FromStr` (parses the keyword string back into the variant, returning
`TypedefError::Schema` for unknown kinds). The layout engines and the
`AlkTypeError::Schema` for unknown kinds). The layout engines and the
validator dispatch on the enum, not on strings.
### Fixed-size types
@@ -90,17 +90,17 @@ computation uses these sizes directly. Read/write is zero-copy pointer
cast for these types.
**`TBoolean` byte representation:** `0x00` = false, `0x01` = true. Other
values are invalid and produce a `TypedefError::Access` on read.
values are invalid and produce a `AlkTypeError::Access` on read.
**`TEnum` binary representation:** A `u32` index into the enum's declared
values, in declaration order. The first declared value is index 0, the
second is index 1, etc. The enum's values are declared via the standard
JSON Schema `"enum"` keyword (e.g., `"enum": ["read", "write", "execute"]`).
The `TypeDef:Enum` custom keyword signals that the type is an enum for
The `AlkType:Enum` custom keyword signals that the type is an enum for
layout purposes; the built-in `enum` keyword provides the value list.
**Design note:** TypeBox's `TEnum` is a string enum (variable-length). The
typedef engine uses a `u32` index instead — a deliberate deviation from
alktype engine uses a `u32` index instead — a deliberate deviation from
TypeBox fidelity in favor of binary efficiency. Most enums have a small
number of variants (e.g., the call protocol's 5 event types); a `u32`
index is compact, fixed-size, and sufficient for any realistic enum. The
@@ -113,7 +113,7 @@ all other fixed-size types. In little-endian mode the index is
### Variable-length types
`TString`, `TBytes`, `TRecord`, and `TTimestamp` have variable byte sizes.
The typedef engine supports three strategies for handling variable-length
The alktype engine supports three strategies for handling variable-length
types in binary layouts, selected by the `encoding` annotation and the
standard JSON Schema `maxLength` keyword:
@@ -169,7 +169,7 @@ consistent byte order for both field values and length prefixes.
**`TBytes`:** Raw bytes — no UTF-8 constraint. The payload is `&[u8]`.
Otherwise identical to `TString` in layout (same three strategies).
**Design note:** `TypeDef:Bytes` is an alktype addition — it does
**Design note:** `AlkType:Bytes` is an alktype addition — it does
not exist in TypeBox's `typedef.ts` (which defines 16 kinds). It is
included because raw byte arrays are a common binary protocol primitive
(SFTP data payloads, channels payloads, tensor data) and are semantically
@@ -178,11 +178,11 @@ bytes with no encoding (not base64, not hex). In the JSON representation
(for validation), TBytes is a string (JSON has no native byte type).
**`TRecord`:** A string-keyed map. The value type is declared via the
schema's `"values"` property (e.g., `"values": { "TypeDef:Float32": true }`).
schema's `"values"` property (e.g., `"values": { "AlkType:Float32": true }`).
Binary layout is a count-prefixed sequence of `(key, value)` pairs:
`[count: u32][key_len: u32][key_bytes][value]...` repeated `count` times.
The count is the number of entries. Each key is a length-prefixed UTF-8
string. Each value is encoded according to its declared `TypeDef:*` kind
string. Each value is encoded according to its declared `AlkType:*` kind
— a `Record<Uint32>` value is 4 raw bytes; a `Record<String>` value is
itself a length-prefixed string; a `Record<Struct>` value is the struct's
fields laid out inline. There is **no separate `value_len` prefix** —
@@ -222,25 +222,25 @@ computation recurses into their properties.
The `schema` module exposes the foundational types and functions every
other module depends on. These are re-exported from the crate root.
### `get_typedef_kind` vs `get_typedef_kind_loose`
### `get_alktype_kind` vs `get_alktype_kind_loose`
The engine recognizes a `TypeDef:*` kind on a schema node two ways,
The engine recognizes a `AlkType:*` kind on a schema node two ways,
because the keyword value may be either a boolean (`true`) or an
annotation object (`{ "encoding": "..." }`):
| Function | Recognizes | Returns |
|----------|------------|---------|
| `get_typedef_kind(node) -> Option<&str>` | Boolean form only (`{ "TypeDef:String": true }`) | The keyword string, e.g. `"TypeDef:String"` |
| `get_typedef_kind_loose(node) -> Option<&str>` | Boolean form **and** object form | The keyword string |
| `get_typedef_kind_enum(node) -> Option<TypeDefKind>` | Boolean form only | The parsed enum variant |
| `get_typedef_kind_loose_enum(node) -> Option<TypeDefKind>` | Boolean form **and** object form | The parsed enum variant |
| `get_alktype_kind(node) -> Option<&str>` | Boolean form only (`{ "AlkType:String": true }`) | The keyword string, e.g. `"AlkType:String"` |
| `get_alktype_kind_loose(node) -> Option<&str>` | Boolean form **and** object form | The keyword string |
| `get_alktype_kind_enum(node) -> Option<AlkTypeKind>` | Boolean form only | The parsed enum variant |
| `get_alktype_kind_loose_enum(node) -> Option<AlkTypeKind>` | Boolean form **and** object form | The parsed enum variant |
The boolean-form-only functions are used by the validator factories
(which reject the object form as a schema error) and the top-level
kind-check in `OffsetMap::compute` / `LayoutBuilder::new` / `SequentialReader::new`
(which require `TypeDef:Struct` at the root). The "loose" variants are
(which require `AlkType:Struct` at the root). The "loose" variants are
used by the layout engines during field traversal, so that a variable-
length field with an `encoding` annotation (`{ "TypeDef:String":
length field with an `encoding` annotation (`{ "AlkType:String":
{ "encoding": "offset-indirect" } }`) is still recognized as a `String`.
### Annotation parsers
@@ -254,7 +254,7 @@ Each schema-level annotation has a dedicated parser that reads it from a
| `parse_align(node) -> Option<usize>` | `"align"` | `None` |
| `parse_max_length(node) -> Option<usize>` | `"maxLength"` | `None` |
| `parse_encoding(keyword_value) -> VariableEncoding` | `"encoding"` (within the keyword's value object) | `VariableEncoding::LengthPrefixed` |
| `parse_discriminator(node) -> Result<DiscriminatorKind, TypedefError>` | `"discriminator"` | (required — returns `TypedefError::Schema` if absent) |
| `parse_discriminator(node) -> Result<DiscriminatorKind, AlkTypeError>` | `"discriminator"` | (required — returns `AlkTypeError::Schema` if absent) |
### Public enums
@@ -262,13 +262,13 @@ Each schema-level annotation has a dedicated parser that reads it from a
pub enum Endian { Little, Big }
pub enum VariableEncoding { LengthPrefixed, OffsetIndirect }
pub enum DiscriminatorKind {
Byte { offset: usize, disc_type: TypeDefKind },
Byte { offset: usize, disc_type: AlkTypeKind },
Field { name: String },
}
```
`DiscriminatorKind::Byte` carries the byte position (`offset`) and the
discriminator's `TypeDef:*` kind (`disc_type`, restricted to `Uint8`/
discriminator's `AlkType:*` kind (`disc_type`, restricted to `Uint8`/
`Uint16`/`Uint32`). `DiscriminatorKind::Field` carries the discriminator
field's name. See [data-access.md](data-access.md) §"TUnion Dispatch" for
how these drive dispatch.
@@ -277,7 +277,7 @@ how these drive dispatch.
| Function | Purpose |
|----------|---------|
| `normalize_refs(schema: &mut Value)` | Walks the schema; rewrites every `"$ref"` whose value is a bare name (no `#` prefix) to `"#/$defs/<name>"`. Idempotent. Runs once at `TypedefEngine::compile` time. |
| `normalize_refs(schema: &mut Value)` | Walks the schema; rewrites every `"$ref"` whose value is a bare name (no `#` prefix) to `"#/$defs/<name>"`. Idempotent. Runs once at `AlkTypeEngine::compile` time. |
| `resolve_ref(root, ref_path) -> Option<&Value>` | Resolves a JSON Pointer `$ref` (e.g. `"#/$defs/Read"`) against the root schema. |
| `resolve_ref_or_inline(node, root) -> Option<&Value>` | If `node` has a `"$ref"`, resolves it against `root`; otherwise returns `node` itself (it's an inline schema). |
@@ -288,22 +288,22 @@ on every `$ref`-bearing node they encounter during traversal.
## jsonschema Custom Keyword Integration
The `jsonschema` crate (v0.46.5, Draft 2020-12) supports custom keywords
via the `with_keyword` API. Each `TypeDef:*` kind is registered as a
via the `with_keyword` API. Each `AlkType:*` kind is registered as a
custom keyword:
```rust
let validator = jsonschema::options()
.with_keyword("TypeDef:Float32", factory)
.with_keyword("TypeDef:Int32", factory)
.with_keyword("TypeDef:Struct", factory)
.with_keyword("AlkType:Float32", factory)
.with_keyword("AlkType:Int32", factory)
.with_keyword("AlkType:Struct", factory)
// ... all 17 kinds
.build(&schema)?;
```
The factory closure receives the parent schema object, the keyword's
value, and the schema path — enabling cross-keyword awareness. The
`TypeDef:Struct` validator, for example, inspects the parent's
`properties` to validate each field against its declared `TypeDef:*` kind.
`AlkType:Struct` validator, for example, inspects the parent's
`properties` to validate each field against its declared `AlkType:*` kind.
Each custom keyword implementation is ~10 lines. The `jsonschema` crate
handles all structural validation (object properties, required fields,
@@ -313,7 +313,7 @@ validator implementations.
This is the same pattern as TypeBox's `TypeRegistry.Set` on the JS side.
Same semantics, different language, same JSON Schema wire format. A
TypeBox schema serialized to JSON feeds into the typedef engine after a
TypeBox schema serialized to JSON feeds into the alktype engine after a
single pre-processing step: normalizing `$ref` values (see below).
## TypeBox Interop
@@ -330,8 +330,8 @@ const TensorRef = Type.Object({
```
serialized to JSON is a standard JSON Schema with `type: "object"`,
`properties`, and `required`. That JSON feeds into the typedef engine
after `$ref` normalization. The `TypeDef:*` custom keywords are added by
`properties`, and `required`. That JSON feeds into the alktype engine
after `$ref` normalization. The `AlkType:*` custom keywords are added by
TypeBox's `TypeRegistry.Set` — they appear in the serialized JSON as
additional properties on the schema object.
@@ -340,11 +340,11 @@ additional properties on the schema object.
TypeBox generates bare-name `$ref` values (e.g., `"$ref": "Read"`),
referencing sibling definitions within the same `$defs` block. The
`jsonschema` crate requires full JSON Pointer paths (e.g.,
`"$ref": "#/$defs/Read"`). The typedef engine normalizes TypeBox-style
`"$ref": "#/$defs/Read"`). The alktype engine normalizes TypeBox-style
refs at schema load time via [`normalize_refs`](#ref-resolution-and-normalization)
— a ~20-line recursive walk that rewrites every bare-name `"$ref"` to
`"#/$defs/<name>"`. The normalization is idempotent — full JSON Pointer
refs pass through unchanged. It runs once at `TypedefEngine::compile`
refs pass through unchanged. It runs once at `AlkTypeEngine::compile`
time, before the schema is passed to `jsonschema` or the offset
computation.
@@ -353,7 +353,7 @@ with `Resource 'Read' is not present in a registry`. Full JSON Pointer
refs (`#/$defs/Read`) resolve correctly. The normalization step bridges
the gap between TypeBox's output and jsonschema's input.
The typedef engine does not depend on TypeBox or any JS toolchain. It
The alktype engine does not depend on TypeBox or any JS toolchain. It
consumes JSON — whether that JSON was authored in TypeBox, generated by
a ujsx component, or hand-written. The schema is the interface.
@@ -367,7 +367,7 @@ decided in [ADR-003](decisions/003-schema-annotations.md).
Schema-level annotation with a default of little-endian:
```json
{ "TypeDef:Struct": true, "endian": "big", "properties": { ... } }
{ "AlkType:Struct": true, "endian": "big", "properties": { ... } }
```
- `"endian": "little"` (default) — read/write in little-endian byte order.
@@ -380,10 +380,10 @@ Both struct-level and field-level, with field-level overriding:
```json
{
"TypeDef:Struct": true,
"AlkType:Struct": true,
"align": 256,
"properties": {
"weight": { "TypeDef:Float32": true, "align": 16 }
"weight": { "AlkType:Float32": true, "align": 16 }
}
}
```
@@ -398,23 +398,23 @@ Both struct-level and field-level, with field-level overriding:
### Variable-length encoding
The typedef engine supports three strategies for variable-length types
The alktype engine supports three strategies for variable-length types
(see §Variable-length types above for full details). The strategy is
selected by the `encoding` annotation and the standard JSON Schema
`maxLength` keyword:
```json
// Strategy 1: Inline length-prefixing (default, shorthand)
{ "TypeDef:String": true }
{ "AlkType:String": true }
// Strategy 1: Explicit inline length-prefixing
{ "TypeDef:String": { "encoding": "length-prefixed" } }
{ "AlkType:String": { "encoding": "length-prefixed" } }
// Strategy 2: Fixed-size reservation (uses standard maxLength)
{ "TypeDef:String": true, "maxLength": 256 }
{ "AlkType:String": true, "maxLength": 256 }
// Strategy 3: Offset indirection (opt-in)
{ "TypeDef:String": { "encoding": "offset-indirect" } }
{ "AlkType:String": { "encoding": "offset-indirect" } }
```
- `"encoding": "length-prefixed"` (default) — 4-byte length prefix at
@@ -428,8 +428,8 @@ selected by the `encoding` annotation and the standard JSON Schema
`{offset: u32, length: u32}` pointing into a separate data region.
The consumer provides the data region separately. Used by metatensor
blob tensors.
- Applies to all variable-length types: `TypeDef:String`, `TypeDef:Bytes`,
`TypeDef:Array`, `TypeDef:Record`, `TypeDef:Timestamp`.
- Applies to all variable-length types: `AlkType:String`, `AlkType:Bytes`,
`AlkType:Array`, `AlkType:Record`, `AlkType:Timestamp`.
### TUnion discriminators
@@ -440,11 +440,11 @@ Two discriminator kinds: byte-offset (protocol dispatch) and field-name
```json
{
"TypeDef:Union": true,
"AlkType:Union": true,
"discriminator": {
"kind": "byte",
"offset": 0,
"type": "TypeDef:Uint8"
"type": "AlkType:Uint8"
},
"mapping": {
"5": { "$ref": "#/$defs/Read" },
@@ -455,8 +455,8 @@ Two discriminator kinds: byte-offset (protocol dispatch) and field-name
```
- `"offset"` — byte position of the discriminator.
- `"type"` — the `TypeDef:*` kind of the discriminator (typically
`TypeDef:Uint8`).
- `"type"` — the `AlkType:*` kind of the discriminator (typically
`AlkType:Uint8`).
- Mapping keys are stringified integers. The variant struct starts at
`offset + discriminator_size`.
@@ -464,7 +464,7 @@ Two discriminator kinds: byte-offset (protocol dispatch) and field-name
```json
{
"TypeDef:Union": true,
"AlkType:Union": true,
"discriminator": {
"kind": "field",
"name": "type"

96
docs/architecture/validation.md
View File

@@ -5,22 +5,22 @@ last_updated: 2026-07-22
# alktype — Validation
The validation layer: custom keyword validators for all 19 `TypeDef:*`
kinds, the `TypedefError` enum, load-time vs access-time validation
strategy, and the `TypedefEngine` as the compiled form of a schema.
The validation layer: custom keyword validators for all 19 `AlkType:*`
kinds, the `AlkTypeError` enum, load-time vs access-time validation
strategy, and the `AlkTypeEngine` as the compiled form of a schema.
## Validation Strategy
Validation is delegated to the `jsonschema` crate (v0.46.5, Draft
2020-12). The typedef engine does not implement its own validation —
it registers custom keyword validators for each `TypeDef:*` kind and
2020-12). The alktype engine does not implement its own validation —
it registers custom keyword validators for each `AlkType:*` kind and
lets `jsonschema` handle the structural validation (object properties,
required fields, array items, enum values).
The strategy is decided in [ADR-004](decisions/004-error-handling-validation-strategy.md):
1. **Load time:** Parse the schema JSON, build the layout engine, build the
jsonschema validator. This is the `TypedefEngine::compile(schema)` constructor.
jsonschema validator. This is the `AlkTypeEngine::compile(schema)` constructor.
2. **Access time:** Use the compiled engine for repeated read/write
operations. Validation is opt-in per operation.
@@ -37,7 +37,7 @@ the correct separation of concerns:
- **Binary access validation** (data access layer): the read/write
functions perform type-level validation at access time — range checks
for integers, UTF-8 validity for strings, buffer bounds checking.
These return `TypedefError::Access` with field paths.
These return `AlkTypeError::Access` with field paths.
The "schema is the format" principle means the same schema describes
both the JSON shape and the binary layout. The jsonschema validator
@@ -47,13 +47,13 @@ buffer into a `Value` tree via the data access layer, then validates
that `Value` against the jsonschema validator. This is a two-step
process, not a single `validate(buffer)` call.
### The `TypedefEngine` struct
### The `AlkTypeEngine` struct
The `TypedefEngine` is the compiled form of a schema. It supports both
The `AlkTypeEngine` is the compiled form of a schema. It supports both
layout modes (ADR-002) via an internal `Layout` enum:
```rust
pub struct TypedefEngine {
pub struct AlkTypeEngine {
layout: Layout, // packed or aligned (private enum)
validator: jsonschema::Validator, // compiled once at load time
endian: Endian, // parsed from the schema's "endian" annotation
@@ -72,8 +72,8 @@ The consumer selects the mode at construction time via `LayoutMode`
enum is private — the engine exposes mode-appropriate accessors instead:
```rust
impl TypedefEngine {
pub fn compile(schema: &mut Value, mode: LayoutMode) -> Result<Self, TypedefError>;
impl AlkTypeEngine {
pub fn compile(schema: &mut Value, mode: LayoutMode) -> Result<Self, AlkTypeError>;
pub fn mode(&self) -> LayoutMode;
pub fn endian(&self) -> Endian;
pub fn offset_map(&self) -> Option<&OffsetMap>; // Some in aligned mode
@@ -94,39 +94,39 @@ The `SequentialReader` (read-side) is not stored — it has mutable cursor
state that the consumer owns, so `sequential_reader()` constructs a fresh
reader on each call (ADR-007).
The `read_field`/`write_field` methods on `TypedefEngine` are the
The `read_field`/`write_field` methods on `AlkTypeEngine` are the
aligned-mode data-access API — see [data-access.md](data-access.md)
§"Higher-level read/write".
## Custom Keyword Validators
Each `TypeDef:*` kind gets a `Keyword` implementation registered via
Each `AlkType:*` kind gets a `Keyword` implementation registered via
`jsonschema::options().with_keyword(...)`. The validators check leaf
type constraints; `jsonschema` handles all structural validation.
### Numeric type validators
**`TypeDef:Float32` / `TypeDef:Float64`:**
**`AlkType:Float32` / `AlkType:Float64`:**
- Value must be a finite number.
- For `Float32`: value must be representable as `f32` (no precision loss
beyond `f32`'s mantissa).
**`TypeDef:Int8` / `TypeDef:Int16` / `TypeDef:Int32`:**
**`AlkType:Int8` / `AlkType:Int16` / `AlkType:Int32`:**
- Value must be an integer within the type's range.
- Int8: -128..127, Int16: -32768..32767, Int32: -2147483648..2147483647.
**`TypeDef:Uint8` / `TypeDef:Uint16` / `TypeDef:Uint32`:**
**`AlkType:Uint8` / `AlkType:Uint16` / `AlkType:Uint32`:**
- Value must be a non-negative integer within the type's range.
- Uint8: 0..255, Uint16: 0..65535, Uint32: 0..4294967295.
### String and binary validators
**`TypeDef:String`:**
**`AlkType:String`:**
- Value must be a valid UTF-8 string.
- If `maxLength` is specified in the schema, the string's byte length
must not exceed it.
**`TypeDef:Bytes`:**
**`AlkType:Bytes`:**
- Value must be a string (JSON represents binary data as a string — JSON
has no native byte type).
- If `maxLength` is specified, the byte length must not exceed it.
@@ -135,34 +135,34 @@ type constraints; `jsonschema` handles all structural validation.
validation) uses a string; the binary representation (for data access)
uses `&[u8]` directly.
**`TypeDef:Enum`:**
- The `TypeDef:Enum` custom keyword signals that the type is an enum for
**`AlkType:Enum`:**
- The `AlkType:Enum` custom keyword signals that the type is an enum for
*layout* purposes (the engine needs to know it's a fixed-size u32 index,
not a variable-length string). The built-in `enum` keyword provides the
value list and handles value-membership validation. The custom keyword
validator is a no-op beyond the built-in check — it exists solely for
the layout engine to recognize the type.
**`TypeDef:Timestamp`:**
**`AlkType:Timestamp`:**
- Value must be a valid RFC 3339 timestamp string (the internet profile
of ISO 8601, e.g., `"2026-07-20T15:30:00Z"`).
### Composite type validators
**`TypeDef:Struct`:**
**`AlkType:Struct`:**
- Value must be an object.
- Each property must match its declared `TypeDef:*` kind.
- Each property must match its declared `AlkType:*` kind.
- Required fields must be present.
- The `jsonschema` crate's built-in `properties` and `required` keywords
handle the structural checks — the custom keyword only needs to
validate that each field's value matches its `TypeDef:*` kind.
validate that each field's value matches its `AlkType:*` kind.
**`TypeDef:Union`:**
**`AlkType:Union`:**
- The discriminator value must be one of the mapping keys.
- The variant struct must match the declared schema for that discriminator
value.
**`TypeDef:Array`:**
**`AlkType:Array`:**
- Value must be an array.
- Each element must match the array's declared element type.
- If `minItems`/`maxItems` is specified, the array length must be within
@@ -170,19 +170,19 @@ type constraints; `jsonschema` handles all structural validation.
### Other validators
**`TypeDef:Boolean`:**
**`AlkType:Boolean`:**
- Value must be `true` or `false`.
**`TypeDef:Record`:**
**`AlkType:Record`:**
- Value must be an object.
- All values must match the record's declared value type (specified via
the `"values"` property in the schema, e.g.,
`"values": { "TypeDef:Float32": true }`).
`"values": { "AlkType:Float32": true }`).
### Validator implementation pattern
Each custom keyword implementation is ~10 lines. Example for
`TypeDef:Float32`:
`AlkType:Float32`:
```rust
struct Float32Validator;
@@ -204,7 +204,7 @@ Registration:
```rust
let validator = jsonschema::options()
.with_keyword("TypeDef:Float32", |parent, value, path| {
.with_keyword("AlkType:Float32", |parent, value, path| {
Ok(Box::new(Float32Validator))
})
.build(&schema)?;
@@ -212,18 +212,18 @@ let validator = jsonschema::options()
The factory closure receives the parent schema object, the keyword's
value, and the schema path. This enables cross-keyword awareness — for
example, a `TypeDef:Struct` validator can inspect the parent's
`properties` to validate each field against its declared `TypeDef:*` kind.
example, a `AlkType:Struct` validator can inspect the parent's
`properties` to validate each field against its declared `AlkType:*` kind.
## TypedefError
## AlkTypeError
A single `TypedefError` enum covers all error conditions across the
A single `AlkTypeError` enum covers all error conditions across the
engine's three phases (schema parsing, offset computation, read/write)
plus validation. Decided in [ADR-004](decisions/004-error-handling-validation-strategy.md).
```rust
pub enum TypedefError {
/// Schema parsing errors (invalid JSON, missing keywords, unknown TypeDef kinds).
pub enum AlkTypeError {
/// Schema parsing errors (invalid JSON, missing keywords, unknown AlkType kinds).
Schema(String),
/// Offset computation errors (field not found, unsupported type).
Offset { field_path: String, reason: String },
@@ -234,8 +234,8 @@ pub enum TypedefError {
}
```
- **`Schema`** — for errors during `TypedefEngine::compile()`. Invalid
JSON, missing required keywords, unknown `TypeDef:*` kinds.
- **`Schema`** — for errors during `AlkTypeEngine::compile()`. Invalid
JSON, missing required keywords, unknown `AlkType:*` kinds.
- **`Offset`** — for errors during offset computation. Field not found
in the schema, type not supported for offset computation, recursive
depth exceeded. Carries the field path.
@@ -244,14 +244,14 @@ pub enum TypedefError {
Carries the field path.
- **`Validation`** — wraps `jsonschema`'s `ValidationError`. The
`'static` lifetime is correct — the validator owns its schema reference
and lives for the lifetime of the `TypedefEngine`.
and lives for the lifetime of the `AlkTypeEngine`.
### Field-path-carrying errors
Read/write and offset errors include the field path for debugging:
```rust
Err(TypedefError::Access {
Err(AlkTypeError::Access {
field_path: "header.version".to_string(),
reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(),
})
@@ -262,7 +262,7 @@ you exactly which field failed and why.
## Validation Timing
### Load time: `TypedefEngine::compile()`
### Load time: `AlkTypeEngine::compile()`
The expensive work happens once at schema load time:
1. Normalize `$ref` values in the schema (`normalize_refs`).
@@ -270,7 +270,7 @@ The expensive work happens once at schema load time:
3. Compute the layout (`LayoutBuilder`/`SequentialReader` for packed, `OffsetMap` for aligned).
4. Build the jsonschema validator (`jsonschema::options().with_keyword(...).build(&schema)?`).
The result is a `TypedefEngine` that can be used for repeated operations.
The result is a `AlkTypeEngine` that can be used for repeated operations.
### Access time: `engine.validate_json(&Value)` / `engine.is_valid_json(&Value)`
@@ -281,7 +281,7 @@ validator is already compiled — these are fast checks against the
compiled validator.
```rust
pub fn validate_json(&self, instance: &Value) -> Result<(), TypedefError>;
pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>;
pub fn is_valid_json(&self, instance: &Value) -> bool;
```
@@ -314,12 +314,12 @@ representation first, then access the binary buffer.
| Decision | ADR | Summary |
|----------|-----|---------|
| Error handling and validation | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Error handling and validation | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Purpose and scope | [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine |
## Open Questions
None specific to validation. The three typedef OQs (OQ-001, OQ-002,
None specific to validation. The three alktype OQs (OQ-001, OQ-002,
OQ-003) are about layout, platform support, and schema construction —
not validation.
@@ -329,7 +329,7 @@ not validation.
custom keyword validators for all 17 kinds
- [ADR-004](decisions/004-error-handling-validation-strategy.md) —
error handling and validation strategy
- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds that the
- [schema-layer.md](schema-layer.md) — the 17 AlkType kinds that the
validators check
- [data-access.md](data-access.md) — read/write functions that operate
on the same buffers