alktype/README.md

# alktype

The binary struct engine: a small Rust crate that takes a BAST (Binary
Abstract Syntax Tree) document and produces an offset map, read/write
functions, and validation — all driven by the schema. The schema is the
format definition; the engine is generic.

`alktype` is a standalone crate with **two dependencies**: `jsonschema`
(for JSON validation and BAST meta-schema validation) and `serde_json`
(for BAST document parsing). No tokio, no platform deps, no `unsafe`.
Compiles to `wasm32-unknown-unknown`.

## What it is

BAST is a JSON document that describes binary data layouts using a
`kind`-based vocabulary with `$defs`/`$ref` for composition. BAST is
itself a valid JSON Schema instance (it has a meta-schema), making it
self-validating, editor-friendly, and trivially consumable from any
language with a JSON parser. See
[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md)
for the normative format spec.

A BAST document serves three roles simultaneously:

| Role | Mechanism | When |
|------|-----------|------|
| **Validation spec (bytes)** | BAST-native validator (recursive walker over the BAST type tree) | Access time (`validate_bytes`) |
| **Validation spec (JSON)** | Standard `jsonschema::Validator` from a consumer-provided JSON Schema | Load time (build validator), access time (`validate_json`) |
| **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map / packed layout) |
| **Data access** | Read/write at computed offsets | Access time (read field, write field) |

No separate format definition, no separate parser, no separate
validator. The BAST document is the single source of truth for the
binary format. Adding a new field to a protocol is adding an entry to
the BAST `fields` array — the engine computes the new offsets
automatically.

This is the same principle as `#[repr(C)]` struct field access, but at
runtime from a portable JSON document instead of at compile time from
language-specific annotations. The BAST document is the ABI contract.

## Usage

Build the BAST document with the fluent Rust builder (ADR-009), compile
it once into an [`AlkTypeEngine`], then read/write fields at computed
offsets:

```rust
use alktype::{AlkTypeEngine, Definitions, Endian, LayoutMode, Schema, FieldValue};

// Channels' 8-byte chunk header: big-endian, packed mode.
let doc = Definitions::new().build_doc("ChunkHeader", Schema::struct_()
    .endian(Endian::Big)
    .field("channel_id", Schema::uint32())
    .field("length",     Schema::uint32()));

// `json_schema: None` — no JSON-validation path needed for a binary-only schema.
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;

// Write a frame into a buffer. For fixed-size structs, the byte
// positions are a direct read off the layout — channel_id at 0,
// length at 4. (For variable-length fields, use LayoutBuilder to
// compute positions from known data sizes.)
let mut buf = vec![0u8; 8];
alktype::data_access::write_u32(&mut buf, 0, 42, "channel_id", Endian::Big)?;
alktype::data_access::write_u32(&mut buf, 4,  7, "length",     Endian::Big)?;

// Validate the bytes against the BAST document in one call.
engine.validate_bytes(&buf)?; // materializes a Value, then runs the BAST-native validator

// Read the frame back sequentially (packed mode is sequential by
// construction — variable-length fields shift subsequent fields).
let mut reader = engine.sequential_reader().expect("packed mode");
let (name, value) = reader.read_next(&buf)?.expect("first field");
assert_eq!(name, "channel_id");
assert_eq!(value, FieldValue::U32(42));
# Ok::<(), alktype::AlkTypeError>(())
```

BAST documents may also be authored as plain `serde_json::json!{...}`
literals and passed directly to `AlkTypeEngine::compile` — the builder
is a construction convenience, not a requirement.

```rust
use alktype::{AlkTypeEngine, LayoutMode};
use serde_json::json;

let doc = json!({
    "$defs": {
        "ChunkHeader": {
            "kind": "struct",
            "endian": "big",
            "fields": [
                { "name": "channel_id", "kind": "uint32" },
                { "name": "length",     "kind": "uint32" }
            ]
        }
    }
});
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;
# Ok::<(), alktype::AlkTypeError>(())
```

## The 19 BAST kinds

| `kind` | Rust type | Size | Notes |
|--------|-----------|-----:|-------|
| `int8` | `i8` | 1 | |
| `int16` | `i16` | 2 | endian-sensitive |
| `int32` | `i32` | 4 | endian-sensitive |
| `int64` | `i64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `uint8` | `u8` | 1 | |
| `uint16` | `u16` | 2 | endian-sensitive |
| `uint32` | `u32` | 4 | endian-sensitive; also the enum/string/bytes length-prefix width |
| `uint64` | `u64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `float32` | `f32` | 4 | endian-sensitive; NaN/inf rejected by validator |
| `float64` | `f64` | 8 | endian-sensitive; NaN/inf rejected by validator |
| `bool` | `bool` | 1 | `0x00`=false, `0x01`=true |
| `enum` | `u32` index | 4 | index into the `values` array; bounds-checked by the BAST-native validator |
| `string` | length-prefixed UTF-8 | 4 + N | `[length: u32][bytes]` by default |
| `bytes` | length-prefixed raw bytes | 4 + N | `[length: u32][bytes]` by default |
| `timestamp` | length-prefixed RFC 3339 | 4 + N | non-strict string check (see inline docs) |
| `struct` | record of fields | composite | nested; field paths are dotted (`"header.version"`) |
| `union` | tagged union | composite | byte-offset or field-name discriminator |
| `array` | repeated element | composite | fixed-size elements with stride; `count` required in v1 (D-BAST-004) |
| `record` | string-keyed map | composite | `[count: u32][key, value]...` |

The 19 kinds map to the `AlkTypeKind` Rust enum. `AlkTypeKind::to_bast_str`/
`from_bast_str` convert between the enum and the lowercase BAST strings
(D-BAST-002). Only `struct`, `union`, and `enum` can appear as named
`$defs` entries; primitives, arrays, and records appear as field/element/
value types via [TypeRef](docs/architecture/bast-format.md#typeref).

## Two layout modes

The consumer selects the layout mode at engine construction time via
`AlkTypeEngine::compile(bast_doc, root_name, mode, json_schema)`. The
same BAST document can be compiled in either mode. Decided in ADR-002.

| Mode | Use case | Read API | Write API |
|------|----------|----------|-----------|
| **Packed** (`LayoutMode::Packed`) | Protocol wire formats (SFTP, channels, TTY) — fields packed with no alignment padding; variable-length fields shift subsequent fields | [`SequentialReader`] (walks fields in order) | [`LayoutBuilder`] (computes positions from known data sizes) |
| **Aligned** (`LayoutMode::Aligned`) | mmap-friendly formats (metatensor, safetensors) — fixed positions with natural alignment padding; variable-length data lives outside the static layout | [`OffsetMap`] (random access by field path) | `OffsetMap` (write at known offsets) |

### Variable-length handling

- **Packed mode**: `[length: u32][data]` inline by default. The
  `LayoutBuilder` takes actual data sizes to compute positions; the
  `SequentialReader` reads the length prefix to find the data extent.
- **Aligned mode**: a 4-byte length prefix sits at a known offset; the
  variable data is not part of the static layout. Offset indirection
  (the metatensor blob pattern: `{offset, length}` pointing into a
  separate data region) is opt-in via the field-level `encoding`
  annotation. Fixed-size reservation via `maxLength` is also supported.

### Union discriminators

`kind: "union"` supports two discriminator kinds (ADR-003):

- **Byte-offset** — a fixed-size integer (`uint8`/`uint16`/`uint32`) at
  a known byte offset. The SFTP `Packet` pattern: byte 0 is the type
  byte, bytes 1..N are the variant struct. Mapping keys are stringified
  integers.
- **Field-name** — a named field within the union. The TypeBox
  `typedef.ts` pattern. Mapping keys are string values matching the
  discriminator field's value. The `fields` array declares the
  discriminator field (D-BAST-005).

Variant `$ref`s are resolved lazily — no compile-time inlining step.

## Endianness

Per-schema, default little-endian. Set `"endian": "big"` on the root
struct (or via `Schema::endian(Endian::Big)`) and the engine byte-swaps
every multi-byte read/write accordingly. Field-level `endian` overrides
the struct default. SFTP consumers specify big-endian; channels' chunk
header is big-endian.

## Validation

Two entry points on [`AlkTypeEngine`], two validators for two input
types (ADR-VAL-SPLIT):

- `validate_bytes(&[u8])` — for raw byte buffers (channels' chunk
  header, SFTP packets). Materializes a `Value` tree from the bytes via
  the layout engine, then runs the **BAST-native validator** — a
  recursive walker over the BAST type tree that checks the value-domain
  constraints the materializer doesn't (integer ranges, `maxLength`,
  timestamp shape, enum index bounds, union variant constraints). No
  `jsonschema` involvement; the BAST document is the complete
  validation spec for bytes (D-BAST-006).
- `validate_json(&Value)` / `is_valid_json(&Value)` — for already-parsed
  JSON (call's `OperationSpec.input_schema` payloads). Validates
  against a **standard `jsonschema::Validator`** compiled at
  `AlkTypeEngine::compile` time from a consumer-provided JSON Schema
  (the `json_schema: Option<&Value>` parameter). BAST is not involved —
  BAST describes bytes, not JSON shape (D-BAST-007).

Both paths return `AlkTypeError::Validation(jsonschema::ValidationError<'static>)`
— one uniform payload, one match arm (D-BAST-009).

Validation is opt-in per operation. High-throughput paths can skip it;
security-sensitive paths can validate every frame. The BAST-native
validator also fixes a v0.1.0 dead constraint: enum index bounds are
now checked (the materializer emits a numeric index; the validator
checks it against `values.len()`).

## BAST document shape

Every BAST document has the same top-level shape:

```json
{ "$defs": { "<TypeName>": { ...TypeDef... }, ... } }
```

- The `$defs` block is **required** (D-BAST-003).
- The **root type name** is a required parameter to
  `AlkTypeEngine::compile(bast_doc, root_name, mode, ...)` (D-BAST-001)
  — it selects which `$defs` entry is the top-level type.
- `$ref` is restricted to `#/$defs/<name>` — one hash lookup, no
  normalization pass.

The BAST meta-schema is embedded in the crate as `BAST_META_SCHEMA`
(re-exported from the crate root) and published at
`https://alk.dev/bast/v1/schema`. Consumers can validate a BAST
document's structure with any JSON Schema validator. See
[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md)
for the full spec.

## Crate independence

`alktype` does **not** depend on any application or networking crate.
It defines its own types (`AlkTypeError`, `AlkTypeEngine`, `FieldValue`,
etc.) and is usable in contexts where networking doesn't exist — CLI
tools, test harnesses, schema-building utilities, and WASM targets. The
upcoming `alkcall` crate (the `alknet-call` + `alknet-channels`
unification) depends on `alktype` for both binary layout and JSON
payload schemas; `alktype` knows nothing about `alkcall`.

## Schemas as untrusted input

The crate treats BAST documents as untrusted input. A malformed
document returns `AlkTypeError::Schema` from any engine path — never a
panic. This matters for hub/spoke topologies where the remote peer
provides the schema (e.g. `alkcall` accepting an `OperationSpec` from
an arbitrary internet peer). Every `unreachable!()` site in production
code was converted to `Err` ahead of v0.1.0 (review #002, L2); the
BAST parser preserves this invariant — overflow-safe arithmetic
(`checked_add`, `usize::try_from`) on all offset/count casts.

## Documentation

Architecture documentation lives under [`docs/architecture/`](docs/architecture/):

- [Overview](docs/architecture/overview.md) — purpose, "schema is the
  format" principle, dependencies, consumers, scope boundaries
- [BAST format](docs/architecture/bast-format.md) — **normative format
  spec**: meta-schema, TypeRef, TypeDef shapes, validation model
- [Schema layer](docs/architecture/schema-layer.md) — the BAST parser
  (`BastDoc`/`BastDef`/`BastType` typed tree), the 19 kinds, the
  `AlkTypeKind` enum
- [Layout engine](docs/architecture/layout-engine.md) — offset
  computation, the two layout modes, alignment, endianness
- [Data access](docs/architecture/data-access.md) — read/write
  functions, TUnion dispatch, field paths, zero-copy access
- [Validation](docs/architecture/validation.md) — the two-validator
  model, `AlkTypeError`, load-time vs access-time validation
- [Builder](docs/architecture/builder.md) — fluent Rust API for
  constructing BAST documents and standard JSON Schemas at runtime
- [Architecture decisions (ADRs)](docs/architecture/decisions/) —
  purpose/scope (ADR-001), BAST format (ADR-BAST), two-validator model
  (ADR-VAL-SPLIT), two layout modes (ADR-002), schema annotations
  (ADR-003), error handling (ADR-004), int64/uint64 kinds (ADR-005),
  non-final inline variable fields (ADR-006), packed-mode read factory
  (ADR-007), TUnion in aligned mode (ADR-008), builder API (ADR-009),
  `validate_bytes` (ADR-010)

## License

Licensed under either of

- Apache License, Version 2.0 ([LICENSE-APACHE](LICENSE-APACHE) or http://www.apache.org/licenses/LICENSE-2.0)
- MIT license ([LICENSE-MIT](LICENSE-MIT) or http://opensource.org/licenses/MIT)

at your option.

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this work by you, as defined in the Apache-2.0 license, shall be dual licensed as above, without any additional terms or conditions.