alkdev/alktype
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Rewrite README for BAST pivot

Browse Source 
The README still described the v0.1.0 custom-keyword JSON Schema format
(`AlkType:*` kinds, `compile(&mut schema, mode)`, single jsonschema
validator). Rewrite it for the BAST-era shipped API:

- What-it-is table: two validation specs (bytes via BAST-native, JSON
  via consumer-provided JSON Schema) instead of one.
- Usage example: `Definitions::new().build_doc(ChunkHeader, ...)` +
  `AlkTypeEngine::compile(&doc, ChunkHeader, LayoutMode::Packed,
  None)`; added a json!-literal variant showing the BAST document
  shape.
- The 19 kinds table: lowercase BAST `kind` strings instead of
  `AlkType:*` keywords; note the enum index bounds fix.
- Two layout modes: updated `compile` signature.
- Variable-length handling: BAST field-level `encoding` annotation.
- Union discriminators: `kind: union`, lazy variant $ref
  resolution, D-BAST-005 fields requirement.
- Endianness: struct-level (was 'top-level schema').
- Validation: two validators (ADR-VAL-SPLIT), BAST-native for bytes,
  standard jsonschema for JSON; uniform AlkTypeError::Validation
  payload (D-BAST-009); enum bounds fix noted.
- New 'BAST document shape' section: $defs required, root_name
  parameter, $ref restricted to #/$defs/<name>, BAST_META_SCHEMA
  re-export.
- Untrusted input: BAST document (was 'schema'); BAST parser
  preserves the no-panic invariant.
- Documentation index: updated for the new/rewritten docs and the
  ADR-BAST / ADR-VAL-SPLIT additions.

Verified both code examples compile and pass against the shipped API
(via a throwaway integration test, since removed).
This commit is contained in:
glm-5.2 committed 2026-08-15 14:13:34 +00:00
1 parent 62270b03ca
commit 562284faf4
1 file changed
+168 -87
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+168 -87
View File
@@ -1,51 +1,61 @@
# alktype
The binary struct engine: a small Rust crate that takes a JSON Schema
with `AlkType:*` custom keywords and produces an offset map, read/write
The binary struct engine: a small Rust crate that takes a BAST (Binary
Abstract Syntax Tree) document and produces an offset map, read/write
functions, and validation — all driven by the schema. The schema is the
format definition; the engine is generic.
`alktype` is a standalone crate with **two dependencies**: `jsonschema`
(for validation) and `serde_json` (for schema parsing). No tokio, no
platform deps, no `unsafe`. Compiles to `wasm32-unknown-unknown`.
(for JSON validation and BAST meta-schema validation) and `serde_json`
(for BAST document parsing). No tokio, no platform deps, no `unsafe`.
Compiles to `wasm32-unknown-unknown`.
## What it is
A JSON Schema annotated with `AlkType:*` custom keywords serves three
roles simultaneously:
BAST is a JSON document that describes binary data layouts using a
`kind`-based vocabulary with `$defs`/`$ref` for composition. BAST is
itself a valid JSON Schema instance (it has a meta-schema), making it
self-validating, editor-friendly, and trivially consumable from any
language with a JSON parser. See
[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md)
for the normative format spec.
A BAST document serves three roles simultaneously:
| Role | Mechanism | When |
|------|-----------|------|
| **Validation spec** | `jsonschema` custom keywords | Load time (build validator), access time (validate buffer) |
| **Validation spec (bytes)** | BAST-native validator (recursive walker over the BAST type tree) | Access time (`validate_bytes`) |
| **Validation spec (JSON)** | Standard `jsonschema::Validator` from a consumer-provided JSON Schema | Load time (build validator), access time (`validate_json`) |
| **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map / packed layout) |
| **Data access** | Read/write at computed offsets | Access time (read field, write field) |
No separate format definition, no separate parser, no separate
validator. The schema is the single source of truth for the binary
format. Adding a new field to a protocol is adding a property to the
schema JSON — the engine computes the new offsets automatically.
validator. The BAST document is the single source of truth for the
binary format. Adding a new field to a protocol is adding an entry to
the BAST `fields` array — the engine computes the new offsets
automatically.
This is the same principle as `#[repr(C)]` struct field access, but at
runtime from a portable JSON Schema instead of at compile time from
language-specific annotations. The schema is the ABI contract.
runtime from a portable JSON document instead of at compile time from
language-specific annotations. The BAST document is the ABI contract.
## Usage
Build the schema with the fluent Rust builder (ADR-009), compile it
once into an [`AlkTypeEngine`], then read/write fields at computed
Build the BAST document with the fluent Rust builder (ADR-009), compile
it once into an [`AlkTypeEngine`], then read/write fields at computed
offsets:
```rust
use alktype::{AlkTypeEngine, Endian, LayoutMode, Schema, FieldValue};
use alktype::{AlkTypeEngine, Definitions, Endian, LayoutMode, Schema, FieldValue};
// Channels' 8-byte chunk header: big-endian, packed mode.
let mut schema = Schema::struct_()
let doc = Definitions::new().build_doc("ChunkHeader", Schema::struct_()
.endian(Endian::Big)
.field("channel_id", Schema::uint32())
.field("length", Schema::uint32())
.build();
.field("length", Schema::uint32()));
let engine = AlkTypeEngine::compile(&mut schema, LayoutMode::Packed)?;
// `json_schema: None` — no JSON-validation path needed for a binary-only schema.
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;
// Write a frame into a buffer. For fixed-size structs, the byte
// positions are a direct read off the layout — channel_id at 0,
@@ -55,8 +65,8 @@ let mut buf = vec![0u8; 8];
alktype::data_access::write_u32(&mut buf, 0, 42, "channel_id", Endian::Big)?;
alktype::data_access::write_u32(&mut buf, 4, 7, "length", Endian::Big)?;
// Validate the bytes against the schema in one call.
engine.validate_bytes(&buf)?; // materializes a Value, then validates
// Validate the bytes against the BAST document in one call.
engine.validate_bytes(&buf)?; // materializes a Value, then runs the BAST-native validator
// Read the frame back sequentially (packed mode is sequential by
// construction — variable-length fields shift subsequent fields).
@@ -67,43 +77,65 @@ assert_eq!(value, FieldValue::U32(42));
# Ok::<(), alktype::AlkTypeError>(())
```
Schemas may also be authored as plain `serde_json::json!{...}` literals
and passed directly to `AlkTypeEngine::compile` — the builder is a
construction convenience, not a requirement.
BAST documents may also be authored as plain `serde_json::json!{...}`
literals and passed directly to `AlkTypeEngine::compile` — the builder
is a construction convenience, not a requirement.
## The 19 `AlkType:*` kinds
```rust
use alktype::{AlkTypeEngine, LayoutMode};
use serde_json::json;
| Kind | Rust type | Size | Notes |
|------|-----------|-----:|-------|
| `AlkType:Int8` | `i8` | 1 | |
| `AlkType:Int16` | `i16` | 2 | endian-sensitive |
| `AlkType:Int32` | `i32` | 4 | endian-sensitive |
| `AlkType:Int64` | `i64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `AlkType:Uint8` | `u8` | 1 | |
| `AlkType:Uint16` | `u16` | 2 | endian-sensitive |
| `AlkType:Uint32` | `u32` | 4 | endian-sensitive; also the enum/string/bytes length-prefix width |
| `AlkType:Uint64` | `u64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `AlkType:Float32` | `f32` | 4 | endian-sensitive; NaN/inf rejected by validator |
| `AlkType:Float64` | `f64` | 8 | endian-sensitive; NaN/inf rejected by validator |
| `AlkType:Boolean` | `bool` | 1 | |
| `AlkType:Enum` | `u32` index | 4 | index into the schema's `"enum"` array |
| `AlkType:String` | length-prefixed UTF-8 | 4 + N | `[length: u32][bytes]` by default |
| `AlkType:Bytes` | length-prefixed raw bytes | 4 + N | `[length: u32][bytes]` by default |
| `AlkType:Timestamp` | length-prefixed RFC 3339 | 4 + N | non-strict string check (see inline docs) |
| `AlkType:Struct` | record of fields | composite | nested; field paths are dotted (`"header.version"`) |
| `AlkType:Union` | tagged union | composite | byte-offset or field-name discriminator |
| `AlkType:Array` | repeated element | composite | fixed-size elements with stride, or variable count |
| `AlkType:Record` | string-keyed map | composite | `[count: u32][key, value]...` |
let doc = json!({
"$defs": {
"ChunkHeader": {
"kind": "struct",
"endian": "big",
"fields": [
{ "name": "channel_id", "kind": "uint32" },
{ "name": "length", "kind": "uint32" }
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?;
# Ok::<(), alktype::AlkTypeError>(())
```
The engine recognizes a kind when the schema object has a key starting
with `AlkType:` whose value is `true` (the boolean shorthand) or an
annotation object (e.g. `{ "AlkType:String": { "encoding": "offset-indirect" } }`).
## The 19 BAST kinds
| `kind` | Rust type | Size | Notes |
|--------|-----------|-----:|-------|
| `int8` | `i8` | 1 | |
| `int16` | `i16` | 2 | endian-sensitive |
| `int32` | `i32` | 4 | endian-sensitive |
| `int64` | `i64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `uint8` | `u8` | 1 | |
| `uint16` | `u16` | 2 | endian-sensitive |
| `uint32` | `u32` | 4 | endian-sensitive; also the enum/string/bytes length-prefix width |
| `uint64` | `u64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) |
| `float32` | `f32` | 4 | endian-sensitive; NaN/inf rejected by validator |
| `float64` | `f64` | 8 | endian-sensitive; NaN/inf rejected by validator |
| `bool` | `bool` | 1 | `0x00`=false, `0x01`=true |
| `enum` | `u32` index | 4 | index into the `values` array; bounds-checked by the BAST-native validator |
| `string` | length-prefixed UTF-8 | 4 + N | `[length: u32][bytes]` by default |
| `bytes` | length-prefixed raw bytes | 4 + N | `[length: u32][bytes]` by default |
| `timestamp` | length-prefixed RFC 3339 | 4 + N | non-strict string check (see inline docs) |
| `struct` | record of fields | composite | nested; field paths are dotted (`"header.version"`) |
| `union` | tagged union | composite | byte-offset or field-name discriminator |
| `array` | repeated element | composite | fixed-size elements with stride; `count` required in v1 (D-BAST-004) |
| `record` | string-keyed map | composite | `[count: u32][key, value]...` |
The 19 kinds map to the `AlkTypeKind` Rust enum. `AlkTypeKind::to_bast_str`/
`from_bast_str` convert between the enum and the lowercase BAST strings
(D-BAST-002). Only `struct`, `union`, and `enum` can appear as named
`$defs` entries; primitives, arrays, and records appear as field/element/
value types via [TypeRef](docs/architecture/bast-format.md#typeref).
## Two layout modes
The consumer selects the layout mode at engine construction time via
`AlkTypeEngine::compile(schema, mode)`. The same schema can be compiled
in either mode. Decided in ADR-002.
`AlkTypeEngine::compile(bast_doc, root_name, mode, json_schema)`. The
same BAST document can be compiled in either mode. Decided in ADR-002.
| Mode | Use case | Read API | Write API |
|------|----------|----------|-----------|
@@ -118,40 +150,82 @@ in either mode. Decided in ADR-002.
- **Aligned mode**: a 4-byte length prefix sits at a known offset; the
variable data is not part of the static layout. Offset indirection
(the metatensor blob pattern: `{offset, length}` pointing into a
separate data region) is opt-in via the `encoding` annotation.
separate data region) is opt-in via the field-level `encoding`
annotation. Fixed-size reservation via `maxLength` is also supported.
### TUnion discriminators
### Union discriminators
`AlkType:Union` supports two discriminator kinds (ADR-003):
`kind: "union"` supports two discriminator kinds (ADR-003):
- **Byte-offset** — a fixed-size integer at a known byte offset. The
SFTP `Packet` pattern: byte 0 is the type byte, bytes 1..N are the
variant struct. Mapping keys are stringified integers.
- **Field-name** — a named field within the struct. The TypeBox
- **Byte-offset** — a fixed-size integer (`uint8`/`uint16`/`uint32`) at
a known byte offset. The SFTP `Packet` pattern: byte 0 is the type
byte, bytes 1..N are the variant struct. Mapping keys are stringified
integers.
- **Field-name** — a named field within the union. The TypeBox
`typedef.ts` pattern. Mapping keys are string values matching the
discriminator field's value.
discriminator field's value. The `fields` array declares the
discriminator field (D-BAST-005).
Variant `$ref`s are resolved lazily — no compile-time inlining step.
## Endianness
Per-schema, default little-endian. Set `"endian": "big"` on the
top-level schema (or via `Schema::endian(Endian::Big)`) and the engine
byte-swaps every multi-byte read/write accordingly. SFTP consumers
specify big-endian; channels' chunk header is big-endian.
Per-schema, default little-endian. Set `"endian": "big"` on the root
struct (or via `Schema::endian(Endian::Big)`) and the engine byte-swaps
every multi-byte read/write accordingly. Field-level `endian` overrides
the struct default. SFTP consumers specify big-endian; channels' chunk
header is big-endian.
## Validation
Two entry points on [`AlkTypeEngine`], one underlying `jsonschema`
validator (ADR-010):
Two entry points on [`AlkTypeEngine`], two validators for two input
types (ADR-VAL-SPLIT):
- `validate_bytes(&[u8])` — for raw byte buffers (channels' chunk
header, SFTP packets). Materializes a `Value` tree from the bytes via
the layout engine, then runs the **BAST-native validator** — a
recursive walker over the BAST type tree that checks the value-domain
constraints the materializer doesn't (integer ranges, `maxLength`,
timestamp shape, enum index bounds, union variant constraints). No
`jsonschema` involvement; the BAST document is the complete
validation spec for bytes (D-BAST-006).
- `validate_json(&Value)` / `is_valid_json(&Value)` — for already-parsed
JSON (call's payload schemas).
- `validate_bytes(&[u8])` — materializes a `Value` tree from the bytes
via the layout engine, then validates that `Value`. Single-call binary
buffer validation.
JSON (call's `OperationSpec.input_schema` payloads). Validates
against a **standard `jsonschema::Validator`** compiled at
`AlkTypeEngine::compile` time from a consumer-provided JSON Schema
(the `json_schema: Option<&Value>` parameter). BAST is not involved —
BAST describes bytes, not JSON shape (D-BAST-007).
The validator is compiled once at load time; access-time validation is
a fast `is_valid()` check. High-throughput paths can skip validation;
security-sensitive paths can validate every frame.
Both paths return `AlkTypeError::Validation(jsonschema::ValidationError<'static>)`
— one uniform payload, one match arm (D-BAST-009).
Validation is opt-in per operation. High-throughput paths can skip it;
security-sensitive paths can validate every frame. The BAST-native
validator also fixes a v0.1.0 dead constraint: enum index bounds are
now checked (the materializer emits a numeric index; the validator
checks it against `values.len()`).
## BAST document shape
Every BAST document has the same top-level shape:
```json
{ "$defs": { "<TypeName>": { ...TypeDef... }, ... } }
```
- The `$defs` block is **required** (D-BAST-003).
- The **root type name** is a required parameter to
`AlkTypeEngine::compile(bast_doc, root_name, mode, ...)` (D-BAST-001)
— it selects which `$defs` entry is the top-level type.
- `$ref` is restricted to `#/$defs/<name>` — one hash lookup, no
normalization pass.
The BAST meta-schema is embedded in the crate as `BAST_META_SCHEMA`
(re-exported from the crate root) and published at
`https://alk.dev/bast/v1/schema`. Consumers can validate a BAST
document's structure with any JSON Schema validator. See
[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md)
for the full spec.
## Crate independence
@@ -165,13 +239,14 @@ payload schemas; `alktype` knows nothing about `alkcall`.
## Schemas as untrusted input
The crate treats schemas as untrusted input. A malformed schema
returns `AlkTypeError::Schema` / `AlkTypeError::Offset` from any
engine path — never a panic. This matters for hub/spoke topologies
where the remote peer provides the schema (e.g. `alkcall` accepting an
`OperationSpec` from an arbitrary internet peer). All `unreachable!()`
sites in production code were converted to `Err` ahead of v0.1.0
(review #002, L2).
The crate treats BAST documents as untrusted input. A malformed
document returns `AlkTypeError::Schema` from any engine path — never a
panic. This matters for hub/spoke topologies where the remote peer
provides the schema (e.g. `alkcall` accepting an `OperationSpec` from
an arbitrary internet peer). Every `unreachable!()` site in production
code was converted to `Err` ahead of v0.1.0 (review #002, L2); the
BAST parser preserves this invariant — overflow-safe arithmetic
(`checked_add`, `usize::try_from`) on all offset/count casts.
## Documentation
@@ -179,20 +254,26 @@ Architecture documentation lives under [`docs/architecture/`](docs/architecture/
- [Overview](docs/architecture/overview.md) — purpose, "schema is the
format" principle, dependencies, consumers, scope boundaries
- [Schema layer](docs/architecture/schema-layer.md) — the 19 kinds,
jsonschema custom keyword integration, schema annotations
- [BAST format](docs/architecture/bast-format.md) — **normative format
spec**: meta-schema, TypeRef, TypeDef shapes, validation model
- [Schema layer](docs/architecture/schema-layer.md) — the BAST parser
(`BastDoc`/`BastDef`/`BastType` typed tree), the 19 kinds, the
`AlkTypeKind` enum
- [Layout engine](docs/architecture/layout-engine.md) — offset
computation, the two layout modes, alignment, endianness
- [Data access](docs/architecture/data-access.md) — read/write
functions, TUnion dispatch, field paths, zero-copy access
- [Validation](docs/architecture/validation.md) — custom keyword
validators, `AlkTypeError`, load-time vs access-time validation
- [Validation](docs/architecture/validation.md) — the two-validator
model, `AlkTypeError`, load-time vs access-time validation
- [Builder](docs/architecture/builder.md) — fluent Rust API for
constructing alktype JSON Schemas at runtime
constructing BAST documents and standard JSON Schemas at runtime
- [Architecture decisions (ADRs)](docs/architecture/decisions/) —
purpose/scope, two layout modes, schema annotations, error handling,
int64/uint64 kinds, packed-mode read factory, TUnion in aligned mode,
builder API, `validate_bytes`
purpose/scope (ADR-001), BAST format (ADR-BAST), two-validator model
(ADR-VAL-SPLIT), two layout modes (ADR-002), schema annotations
(ADR-003), error handling (ADR-004), int64/uint64 kinds (ADR-005),
non-final inline variable fields (ADR-006), packed-mode read factory
(ADR-007), TUnion in aligned mode (ADR-008), builder API (ADR-009),
`validate_bytes` (ADR-010)
## License