diff --git a/README.md b/README.md index 11d05c5..50ff9f5 100644 --- a/README.md +++ b/README.md @@ -1,51 +1,61 @@ # alktype -The binary struct engine: a small Rust crate that takes a JSON Schema -with `AlkType:*` custom keywords and produces an offset map, read/write +The binary struct engine: a small Rust crate that takes a BAST (Binary +Abstract Syntax Tree) document and produces an offset map, read/write functions, and validation — all driven by the schema. The schema is the format definition; the engine is generic. `alktype` is a standalone crate with **two dependencies**: `jsonschema` -(for validation) and `serde_json` (for schema parsing). No tokio, no -platform deps, no `unsafe`. Compiles to `wasm32-unknown-unknown`. +(for JSON validation and BAST meta-schema validation) and `serde_json` +(for BAST document parsing). No tokio, no platform deps, no `unsafe`. +Compiles to `wasm32-unknown-unknown`. ## What it is -A JSON Schema annotated with `AlkType:*` custom keywords serves three -roles simultaneously: +BAST is a JSON document that describes binary data layouts using a +`kind`-based vocabulary with `$defs`/`$ref` for composition. BAST is +itself a valid JSON Schema instance (it has a meta-schema), making it +self-validating, editor-friendly, and trivially consumable from any +language with a JSON parser. See +[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md) +for the normative format spec. + +A BAST document serves three roles simultaneously: | Role | Mechanism | When | |------|-----------|------| -| **Validation spec** | `jsonschema` custom keywords | Load time (build validator), access time (validate buffer) | +| **Validation spec (bytes)** | BAST-native validator (recursive walker over the BAST type tree) | Access time (`validate_bytes`) | +| **Validation spec (JSON)** | Standard `jsonschema::Validator` from a consumer-provided JSON Schema | Load time (build validator), access time (`validate_json`) | | **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map / packed layout) | | **Data access** | Read/write at computed offsets | Access time (read field, write field) | No separate format definition, no separate parser, no separate -validator. The schema is the single source of truth for the binary -format. Adding a new field to a protocol is adding a property to the -schema JSON — the engine computes the new offsets automatically. +validator. The BAST document is the single source of truth for the +binary format. Adding a new field to a protocol is adding an entry to +the BAST `fields` array — the engine computes the new offsets +automatically. This is the same principle as `#[repr(C)]` struct field access, but at -runtime from a portable JSON Schema instead of at compile time from -language-specific annotations. The schema is the ABI contract. +runtime from a portable JSON document instead of at compile time from +language-specific annotations. The BAST document is the ABI contract. ## Usage -Build the schema with the fluent Rust builder (ADR-009), compile it -once into an [`AlkTypeEngine`], then read/write fields at computed +Build the BAST document with the fluent Rust builder (ADR-009), compile +it once into an [`AlkTypeEngine`], then read/write fields at computed offsets: ```rust -use alktype::{AlkTypeEngine, Endian, LayoutMode, Schema, FieldValue}; +use alktype::{AlkTypeEngine, Definitions, Endian, LayoutMode, Schema, FieldValue}; // Channels' 8-byte chunk header: big-endian, packed mode. -let mut schema = Schema::struct_() +let doc = Definitions::new().build_doc("ChunkHeader", Schema::struct_() .endian(Endian::Big) .field("channel_id", Schema::uint32()) - .field("length", Schema::uint32()) - .build(); + .field("length", Schema::uint32())); -let engine = AlkTypeEngine::compile(&mut schema, LayoutMode::Packed)?; +// `json_schema: None` — no JSON-validation path needed for a binary-only schema. +let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?; // Write a frame into a buffer. For fixed-size structs, the byte // positions are a direct read off the layout — channel_id at 0, @@ -55,8 +65,8 @@ let mut buf = vec![0u8; 8]; alktype::data_access::write_u32(&mut buf, 0, 42, "channel_id", Endian::Big)?; alktype::data_access::write_u32(&mut buf, 4, 7, "length", Endian::Big)?; -// Validate the bytes against the schema in one call. -engine.validate_bytes(&buf)?; // materializes a Value, then validates +// Validate the bytes against the BAST document in one call. +engine.validate_bytes(&buf)?; // materializes a Value, then runs the BAST-native validator // Read the frame back sequentially (packed mode is sequential by // construction — variable-length fields shift subsequent fields). @@ -67,43 +77,65 @@ assert_eq!(value, FieldValue::U32(42)); # Ok::<(), alktype::AlkTypeError>(()) ``` -Schemas may also be authored as plain `serde_json::json!{...}` literals -and passed directly to `AlkTypeEngine::compile` — the builder is a -construction convenience, not a requirement. +BAST documents may also be authored as plain `serde_json::json!{...}` +literals and passed directly to `AlkTypeEngine::compile` — the builder +is a construction convenience, not a requirement. -## The 19 `AlkType:*` kinds +```rust +use alktype::{AlkTypeEngine, LayoutMode}; +use serde_json::json; -| Kind | Rust type | Size | Notes | -|------|-----------|-----:|-------| -| `AlkType:Int8` | `i8` | 1 | | -| `AlkType:Int16` | `i16` | 2 | endian-sensitive | -| `AlkType:Int32` | `i32` | 4 | endian-sensitive | -| `AlkType:Int64` | `i64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) | -| `AlkType:Uint8` | `u8` | 1 | | -| `AlkType:Uint16` | `u16` | 2 | endian-sensitive | -| `AlkType:Uint32` | `u32` | 4 | endian-sensitive; also the enum/string/bytes length-prefix width | -| `AlkType:Uint64` | `u64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) | -| `AlkType:Float32` | `f32` | 4 | endian-sensitive; NaN/inf rejected by validator | -| `AlkType:Float64` | `f64` | 8 | endian-sensitive; NaN/inf rejected by validator | -| `AlkType:Boolean` | `bool` | 1 | | -| `AlkType:Enum` | `u32` index | 4 | index into the schema's `"enum"` array | -| `AlkType:String` | length-prefixed UTF-8 | 4 + N | `[length: u32][bytes]` by default | -| `AlkType:Bytes` | length-prefixed raw bytes | 4 + N | `[length: u32][bytes]` by default | -| `AlkType:Timestamp` | length-prefixed RFC 3339 | 4 + N | non-strict string check (see inline docs) | -| `AlkType:Struct` | record of fields | composite | nested; field paths are dotted (`"header.version"`) | -| `AlkType:Union` | tagged union | composite | byte-offset or field-name discriminator | -| `AlkType:Array` | repeated element | composite | fixed-size elements with stride, or variable count | -| `AlkType:Record` | string-keyed map | composite | `[count: u32][key, value]...` | +let doc = json!({ + "$defs": { + "ChunkHeader": { + "kind": "struct", + "endian": "big", + "fields": [ + { "name": "channel_id", "kind": "uint32" }, + { "name": "length", "kind": "uint32" } + ] + } + } +}); +let engine = AlkTypeEngine::compile(&doc, "ChunkHeader", LayoutMode::Packed, None)?; +# Ok::<(), alktype::AlkTypeError>(()) +``` -The engine recognizes a kind when the schema object has a key starting -with `AlkType:` whose value is `true` (the boolean shorthand) or an -annotation object (e.g. `{ "AlkType:String": { "encoding": "offset-indirect" } }`). +## The 19 BAST kinds + +| `kind` | Rust type | Size | Notes | +|--------|-----------|-----:|-------| +| `int8` | `i8` | 1 | | +| `int16` | `i16` | 2 | endian-sensitive | +| `int32` | `i32` | 4 | endian-sensitive | +| `int64` | `i64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) | +| `uint8` | `u8` | 1 | | +| `uint16` | `u16` | 2 | endian-sensitive | +| `uint32` | `u32` | 4 | endian-sensitive; also the enum/string/bytes length-prefix width | +| `uint64` | `u64` | 8 | endian-sensitive; JSON precision caveat (ADR-005) | +| `float32` | `f32` | 4 | endian-sensitive; NaN/inf rejected by validator | +| `float64` | `f64` | 8 | endian-sensitive; NaN/inf rejected by validator | +| `bool` | `bool` | 1 | `0x00`=false, `0x01`=true | +| `enum` | `u32` index | 4 | index into the `values` array; bounds-checked by the BAST-native validator | +| `string` | length-prefixed UTF-8 | 4 + N | `[length: u32][bytes]` by default | +| `bytes` | length-prefixed raw bytes | 4 + N | `[length: u32][bytes]` by default | +| `timestamp` | length-prefixed RFC 3339 | 4 + N | non-strict string check (see inline docs) | +| `struct` | record of fields | composite | nested; field paths are dotted (`"header.version"`) | +| `union` | tagged union | composite | byte-offset or field-name discriminator | +| `array` | repeated element | composite | fixed-size elements with stride; `count` required in v1 (D-BAST-004) | +| `record` | string-keyed map | composite | `[count: u32][key, value]...` | + +The 19 kinds map to the `AlkTypeKind` Rust enum. `AlkTypeKind::to_bast_str`/ +`from_bast_str` convert between the enum and the lowercase BAST strings +(D-BAST-002). Only `struct`, `union`, and `enum` can appear as named +`$defs` entries; primitives, arrays, and records appear as field/element/ +value types via [TypeRef](docs/architecture/bast-format.md#typeref). ## Two layout modes The consumer selects the layout mode at engine construction time via -`AlkTypeEngine::compile(schema, mode)`. The same schema can be compiled -in either mode. Decided in ADR-002. +`AlkTypeEngine::compile(bast_doc, root_name, mode, json_schema)`. The +same BAST document can be compiled in either mode. Decided in ADR-002. | Mode | Use case | Read API | Write API | |------|----------|----------|-----------| @@ -118,40 +150,82 @@ in either mode. Decided in ADR-002. - **Aligned mode**: a 4-byte length prefix sits at a known offset; the variable data is not part of the static layout. Offset indirection (the metatensor blob pattern: `{offset, length}` pointing into a - separate data region) is opt-in via the `encoding` annotation. + separate data region) is opt-in via the field-level `encoding` + annotation. Fixed-size reservation via `maxLength` is also supported. -### TUnion discriminators +### Union discriminators -`AlkType:Union` supports two discriminator kinds (ADR-003): +`kind: "union"` supports two discriminator kinds (ADR-003): -- **Byte-offset** — a fixed-size integer at a known byte offset. The - SFTP `Packet` pattern: byte 0 is the type byte, bytes 1..N are the - variant struct. Mapping keys are stringified integers. -- **Field-name** — a named field within the struct. The TypeBox +- **Byte-offset** — a fixed-size integer (`uint8`/`uint16`/`uint32`) at + a known byte offset. The SFTP `Packet` pattern: byte 0 is the type + byte, bytes 1..N are the variant struct. Mapping keys are stringified + integers. +- **Field-name** — a named field within the union. The TypeBox `typedef.ts` pattern. Mapping keys are string values matching the - discriminator field's value. + discriminator field's value. The `fields` array declares the + discriminator field (D-BAST-005). + +Variant `$ref`s are resolved lazily — no compile-time inlining step. ## Endianness -Per-schema, default little-endian. Set `"endian": "big"` on the -top-level schema (or via `Schema::endian(Endian::Big)`) and the engine -byte-swaps every multi-byte read/write accordingly. SFTP consumers -specify big-endian; channels' chunk header is big-endian. +Per-schema, default little-endian. Set `"endian": "big"` on the root +struct (or via `Schema::endian(Endian::Big)`) and the engine byte-swaps +every multi-byte read/write accordingly. Field-level `endian` overrides +the struct default. SFTP consumers specify big-endian; channels' chunk +header is big-endian. ## Validation -Two entry points on [`AlkTypeEngine`], one underlying `jsonschema` -validator (ADR-010): +Two entry points on [`AlkTypeEngine`], two validators for two input +types (ADR-VAL-SPLIT): +- `validate_bytes(&[u8])` — for raw byte buffers (channels' chunk + header, SFTP packets). Materializes a `Value` tree from the bytes via + the layout engine, then runs the **BAST-native validator** — a + recursive walker over the BAST type tree that checks the value-domain + constraints the materializer doesn't (integer ranges, `maxLength`, + timestamp shape, enum index bounds, union variant constraints). No + `jsonschema` involvement; the BAST document is the complete + validation spec for bytes (D-BAST-006). - `validate_json(&Value)` / `is_valid_json(&Value)` — for already-parsed - JSON (call's payload schemas). -- `validate_bytes(&[u8])` — materializes a `Value` tree from the bytes - via the layout engine, then validates that `Value`. Single-call binary - buffer validation. + JSON (call's `OperationSpec.input_schema` payloads). Validates + against a **standard `jsonschema::Validator`** compiled at + `AlkTypeEngine::compile` time from a consumer-provided JSON Schema + (the `json_schema: Option<&Value>` parameter). BAST is not involved — + BAST describes bytes, not JSON shape (D-BAST-007). -The validator is compiled once at load time; access-time validation is -a fast `is_valid()` check. High-throughput paths can skip validation; -security-sensitive paths can validate every frame. +Both paths return `AlkTypeError::Validation(jsonschema::ValidationError<'static>)` +— one uniform payload, one match arm (D-BAST-009). + +Validation is opt-in per operation. High-throughput paths can skip it; +security-sensitive paths can validate every frame. The BAST-native +validator also fixes a v0.1.0 dead constraint: enum index bounds are +now checked (the materializer emits a numeric index; the validator +checks it against `values.len()`). + +## BAST document shape + +Every BAST document has the same top-level shape: + +```json +{ "$defs": { "": { ...TypeDef... }, ... } } +``` + +- The `$defs` block is **required** (D-BAST-003). +- The **root type name** is a required parameter to + `AlkTypeEngine::compile(bast_doc, root_name, mode, ...)` (D-BAST-001) + — it selects which `$defs` entry is the top-level type. +- `$ref` is restricted to `#/$defs/` — one hash lookup, no + normalization pass. + +The BAST meta-schema is embedded in the crate as `BAST_META_SCHEMA` +(re-exported from the crate root) and published at +`https://alk.dev/bast/v1/schema`. Consumers can validate a BAST +document's structure with any JSON Schema validator. See +[`docs/architecture/bast-format.md`](docs/architecture/bast-format.md) +for the full spec. ## Crate independence @@ -165,13 +239,14 @@ payload schemas; `alktype` knows nothing about `alkcall`. ## Schemas as untrusted input -The crate treats schemas as untrusted input. A malformed schema -returns `AlkTypeError::Schema` / `AlkTypeError::Offset` from any -engine path — never a panic. This matters for hub/spoke topologies -where the remote peer provides the schema (e.g. `alkcall` accepting an -`OperationSpec` from an arbitrary internet peer). All `unreachable!()` -sites in production code were converted to `Err` ahead of v0.1.0 -(review #002, L2). +The crate treats BAST documents as untrusted input. A malformed +document returns `AlkTypeError::Schema` from any engine path — never a +panic. This matters for hub/spoke topologies where the remote peer +provides the schema (e.g. `alkcall` accepting an `OperationSpec` from +an arbitrary internet peer). Every `unreachable!()` site in production +code was converted to `Err` ahead of v0.1.0 (review #002, L2); the +BAST parser preserves this invariant — overflow-safe arithmetic +(`checked_add`, `usize::try_from`) on all offset/count casts. ## Documentation @@ -179,20 +254,26 @@ Architecture documentation lives under [`docs/architecture/`](docs/architecture/ - [Overview](docs/architecture/overview.md) — purpose, "schema is the format" principle, dependencies, consumers, scope boundaries -- [Schema layer](docs/architecture/schema-layer.md) — the 19 kinds, - jsonschema custom keyword integration, schema annotations +- [BAST format](docs/architecture/bast-format.md) — **normative format + spec**: meta-schema, TypeRef, TypeDef shapes, validation model +- [Schema layer](docs/architecture/schema-layer.md) — the BAST parser + (`BastDoc`/`BastDef`/`BastType` typed tree), the 19 kinds, the + `AlkTypeKind` enum - [Layout engine](docs/architecture/layout-engine.md) — offset computation, the two layout modes, alignment, endianness - [Data access](docs/architecture/data-access.md) — read/write functions, TUnion dispatch, field paths, zero-copy access -- [Validation](docs/architecture/validation.md) — custom keyword - validators, `AlkTypeError`, load-time vs access-time validation +- [Validation](docs/architecture/validation.md) — the two-validator + model, `AlkTypeError`, load-time vs access-time validation - [Builder](docs/architecture/builder.md) — fluent Rust API for - constructing alktype JSON Schemas at runtime + constructing BAST documents and standard JSON Schemas at runtime - [Architecture decisions (ADRs)](docs/architecture/decisions/) — - purpose/scope, two layout modes, schema annotations, error handling, - int64/uint64 kinds, packed-mode read factory, TUnion in aligned mode, - builder API, `validate_bytes` + purpose/scope (ADR-001), BAST format (ADR-BAST), two-validator model + (ADR-VAL-SPLIT), two layout modes (ADR-002), schema annotations + (ADR-003), error handling (ADR-004), int64/uint64 kinds (ADR-005), + non-final inline variable fields (ADR-006), packed-mode read factory + (ADR-007), TUnion in aligned mode (ADR-008), builder API (ADR-009), + `validate_bytes` (ADR-010) ## License