docs(architecture): add alknet-typedef crate specs, ADRs 095-098, and OQs 069-071
Add the alknet-typedef architecture specification — the binary struct engine that takes JSON Schema with TypeDef:* custom keywords and produces offset maps, read/write functions, and validation. Specs (docs/architecture/crates/typedef/): - overview.md: purpose, 'schema is the format' principle, consumers, scope - schema-layer.md: 17 TypeDef:* kinds, jsonschema integration, annotations - layout-engine.md: two layout modes, three variable-length strategies - data-access.md: read/write, TUnion dispatch, field paths, zero-copy - validation.md: custom keyword validators, TypedefError, TypedefEngine ADRs: - 095: Purpose, scope, and the jsonschema engine - 096: Two layout modes — packed sequential vs aligned static - 097: Schema annotations — endianness, alignment, encoding, TUnion - 098: Error handling and validation strategy OQs (deferred(scope)): - 069: Arrays of variable-length-element structs - 070: no_std + alloc support - 071: Builder API for schema construction Index updates: README doc table + ADR table, open-questions.md theme table + Deferred/Blocked section, overview.md crate graph. Grounded in the alknet-typedef POC (26 tests passing) and the call-channels-unification research. Reviewed by architecture-reviewer; all critical issues, warnings, and suggestions addressed.
This commit is contained in:
1 parent
bf0f827bf4
commit
85c5590001
16 files changed
+2230
-1
No files matched your search
@@ -0,0 +1,106 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef
|
||||
|
||||
The binary struct engine: a small Rust crate that takes a JSON Schema
|
||||
with `TypeDef:*` custom keywords and produces an offset map, read/write
|
||||
functions, and validation — all driven by the schema. The schema is the
|
||||
format definition; the engine is generic.
|
||||
|
||||
## Documents
|
||||
|
||||
| Document | Status | Description |
|
||||
|----------|--------|-------------|
|
||||
| [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
|
||||
| [schema-layer.md](schema-layer.md) | draft | The 17 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
|
||||
| [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
|
||||
| [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
|
||||
| [validation.md](validation.md) | draft | Custom keyword validators for all 16 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation, `TypedefEngine` |
|
||||
|
||||
## Applicable ADRs
|
||||
|
||||
| ADR | Title | Relevance |
|
||||
|-----|-------|-----------|
|
||||
| [095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
|
||||
| [096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` |
|
||||
| [097](../../decisions/097-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations |
|
||||
| [098](../../decisions/098-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors |
|
||||
|
||||
## Relevant Open Questions
|
||||
|
||||
| OQ | Title | Status | Relevance |
|
||||
|----|-------|--------|-----------|
|
||||
| OQ-069 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it |
|
||||
| OQ-070 | `no_std` + `alloc` support | deferred(scope) | Target `std` for v1; blocked on an embedded use case |
|
||||
| OQ-071 | Builder API for schema construction | deferred(scope) | Schemas are authored in TypeBox or hand-written JSON for v1; blocked on a concrete need |
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
1. **The schema is the format.** A JSON Schema with `TypeDef:*` custom
|
||||
keywords is both the validation spec and the layout spec. No separate
|
||||
format definition, no separate parser, no separate validator. One
|
||||
schema, three uses: validate, compute offsets, access data. See
|
||||
[overview.md](overview.md) and [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
2. **jsonschema is the validation engine, not a custom engine.** The
|
||||
`jsonschema` crate (v0.46.5, Draft 2020-12) handles validation with
|
||||
custom keyword support. The novel code is the offset computation, not
|
||||
the validation. This eliminates ~14,000 lines of hand-rolled schema
|
||||
engines (typebox-rs, alktype). See [schema-layer.md](schema-layer.md)
|
||||
and [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
3. **Two layout modes for two use cases.** Packed sequential
|
||||
(`LayoutBuilder`/`SequentialReader`) for protocol wire formats (SFTP,
|
||||
channels, TTY). Aligned static (`OffsetMap`) for mmap-friendly formats
|
||||
(metatensor). The consumer selects the mode; the schema is the same.
|
||||
See [layout-engine.md](layout-engine.md) and
|
||||
[ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md).
|
||||
|
||||
4. **Variable-length types default to inline length-prefixing.**
|
||||
`[length: u32][data]` is the universal pattern used by channels, SFTP,
|
||||
TTY, and most binary protocols. Offset indirection (the metatensor
|
||||
blob tensor pattern) is opt-in via the `encoding` annotation. See
|
||||
[layout-engine.md](layout-engine.md) and
|
||||
[ADR-097](../../decisions/097-schema-annotations.md).
|
||||
|
||||
5. **TUnion supports both byte-offset and field-name discriminators.**
|
||||
Byte-offset for protocol dispatch (SFTP type bytes, call protocol
|
||||
event types). Field-name for the typedef.ts string pattern. See
|
||||
[data-access.md](data-access.md) and
|
||||
[ADR-097](../../decisions/097-schema-annotations.md).
|
||||
|
||||
6. **Endianness is per-schema, default little-endian.** The engine reads
|
||||
the `"endian"` annotation and byte-swaps accordingly. SFTP consumers
|
||||
specify `"endian": "big"`. See [layout-engine.md](layout-engine.md)
|
||||
and [ADR-097](../../decisions/097-schema-annotations.md).
|
||||
|
||||
7. **Validation is opt-in, built once at load time.** The jsonschema
|
||||
validator is compiled once at schema load time. Access-time validation
|
||||
is a fast `is_valid()` check. High-throughput paths can skip
|
||||
validation; security-sensitive paths can validate every frame. See
|
||||
[validation.md](validation.md) and
|
||||
[ADR-098](../../decisions/098-error-handling-validation-strategy.md).
|
||||
|
||||
8. **Not a serialization framework.** The typedef engine is not a
|
||||
general-purpose serde replacement. It operates on raw byte buffers at
|
||||
computed offsets — no intermediate `Value` tree, no reflection, no
|
||||
dynamic dispatch per field. For JSON data, use serde. For binary data
|
||||
with a known schema, use typedef. See [overview.md](overview.md) and
|
||||
[ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/alknet-typedef/findings.md` — POC results (26 tests
|
||||
passing, two layout modes, TUnion dispatch, endianness)
|
||||
- `docs/research/call-channels-unification/findings.md` §"alknet-typedef:
|
||||
JSON Schema as the binary struct engine" — the origin of this research
|
||||
thread
|
||||
- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox
|
||||
schema kinds (619 lines)
|
||||
- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12)
|
||||
- `/workspace/alknet-typedef-poc/` — the POC code (disposable)
|
||||
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef
|
||||
- `/workspace/@alkimiadev/alktype/` — prior attempt, replaced by typedef
|
||||
@@ -0,0 +1,224 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef — Data Access
|
||||
|
||||
The data access layer: read/write functions, TUnion dispatch, field paths,
|
||||
zero-copy access for fixed-size types, and length-prefix reading for
|
||||
variable-length types. This is the consumer-facing API — given a compiled
|
||||
`TypedefEngine` and a byte buffer, read and write fields at
|
||||
schema-computed offsets.
|
||||
|
||||
## Read/Write Model
|
||||
|
||||
The typedef engine operates on raw byte buffers (`&[u8]` for reading,
|
||||
`&mut [u8]` for writing). There is no intermediate `Value` tree, no
|
||||
reflection, no dynamic dispatch per field. The engine uses the offset map
|
||||
(or `LayoutBuilder`/`SequentialReader`) to locate fields, then performs
|
||||
typed access at the computed positions.
|
||||
|
||||
### Fixed-size types
|
||||
|
||||
Fixed-size types (`TFloat32`, `TInt32`, `TUint8`, etc.) are accessed via
|
||||
zero-copy pointer casts:
|
||||
|
||||
```rust
|
||||
// Read a u32 at a known offset
|
||||
fn read_u32(buffer: &[u8], offset: usize, endian: Endian) -> u32 {
|
||||
let bytes: [u8; 4] = buffer[offset..offset+4].try_into().unwrap();
|
||||
match endian {
|
||||
Endian::Little => u32::from_le_bytes(bytes),
|
||||
Endian::Big => u32::from_be_bytes(bytes),
|
||||
}
|
||||
}
|
||||
|
||||
// Write a u32 at a known offset
|
||||
fn write_u32(buffer: &mut [u8], offset: usize, value: u32, endian: Endian) {
|
||||
let bytes = match endian {
|
||||
Endian::Little => value.to_le_bytes(),
|
||||
Endian::Big => value.to_be_bytes(),
|
||||
};
|
||||
buffer[offset..offset+4].copy_from_slice(&bytes);
|
||||
}
|
||||
```
|
||||
|
||||
The engine applies endianness at access time based on the schema's
|
||||
`"endian"` annotation (ADR-097). The offset computation is
|
||||
endian-agnostic.
|
||||
|
||||
### Variable-length types (inline length-prefixing)
|
||||
|
||||
For variable-length types with inline length-prefixing (the default):
|
||||
|
||||
```rust
|
||||
// Read a length-prefixed string
|
||||
fn read_string<'a>(buffer: &'a [u8], offset: usize) -> &'a str {
|
||||
let len = u32::from_le_bytes(buffer[offset..offset+4].try_into().unwrap()) as usize;
|
||||
std::str::from_utf8(&buffer[offset+4..offset+4+len]).unwrap()
|
||||
}
|
||||
|
||||
// Write a length-prefixed string
|
||||
fn write_string(buffer: &mut [u8], offset: usize, value: &str) {
|
||||
let data = value.as_bytes();
|
||||
buffer[offset..offset+4].copy_from_slice(&(data.len() as u32).to_le_bytes());
|
||||
buffer[offset+4..offset+4+data.len()].copy_from_slice(data);
|
||||
}
|
||||
```
|
||||
|
||||
The engine reads the 4-byte length prefix at the field's offset, then
|
||||
slices the data that follows. For writing, the engine writes the length
|
||||
prefix + data.
|
||||
|
||||
In packed sequential mode, the `SequentialReader` uses the length prefix
|
||||
to determine the position of the next field. In aligned static mode, the
|
||||
`OffsetMap` records the position of the length prefix; the variable data
|
||||
is accessed separately.
|
||||
|
||||
### Variable-length types (offset indirection)
|
||||
|
||||
For variable-length types with offset indirection (opt-in):
|
||||
|
||||
```rust
|
||||
// Read an offset-indirect string
|
||||
fn read_string_indirect(data_region: &[u8], offset: usize) -> &str {
|
||||
let ptr_offset = u32::from_le_bytes(data_region[offset..offset+4].try_into().unwrap()) as usize;
|
||||
let ptr_length = u32::from_le_bytes(data_region[offset+4..offset+8].try_into().unwrap()) as usize;
|
||||
std::str::from_utf8(&data_region[ptr_offset..ptr_offset+ptr_length]).unwrap()
|
||||
}
|
||||
```
|
||||
|
||||
The field is a struct `{offset: u32, length: u32}` at a known position
|
||||
in the `OffsetMap`. The consumer provides the data region separately; the
|
||||
engine reads the offset and length, then slices the data region.
|
||||
|
||||
## TUnion Dispatch
|
||||
|
||||
TUnion dispatch reads the discriminator value, looks up the variant
|
||||
schema, and then reads the variant's fields. The dispatch mechanism
|
||||
differs by discriminator kind (ADR-097).
|
||||
|
||||
### Byte-offset discriminator
|
||||
|
||||
```rust
|
||||
fn read_union(buffer: &[u8], schema: &Value) -> Result<Value, TypedefError> {
|
||||
let disc = &schema["discriminator"];
|
||||
let offset = disc["offset"].as_u64().unwrap() as usize;
|
||||
let disc_type = disc["type"].as_str().unwrap(); // e.g., "TypeDef:Uint8"
|
||||
|
||||
// Read the discriminator value
|
||||
let disc_value: u8 = read_u8(buffer, offset);
|
||||
let key = disc_value.to_string(); // "5", "6", "101"
|
||||
|
||||
// Look up the variant schema
|
||||
let mapping = &schema["mapping"];
|
||||
let variant_schema = &mapping[&key];
|
||||
|
||||
// Read the variant struct starting at offset + discriminator_size
|
||||
let variant_offset = offset + 1; // discriminator_size for Uint8
|
||||
read_struct(buffer, variant_offset, variant_schema)
|
||||
}
|
||||
```
|
||||
|
||||
The discriminator is a fixed-size integer at a known byte offset. The
|
||||
mapping keys are stringified integers. The variant struct starts at
|
||||
`offset + discriminator_size`.
|
||||
|
||||
This is the SFTP `Packet` enum pattern — byte 0 is the type byte, bytes
|
||||
1..N are the variant struct. The call protocol's 5 event types
|
||||
(`call.requested` → 0x01, etc.) use the same pattern.
|
||||
|
||||
### Field-name discriminator
|
||||
|
||||
```rust
|
||||
fn read_union_field(buffer: &[u8], schema: &Value) -> Result<Value, TypedefError> {
|
||||
let disc = &schema["discriminator"];
|
||||
let field_name = disc["name"].as_str().unwrap(); // e.g., "type"
|
||||
|
||||
// Read the discriminator field like any other field
|
||||
let disc_value = read_field(buffer, field_name, schema)?;
|
||||
|
||||
// Look up the variant schema
|
||||
let mapping = &schema["mapping"];
|
||||
let variant_schema = &mapping[disc_value.as_str().unwrap()];
|
||||
|
||||
// Read the variant struct
|
||||
read_struct(buffer, variant_offset, variant_schema)
|
||||
}
|
||||
```
|
||||
|
||||
The discriminator is a named field within the struct. Its offset is
|
||||
computed like any other field. The mapping keys are string values.
|
||||
|
||||
## Field Paths
|
||||
|
||||
Fields are addressed by dotted paths: `"header.version"`, `"payload.data"`.
|
||||
The `OffsetMap` stores fully-qualified paths. The read/write functions
|
||||
accept a field path and look up the byte range:
|
||||
|
||||
```rust
|
||||
fn read_f32(&self, buffer: &[u8], field_path: &str) -> Result<f32, TypedefError> {
|
||||
let range = self.offset_map.get(field_path)
|
||||
.ok_or_else(|| TypedefError::Offset {
|
||||
field_path: field_path.to_string(),
|
||||
reason: "field not found in offset map".to_string(),
|
||||
})?;
|
||||
if buffer.len() < range.end {
|
||||
return Err(TypedefError::Access {
|
||||
field_path: field_path.to_string(),
|
||||
reason: format!("buffer too short: need {} bytes, have {}", range.end, buffer.len()),
|
||||
});
|
||||
}
|
||||
Ok(read_f32_raw(buffer, range.start, self.endian))
|
||||
}
|
||||
```
|
||||
|
||||
Nested structs produce nested field paths. The offset computation
|
||||
propagates the field path prefix during recursion, so the `OffsetMap`
|
||||
contains entries like `"header.version"` and `"header.magic"`.
|
||||
|
||||
## Zero-Copy Access
|
||||
|
||||
For fixed-size types, the engine provides zero-copy access — the consumer
|
||||
gets a reference to the bytes in the buffer, not a copy. This is
|
||||
important for performance-sensitive paths (metatensor tensor access,
|
||||
high-throughput protocol parsing).
|
||||
|
||||
For variable-length types with inline length-prefixing, the engine
|
||||
returns a slice of the buffer — the string or byte array data is not
|
||||
copied. The consumer gets a `&str` or `&[u8]` that borrows from the
|
||||
input buffer.
|
||||
|
||||
For offset-indirect types, the consumer provides the data region; the
|
||||
engine returns a slice of that region.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Read/write errors carry the field path for debugging. See
|
||||
[ADR-098](../../decisions/098-error-handling-validation-strategy.md) and
|
||||
[validation.md](validation.md) for the full error model.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Determines whether offsets are fixed (OffsetMap) or sequential (SequentialReader) |
|
||||
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness, encoding, and TUnion discriminator shapes that control data access |
|
||||
| Error handling | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | Field-path-carrying errors for read/write operations |
|
||||
|
||||
## Open Questions
|
||||
|
||||
See [open-questions.md](../../open-questions.md) for full details.
|
||||
|
||||
- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs
|
||||
— affects the sequential walking logic for array access.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1
|
||||
(read/write round-trip) and POC 2 (SFTP byte-identical round-trip)
|
||||
- [layout-engine.md](layout-engine.md) — offset computation that produces
|
||||
the positions this layer reads/writes at
|
||||
- [validation.md](validation.md) — validation that runs on the same
|
||||
buffers
|
||||
@@ -0,0 +1,254 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef — Layout Engine
|
||||
|
||||
The layout engine: offset computation, the two layout modes (packed
|
||||
sequential vs aligned static), alignment, endianness, and variable-length
|
||||
field handling. This is the novel code — the recursive walk of the schema
|
||||
JSON that computes byte positions for each field.
|
||||
|
||||
## The Two Layout Modes
|
||||
|
||||
The POCs surfaced that protocols and mmap-friendly formats need different
|
||||
layout strategies. This is the most important architectural finding —
|
||||
decided in [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md).
|
||||
|
||||
### Mode 1: Packed sequential (protocol wire formats)
|
||||
|
||||
Fields are packed with no alignment padding. Variable-length fields shift
|
||||
all subsequent fields. Used by SFTP, channels, TTY, and most binary
|
||||
protocols.
|
||||
|
||||
**Components:**
|
||||
|
||||
- **`LayoutBuilder`** — takes a schema and actual data sizes for
|
||||
variable-length fields, computes byte positions for each field in a
|
||||
packed layout. Used at write time when the consumer knows the data
|
||||
sizes upfront.
|
||||
- **`SequentialReader`** — walks a buffer field-by-field according to the
|
||||
schema, reading length prefixes to determine variable-length data
|
||||
positions. Used at read time when the consumer is parsing an incoming
|
||||
frame.
|
||||
|
||||
**How it works:**
|
||||
|
||||
For a struct with fields `[u8, u32, string]`:
|
||||
|
||||
```
|
||||
LayoutBuilder (write):
|
||||
field[0] u8: offset 0, size 1
|
||||
field[1] u32: offset 1, size 4
|
||||
field[2] string: offset 5, size 4 (length prefix) + data_len
|
||||
total: 9 + data_len
|
||||
|
||||
SequentialReader (read):
|
||||
read u8 at offset 0
|
||||
read u32 at offset 1
|
||||
read u32 length prefix at offset 5 → data_len
|
||||
read string data at offset 9, length data_len
|
||||
next field at offset 9 + data_len
|
||||
```
|
||||
|
||||
There is no alignment padding. The `u32` at offset 1 is unaligned — this
|
||||
is correct for protocol wire formats, which pack fields tightly.
|
||||
|
||||
**Variable-length fields in packed mode:**
|
||||
|
||||
The `LayoutBuilder` takes actual data sizes for variable-length fields
|
||||
to compute correct positions for subsequent fields. The consumer must
|
||||
know the data sizes before writing — this is inherent to packed layouts.
|
||||
|
||||
The `SequentialReader` reads each field's length prefix to determine the
|
||||
data extent and the position of the next field. The reader walks the
|
||||
buffer sequentially; it cannot jump to field N without reading fields
|
||||
0..N-1 first.
|
||||
|
||||
### Mode 2: Aligned static (mmap-friendly formats)
|
||||
|
||||
Fields have fixed positions with natural alignment padding.
|
||||
Variable-length fields get a 4-byte length prefix at a known offset; the
|
||||
variable data is not included in the static layout. Used by metatensor
|
||||
and safetensors.
|
||||
|
||||
**Component:**
|
||||
|
||||
- **`OffsetMap`** — walks the schema once, computes fixed byte positions
|
||||
for each field based on type sizes and alignment. The output is a flat
|
||||
table of `(field_path, byte_range)` pairs. Used for both read and write
|
||||
at known offsets.
|
||||
|
||||
**How it works:**
|
||||
|
||||
For a struct with fields `[u8, u32, f32]` and natural alignment:
|
||||
|
||||
```
|
||||
OffsetMap:
|
||||
field[0] u8: offset 0, size 1
|
||||
field[1] u32: offset 4, size 4 (3 bytes padding after u8)
|
||||
field[2] f32: offset 8, size 4
|
||||
total: 12 (struct aligned to 4)
|
||||
```
|
||||
|
||||
The `u32` is aligned to offset 4 (its natural alignment). The consumer
|
||||
can read `field[1]` at offset 4 without reading `field[0]` first — random
|
||||
access by field path.
|
||||
|
||||
**Variable-length fields in aligned mode:**
|
||||
|
||||
Variable-length fields get a 4-byte length prefix at a known offset. The
|
||||
variable data lives outside the static layout — either immediately after
|
||||
the fixed fields (inline length-prefixing) or in a separate data region
|
||||
(offset indirection). The `OffsetMap` records the position of the length
|
||||
prefix (or the `{offset, length}` pair for offset-indirect fields).
|
||||
|
||||
For inline length-prefixing, the variable data follows the fixed fields
|
||||
but is not included in the `OffsetMap`'s field ranges. The consumer reads
|
||||
the length prefix from the `OffsetMap`'s known offset, then slices the
|
||||
data region.
|
||||
|
||||
For offset indirection, the field is a struct `{offset: u32, length: u32}`
|
||||
at a known position in the `OffsetMap`. The consumer reads the offset and
|
||||
length, then slices the separate data region.
|
||||
|
||||
## Offset Computation Algorithm
|
||||
|
||||
The offset computation is a recursive walk of the schema JSON. The
|
||||
algorithm is the same for both modes; the difference is whether alignment
|
||||
padding is inserted between fields.
|
||||
|
||||
### Fixed-size types
|
||||
|
||||
For each fixed-size type, the algorithm:
|
||||
1. Determines the type's byte size from the `TypeDef:*` kind.
|
||||
2. In aligned mode: inserts padding to satisfy the type's alignment
|
||||
(or the field's `align` annotation, or the struct's `align` default).
|
||||
3. Records the field's `(start, end)` range.
|
||||
4. Advances the current offset by the type's size.
|
||||
|
||||
### Composite types
|
||||
|
||||
**`TStruct`:** Recurse into the struct's `properties`. The inner fields
|
||||
are computed relative to the struct's start offset. The struct's total
|
||||
size is the sum of its fields' sizes (plus alignment padding in aligned
|
||||
mode). The struct itself may have an `align` annotation that rounds up
|
||||
its total size.
|
||||
|
||||
**`TUnion`:** The discriminator occupies `offset..offset + discriminator_size`
|
||||
bytes. For byte-offset discriminators, the variant struct starts at
|
||||
`offset + discriminator_size`. For field-name discriminators, the
|
||||
discriminator is just another field — its offset is computed like any
|
||||
other field, and the variant struct follows at the end of the
|
||||
discriminator field.
|
||||
|
||||
In aligned static mode, the union's total size is `discriminator_size +
|
||||
max(variant_sizes)`, where variant sizes are computed from the schema
|
||||
(variable-length data lives outside the static layout).
|
||||
|
||||
In packed sequential mode, variant sizes depend on the actual sizes of
|
||||
variable-length fields within each variant, which aren't known at schema
|
||||
time. The `LayoutBuilder` takes the actual variant discriminator value
|
||||
and data sizes at write time, computes the size of the selected variant,
|
||||
and uses that for the union's total size. The `SequentialReader` reads
|
||||
the discriminator first, looks up the variant schema, then reads the
|
||||
variant struct sequentially — it doesn't need to know the union's total
|
||||
size upfront.
|
||||
|
||||
**`TArray` of fixed-size elements:** Element stride = element size (plus
|
||||
alignment padding in aligned mode). Element `i` starts at
|
||||
`array_offset + i × stride`. The array's total size is `count × stride`.
|
||||
|
||||
**`TArray` of variable-length-element structs:** Deferred for v1
|
||||
(OQ-069).
|
||||
|
||||
### Variable-length types
|
||||
|
||||
The typedef engine supports three strategies for variable-length types
|
||||
(see [schema-layer.md](schema-layer.md) §Variable-length types and
|
||||
[ADR-097](../../decisions/097-schema-annotations.md) §3 for the full
|
||||
annotation shapes).
|
||||
|
||||
**Strategy 1: Inline length-prefixing (default).**
|
||||
1. Records the position of the 4-byte length prefix.
|
||||
2. In aligned mode: the length prefix is aligned; the variable data is
|
||||
not included in the static layout.
|
||||
3. In packed mode: the `LayoutBuilder` takes the actual data size to
|
||||
compute the length prefix value and the position of subsequent fields.
|
||||
The `SequentialReader` reads the length prefix to determine the data
|
||||
extent and the position of the next field.
|
||||
|
||||
**Strategy 2: Fixed-size reservation (`maxLength`).**
|
||||
1. In aligned static mode: reserves `maxLength` bytes at a fixed offset.
|
||||
Data shorter than `maxLength` is zero-padded. Subsequent fields have
|
||||
known, unchanging offsets — the field is fixed-size from the layout
|
||||
perspective. This is the database `VARCHAR(N)` pattern.
|
||||
2. In packed sequential mode: `maxLength` is a validation constraint
|
||||
only. The engine uses strategy 1 (inline length-prefixing) because
|
||||
protocols don't benefit from fixed-size reservation.
|
||||
|
||||
**Strategy 3: Offset indirection (`"encoding": "offset-indirect"`).**
|
||||
1. The field is a struct `{offset: u32, length: u32}`.
|
||||
2. The `OffsetMap` records the position of this struct.
|
||||
3. The consumer provides the data region separately. This is the
|
||||
metatensor blob tensor pattern — the index struct lives in one region,
|
||||
the blob data lives in another.
|
||||
|
||||
### Nested structs and field paths
|
||||
|
||||
Nested structs produce dotted field paths: `header.version`,
|
||||
`header.magic`. The offset computation propagates the field path prefix
|
||||
during recursion. The `OffsetMap` stores fully-qualified paths.
|
||||
|
||||
### Endianness
|
||||
|
||||
Endianness is per-schema (ADR-097). The offset computation is
|
||||
endian-agnostic — it computes byte positions, not byte values. The
|
||||
read/write functions apply endianness when converting between bytes and
|
||||
typed values. The engine reads the `"endian"` annotation from the schema
|
||||
and byte-swaps accordingly.
|
||||
|
||||
## Mode Selection
|
||||
|
||||
The consumer selects the mode at engine construction time. The choice is
|
||||
determined by the use case, not by the schema:
|
||||
|
||||
- **Protocol consumer** (SFTP, binary call frames, TTY negotiation):
|
||||
uses `LayoutBuilder` for writing and `SequentialReader` for reading.
|
||||
- **mmap consumer** (metatensor): uses `OffsetMap` for both reading and
|
||||
writing.
|
||||
|
||||
The same schema can be used in either mode. A schema describing an SFTP
|
||||
packet can be consumed by a `SequentialReader` (for parsing incoming
|
||||
frames) and a `LayoutBuilder` (for constructing outgoing frames). A schema
|
||||
describing a metatensor layout can be consumed by an `OffsetMap` (for
|
||||
mmap access).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential for protocols; aligned static for mmap formats |
|
||||
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness, alignment, encoding annotations that control layout behavior |
|
||||
|
||||
## Open Questions
|
||||
|
||||
See [open-questions.md](../../open-questions.md) for full details.
|
||||
|
||||
- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs
|
||||
— requires lazy walking logic; blocked on a concrete consumer that
|
||||
needs it.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/alknet-typedef/findings.md` §"POC Results" — POC 1
|
||||
(aligned OffsetMap) and POC 2 (packed LayoutBuilder/SequentialReader)
|
||||
- [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) —
|
||||
the two layout modes decision
|
||||
- [ADR-097](../../decisions/097-schema-annotations.md) — schema
|
||||
annotations
|
||||
- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds and their
|
||||
byte sizes
|
||||
- [data-access.md](data-access.md) — read/write functions that use the
|
||||
computed offsets
|
||||
@@ -0,0 +1,198 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef — Overview
|
||||
|
||||
The binary struct engine: a small Rust crate that takes a JSON Schema
|
||||
with `TypeDef:*` custom keywords and produces an offset map, read/write
|
||||
functions, and validation — all driven by the schema. The schema is the
|
||||
format definition; the engine is generic.
|
||||
|
||||
This document covers the crate's purpose, the "schema is the format"
|
||||
principle, its dependency edges, consumers, and scope boundaries.
|
||||
Component details are in the sibling documents.
|
||||
|
||||
## What
|
||||
|
||||
`alknet-typedef` is a library crate that consumes JSON Schemas annotated
|
||||
with `TypeDef:*` custom keywords (the same kinds defined in TypeBox's
|
||||
`typedef.ts`) and produces three capabilities:
|
||||
|
||||
1. **An offset map** — walks the schema, computes byte offsets for each
|
||||
field based on type sizes, field order, and alignment.
|
||||
2. **Read/write functions** — given a `&[u8]` buffer and a field path,
|
||||
read the field's bytes at its offset (zero-copy for fixed-size types).
|
||||
Given a `&mut [u8]` buffer, write a value at its offset.
|
||||
3. **Validation** — via `jsonschema` custom keywords, validates that a
|
||||
buffer's bytes match the schema's type constraints.
|
||||
|
||||
The heavy lifting is done by the `jsonschema` crate (validation) and
|
||||
`serde_json` (schema parsing). The novel code is the offset computation
|
||||
— a recursive walk of the schema JSON that computes byte positions for
|
||||
each field. The custom keyword implementations are ~10 lines each.
|
||||
|
||||
The crate is ~1,900 lines (POC verified, 26 tests passing). It replaces
|
||||
two prior attempts that built their own jsonschema engines — typebox-rs
|
||||
(~8,400 lines) and alktype (~5,600 lines) — with `jsonschema` + an
|
||||
offset map + ~50 lines of custom keyword implementations. See
|
||||
[ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
## Why
|
||||
|
||||
The crate's purpose is to be the binary struct engine for every alknet
|
||||
component that reads or writes binary data at computed offsets. Instead
|
||||
of per-protocol serde structs (russh-sftp's 29 packet types), per-handler
|
||||
wire format code (TTY's 5-byte format parser), or per-format offset
|
||||
computation (metatensor's tensor access), all of these become instances
|
||||
of the same engine with different schemas.
|
||||
|
||||
The guiding insight:
|
||||
|
||||
> **The schema is the format.** A JSON Schema with `TypeDef:Float32`,
|
||||
> `TypeDef:Struct`, `TypeDef:Union` etc. is both the validation spec and
|
||||
> the layout spec. No separate format definition, no separate parser, no
|
||||
> separate validator. One schema, three uses: validate, compute offsets,
|
||||
> access data.
|
||||
|
||||
This is the convergence of three threads identified in the
|
||||
call-channels-unification research: the `typedef.ts` schema kinds from
|
||||
TypeBox, the russh-sftp protocol packets, and the metatensor format. The
|
||||
common pattern: a JSON Schema describes the shape of binary data, and
|
||||
the binary data is the struct's bytes at computed offsets.
|
||||
|
||||
The crate was bumped up in the timeline when the call-channels-unification
|
||||
research surfaced that channels, TTY, and the binary call protocol are
|
||||
all variations on the same wire-format family — `[discriminant][length][payload]`.
|
||||
The typedef engine makes the "channels is call with a binary data plane"
|
||||
unification concrete: the binary data plane's wire format is the call
|
||||
protocol's own schema system, just binary-encoded. The `channel_open`
|
||||
marker says "use binary framing"; the typedef engine says "here's how to
|
||||
read/write the binary payload."
|
||||
|
||||
## The "Schema Is the Format" Principle
|
||||
|
||||
A JSON Schema with `TypeDef:*` custom keywords serves three roles
|
||||
simultaneously:
|
||||
|
||||
| Role | Mechanism | When |
|
||||
|------|-----------|------|
|
||||
| **Validation spec** | `jsonschema` custom keywords | Load time (build validator), access time (validate buffer) |
|
||||
| **Layout spec** | Offset computation from type sizes + field order | Load time (build offset map) |
|
||||
| **Data access** | Read/write at computed offsets | Access time (read field, write field) |
|
||||
|
||||
No separate format definition, no separate parser, no separate validator.
|
||||
The schema is the single source of truth for the binary format. Adding a
|
||||
new field to a protocol is adding a property to the schema JSON — the
|
||||
engine computes the new offsets automatically.
|
||||
|
||||
This is the same principle as `#[repr(C)]` struct field access, but at
|
||||
runtime from a portable JSON Schema instead of at compile-time from
|
||||
language-specific annotations. The schema is the ABI contract.
|
||||
|
||||
## Dependencies
|
||||
|
||||
```
|
||||
alknet-typedef
|
||||
├── jsonschema (v0.46.5, Draft 2020-12) — validation engine, custom keyword support
|
||||
├── serde_json (with preserve_order) — schema parsing; field order is load-bearing
|
||||
└── (no tokio, no platform deps) — WASM-clean by construction
|
||||
```
|
||||
|
||||
`alknet-typedef` is dependency-light: `jsonschema` + `serde_json` only.
|
||||
No tokio, no platform deps. Compiles to `wasm32-unknown-unknown` for
|
||||
browser use. The `jsonschema` crate is already in the workspace at
|
||||
`/workspace/jsonschema/` but not yet used by any alknet crate — typedef
|
||||
is the first consumer.
|
||||
|
||||
`serde_json` requires the `preserve_order` feature because field order
|
||||
is load-bearing for binary layouts. The order of properties in the
|
||||
schema JSON determines the order of fields in the binary struct.
|
||||
|
||||
## Consumers
|
||||
|
||||
| Consumer | Schema describes | Engine provides |
|
||||
|----------|-----------------|-----------------|
|
||||
| russh-sftp | 29 packet structs + Packet union (byte discriminator) | Read/write SFTP frames from bytes |
|
||||
| metatensor | Model layout (ConvNet struct, tensor refs) | Offset map for mmap'd tensor access |
|
||||
| binary call frames | `call.requested` / `call.responded` / etc. structs | Read/write binary call frames |
|
||||
| TTY negotiation | `NegotiateRequest` / `NegotiateResponse` structs | Read/write TTY control frames |
|
||||
| channels wire | `ChunkHeader { channel_id, length }` | Already trivial (8 bytes, no schema needed) |
|
||||
|
||||
The russh-sftp case is the most instructive and the highest-value POC
|
||||
target. The `Packet` enum's `TryFrom<&mut Bytes>` impl is a hand-written
|
||||
dispatch on a type byte followed by serde deserialization. Under typedef,
|
||||
the dispatch is `TUnion` with a byte-offset discriminator — the schema
|
||||
says "byte 0 is the discriminator, bytes 1..N are the variant struct."
|
||||
The engine reads the discriminator, looks up the variant schema, computes
|
||||
offsets, reads fields. Same result, no per-packet-type code.
|
||||
|
||||
## Scope Boundaries (What This Is Not)
|
||||
|
||||
These boundaries are decided in [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
- **Not metatensor.** typedef is the binary struct *engine*. Metatensor
|
||||
is a *format* (8-byte header + JSON header + binary data) that uses the
|
||||
typedef engine for its offset computation and tensor access.
|
||||
- **Not a Value system.** TypeBox's `Value.Diff`, `Value.Migrate`,
|
||||
`Value.Convert` — schema evolution — is out of scope for v1. The engine
|
||||
should not do anything that explicitly blocks adding a Value system
|
||||
later.
|
||||
- **Not a code generator.** typebox-rs's `codegen/` module is a separate
|
||||
concern. The typedef engine consumes schemas; it does not generate them.
|
||||
- **Not a schema builder.** The typedef engine does not provide a fluent
|
||||
API for constructing schemas. Schemas are plain JSON — authored in
|
||||
TypeBox, generated by ujsx components, or hand-written. A builder API
|
||||
is deferred (OQ-071).
|
||||
- **Not a serialization framework.** The typedef engine is not a
|
||||
general-purpose serde replacement. It operates on raw byte buffers at
|
||||
computed offsets — no intermediate `Value` tree, no reflection, no
|
||||
dynamic dispatch per field. For JSON data, use serde. For binary data
|
||||
with a known schema, use typedef.
|
||||
|
||||
## Architecture (component pointers)
|
||||
|
||||
- **[schema-layer.md](schema-layer.md)** — the 17 `TypeDef:*` kinds,
|
||||
jsonschema custom keyword integration, TypeBox interop, schema
|
||||
annotations (endianness, alignment, encoding, TUnion discriminators).
|
||||
- **[layout-engine.md](layout-engine.md)** — offset computation, the two
|
||||
layout modes (packed sequential vs aligned static), alignment,
|
||||
endianness, variable-length field handling.
|
||||
- **[data-access.md](data-access.md)** — read/write functions, TUnion
|
||||
dispatch, field paths, zero-copy access for fixed-size types,
|
||||
length-prefix reading for variable-length types.
|
||||
- **[validation.md](validation.md)** — custom keyword validators for all
|
||||
16 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time
|
||||
validation, `TypedefEngine` as the compiled form of a schema.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Purpose, scope, and the jsonschema engine | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
|
||||
| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential (`LayoutBuilder`/`SequentialReader`) for protocols; aligned static (`OffsetMap`) for mmap formats |
|
||||
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name) |
|
||||
| Error handling and validation | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
|
||||
|
||||
## Open Questions
|
||||
|
||||
See [open-questions.md](../../open-questions.md) for full details.
|
||||
|
||||
- **OQ-069** (deferred(scope)): Arrays of variable-length-element structs.
|
||||
- **OQ-070** (deferred(scope)): `no_std` + `alloc` support.
|
||||
- **OQ-071** (deferred(scope)): Builder API for schema construction.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/alknet-typedef/findings.md` — POC results (26 tests
|
||||
passing, two layout modes, TUnion dispatch, endianness)
|
||||
- `docs/research/call-channels-unification/findings.md` §"alknet-typedef:
|
||||
JSON Schema as the binary struct engine" — the origin of this research
|
||||
thread
|
||||
- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox
|
||||
schema kinds (619 lines)
|
||||
- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12)
|
||||
- `/workspace/alknet-typedef-poc/` — the POC code (disposable)
|
||||
- `/workspace/@alkimiadev/typebox-rs/` — prior attempt, replaced by typedef
|
||||
- `/workspace/@alkimiadev/alktype/` — prior attempt, replaced by typedef
|
||||
@@ -0,0 +1,348 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef — Schema Layer
|
||||
|
||||
The schema layer: the 16 `TypeDef:*` custom type kinds, their mapping to
|
||||
Rust types and byte sizes, the `jsonschema` custom keyword integration,
|
||||
TypeBox interop, and the concrete JSON shapes for schema-level annotations.
|
||||
|
||||
## The 17 TypeDef Kinds
|
||||
|
||||
These are the custom schema kinds defined in TypeBox's `typedef.ts`
|
||||
(`/workspace/@alkdev/typebox/example/typedef/typedef.ts`, 619 lines) and
|
||||
ported to Rust via `jsonschema` custom keywords. Each kind carries binary
|
||||
layout semantics — a known byte size (for fixed-size types) or a known
|
||||
encoding strategy (for variable-length types).
|
||||
|
||||
| Kind | TypeBox key | Rust type | Size | Category |
|
||||
|------|-------------|-----------|------|----------|
|
||||
| `TFloat32` | `TypeDef:Float32` | `f32` | 4 | fixed |
|
||||
| `TFloat64` | `TypeDef:Float64` | `f64` | 8 | fixed |
|
||||
| `TInt8` | `TypeDef:Int8` | `i8` | 1 | fixed |
|
||||
| `TInt16` | `TypeDef:Int16` | `i16` | 2 | fixed |
|
||||
| `TInt32` | `TypeDef:Int32` | `i32` | 4 | fixed |
|
||||
| `TUint8` | `TypeDef:Uint8` | `u8` | 1 | fixed |
|
||||
| `TUint16` | `TypeDef:Uint16` | `u16` | 2 | fixed |
|
||||
| `TUint32` | `TypeDef:Uint32` | `u32` | 4 | fixed |
|
||||
| `TBoolean` | `TypeDef:Boolean` | `bool` (0x00=false, 0x01=true) | 1 | fixed |
|
||||
| `TString` | `TypeDef:String` | length-prefixed UTF-8 | variable | variable |
|
||||
| `TBytes` | `TypeDef:Bytes` | length-prefixed raw bytes | variable | variable |
|
||||
| `TStruct` | `TypeDef:Struct` | record of fields | sum of field sizes | composite |
|
||||
| `TUnion` | `TypeDef:Union` | tagged union | discriminator + variant | composite |
|
||||
| `TArray` | `TypeDef:Array` | repeated element | count × element size | composite |
|
||||
| `TEnum` | `TypeDef:Enum` | u32 index into enum values | 4 (fixed) | fixed |
|
||||
| `TRecord` | `TypeDef:Record` | count-prefixed sequence of (key, value) pairs | variable | variable |
|
||||
| `TTimestamp` | `TypeDef:Timestamp` | length-prefixed RFC 3339 string | variable | variable |
|
||||
|
||||
### Fixed-size types
|
||||
|
||||
`TFloat32`, `TFloat64`, `TInt8`, `TInt16`, `TInt32`, `TUint8`, `TUint16`,
|
||||
`TUint32`, `TBoolean`, and `TEnum` have known byte sizes. The offset
|
||||
computation uses these sizes directly. Read/write is zero-copy pointer
|
||||
cast for these types.
|
||||
|
||||
**`TBoolean` byte representation:** `0x00` = false, `0x01` = true. Other
|
||||
values are invalid and produce a `TypedefError::Access` on read.
|
||||
|
||||
**`TEnum` binary representation:** A `u32` index into the enum's declared
|
||||
values, in declaration order. The first declared value is index 0, the
|
||||
second is index 1, etc. The enum's values are declared via the standard
|
||||
JSON Schema `"enum"` keyword (e.g., `"enum": ["read", "write", "execute"]`).
|
||||
The `TypeDef:Enum` custom keyword signals that the type is an enum for
|
||||
layout purposes; the built-in `enum` keyword provides the value list.
|
||||
The `u32` index is always little-endian (enum indices are not protocol
|
||||
data — they are internal to the schema). See [ADR-097](../../decisions/097-schema-annotations.md).
|
||||
|
||||
### Variable-length types
|
||||
|
||||
`TString`, `TBytes`, `TRecord`, and `TTimestamp` have variable byte sizes.
|
||||
The typedef engine supports three strategies for handling variable-length
|
||||
types in binary layouts, selected by the `encoding` annotation and the
|
||||
standard JSON Schema `maxLength` keyword:
|
||||
|
||||
| Strategy | Encoding annotation | Layout behavior | Use case |
|
||||
|----------|-------------------|-----------------|----------|
|
||||
| **Inline length-prefixed** | `"length-prefixed"` (default) | `[length: u32][data]`; shifts subsequent fields in packed mode | Protocol wire formats (SFTP, channels, TTY) |
|
||||
| **Fixed-size reservation** | (none — uses `maxLength`) | `[data: maxLength bytes]`, zero-padded; fixed offset in aligned mode | mmap-friendly formats where max size is known (database `VARCHAR(N)` pattern) |
|
||||
| **Offset indirection** | `"offset-indirect"` | `{offset: u32, length: u32}` pointing into a separate data region | Blob tensors, metatensor variable-length data (the blob tensor pattern) |
|
||||
|
||||
**Strategy 1: Inline length-prefixing (default).** The field's fixed
|
||||
portion is a 4-byte length prefix at a computed offset. The variable data
|
||||
follows immediately after. In packed sequential mode, the length prefix
|
||||
determines the position of subsequent fields. In aligned static mode, the
|
||||
length prefix is at a known offset; the variable data is not included in
|
||||
the static layout. This is the universal pattern used by channels, SFTP,
|
||||
TTY, and most binary protocols.
|
||||
|
||||
**Strategy 2: Fixed-size reservation.** When a variable-length field
|
||||
declares `maxLength` (a standard JSON Schema keyword), the engine reserves
|
||||
`maxLength` bytes at a fixed offset in aligned static mode. Data shorter
|
||||
than `maxLength` is zero-padded; data longer than `maxLength` is a
|
||||
validation error. This makes the field fixed-size from the layout
|
||||
perspective — subsequent fields have known, unchanging offsets. This is
|
||||
the database `VARCHAR(N)` pattern and the metatensor struct-tensor
|
||||
pattern for fields with known maximum sizes.
|
||||
|
||||
In packed sequential mode, `maxLength` is a validation constraint only —
|
||||
the engine still uses inline length-prefixing (strategy 1) because
|
||||
protocols don't benefit from fixed-size reservation.
|
||||
|
||||
**Strategy 3: Offset indirection.** The field is a struct
|
||||
`{offset: u32, length: u32}` at a known position. The consumer provides
|
||||
the data region separately; the engine reads the offset and length, then
|
||||
slices the data region. This is the metatensor blob tensor pattern — the
|
||||
index struct lives in one region, the blob data lives in another. Enables
|
||||
mmap-friendly random access to variable-length data without parsing
|
||||
length prefixes and without reserving worst-case space.
|
||||
|
||||
**Default strategy selection:**
|
||||
- In packed sequential mode: always strategy 1 (inline length-prefixing).
|
||||
`maxLength` is a validation constraint only.
|
||||
- In aligned static mode: strategy 2 (fixed-size reservation) if
|
||||
`maxLength` is declared; strategy 3 (offset indirection) if
|
||||
`"encoding": "offset-indirect"` is declared; strategy 1 (inline
|
||||
length-prefixing) otherwise.
|
||||
|
||||
**Length prefix endianness:** The 4-byte length prefix (strategies 1 and 3)
|
||||
respects the schema's `"endian"` annotation (ADR-097). In little-endian
|
||||
mode, the length is `u32::from_le_bytes`. In big-endian mode, the length
|
||||
is `u32::from_be_bytes`. This ensures SFTP consumers (big-endian) have
|
||||
consistent byte order for both field values and length prefixes.
|
||||
|
||||
**`TBytes`:** Raw bytes — no UTF-8 constraint. The payload is `&[u8]`.
|
||||
Otherwise identical to `TString` in layout (same three strategies).
|
||||
|
||||
**`TRecord`:** A string-keyed map. Binary layout is a count-prefixed
|
||||
sequence of `(key, value)` pairs: `[count: u32][key_len: u32][key_bytes]
|
||||
[value_len: u32][value_bytes]...`. The count is the number of entries.
|
||||
Each key is a length-prefixed UTF-8 string. Each value is the record's
|
||||
declared value type (specified via the `"values"` property in the schema,
|
||||
e.g., `"values": { "TypeDef:Float32": true }`). The count prefix respects
|
||||
the schema's endianness. In aligned static mode with `maxLength`, the
|
||||
entire record is reserved at `maxLength` bytes (zero-padded).
|
||||
|
||||
**`TTimestamp`:** An RFC 3339 timestamp string (the internet profile of
|
||||
ISO 8601). Stored as a length-prefixed UTF-8 string (strategy 1) or
|
||||
fixed-size reservation (strategy 2 with `maxLength`). The engine does not
|
||||
parse or validate the timestamp format beyond UTF-8 — the jsonschema
|
||||
validator checks RFC 3339 conformance at the JSON level.
|
||||
|
||||
`TArray` is variable-length when the element type is variable-length or
|
||||
when the count is not known at schema time. For fixed-size element arrays
|
||||
with a known count, the size is `element_size × count`.
|
||||
|
||||
**`TArray` count declaration:** The array count is declared via the
|
||||
standard JSON Schema `"minItems"` and `"maxItems"` keywords. When
|
||||
`minItems == maxItems`, the array has a fixed count known at schema time.
|
||||
When they differ or are absent, the count is variable and the array uses
|
||||
a length-prefixed encoding: `[count: u32][element_0]...[element_N]`.
|
||||
The count prefix respects the schema's endianness.
|
||||
|
||||
### Composite types
|
||||
|
||||
`TStruct` and `TUnion` are composite — their size is the sum of their
|
||||
fields' sizes (plus alignment padding in aligned static mode). The offset
|
||||
computation recurses into their properties.
|
||||
|
||||
## jsonschema Custom Keyword Integration
|
||||
|
||||
The `jsonschema` crate (v0.46.5, Draft 2020-12) supports custom keywords
|
||||
via the `with_keyword` API. Each `TypeDef:*` kind is registered as a
|
||||
custom keyword:
|
||||
|
||||
```rust
|
||||
let validator = jsonschema::options()
|
||||
.with_keyword("TypeDef:Float32", factory)
|
||||
.with_keyword("TypeDef:Int32", factory)
|
||||
.with_keyword("TypeDef:Struct", factory)
|
||||
// ... all 17 kinds
|
||||
.build(&schema)?;
|
||||
```
|
||||
|
||||
The factory closure receives the parent schema object, the keyword's
|
||||
value, and the schema path — enabling cross-keyword awareness. The
|
||||
`TypeDef:Struct` validator, for example, inspects the parent's
|
||||
`properties` to validate each field against its declared `TypeDef:*` kind.
|
||||
|
||||
Each custom keyword implementation is ~10 lines. The `jsonschema` crate
|
||||
handles all structural validation (object properties, required fields,
|
||||
array items, enum values) — the custom keywords only need to validate
|
||||
the leaf type constraints. See [validation.md](validation.md) for the
|
||||
validator implementations.
|
||||
|
||||
This is the same pattern as TypeBox's `TypeRegistry.Set` on the JS side.
|
||||
Same semantics, different language, same JSON Schema wire format. A
|
||||
TypeBox schema serialized to JSON feeds directly into
|
||||
`jsonschema::validator_for(&schema)` on the Rust side — zero translation.
|
||||
|
||||
## TypeBox Interop
|
||||
|
||||
TypeBox modules render to standard JSON Schema under `$defs`. A TypeBox
|
||||
schema like:
|
||||
|
||||
```typescript
|
||||
const TensorRef = Type.Object({
|
||||
dtype: Type.Union([Type.Literal("F32"), Type.Literal("I16")]),
|
||||
shape: Type.Array(Type.Number()),
|
||||
data_offsets: Type.Tuple([Type.Number(), Type.Number()])
|
||||
});
|
||||
```
|
||||
|
||||
serialized to JSON is a standard JSON Schema with `type: "object"`,
|
||||
`properties`, and `required`. That JSON feeds directly into the typedef
|
||||
engine. The `TypeDef:*` custom keywords are added by TypeBox's
|
||||
`TypeRegistry.Set` — they appear in the serialized JSON as additional
|
||||
properties on the schema object.
|
||||
|
||||
The typedef engine does not depend on TypeBox or any JS toolchain. It
|
||||
consumes JSON — whether that JSON was authored in TypeBox, generated by
|
||||
a ujsx component, or hand-written. The schema is the interface.
|
||||
|
||||
## Schema Annotations
|
||||
|
||||
Schema-level annotations control binary layout behavior. These are
|
||||
decided in [ADR-097](../../decisions/097-schema-annotations.md).
|
||||
|
||||
### Endianness
|
||||
|
||||
Schema-level annotation with a default of little-endian:
|
||||
|
||||
```json
|
||||
{ "TypeDef:Struct": true, "endian": "big", "properties": { ... } }
|
||||
```
|
||||
|
||||
- `"endian": "little"` (default) — read/write in little-endian byte order.
|
||||
- `"endian": "big"` — read/write in big-endian byte order.
|
||||
- Applies to the entire schema and all nested types.
|
||||
|
||||
### Alignment
|
||||
|
||||
Both struct-level and field-level, with field-level overriding:
|
||||
|
||||
```json
|
||||
{
|
||||
"TypeDef:Struct": true,
|
||||
"align": 256,
|
||||
"properties": {
|
||||
"weight": { "TypeDef:Float32": true, "align": 16 }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- Struct-level `"align"` sets the default for all fields.
|
||||
- Field-level `"align"` overrides the struct default.
|
||||
- Default alignment: 1 for u8/bool, 2 for u16/i16, 4 for u32/i32/f32,
|
||||
8 for u64/i64/f64, max field alignment for structs.
|
||||
- Only meaningful in aligned static mode (ADR-096). Ignored in packed
|
||||
sequential mode.
|
||||
|
||||
### Variable-length encoding
|
||||
|
||||
The typedef engine supports three strategies for variable-length types
|
||||
(see §Variable-length types above for full details). The strategy is
|
||||
selected by the `encoding` annotation and the standard JSON Schema
|
||||
`maxLength` keyword:
|
||||
|
||||
```json
|
||||
// Strategy 1: Inline length-prefixing (default, shorthand)
|
||||
{ "TypeDef:String": true }
|
||||
|
||||
// Strategy 1: Explicit inline length-prefixing
|
||||
{ "TypeDef:String": { "encoding": "length-prefixed" } }
|
||||
|
||||
// Strategy 2: Fixed-size reservation (uses standard maxLength)
|
||||
{ "TypeDef:String": true, "maxLength": 256 }
|
||||
|
||||
// Strategy 3: Offset indirection (opt-in)
|
||||
{ "TypeDef:String": { "encoding": "offset-indirect" } }
|
||||
```
|
||||
|
||||
- `"encoding": "length-prefixed"` (default) — 4-byte length prefix at
|
||||
computed offset, variable data follows immediately. Used by protocol
|
||||
wire formats.
|
||||
- `maxLength` (standard JSON Schema keyword) — in aligned static mode,
|
||||
reserves `maxLength` bytes at a fixed offset (zero-padded). Makes the
|
||||
field fixed-size from the layout perspective. In packed sequential
|
||||
mode, `maxLength` is a validation constraint only.
|
||||
- `"encoding": "offset-indirect"` — field is a struct
|
||||
`{offset: u32, length: u32}` pointing into a separate data region.
|
||||
The consumer provides the data region separately. Used by metatensor
|
||||
blob tensors.
|
||||
- Applies to all variable-length types: `TypeDef:String`, `TypeDef:Bytes`,
|
||||
`TypeDef:Array`, `TypeDef:Record`, `TypeDef:Timestamp`.
|
||||
|
||||
### TUnion discriminators
|
||||
|
||||
Two discriminator kinds: byte-offset (protocol dispatch) and field-name
|
||||
(typedef.ts pattern).
|
||||
|
||||
**Byte-offset discriminator** (SFTP type bytes, call protocol event types):
|
||||
|
||||
```json
|
||||
{
|
||||
"TypeDef:Union": true,
|
||||
"discriminator": {
|
||||
"kind": "byte",
|
||||
"offset": 0,
|
||||
"type": "TypeDef:Uint8"
|
||||
},
|
||||
"mapping": {
|
||||
"5": { "$ref": "#/$defs/Read" },
|
||||
"6": { "$ref": "#/$defs/Write" },
|
||||
"101": { "$ref": "#/$defs/Status" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `"offset"` — byte position of the discriminator.
|
||||
- `"type"` — the `TypeDef:*` kind of the discriminator (typically
|
||||
`TypeDef:Uint8`).
|
||||
- Mapping keys are stringified integers. The variant struct starts at
|
||||
`offset + discriminator_size`.
|
||||
|
||||
**Field-name discriminator** (typedef.ts pattern):
|
||||
|
||||
```json
|
||||
{
|
||||
"TypeDef:Union": true,
|
||||
"discriminator": {
|
||||
"kind": "field",
|
||||
"name": "type"
|
||||
},
|
||||
"mapping": {
|
||||
"read": { "$ref": "#/$defs/Read" },
|
||||
"write": { "$ref": "#/$defs/Write" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
- `"name"` — the field name holding the discriminator value.
|
||||
- Mapping keys are string values matching the discriminator field's value.
|
||||
- The discriminator field is just another field in the struct.
|
||||
|
||||
Mapping values may be either inline schemas or `$ref` pointers. Both work.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Concrete JSON shapes for endianness, alignment, encoding, and TUnion discriminators |
|
||||
| Purpose and scope | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine; "schema is the format" principle |
|
||||
|
||||
## Open Questions
|
||||
|
||||
See [open-questions.md](../../open-questions.md) for full details.
|
||||
|
||||
- **OQ-071** (deferred(scope)): Builder API for schema construction.
|
||||
|
||||
## References
|
||||
|
||||
- `/workspace/@alkdev/typebox/example/typedef/typedef.ts` — the TypeBox
|
||||
schema kinds (619 lines)
|
||||
- `/workspace/jsonschema/` — the jsonschema crate (v0.46.5, Draft 2020-12)
|
||||
- [ADR-097](../../decisions/097-schema-annotations.md) — schema
|
||||
annotation shapes
|
||||
- [validation.md](validation.md) — custom keyword validator implementations
|
||||
@@ -0,0 +1,292 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-07-20
|
||||
---
|
||||
|
||||
# alknet-typedef — Validation
|
||||
|
||||
The validation layer: custom keyword validators for all 17 `TypeDef:*`
|
||||
kinds, the `TypedefError` enum, load-time vs access-time validation
|
||||
strategy, and the `TypedefEngine` as the compiled form of a schema.
|
||||
|
||||
## Validation Strategy
|
||||
|
||||
Validation is delegated to the `jsonschema` crate (v0.46.5, Draft
|
||||
2020-12). The typedef engine does not implement its own validation —
|
||||
it registers custom keyword validators for each `TypeDef:*` kind and
|
||||
lets `jsonschema` handle the structural validation (object properties,
|
||||
required fields, array items, enum values).
|
||||
|
||||
The strategy is decided in [ADR-098](../../decisions/098-error-handling-validation-strategy.md):
|
||||
|
||||
1. **Load time:** Parse the schema JSON, build the layout engine, build the
|
||||
jsonschema validator. This is the `TypedefEngine::compile(schema)` constructor.
|
||||
2. **Access time:** Use the compiled engine for repeated read/write
|
||||
operations. Validation is opt-in per operation.
|
||||
|
||||
### What validation validates
|
||||
|
||||
The jsonschema validator operates on `serde_json::Value` instances — it
|
||||
validates JSON representations of data, not raw byte buffers. This is
|
||||
the correct separation of concerns:
|
||||
|
||||
- **JSON validation** (jsonschema): validates that a JSON document
|
||||
conforms to the schema. Used for validating hand-written schemas,
|
||||
TypeBox output, JSON payloads, or the JSON representation of a binary
|
||||
struct after deserialization.
|
||||
- **Binary access validation** (data access layer): the read/write
|
||||
functions perform type-level validation at access time — range checks
|
||||
for integers, UTF-8 validity for strings, buffer bounds checking.
|
||||
These return `TypedefError::Access` with field paths.
|
||||
|
||||
The "schema is the format" principle means the same schema describes
|
||||
both the JSON shape and the binary layout. The jsonschema validator
|
||||
checks the JSON shape; the data access layer checks the binary layout.
|
||||
A consumer that wants to validate a binary buffer end-to-end reads the
|
||||
buffer into a `Value` tree via the data access layer, then validates
|
||||
that `Value` against the jsonschema validator. This is a two-step
|
||||
process, not a single `validate(buffer)` call.
|
||||
|
||||
### The `TypedefEngine` struct
|
||||
|
||||
The `TypedefEngine` is the compiled form of a schema. It supports both
|
||||
layout modes (ADR-096) via an internal enum:
|
||||
|
||||
```rust
|
||||
pub struct TypedefEngine {
|
||||
layout: Layout, // packed or aligned (see below)
|
||||
validator: jsonschema::Validator, // compiled once at load time
|
||||
}
|
||||
|
||||
enum Layout {
|
||||
Packed {
|
||||
builder: LayoutBuilder,
|
||||
reader: SequentialReader,
|
||||
},
|
||||
Aligned {
|
||||
offset_map: OffsetMap,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The consumer selects the mode at construction time. The `Layout` enum
|
||||
ensures the engine always has the correct layout strategy for the
|
||||
consumer's use case — a protocol consumer gets `Packed`, an mmap
|
||||
consumer gets `Aligned`. The validator is mode-agnostic (it operates on
|
||||
`Value`, not raw bytes).
|
||||
|
||||
## Custom Keyword Validators
|
||||
|
||||
Each `TypeDef:*` kind gets a `Keyword` implementation registered via
|
||||
`jsonschema::options().with_keyword(...)`. The validators check leaf
|
||||
type constraints; `jsonschema` handles all structural validation.
|
||||
|
||||
### Numeric type validators
|
||||
|
||||
**`TypeDef:Float32` / `TypeDef:Float64`:**
|
||||
- Value must be a finite number.
|
||||
- For `Float32`: value must be representable as `f32` (no precision loss
|
||||
beyond `f32`'s mantissa).
|
||||
|
||||
**`TypeDef:Int8` / `TypeDef:Int16` / `TypeDef:Int32`:**
|
||||
- Value must be an integer within the type's range.
|
||||
- Int8: -128..127, Int16: -32768..32767, Int32: -2147483648..2147483647.
|
||||
|
||||
**`TypeDef:Uint8` / `TypeDef:Uint16` / `TypeDef:Uint32`:**
|
||||
- Value must be a non-negative integer within the type's range.
|
||||
- Uint8: 0..255, Uint16: 0..65535, Uint32: 0..4294967295.
|
||||
|
||||
### String and binary validators
|
||||
|
||||
**`TypeDef:String`:**
|
||||
- Value must be a valid UTF-8 string.
|
||||
- If `maxLength` is specified in the schema, the string's byte length
|
||||
must not exceed it.
|
||||
|
||||
**`TypeDef:Bytes`:**
|
||||
- Value must be a string (JSON represents binary data as a string).
|
||||
- If `maxLength` is specified, the byte length must not exceed it.
|
||||
|
||||
**`TypeDef:Enum`:**
|
||||
- The `TypeDef:Enum` custom keyword signals that the type is an enum for
|
||||
*layout* purposes (the engine needs to know it's a fixed-size u32 index,
|
||||
not a variable-length string). The built-in `enum` keyword provides the
|
||||
value list and handles value-membership validation. The custom keyword
|
||||
validator is a no-op beyond the built-in check — it exists solely for
|
||||
the layout engine to recognize the type.
|
||||
|
||||
**`TypeDef:Timestamp`:**
|
||||
- Value must be a valid RFC 3339 timestamp string (the internet profile
|
||||
of ISO 8601, e.g., `"2026-07-20T15:30:00Z"`).
|
||||
|
||||
### Composite type validators
|
||||
|
||||
**`TypeDef:Struct`:**
|
||||
- Value must be an object.
|
||||
- Each property must match its declared `TypeDef:*` kind.
|
||||
- Required fields must be present.
|
||||
- The `jsonschema` crate's built-in `properties` and `required` keywords
|
||||
handle the structural checks — the custom keyword only needs to
|
||||
validate that each field's value matches its `TypeDef:*` kind.
|
||||
|
||||
**`TypeDef:Union`:**
|
||||
- The discriminator value must be one of the mapping keys.
|
||||
- The variant struct must match the declared schema for that discriminator
|
||||
value.
|
||||
|
||||
**`TypeDef:Array`:**
|
||||
- Value must be an array.
|
||||
- Each element must match the array's declared element type.
|
||||
- If `minItems`/`maxItems` is specified, the array length must be within
|
||||
bounds.
|
||||
|
||||
### Other validators
|
||||
|
||||
**`TypeDef:Boolean`:**
|
||||
- Value must be `true` or `false`.
|
||||
|
||||
**`TypeDef:Record`:**
|
||||
- Value must be an object.
|
||||
- All values must match the record's declared value type (specified via
|
||||
the `"values"` property in the schema, e.g.,
|
||||
`"values": { "TypeDef:Float32": true }`).
|
||||
|
||||
### Validator implementation pattern
|
||||
|
||||
Each custom keyword implementation is ~10 lines. Example for
|
||||
`TypeDef:Float32`:
|
||||
|
||||
```rust
|
||||
struct Float32Validator;
|
||||
|
||||
impl Keyword for Float32Validator {
|
||||
fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> {
|
||||
match instance {
|
||||
Value::Number(n) if n.as_f64().map_or(false, |f| f.is_finite()) => Ok(()),
|
||||
_ => Err(ValidationError::custom("expected finite f32-compatible number")),
|
||||
}
|
||||
}
|
||||
fn is_valid(&self, instance: &Value) -> bool {
|
||||
instance.as_f64().map_or(false, |f| f.is_finite())
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Registration:
|
||||
|
||||
```rust
|
||||
let validator = jsonschema::options()
|
||||
.with_keyword("TypeDef:Float32", |parent, value, path| {
|
||||
Ok(Box::new(Float32Validator))
|
||||
})
|
||||
.build(&schema)?;
|
||||
```
|
||||
|
||||
The factory closure receives the parent schema object, the keyword's
|
||||
value, and the schema path. This enables cross-keyword awareness — for
|
||||
example, a `TypeDef:Struct` validator can inspect the parent's
|
||||
`properties` to validate each field against its declared `TypeDef:*` kind.
|
||||
|
||||
## TypedefError
|
||||
|
||||
A single `TypedefError` enum covers all error conditions across the
|
||||
engine's three phases (schema parsing, offset computation, read/write)
|
||||
plus validation. Decided in [ADR-098](../../decisions/098-error-handling-validation-strategy.md).
|
||||
|
||||
```rust
|
||||
pub enum TypedefError {
|
||||
/// Schema parsing errors (invalid JSON, missing keywords, unknown TypeDef kinds).
|
||||
Schema(String),
|
||||
/// Offset computation errors (field not found, unsupported type).
|
||||
Offset { field_path: String, reason: String },
|
||||
/// Read/write errors (buffer too short, invalid UTF-8, value out of range).
|
||||
Access { field_path: String, reason: String },
|
||||
/// Validation errors (delegated to jsonschema).
|
||||
Validation(ValidationError<'static>),
|
||||
}
|
||||
```
|
||||
|
||||
- **`Schema`** — for errors during `TypedefEngine::compile()`. Invalid
|
||||
JSON, missing required keywords, unknown `TypeDef:*` kinds.
|
||||
- **`Offset`** — for errors during offset computation. Field not found
|
||||
in the schema, type not supported for offset computation, recursive
|
||||
depth exceeded. Carries the field path.
|
||||
- **`Access`** — for errors during read/write. Buffer too short, invalid
|
||||
UTF-8 in a string field, value out of range for the target type.
|
||||
Carries the field path.
|
||||
- **`Validation`** — wraps `jsonschema`'s `ValidationError`. The
|
||||
`'static` lifetime is correct — the validator owns its schema reference
|
||||
and lives for the lifetime of the `TypedefEngine`.
|
||||
|
||||
### Field-path-carrying errors
|
||||
|
||||
Read/write and offset errors include the field path for debugging:
|
||||
|
||||
```rust
|
||||
Err(TypedefError::Access {
|
||||
field_path: "header.version".to_string(),
|
||||
reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(),
|
||||
})
|
||||
```
|
||||
|
||||
This makes debugging binary format issues tractable — the error tells
|
||||
you exactly which field failed and why.
|
||||
|
||||
## Validation Timing
|
||||
|
||||
### Load time: `TypedefEngine::compile()`
|
||||
|
||||
The expensive work happens once at schema load time:
|
||||
1. Parse the schema JSON (`serde_json::from_str` with `preserve_order`).
|
||||
2. Compute the offset map (or `LayoutBuilder`/`SequentialReader`).
|
||||
3. Build the jsonschema validator (`jsonschema::options().with_keyword(...).build(&schema)?`).
|
||||
|
||||
The result is a `TypedefEngine` that can be used for repeated operations.
|
||||
|
||||
### Access time: `engine.validate(buffer)`
|
||||
|
||||
Validation is opt-in per operation. The consumer calls
|
||||
`engine.validate(buffer)` when validation is desired. The jsonschema
|
||||
validator is already compiled — `is_valid()` is a fast check against
|
||||
the compiled validator.
|
||||
|
||||
High-throughput paths can skip validation. Security-sensitive paths
|
||||
(parsing incoming frames from untrusted peers) can validate every frame.
|
||||
The choice is the consumer's.
|
||||
|
||||
## Relationship to Read/Write
|
||||
|
||||
Validation and data access are independent operations on the same buffer.
|
||||
The consumer can:
|
||||
|
||||
1. Validate a buffer to ensure it conforms to the schema.
|
||||
2. Read fields from the buffer at computed offsets.
|
||||
3. Both — validate first, then read (defense in depth).
|
||||
|
||||
The engine does not couple validation and access. A consumer that trusts
|
||||
its data source can skip validation and go straight to read/write. A
|
||||
consumer that parses untrusted input can validate first, then access.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Error handling and validation | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
|
||||
| Purpose and scope | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine |
|
||||
|
||||
## Open Questions
|
||||
|
||||
None specific to validation. The three typedef OQs (OQ-069, OQ-070,
|
||||
OQ-071) are about layout, platform support, and schema construction —
|
||||
not validation.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/alknet-typedef/findings.md` §"Validation" — the POC's
|
||||
custom keyword validators for all 17 kinds
|
||||
- [ADR-098](../../decisions/098-error-handling-validation-strategy.md) —
|
||||
error handling and validation strategy
|
||||
- [schema-layer.md](schema-layer.md) — the 17 TypeDef kinds that the
|
||||
validators check
|
||||
- [data-access.md](data-access.md) — read/write functions that operate
|
||||
on the same buffers
|
||||
Reference in new issue
Block a user