docs(typedef): sync specs with ADRs 099-102 (19 kinds, aligned-mode restrictions, read factory)

- schema-layer.md: 17→19 kinds, add Int64/Uint64 to table + TypeDefKind
  enum, update method tables (12 fixed-size kinds, 8-byte alignment for
  i64/u64/f64), add ADR-099 to design decisions.
- layout-engine.md: document ADR-100 (non-final inline variable field
  rejection), ADR-102 (TUnion rejected in aligned mode), ADR-101
  (sequential_reader returns owned reader). Update TUnion section to
  note aligned-mode rejection. Add ADRs to design decisions table.
- data-access.md: add I64/U64 to FieldValue enum, update
  sequential_reader() signature (factory, ADR-101).
- validation.md: 17→19 kinds, update TypedefEngine struct (no stored
  reader in Layout::Packed), update sequential_reader() signature.
- overview.md: 17→19 kinds, add ADRs 099-102 to design decisions.
- README.md (typedef): 17→19 kinds, add ADRs 099-102 to applicable ADRs.
- README.md (top-level): add ADRs 099-102 to ADR table, update typedef
  doc descriptions (17→19 kinds).
This commit is contained in:
2026-07-22 09:37:12 +00:00
parent 51806f4469
commit a5d5d037dd
7 changed files with 105 additions and 50 deletions

View File

@@ -298,10 +298,10 @@ adapter location map is now consistent: all HTTP-backed adapters
| [crates/channels/channel-client.md](crates/channels/channel-client.md) | draft | `ChannelClient` — client side of a channels connection, transport-agnostic `from_connection` primary; dial lives in `AlknetClient` (ADR-089); bidirectionality preserved |
| [crates/typedef/README.md](crates/typedef/README.md) | draft | alknet-typedef crate — binary struct engine; JSON Schema with `TypeDef:*` custom keywords → offset map + read/write + validation |
| [crates/typedef/overview.md](crates/typedef/overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
| [crates/typedef/schema-layer.md](crates/typedef/schema-layer.md) | draft | The 17 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [crates/typedef/schema-layer.md](crates/typedef/schema-layer.md) | draft | The 19 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [crates/typedef/layout-engine.md](crates/typedef/layout-engine.md) | draft | Offset computation, two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
| [crates/typedef/data-access.md](crates/typedef/data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
| [crates/typedef/validation.md](crates/typedef/validation.md) | draft | Custom keyword validators for all 17 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation |
| [crates/typedef/validation.md](crates/typedef/validation.md) | draft | Custom keyword validators for all 19 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation |
## ADR Table
@@ -405,6 +405,10 @@ adapter location map is now consistent: all HTTP-backed adapters
| [096](decisions/096-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | Accepted |
| [097](decisions/097-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, and TUnion Discriminators | Accepted |
| [098](decisions/098-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | Accepted |
| [099](decisions/099-int64-uint64-first-class-kinds.md) | Int64/Uint64 as First-Class Kinds | Accepted |
| [100](decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode | Accepted |
| [101](decisions/101-packed-mode-read-factory.md) | Packed-Mode Read API — Engine as SequentialReader Factory | Accepted |
| [102](decisions/102-reject-tunion-in-aligned-mode.md) | Reject TUnion in Aligned Mode for v1 | Accepted |
## Open Questions

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef
@@ -15,10 +15,10 @@ format definition; the engine is generic.
| Document | Status | Description |
|----------|--------|-------------|
| [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
| [schema-layer.md](schema-layer.md) | draft | The 17 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [schema-layer.md](schema-layer.md) | draft | The 19 `TypeDef:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
| [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
| [validation.md](validation.md) | draft | Custom keyword validators for all 17 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation, `TypedefEngine` |
| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time validation, `TypedefEngine` |
## Applicable ADRs
@@ -28,6 +28,10 @@ format definition; the engine is generic.
| [096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` |
| [097](../../decisions/097-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations |
| [098](../../decisions/098-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors |
| [099](../../decisions/099-int64-uint64-first-class-kinds.md) | Int64/Uint64 as First-Class Kinds | 64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat |
| [100](../../decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode | Prevents silent data corruption (inline variable data clobbering subsequent fields) |
| [101](../../decisions/101-packed-mode-read-factory.md) | Packed-Mode Read API — Engine as SequentialReader Factory | `engine.sequential_reader()` returns an owned reader, not a reference |
| [102](../../decisions/102-reject-tunion-in-aligned-mode.md) | Reject TUnion in Aligned Mode for v1 | Unions are the protocol pattern; aligned-mode union semantics were broken |
## Relevant Open Questions

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef — Data Access
@@ -34,8 +34,8 @@ borrows from the input buffer for variable-length kinds (zero-copy).
```rust
pub enum FieldValue<'a> {
I8(i8), I16(i16), I32(i32),
U8(u8), U16(u16), U32(u32),
I8(i8), I16(i16), I32(i32), I64(i64),
U8(u8), U16(u16), U32(u32), U64(u64),
F32(f32), F64(f64),
Bool(bool),
Enum(u32), // u32 index into the schema's "enum" array
@@ -78,6 +78,11 @@ impl TypedefEngine {
-> Result<FieldValue<'a>, TypedefError>;
pub fn write_field(&self, buffer: &mut [u8], field_path: &str,
value: &FieldValue<'_>) -> Result<(), TypedefError>;
// Packed mode: returns an owned fresh SequentialReader (ADR-101).
// Each call returns a new reader with the cursor at position 0.
// The consumer owns the reader and drives read_next/read_field/reset.
pub fn sequential_reader(&self) -> Option<SequentialReader>;
}
impl SequentialReader {

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef — Layout Engine
@@ -104,6 +104,18 @@ For offset indirection, the field is a struct `{offset: u32, length: u32}`
at a known position in the `OffsetMap`. The consumer reads the offset and
length, then slices the separate data region.
### Inline length-prefixing in aligned mode — non-final field restriction
Inline length-prefixed variable fields in aligned mode are only allowed
as the **last field** in their struct. A non-final inline
length-prefixed variable field is rejected at `OffsetMap::compute` time
with a `TypedefError::Offset` — the `OffsetMap` reserves only 4 bytes
(the length prefix), but `data_access::write_string` writes prefix +
data inline, which would clobber subsequent fields. Non-final variable
fields must use `maxLength` (fixed-size reservation) or
`"encoding": "offset-indirect"`. See
[ADR-100](../../decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md).
## Offset Computation Algorithm
The offset computation is a recursive walk of the schema JSON. The
@@ -127,25 +139,27 @@ size is the sum of its fields' sizes (plus alignment padding in aligned
mode). The struct itself may have an `align` annotation that rounds up
its total size.
**`TUnion`:** The discriminator occupies `offset..offset + discriminator_size`
bytes. For byte-offset discriminators, the variant struct starts at
`offset + discriminator_size`. For field-name discriminators, the
discriminator is just another field — its offset is computed like any
other field, and the variant struct follows at the end of the
discriminator field.
**`TUnion`:** TUnion is supported in packed sequential mode only. In
aligned static mode, `OffsetMap::compute` rejects `TUnion` fields with
`TypedefError::Offset` — see
[ADR-102](../../decisions/102-reject-tunion-in-aligned-mode.md). Unions
are the protocol dispatch pattern (SFTP type bytes, call protocol event
types); mmap-friendly formats use structs and arrays, not tagged unions.
In aligned static mode, the union's total size is `discriminator_size +
max(variant_sizes)`, where variant sizes are computed from the schema
(variable-length data lives outside the static layout).
In packed sequential mode, the discriminator occupies
`offset..offset + discriminator_size` bytes. For byte-offset
discriminators, the variant struct starts at `offset + discriminator_size`.
For field-name discriminators, the discriminator is just another field —
its offset is computed like any other field, and the variant struct
follows at the end of the discriminator field.
In packed sequential mode, variant sizes depend on the actual sizes of
variable-length fields within each variant, which aren't known at schema
time. The `LayoutBuilder` takes the actual variant discriminator value
and data sizes at write time, computes the size of the selected variant,
and uses that for the union's total size. The `SequentialReader` reads
the discriminator first, looks up the variant schema, then reads the
variant struct sequentially — it doesn't need to know the union's total
size upfront.
Variant sizes depend on the actual sizes of variable-length fields within
each variant, which aren't known at schema time. The `LayoutBuilder`
takes the actual variant discriminator value and data sizes at write time,
computes the size of the selected variant, and uses that for the union's
total size. The `SequentialReader` reads the discriminator first, looks
up the variant schema, then reads the variant struct sequentially — it
doesn't need to know the union's total size upfront.
**`TArray` of fixed-size elements:** Element stride = element size (plus
alignment padding in aligned mode). Element `i` starts at
@@ -233,9 +247,13 @@ describing a metatensor layout can be consumed by an `OffsetMap` (for
mmap access).
`TypedefEngine` exposes mode-appropriate accessors: `engine.offset_map()`
returns `Some` in aligned mode and `None` in packed mode;
`engine.layout_builder()` and `engine.sequential_reader()` return `Some`
in packed mode and `None` in aligned mode. See [validation.md](validation.md)
returns `Some(&OffsetMap)` in aligned mode and `None` in packed mode;
`engine.layout_builder()` returns `Some(&LayoutBuilder)` in packed mode
and `None` in aligned mode. `engine.sequential_reader()` returns
`Option<SequentialReader>` (an owned fresh reader, not a reference — the
reader has mutable cursor state that the consumer owns; see
[ADR-101](../../decisions/101-packed-mode-read-factory.md)) in packed
mode and `None` in aligned mode. See [validation.md](validation.md)
§"The TypedefEngine struct" for the engine API.
## Public Types
@@ -316,6 +334,9 @@ order (schema `properties` order, nested struct fields appearing inline).
|----------|-----|---------|
| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential for protocols; aligned static for mmap formats |
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness, alignment, encoding annotations that control layout behavior |
| Non-final inline variable fields | [ADR-100](../../decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Rejected in aligned mode (would clobber subsequent fields); use `maxLength` or `offset-indirect` |
| Packed-mode read factory | [ADR-101](../../decisions/101-packed-mode-read-factory.md) | `engine.sequential_reader()` returns an owned fresh reader, not a reference |
| TUnion in aligned mode | [ADR-102](../../decisions/102-reject-tunion-in-aligned-mode.md) | Rejected for v1 (broken semantics; no current consumer needs it) |
## Open Questions

View File

@@ -1,6 +1,6 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef — Overview
@@ -18,7 +18,8 @@ Component details are in the sibling documents.
`alknet-typedef` is a library crate that consumes JSON Schemas annotated
with `TypeDef:*` custom keywords (the same kinds defined in TypeBox's
`typedef.ts`) and produces three capabilities:
`typedef.ts`, plus `TypeDef:Bytes`, `TypeDef:Int64`, and `TypeDef:Uint64`
as alknet-typedef additions) and produces three capabilities:
1. **An offset map** — walks the schema, computes byte offsets for each
field based on type sizes, field order, and alignment.
@@ -153,7 +154,7 @@ These boundaries are decided in [ADR-095](../../decisions/095-alknet-typedef-pur
## Architecture (component pointers)
- **[schema-layer.md](schema-layer.md)** — the 17 `TypeDef:*` kinds,
- **[schema-layer.md](schema-layer.md)** — the 19 `TypeDef:*` kinds,
jsonschema custom keyword integration, TypeBox interop, schema
annotations (endianness, alignment, encoding, TUnion discriminators).
- **[layout-engine.md](layout-engine.md)** — offset computation, the two
@@ -163,7 +164,7 @@ These boundaries are decided in [ADR-095](../../decisions/095-alknet-typedef-pur
dispatch, field paths, zero-copy access for fixed-size types,
length-prefix reading for variable-length types.
- **[validation.md](validation.md)** — custom keyword validators for all
17 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time
19 `TypeDef:*` kinds, `TypedefError`, load-time vs access-time
validation, `TypedefEngine` as the compiled form of a schema.
## Design Decisions
@@ -174,6 +175,10 @@ These boundaries are decided in [ADR-095](../../decisions/095-alknet-typedef-pur
| Two layout modes | [ADR-096](../../decisions/096-two-layout-modes-packed-vs-aligned.md) | Packed sequential (`LayoutBuilder`/`SequentialReader`) for protocols; aligned static (`OffsetMap`) for mmap formats |
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Endianness (schema-level, default LE), alignment (struct + field-level), encoding (length-prefixed vs offset-indirect), TUnion discriminators (byte-offset vs field-name) |
| Error handling and validation | [ADR-098](../../decisions/098-error-handling-validation-strategy.md) | `TypedefError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
| Int64/Uint64 kinds | [ADR-099](../../decisions/099-int64-uint64-first-class-kinds.md) | 64-bit integers as first-class kinds (SFTP offsets, metatensor data_offsets) |
| Non-final inline variable fields | [ADR-100](../../decisions/100-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Rejected in aligned mode (would clobber subsequent fields) |
| Packed-mode read factory | [ADR-101](../../decisions/101-packed-mode-read-factory.md) | `engine.sequential_reader()` returns an owned fresh reader |
| TUnion in aligned mode | [ADR-102](../../decisions/102-reject-tunion-in-aligned-mode.md) | Rejected for v1 (broken semantics; no current consumer needs it) |
## Open Questions

View File

@@ -1,15 +1,16 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef — Schema Layer
The schema layer: the 17 `TypeDef:*` custom type kinds, their mapping to
The schema layer: the 19 `TypeDef:*` custom type kinds, their mapping to
Rust types and byte sizes, the `jsonschema` custom keyword integration,
TypeBox interop, and the concrete JSON shapes for schema-level annotations.
TypeBox interop, and the concrete JSON shapes for schema-level
annotations.
## The 17 TypeDef Kinds
## The 19 TypeDef Kinds
These are the custom schema kinds defined in TypeBox's `typedef.ts`
(`/workspace/@alkdev/typebox/example/typedef/typedef.ts`, 619 lines) and
@@ -24,9 +25,11 @@ encoding strategy (for variable-length types).
| `TInt8` | `TypeDef:Int8` | `i8` | 1 | fixed |
| `TInt16` | `TypeDef:Int16` | `i16` | 2 | fixed |
| `TInt32` | `TypeDef:Int32` | `i32` | 4 | fixed |
| `TInt64` | `TypeDef:Int64` | `i64` | 8 | fixed |
| `TUint8` | `TypeDef:Uint8` | `u8` | 1 | fixed |
| `TUint16` | `TypeDef:Uint16` | `u16` | 2 | fixed |
| `TUint32` | `TypeDef:Uint32` | `u32` | 4 | fixed |
| `TUint64` | `TypeDef:Uint64` | `u64` | 8 | fixed |
| `TBoolean` | `TypeDef:Boolean` | `bool` (0x00=false, 0x01=true) | 1 | fixed |
| `TString` | `TypeDef:String` | length-prefixed UTF-8 | variable | variable |
| `TBytes` | `TypeDef:Bytes` | length-prefixed raw bytes | variable | variable |
@@ -37,9 +40,15 @@ encoding strategy (for variable-length types).
| `TRecord` | `TypeDef:Record` | count-prefixed sequence of (key, value) pairs | variable | variable |
| `TTimestamp` | `TypeDef:Timestamp` | length-prefixed RFC 3339 string | variable | variable |
`TypeDef:Int64` and `TypeDef:Uint64` are alknet-typedef additions —
TypeBox's `typedef.ts` tops out at 32-bit integers. They are required by
the primary POC targets: SFTP `Read`/`Write` packets have `offset: u64`,
and metatensor `data_offsets` are `u64`. See
[ADR-099](../../decisions/099-int64-uint64-first-class-kinds.md).
### The `TypeDefKind` enum
The engine represents the 17 kinds as a Rust enum — `TypeDefKind` — with
The engine represents the 19 kinds as a Rust enum — `TypeDefKind` — with
one variant per kind (`TypeDefKind::Float32`, `TypeDefKind::Struct`, etc.).
The enum provides compile-time exhaustiveness checking and integer
discriminant dispatch (a jump table) instead of string comparison at
@@ -47,8 +56,8 @@ every field access. It is `pub` and re-exported from the crate root.
```rust
pub enum TypeDefKind {
Int8, Int16, Int32,
Uint8, Uint16, Uint32,
Int8, Int16, Int32, Int64,
Uint8, Uint16, Uint32, Uint64,
Float32, Float64,
Boolean, Enum,
String, Bytes, Timestamp,
@@ -62,8 +71,8 @@ The enum carries the kind's binary-layout metadata as inherent methods:
|--------|---------|-------|
| `as_str(self)` | `&'static str` | The JSON Schema keyword, e.g. `"TypeDef:Uint8"` |
| `type_size(self)` | `Option<usize>` | `Some(N)` for fixed-size kinds; `None` for variable/composite |
| `natural_alignment(self)` | `usize` | 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/enum, 8 for f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array |
| `is_fixed_size(self)` | `bool` | True for the 10 fixed-size primitive kinds |
| `natural_alignment(self)` | `usize` | 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix), 1 for struct/union/array |
| `is_fixed_size(self)` | `bool` | True for the 12 fixed-size primitive kinds |
| `is_composite(self)` | `bool` | True for Struct, Union, Array, Record |
| `is_variable_length(self)` | `bool` | True for String, Bytes, Timestamp, Record |
| `needs_endian(self)` | `bool` | True for kinds whose read/write takes an `Endian` parameter |
@@ -382,8 +391,8 @@ Both struct-level and field-level, with field-level overriding:
- Struct-level `"align"` sets the default for all fields.
- Field-level `"align"` overrides the struct default.
- Default alignment: 1 for u8/i8/bool, 2 for u16/i16, 4 for u32/i32/f32/
enum, 8 for f64, 4 for variable-length (the u32 length prefix), 1 for
struct/union/array.
enum, 8 for u64/i64/f64, 4 for variable-length (the u32 length prefix),
1 for struct/union/array.
- Only meaningful in aligned static mode (ADR-096). Ignored in packed
sequential mode.
@@ -478,6 +487,7 @@ Mapping values may be either inline schemas or `$ref` pointers. Both work.
| Decision | ADR | Summary |
|----------|-----|---------|
| Schema annotations | [ADR-097](../../decisions/097-schema-annotations.md) | Concrete JSON shapes for endianness, alignment, encoding, and TUnion discriminators |
| Int64/Uint64 kinds | [ADR-099](../../decisions/099-int64-uint64-first-class-kinds.md) | 64-bit integers as first-class kinds (required by SFTP offsets and metatensor data_offsets) |
| Purpose and scope | [ADR-095](../../decisions/095-alknet-typedef-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine; "schema is the format" principle |
## Open Questions

View File

@@ -1,11 +1,11 @@
---
status: draft
last_updated: 2026-07-21
last_updated: 2026-07-22
---
# alknet-typedef — Validation
The validation layer: custom keyword validators for all 17 `TypeDef:*`
The validation layer: custom keyword validators for all 19 `TypeDef:*`
kinds, the `TypedefError` enum, load-time vs access-time validation
strategy, and the `TypedefEngine` as the compiled form of a schema.
@@ -62,7 +62,7 @@ pub struct TypedefEngine {
// Private — the consumer selects via LayoutMode at compile time.
enum Layout {
Packed { builder: LayoutBuilder, reader: SequentialReader },
Packed { builder: LayoutBuilder },
Aligned { offset_map: OffsetMap },
}
```
@@ -78,15 +78,21 @@ impl TypedefEngine {
pub fn endian(&self) -> Endian;
pub fn offset_map(&self) -> Option<&OffsetMap>; // Some in aligned mode
pub fn layout_builder(&self) -> Option<&LayoutBuilder>; // Some in packed mode
pub fn sequential_reader(&self) -> Option<&SequentialReader>; // Some in packed mode
pub fn sequential_reader(&self) -> Option<SequentialReader>; // owned fresh reader (ADR-101)
}
```
`compile` takes `&mut Value` because it normalizes `$ref` values in place
(via [`normalize_refs`](schema-layer.md#ref-resolution-and-normalization))
before computing the layout and building the validator. The `schema`
field retains the normalized schema for `read_field`'s kind lookup. The
validator is mode-agnostic (it operates on `Value`, not raw bytes).
field retains the normalized schema for `read_field`'s kind lookup and
for `sequential_reader()`'s factory construction. The validator is
mode-agnostic (it operates on `Value`, not raw bytes).
The `Layout::Packed` variant stores only the `LayoutBuilder` (write-side).
The `SequentialReader` (read-side) is not stored — it has mutable cursor
state that the consumer owns, so `sequential_reader()` constructs a fresh
reader on each call (ADR-101).
The `read_field`/`write_field` methods on `TypedefEngine` are the
aligned-mode data-access API — see [data-access.md](data-access.md)