docs(typedef): ADRs 099-102 — Int64/Uint64, aligned-mode restrictions, read factory

ADR-099: Int64/Uint64 as first-class kinds. The POC targets (SFTP
offset: u64, metatensor data_offsets: u64) require 64-bit integers.
The prior Uint64 addition was removed because it was half-finished;
this ADR specifies the complete addition (layout + validator + API).

ADR-100: Reject non-final inline length-prefixed variable fields in
aligned mode. The OffsetMap reserves only 4 bytes (the length prefix),
but write_string writes prefix+data inline — clobbering subsequent
fields. Non-final variable fields must use maxLength or offset-indirect.

ADR-101: Packed-mode read API — engine as SequentialReader factory.
engine.sequential_reader() returned &SequentialReader but read_next
needs &mut self — dead API. Now returns an owned fresh reader.

ADR-102: Reject TUnion in aligned mode for v1. The aligned-mode union
code had three bugs (no variant offsets, first-variant discriminator
offset, misaligned variant region). Unions are the protocol pattern;
mmap formats use structs/arrays. Reversible when a consumer needs it.
This commit is contained in:
glm-5.2 committed 2026-07-22 09:36:55 +00:00
1 parent 7603f98799
commit c819d99f1d
4 files changed
+440

No files matched your search

@@ -0,0 +1,114 @@
# ADR-099: Int64/Uint64 as First-Class Kinds
## Status
Accepted
## Context
The typedef engine's kind set (ADR-095, ADR-097) tops out at 32-bit
integers. The POC included `u64` read/write primitives, and the
call-channels-unification research's own SFTP schema example uses
`"TypeDef:Uint64"` for the `offset` field (`Read`/`Write` packets have
`offset: u64`). Metatensor/safetensors `data_offsets` are also `u64`.
A `TypeDef:Uint64` variant was added to the `TypeDefKind` enum during
implementation (the task decomposition correctly identified the gap),
but without an ADR the addition was half-finished: `type_size()`
returned `None`, the layout engines couldn't compute offsets for it, and
the validator didn't register a `TypeDef:Uint64` keyword. The variant
was then removed (commit `14d9cf2`) on the grounds that it was
unintended and a latent panic — but the underlying gap is real: SFTP and
metatensor, the two primary POC targets, both require 64-bit integers.
The presumed reason 64-bit integers were left out of the original
specification is a JSON-level concern: `serde_json::Number` loses
precision past 2^53 when parsing from JSON text. This is a
*validation-layer* caveat, not a *layout-layer* one — the binary layout
is 8 raw bytes, and `from_le_bytes`/`from_be_bytes` work correctly for
the full `u64`/`i64` range. The validation concern is handled by
accepting integer-form JSON values (the `jsonschema` crate's
`as_i64`/`as_u64` methods handle the common range; values past 2^53 are
a JSON representation limitation, not a typedef limitation).
## Decision
**Add `TypeDef:Int64` and `TypeDef:Uint64` as first-class kinds.**
Both are fixed-size (8 bytes), with natural alignment 8. They follow
the schema's endianness annotation like all other fixed-size types.
Read/write is via `data_access::read_i64`/`write_i64`/`read_u64`/
`write_u64` (endian-aware, 8 bytes).
### Kind table additions
| Kind | TypeBox key | Rust type | Size | Alignment |
|------|-------------|-----------|------|-----------|
| `TInt64` | `TypeDef:Int64` | `i64` | 8 | 8 |
| `TUint64` | `TypeDef:Uint64` | `u64` | 8 | 8 |
### Validation
The custom keyword validators check:
- `TypeDef:Int64`: value must be an integer in `i64::MIN..=i64::MAX`
(`-9223372036854775808` to `9223372036854775807`).
- `TypeDef:Uint64`: value must be a non-negative integer in
`0..=u64::MAX` (`0` to `18446744073709551615`).
The `jsonschema` crate's `as_i64`/`as_u64` handle the common range.
JSON numbers past 2^53 lose precision in the JSON representation —
this is a JSON limitation, not a typedef limitation. The binary
representation (8 raw bytes) is always exact. A consumer that needs
to validate the full 64-bit range from JSON should provide the value
as a JSON integer (which `serde_json` preserves for values up to
`u64::MAX`/`i64::MIN` when the `arbitrary_precision` feature is
enabled, or when the value fits in `i64`/`u64` without the feature).
### `FieldValue` additions
`FieldValue::I64(i64)` and `FieldValue::U64(u64)` are added to the
unified return type. The `SequentialReader`, `TypedefEngine::read_field`,
and `TypedefEngine::write_field` dispatch on the new kinds.
### Kind count
The engine now has **19** first-class kinds (17 + Int64 + Uint64).
`TypeDefKind::is_fixed_size()` returns `true` for both new kinds.
`type_size()` returns `Some(8)`. `natural_alignment()` returns `8`.
`needs_endian()` returns `true`.
## Consequences
### Positive
- **Unblocks the two primary POC targets.** SFTP `Read`/`Write` packets
(`offset: u64`) and metatensor `data_offsets` (`u64`) are now
expressible in typedef schemas.
- **Completes the half-finished addition.** The `TypeDefKind` enum,
`data_access` primitives, and `FieldValue` variants for 64-bit
integers now have matching layout, validator, and engine support.
- **No new design surface.** Int64/Uint64 are fixed-size types that
follow all existing patterns (endianness, alignment, zero-copy
read/write). They are mechanical additions.
### Negative
- **JSON precision caveat.** Values past 2^53 lose precision in the
JSON representation (not in the binary representation). This is a
JSON limitation, not a typedef limitation, but it means the
validation layer cannot perfectly round-trip the full 64-bit range
through JSON `Number` without `arbitrary_precision`. In practice,
SFTP offsets and tensor data offsets are well within 2^53.
- **Two more kinds to maintain.** The kind table, validator
registration, `FieldValue` enum, and dispatch arms all grow by two
variants. This is the cost of completeness.
## References
- `docs/research/call-channels-unification/findings.md` §"russh-sftp" —
the SFTP schema with `"offset": { "TypeDef:Uint64": true }`
- `docs/research/alknet-typedef/findings.md` §"POC 1" — the POC included
u64 read/write
- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) —
purpose and scope (the kind set)
- [ADR-097](097-schema-annotations.md) — schema annotations
(endianness applies to the new kinds)
@@ -0,0 +1,112 @@
# ADR-100: Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode
## Status
Accepted
## Context
The aligned static layout mode (ADR-096) is designed for mmap-friendly
formats: fields have fixed positions with natural alignment padding,
enabling random access by field path without parsing preceding fields.
The spec (layout-engine.md) says variable-length fields in aligned mode
get a 4-byte length prefix at a known offset, and "the variable data
lives outside the static layout — either immediately after the fixed
fields (inline length-prefixing) or in a separate data region (offset
indirection)."
The implementation has a bug: `OffsetMap::compute` reserves only 4 bytes
for an inline length-prefixed variable field (the length prefix), but
`TypedefEngine::write_field` for a `String`/`Bytes` field calls
`data_access::write_string` at `range.start`, which writes
`[4-byte length][data]` inline — clobbering every subsequent field. The
`read_field` path has the mirror behavior (reads inline), so the engine
is self-consistent but only works correctly when the variable field is
the last field in the struct (no subsequent field to clobber).
Concretely, `{name: String, id: Uint32}` in aligned mode maps
`name → 0..4`, `id → 4..8`. Writing `"hello"` to `name` writes
`[5,0,0,0,h,e,l,l,o]` at offset 0, overwriting `id`'s range with
`hello`. All existing tests happen to put the variable field last, so
the bug is latent.
The spec's "data region after fixed fields" model (where variable data
lives after all fixed fields) is the correct design for aligned mode,
but implementing it would require a two-region layout (fixed fields +
variable data region) with the `OffsetMap` tracking both the prefix
position and the data position. This is a significant design addition
for a use case that doesn't exist yet — real aligned-format consumers
(metatensor, safetensors) use `maxLength` reservation or
`offset-indirect` encoding for variable data, not inline
length-prefixing.
## Decision
**Reject non-final inline length-prefixed variable fields in aligned
static mode at `OffsetMap::compute` time.**
A variable-length field (`TypeDef:String`, `TypeDef:Bytes`,
`TypeDef:Timestamp`, `TypeDef:Record`) in aligned static mode that uses
the default inline length-prefixing strategy (no `maxLength`, no
`offset-indirect`) must be the last field in its struct. If a non-final
inline length-prefixed variable field is encountered,
`OffsetMap::compute` returns `TypedefError::Offset` with a message
explaining that non-final variable fields in aligned mode require
`maxLength` (fixed-size reservation) or `"encoding": "offset-indirect"`
(offset indirection).
This is a validation-time rejection (schema load time), not a runtime
check. The consumer learns about the problem when compiling the schema,
not when writing data.
### What is NOT rejected
- Inline length-prefixed variable fields that are the last field in
their struct — these are fine (no subsequent field to clobber).
- `maxLength` reservation and `offset-indirect` encoding in any
position — these make the field fixed-size from the layout
perspective (known size at a known offset), so they don't clobber.
- Inline length-prefixed variable fields in packed sequential mode —
packed mode doesn't have fixed offsets; variable fields shift
subsequent fields by design.
## Consequences
### Positive
- **Eliminates a silent data-corruption bug.** A consumer that writes
a non-final string in aligned mode currently clobbers subsequent
fields with no error. After this fix, the schema is rejected at
compile time.
- **Matches real aligned-format usage.** mmap-friendly formats use
`maxLength` or `offset-indirect` for variable data; inline
length-prefixing in aligned mode is only meaningful as the last
field.
- **Simple to implement.** A single check in `compute_struct` (is this
variable field non-final and using inline length-prefixing? → reject).
No two-region layout needed.
- **Defers the two-region design without blocking consumers.** If a
future consumer needs inline length-prefixing in non-final position
in aligned mode, the two-region layout can be implemented then. The
rejection is reversible (remove the check, add the two-region logic).
### Negative
- **A schema that worked before (silently corrupting data) now fails
at compile time.** This is the correct behavior — the schema was
always broken, it just wasn't caught.
- **The "data region after fixed fields" model from the spec is not
implemented.** A consumer that wants inline variable data in a
non-final position must use packed mode or wait for the two-region
layout. This is acceptable for v1 — no current consumer needs it.
## References
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout
modes (aligned static mode's variable-length handling)
- [ADR-097](097-schema-annotations.md) — the three variable-length
encoding strategies (`maxLength`, `offset-indirect`, inline
length-prefixing)
- `docs/architecture/crates/typedef/layout-engine.md` §"Variable-length
fields in aligned mode" — the spec's "data region after fixed fields"
description
@@ -0,0 +1,103 @@
# ADR-101: Packed-Mode Read API — Engine as SequentialReader Factory
## Status
Accepted
## Context
`TypedefEngine` stores a `SequentialReader` inside its `Layout::Packed`
variant. The engine exposes it via
`engine.sequential_reader() -> Option<&SequentialReader>`.
The problem: `SequentialReader`'s read methods (`read_next`,
`read_field`, `reset`) all take `&mut self` — they mutate the reader's
internal cursor (`field_index`, `position`). But the engine hands out
`&SequentialReader` (a shared reference), which cannot be used to call
`&mut self` methods. The accessor can only give the consumer
`position()` and `endian()` (the `&self` methods) — the actual read
API is unreachable.
This makes the engine's packed read-side dead API. A consumer that
wants to read a packed buffer must construct their own
`SequentialReader::new(&schema)` from the schema, bypassing the engine
entirely. The stored reader is dead weight.
Three options were considered:
1. **Factory method** — the engine provides a method that returns an
owned fresh `SequentialReader` (reconstructed from the stored
schema). The consumer owns the reader and drives it with `&mut self`.
2. **Interior mutability** — wrap the reader in `Mutex` or `RefCell`
so `&SequentialReader` can be upgraded to `&mut`. Adds overhead and
complexity for mutable cursor state that the consumer legitimately
wants to own.
3. **`sequential_reader_mut()`** — return `&mut SequentialReader`.
Requires `&mut self` on the engine, which is overly restrictive
(the consumer may share the engine across threads or hold it behind
an `Arc`).
## Decision
**The engine is a `SequentialReader` factory.** Replace
`sequential_reader() -> Option<&SequentialReader>` with
`sequential_reader() -> Option<SequentialReader>` — the method returns
an owned fresh reader, reconstructed from the stored schema.
```rust
impl TypedefEngine {
/// Construct a fresh SequentialReader for packed-mode reads.
/// Returns None if compiled in aligned mode.
pub fn sequential_reader(&self) -> Option<SequentialReader>;
}
```
Each call returns a new reader with the cursor at position 0. The
consumer owns the reader and calls `read_next`/`read_field`/`reset` on
it directly. The engine still stores its own reader (used for schema
validation during construction), but no longer exposes it by
reference.
The same applies to `LayoutBuilder`: `layout_builder()` returns
`Option<&LayoutBuilder>` which is fine — `LayoutBuilder::build` takes
`&self`, so the shared reference is usable. No change needed for the
write-side.
### Cost
`SequentialReader::new` clones the top-level struct's field schemas (a
`Vec<(String, Value)>` of the `properties` entries) and clones the
schema itself. This is cheap — a struct has a small number of fields
(SFTP's largest packet has 5). The construction cost is negligible
compared to the cost of reading a buffer.
## Consequences
### Positive
- **The packed read API is now usable.** A consumer calls
`engine.sequential_reader()` to get an owned reader and drives it
directly. No dead API.
- **No interior mutability overhead.** The reader's mutable cursor
state is owned by the consumer, not shared through a lock.
- **Thread-safe engine.** The engine remains `Send + Sync` (it only
exposes `&self` methods). The reader is owned by the calling thread.
- **Simple.** One method signature change. The stored reader in
`Layout::Packed` can be removed (it was only used for schema
validation during construction, which is done by the time the
consumer calls `sequential_reader()`).
### Negative
- **Each call to `sequential_reader()` allocates a new reader.** The
cost is a `Vec` of field schemas + a schema clone. Acceptable for
the use case (one reader per buffer read).
- **The engine no longer holds a live reader.** If a future use case
needs to share a reader's cursor state across calls, the consumer
must manage that themselves. This is the correct separation — cursor
state is consumer-owned, not engine-owned.
## References
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — packed
sequential mode (`SequentialReader` as the read-side)
- `docs/architecture/crates/typedef/data-access.md` §"Higher-level
read/write" — the `SequentialReader` API
@@ -0,0 +1,111 @@
# ADR-102: Reject TUnion in Aligned Mode for v1
## Status
Accepted
## Context
The aligned static layout mode (ADR-096) computes fixed byte positions
for each field, enabling random access by field path. `TUnion` in
aligned mode has three implementation problems:
1. **No variant field offsets.** Only the `__discriminator` byte range
is recorded in the `OffsetMap`. Variant field offsets are not
available anywhere in aligned mode — the consumer must recompute
them by hand. This makes `TypedefEngine::read_field` on a union
variant field impossible.
2. **`find_discriminator_field` takes the first variant's offset.** For
a field-name discriminator, the code probes the first variant that
contains the discriminator field and records that offset globally.
If variants order fields differently, the discriminator sits at
different offsets per variant and the recorded range is silently
wrong. The code should validate that the offset is identical across
all variants (or require the discriminator field to be first).
3. **Byte-discriminator union total misaligns the variant.** The union
total is `disc_off + disc_size + variant_max_size`, but the variant
was probed from offset 0 with alignment. A `u8` discriminator
before a `u32`-bearing variant produces a variant region that
starts at an unaligned offset in a mode whose entire purpose is
alignment.
The real question is whether `TUnion` in aligned mode is even needed.
The two consumer profiles are:
- **Protocol consumers** (SFTP, call protocol event types): use packed
sequential mode. `TUnion` with byte-offset discriminators is the
core dispatch mechanism. This is well-supported.
- **mmap consumers** (metatensor, safetensors): use aligned static
mode. These formats are structs and arrays of structs — they don't
use tagged unions. A tensor file has a header struct with tensor
descriptors, not a "which variant is this?" dispatch.
`TUnion` in aligned mode is a combination that no current or planned
consumer needs. Shipping broken semantics for an unused use case is
worse than rejecting it clearly.
## Decision
**Reject `TUnion` in aligned static mode for v1.**
`OffsetMap::compute` returns `TypedefError::Offset` when it encounters
a `TypeDef:Union` field, with a message explaining that unions are not
supported in aligned mode and the consumer should use packed mode (or
restructure as a struct with an explicit discriminator field).
This is a schema-load-time rejection. The consumer learns about the
problem when compiling the schema, not at runtime.
### What is NOT rejected
- `TUnion` in packed sequential mode — this is the core use case
(SFTP `Packet` dispatch, call protocol event types) and is fully
supported by `LayoutBuilder` and `SequentialReader`.
- `TStruct`, `TArray`, and all primitive kinds in aligned mode — these
are the mmap-format primitives and are fully supported.
### Reversal
This is a two-way door. If a future mmap-format consumer needs tagged
unions in aligned mode, the rejection can be lifted and the three
implementation problems fixed. The fix would require:
- Recording per-variant field offsets in the `OffsetMap` (which
variant's offsets to record when variants have different layouts?).
- Validating that field-name discriminators have identical offsets
across all variants.
- Aligning the variant region correctly after the byte discriminator.
These are design questions that should be answered when the use case
arrives, not speculatively now.
## Consequences
### Positive
- **No broken semantics shipped.** The three implementation problems
are removed from the API surface rather than silently producing
wrong offsets.
- **Clear scope boundary.** Aligned mode is for structs and arrays;
packed mode is for protocols (including union dispatch). The
consumer chooses the mode based on the use case.
- **Reversible.** When a real consumer needs aligned-mode unions, the
rejection is lifted and the design questions are worked through with
a concrete use case.
### Negative
- **A schema with a `TUnion` field cannot be compiled in aligned
mode.** A consumer that wants both aligned layout and union dispatch
must use packed mode or restructure. No current consumer needs this.
- **The aligned-mode union code in `offset_map.rs` is dead.** It can
be removed or left as a reference for when the rejection is lifted.
Removing it is cleaner.
## References
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout
modes
- [ADR-097](097-schema-annotations.md) §4 — TUnion discriminators
- `docs/architecture/crates/typedef/layout-engine.md` §"TUnion" —
aligned-mode union sizing