docs(typedef): ADRs 099-102 — Int64/Uint64, aligned-mode restrictions, read factory
ADR-099: Int64/Uint64 as first-class kinds. The POC targets (SFTP offset: u64, metatensor data_offsets: u64) require 64-bit integers. The prior Uint64 addition was removed because it was half-finished; this ADR specifies the complete addition (layout + validator + API). ADR-100: Reject non-final inline length-prefixed variable fields in aligned mode. The OffsetMap reserves only 4 bytes (the length prefix), but write_string writes prefix+data inline — clobbering subsequent fields. Non-final variable fields must use maxLength or offset-indirect. ADR-101: Packed-mode read API — engine as SequentialReader factory. engine.sequential_reader() returned &SequentialReader but read_next needs &mut self — dead API. Now returns an owned fresh reader. ADR-102: Reject TUnion in aligned mode for v1. The aligned-mode union code had three bugs (no variant offsets, first-variant discriminator offset, misaligned variant region). Unions are the protocol pattern; mmap formats use structs/arrays. Reversible when a consumer needs it.
This commit is contained in:
1 parent
7603f98799
commit
c819d99f1d
4 files changed
+440
No files matched your search
@@ -0,0 +1,114 @@
|
||||
# ADR-099: Int64/Uint64 as First-Class Kinds
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The typedef engine's kind set (ADR-095, ADR-097) tops out at 32-bit
|
||||
integers. The POC included `u64` read/write primitives, and the
|
||||
call-channels-unification research's own SFTP schema example uses
|
||||
`"TypeDef:Uint64"` for the `offset` field (`Read`/`Write` packets have
|
||||
`offset: u64`). Metatensor/safetensors `data_offsets` are also `u64`.
|
||||
|
||||
A `TypeDef:Uint64` variant was added to the `TypeDefKind` enum during
|
||||
implementation (the task decomposition correctly identified the gap),
|
||||
but without an ADR the addition was half-finished: `type_size()`
|
||||
returned `None`, the layout engines couldn't compute offsets for it, and
|
||||
the validator didn't register a `TypeDef:Uint64` keyword. The variant
|
||||
was then removed (commit `14d9cf2`) on the grounds that it was
|
||||
unintended and a latent panic — but the underlying gap is real: SFTP and
|
||||
metatensor, the two primary POC targets, both require 64-bit integers.
|
||||
|
||||
The presumed reason 64-bit integers were left out of the original
|
||||
specification is a JSON-level concern: `serde_json::Number` loses
|
||||
precision past 2^53 when parsing from JSON text. This is a
|
||||
*validation-layer* caveat, not a *layout-layer* one — the binary layout
|
||||
is 8 raw bytes, and `from_le_bytes`/`from_be_bytes` work correctly for
|
||||
the full `u64`/`i64` range. The validation concern is handled by
|
||||
accepting integer-form JSON values (the `jsonschema` crate's
|
||||
`as_i64`/`as_u64` methods handle the common range; values past 2^53 are
|
||||
a JSON representation limitation, not a typedef limitation).
|
||||
|
||||
## Decision
|
||||
|
||||
**Add `TypeDef:Int64` and `TypeDef:Uint64` as first-class kinds.**
|
||||
|
||||
Both are fixed-size (8 bytes), with natural alignment 8. They follow
|
||||
the schema's endianness annotation like all other fixed-size types.
|
||||
Read/write is via `data_access::read_i64`/`write_i64`/`read_u64`/
|
||||
`write_u64` (endian-aware, 8 bytes).
|
||||
|
||||
### Kind table additions
|
||||
|
||||
| Kind | TypeBox key | Rust type | Size | Alignment |
|
||||
|------|-------------|-----------|------|-----------|
|
||||
| `TInt64` | `TypeDef:Int64` | `i64` | 8 | 8 |
|
||||
| `TUint64` | `TypeDef:Uint64` | `u64` | 8 | 8 |
|
||||
|
||||
### Validation
|
||||
|
||||
The custom keyword validators check:
|
||||
- `TypeDef:Int64`: value must be an integer in `i64::MIN..=i64::MAX`
|
||||
(`-9223372036854775808` to `9223372036854775807`).
|
||||
- `TypeDef:Uint64`: value must be a non-negative integer in
|
||||
`0..=u64::MAX` (`0` to `18446744073709551615`).
|
||||
|
||||
The `jsonschema` crate's `as_i64`/`as_u64` handle the common range.
|
||||
JSON numbers past 2^53 lose precision in the JSON representation —
|
||||
this is a JSON limitation, not a typedef limitation. The binary
|
||||
representation (8 raw bytes) is always exact. A consumer that needs
|
||||
to validate the full 64-bit range from JSON should provide the value
|
||||
as a JSON integer (which `serde_json` preserves for values up to
|
||||
`u64::MAX`/`i64::MIN` when the `arbitrary_precision` feature is
|
||||
enabled, or when the value fits in `i64`/`u64` without the feature).
|
||||
|
||||
### `FieldValue` additions
|
||||
|
||||
`FieldValue::I64(i64)` and `FieldValue::U64(u64)` are added to the
|
||||
unified return type. The `SequentialReader`, `TypedefEngine::read_field`,
|
||||
and `TypedefEngine::write_field` dispatch on the new kinds.
|
||||
|
||||
### Kind count
|
||||
|
||||
The engine now has **19** first-class kinds (17 + Int64 + Uint64).
|
||||
`TypeDefKind::is_fixed_size()` returns `true` for both new kinds.
|
||||
`type_size()` returns `Some(8)`. `natural_alignment()` returns `8`.
|
||||
`needs_endian()` returns `true`.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **Unblocks the two primary POC targets.** SFTP `Read`/`Write` packets
|
||||
(`offset: u64`) and metatensor `data_offsets` (`u64`) are now
|
||||
expressible in typedef schemas.
|
||||
- **Completes the half-finished addition.** The `TypeDefKind` enum,
|
||||
`data_access` primitives, and `FieldValue` variants for 64-bit
|
||||
integers now have matching layout, validator, and engine support.
|
||||
- **No new design surface.** Int64/Uint64 are fixed-size types that
|
||||
follow all existing patterns (endianness, alignment, zero-copy
|
||||
read/write). They are mechanical additions.
|
||||
|
||||
### Negative
|
||||
|
||||
- **JSON precision caveat.** Values past 2^53 lose precision in the
|
||||
JSON representation (not in the binary representation). This is a
|
||||
JSON limitation, not a typedef limitation, but it means the
|
||||
validation layer cannot perfectly round-trip the full 64-bit range
|
||||
through JSON `Number` without `arbitrary_precision`. In practice,
|
||||
SFTP offsets and tensor data offsets are well within 2^53.
|
||||
- **Two more kinds to maintain.** The kind table, validator
|
||||
registration, `FieldValue` enum, and dispatch arms all grow by two
|
||||
variants. This is the cost of completeness.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/call-channels-unification/findings.md` §"russh-sftp" —
|
||||
the SFTP schema with `"offset": { "TypeDef:Uint64": true }`
|
||||
- `docs/research/alknet-typedef/findings.md` §"POC 1" — the POC included
|
||||
u64 read/write
|
||||
- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) —
|
||||
purpose and scope (the kind set)
|
||||
- [ADR-097](097-schema-annotations.md) — schema annotations
|
||||
(endianness applies to the new kinds)
|
||||
+112
@@ -0,0 +1,112 @@
|
||||
# ADR-100: Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The aligned static layout mode (ADR-096) is designed for mmap-friendly
|
||||
formats: fields have fixed positions with natural alignment padding,
|
||||
enabling random access by field path without parsing preceding fields.
|
||||
|
||||
The spec (layout-engine.md) says variable-length fields in aligned mode
|
||||
get a 4-byte length prefix at a known offset, and "the variable data
|
||||
lives outside the static layout — either immediately after the fixed
|
||||
fields (inline length-prefixing) or in a separate data region (offset
|
||||
indirection)."
|
||||
|
||||
The implementation has a bug: `OffsetMap::compute` reserves only 4 bytes
|
||||
for an inline length-prefixed variable field (the length prefix), but
|
||||
`TypedefEngine::write_field` for a `String`/`Bytes` field calls
|
||||
`data_access::write_string` at `range.start`, which writes
|
||||
`[4-byte length][data]` inline — clobbering every subsequent field. The
|
||||
`read_field` path has the mirror behavior (reads inline), so the engine
|
||||
is self-consistent but only works correctly when the variable field is
|
||||
the last field in the struct (no subsequent field to clobber).
|
||||
|
||||
Concretely, `{name: String, id: Uint32}` in aligned mode maps
|
||||
`name → 0..4`, `id → 4..8`. Writing `"hello"` to `name` writes
|
||||
`[5,0,0,0,h,e,l,l,o]` at offset 0, overwriting `id`'s range with
|
||||
`hello`. All existing tests happen to put the variable field last, so
|
||||
the bug is latent.
|
||||
|
||||
The spec's "data region after fixed fields" model (where variable data
|
||||
lives after all fixed fields) is the correct design for aligned mode,
|
||||
but implementing it would require a two-region layout (fixed fields +
|
||||
variable data region) with the `OffsetMap` tracking both the prefix
|
||||
position and the data position. This is a significant design addition
|
||||
for a use case that doesn't exist yet — real aligned-format consumers
|
||||
(metatensor, safetensors) use `maxLength` reservation or
|
||||
`offset-indirect` encoding for variable data, not inline
|
||||
length-prefixing.
|
||||
|
||||
## Decision
|
||||
|
||||
**Reject non-final inline length-prefixed variable fields in aligned
|
||||
static mode at `OffsetMap::compute` time.**
|
||||
|
||||
A variable-length field (`TypeDef:String`, `TypeDef:Bytes`,
|
||||
`TypeDef:Timestamp`, `TypeDef:Record`) in aligned static mode that uses
|
||||
the default inline length-prefixing strategy (no `maxLength`, no
|
||||
`offset-indirect`) must be the last field in its struct. If a non-final
|
||||
inline length-prefixed variable field is encountered,
|
||||
`OffsetMap::compute` returns `TypedefError::Offset` with a message
|
||||
explaining that non-final variable fields in aligned mode require
|
||||
`maxLength` (fixed-size reservation) or `"encoding": "offset-indirect"`
|
||||
(offset indirection).
|
||||
|
||||
This is a validation-time rejection (schema load time), not a runtime
|
||||
check. The consumer learns about the problem when compiling the schema,
|
||||
not when writing data.
|
||||
|
||||
### What is NOT rejected
|
||||
|
||||
- Inline length-prefixed variable fields that are the last field in
|
||||
their struct — these are fine (no subsequent field to clobber).
|
||||
- `maxLength` reservation and `offset-indirect` encoding in any
|
||||
position — these make the field fixed-size from the layout
|
||||
perspective (known size at a known offset), so they don't clobber.
|
||||
- Inline length-prefixed variable fields in packed sequential mode —
|
||||
packed mode doesn't have fixed offsets; variable fields shift
|
||||
subsequent fields by design.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **Eliminates a silent data-corruption bug.** A consumer that writes
|
||||
a non-final string in aligned mode currently clobbers subsequent
|
||||
fields with no error. After this fix, the schema is rejected at
|
||||
compile time.
|
||||
- **Matches real aligned-format usage.** mmap-friendly formats use
|
||||
`maxLength` or `offset-indirect` for variable data; inline
|
||||
length-prefixing in aligned mode is only meaningful as the last
|
||||
field.
|
||||
- **Simple to implement.** A single check in `compute_struct` (is this
|
||||
variable field non-final and using inline length-prefixing? → reject).
|
||||
No two-region layout needed.
|
||||
- **Defers the two-region design without blocking consumers.** If a
|
||||
future consumer needs inline length-prefixing in non-final position
|
||||
in aligned mode, the two-region layout can be implemented then. The
|
||||
rejection is reversible (remove the check, add the two-region logic).
|
||||
|
||||
### Negative
|
||||
|
||||
- **A schema that worked before (silently corrupting data) now fails
|
||||
at compile time.** This is the correct behavior — the schema was
|
||||
always broken, it just wasn't caught.
|
||||
- **The "data region after fixed fields" model from the spec is not
|
||||
implemented.** A consumer that wants inline variable data in a
|
||||
non-final position must use packed mode or wait for the two-region
|
||||
layout. This is acceptable for v1 — no current consumer needs it.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout
|
||||
modes (aligned static mode's variable-length handling)
|
||||
- [ADR-097](097-schema-annotations.md) — the three variable-length
|
||||
encoding strategies (`maxLength`, `offset-indirect`, inline
|
||||
length-prefixing)
|
||||
- `docs/architecture/crates/typedef/layout-engine.md` §"Variable-length
|
||||
fields in aligned mode" — the spec's "data region after fixed fields"
|
||||
description
|
||||
@@ -0,0 +1,103 @@
|
||||
# ADR-101: Packed-Mode Read API — Engine as SequentialReader Factory
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
`TypedefEngine` stores a `SequentialReader` inside its `Layout::Packed`
|
||||
variant. The engine exposes it via
|
||||
`engine.sequential_reader() -> Option<&SequentialReader>`.
|
||||
|
||||
The problem: `SequentialReader`'s read methods (`read_next`,
|
||||
`read_field`, `reset`) all take `&mut self` — they mutate the reader's
|
||||
internal cursor (`field_index`, `position`). But the engine hands out
|
||||
`&SequentialReader` (a shared reference), which cannot be used to call
|
||||
`&mut self` methods. The accessor can only give the consumer
|
||||
`position()` and `endian()` (the `&self` methods) — the actual read
|
||||
API is unreachable.
|
||||
|
||||
This makes the engine's packed read-side dead API. A consumer that
|
||||
wants to read a packed buffer must construct their own
|
||||
`SequentialReader::new(&schema)` from the schema, bypassing the engine
|
||||
entirely. The stored reader is dead weight.
|
||||
|
||||
Three options were considered:
|
||||
1. **Factory method** — the engine provides a method that returns an
|
||||
owned fresh `SequentialReader` (reconstructed from the stored
|
||||
schema). The consumer owns the reader and drives it with `&mut self`.
|
||||
2. **Interior mutability** — wrap the reader in `Mutex` or `RefCell`
|
||||
so `&SequentialReader` can be upgraded to `&mut`. Adds overhead and
|
||||
complexity for mutable cursor state that the consumer legitimately
|
||||
wants to own.
|
||||
3. **`sequential_reader_mut()`** — return `&mut SequentialReader`.
|
||||
Requires `&mut self` on the engine, which is overly restrictive
|
||||
(the consumer may share the engine across threads or hold it behind
|
||||
an `Arc`).
|
||||
|
||||
## Decision
|
||||
|
||||
**The engine is a `SequentialReader` factory.** Replace
|
||||
`sequential_reader() -> Option<&SequentialReader>` with
|
||||
`sequential_reader() -> Option<SequentialReader>` — the method returns
|
||||
an owned fresh reader, reconstructed from the stored schema.
|
||||
|
||||
```rust
|
||||
impl TypedefEngine {
|
||||
/// Construct a fresh SequentialReader for packed-mode reads.
|
||||
/// Returns None if compiled in aligned mode.
|
||||
pub fn sequential_reader(&self) -> Option<SequentialReader>;
|
||||
}
|
||||
```
|
||||
|
||||
Each call returns a new reader with the cursor at position 0. The
|
||||
consumer owns the reader and calls `read_next`/`read_field`/`reset` on
|
||||
it directly. The engine still stores its own reader (used for schema
|
||||
validation during construction), but no longer exposes it by
|
||||
reference.
|
||||
|
||||
The same applies to `LayoutBuilder`: `layout_builder()` returns
|
||||
`Option<&LayoutBuilder>` which is fine — `LayoutBuilder::build` takes
|
||||
`&self`, so the shared reference is usable. No change needed for the
|
||||
write-side.
|
||||
|
||||
### Cost
|
||||
|
||||
`SequentialReader::new` clones the top-level struct's field schemas (a
|
||||
`Vec<(String, Value)>` of the `properties` entries) and clones the
|
||||
schema itself. This is cheap — a struct has a small number of fields
|
||||
(SFTP's largest packet has 5). The construction cost is negligible
|
||||
compared to the cost of reading a buffer.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **The packed read API is now usable.** A consumer calls
|
||||
`engine.sequential_reader()` to get an owned reader and drives it
|
||||
directly. No dead API.
|
||||
- **No interior mutability overhead.** The reader's mutable cursor
|
||||
state is owned by the consumer, not shared through a lock.
|
||||
- **Thread-safe engine.** The engine remains `Send + Sync` (it only
|
||||
exposes `&self` methods). The reader is owned by the calling thread.
|
||||
- **Simple.** One method signature change. The stored reader in
|
||||
`Layout::Packed` can be removed (it was only used for schema
|
||||
validation during construction, which is done by the time the
|
||||
consumer calls `sequential_reader()`).
|
||||
|
||||
### Negative
|
||||
|
||||
- **Each call to `sequential_reader()` allocates a new reader.** The
|
||||
cost is a `Vec` of field schemas + a schema clone. Acceptable for
|
||||
the use case (one reader per buffer read).
|
||||
- **The engine no longer holds a live reader.** If a future use case
|
||||
needs to share a reader's cursor state across calls, the consumer
|
||||
must manage that themselves. This is the correct separation — cursor
|
||||
state is consumer-owned, not engine-owned.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — packed
|
||||
sequential mode (`SequentialReader` as the read-side)
|
||||
- `docs/architecture/crates/typedef/data-access.md` §"Higher-level
|
||||
read/write" — the `SequentialReader` API
|
||||
@@ -0,0 +1,111 @@
|
||||
# ADR-102: Reject TUnion in Aligned Mode for v1
|
||||
|
||||
## Status
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
The aligned static layout mode (ADR-096) computes fixed byte positions
|
||||
for each field, enabling random access by field path. `TUnion` in
|
||||
aligned mode has three implementation problems:
|
||||
|
||||
1. **No variant field offsets.** Only the `__discriminator` byte range
|
||||
is recorded in the `OffsetMap`. Variant field offsets are not
|
||||
available anywhere in aligned mode — the consumer must recompute
|
||||
them by hand. This makes `TypedefEngine::read_field` on a union
|
||||
variant field impossible.
|
||||
|
||||
2. **`find_discriminator_field` takes the first variant's offset.** For
|
||||
a field-name discriminator, the code probes the first variant that
|
||||
contains the discriminator field and records that offset globally.
|
||||
If variants order fields differently, the discriminator sits at
|
||||
different offsets per variant and the recorded range is silently
|
||||
wrong. The code should validate that the offset is identical across
|
||||
all variants (or require the discriminator field to be first).
|
||||
|
||||
3. **Byte-discriminator union total misaligns the variant.** The union
|
||||
total is `disc_off + disc_size + variant_max_size`, but the variant
|
||||
was probed from offset 0 with alignment. A `u8` discriminator
|
||||
before a `u32`-bearing variant produces a variant region that
|
||||
starts at an unaligned offset in a mode whose entire purpose is
|
||||
alignment.
|
||||
|
||||
The real question is whether `TUnion` in aligned mode is even needed.
|
||||
The two consumer profiles are:
|
||||
|
||||
- **Protocol consumers** (SFTP, call protocol event types): use packed
|
||||
sequential mode. `TUnion` with byte-offset discriminators is the
|
||||
core dispatch mechanism. This is well-supported.
|
||||
- **mmap consumers** (metatensor, safetensors): use aligned static
|
||||
mode. These formats are structs and arrays of structs — they don't
|
||||
use tagged unions. A tensor file has a header struct with tensor
|
||||
descriptors, not a "which variant is this?" dispatch.
|
||||
|
||||
`TUnion` in aligned mode is a combination that no current or planned
|
||||
consumer needs. Shipping broken semantics for an unused use case is
|
||||
worse than rejecting it clearly.
|
||||
|
||||
## Decision
|
||||
|
||||
**Reject `TUnion` in aligned static mode for v1.**
|
||||
|
||||
`OffsetMap::compute` returns `TypedefError::Offset` when it encounters
|
||||
a `TypeDef:Union` field, with a message explaining that unions are not
|
||||
supported in aligned mode and the consumer should use packed mode (or
|
||||
restructure as a struct with an explicit discriminator field).
|
||||
|
||||
This is a schema-load-time rejection. The consumer learns about the
|
||||
problem when compiling the schema, not at runtime.
|
||||
|
||||
### What is NOT rejected
|
||||
|
||||
- `TUnion` in packed sequential mode — this is the core use case
|
||||
(SFTP `Packet` dispatch, call protocol event types) and is fully
|
||||
supported by `LayoutBuilder` and `SequentialReader`.
|
||||
- `TStruct`, `TArray`, and all primitive kinds in aligned mode — these
|
||||
are the mmap-format primitives and are fully supported.
|
||||
|
||||
### Reversal
|
||||
|
||||
This is a two-way door. If a future mmap-format consumer needs tagged
|
||||
unions in aligned mode, the rejection can be lifted and the three
|
||||
implementation problems fixed. The fix would require:
|
||||
- Recording per-variant field offsets in the `OffsetMap` (which
|
||||
variant's offsets to record when variants have different layouts?).
|
||||
- Validating that field-name discriminators have identical offsets
|
||||
across all variants.
|
||||
- Aligning the variant region correctly after the byte discriminator.
|
||||
|
||||
These are design questions that should be answered when the use case
|
||||
arrives, not speculatively now.
|
||||
|
||||
## Consequences
|
||||
|
||||
### Positive
|
||||
|
||||
- **No broken semantics shipped.** The three implementation problems
|
||||
are removed from the API surface rather than silently producing
|
||||
wrong offsets.
|
||||
- **Clear scope boundary.** Aligned mode is for structs and arrays;
|
||||
packed mode is for protocols (including union dispatch). The
|
||||
consumer chooses the mode based on the use case.
|
||||
- **Reversible.** When a real consumer needs aligned-mode unions, the
|
||||
rejection is lifted and the design questions are worked through with
|
||||
a concrete use case.
|
||||
|
||||
### Negative
|
||||
|
||||
- **A schema with a `TUnion` field cannot be compiled in aligned
|
||||
mode.** A consumer that wants both aligned layout and union dispatch
|
||||
must use packed mode or restructure. No current consumer needs this.
|
||||
- **The aligned-mode union code in `offset_map.rs` is dead.** It can
|
||||
be removed or left as a reference for when the rejection is lifted.
|
||||
Removing it is cleaner.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout
|
||||
modes
|
||||
- [ADR-097](097-schema-annotations.md) §4 — TUnion discriminators
|
||||
- `docs/architecture/crates/typedef/layout-engine.md` §"TUnion" —
|
||||
aligned-mode union sizing
|
||||
Reference in new issue
Block a user