glm-5.2
7603f98799
The code introduced concrete public types and a unified API during implementation that the specs described only conceptually. Sync the specs to match the code: - schema-layer.md: document the TypeDefKind enum and its inherent methods; add a Schema-Layer Public API section (get_typedef_kind vs get_typedef_kind_loose, annotation parsers, Endian/VariableEncoding/ DiscriminatorKind, normalize_refs/resolve_ref/resolve_ref_or_inline); fix the TRecord layout (values are encoded by their declared kind, not universally value_len-prefixed); fix the alignment default list. - layout-engine.md: document the LayoutMode enum and the ByteRange/ FieldPosition/PackedLayout/OffsetMap public types with their actual signatures; update the LayoutBuilder/SequentialReader/OffsetMap component descriptions with the real new/build/compute signatures. - data-access.md: document the FieldValue enum; add a Higher-level read/write section (TypedefEngine::read_field/write_field, SequentialReader::read_next/read_field); fix primitive signatures to include field_path and Endian; replace the wrong read_union_discriminator pseudo-code with the actual tunion module API (read_byte_discriminator/read_field_discriminator/resolve_variant/ discriminator_size) and the UnionDispatch struct. - validation.md: fix the TypedefEngine struct (add endian/schema fields, mark Layout as private); add the real compile signature (&mut Value, LayoutMode) and mode-appropriate accessors; fix engine.validate(buffer) -> engine.validate_json(&Value)/is_valid_json (the validator operates on serde_json::Value, not byte buffers — matches ADR-098). - overview.md: remove the stale ~1,900 lines / 26 tests line count. - ADR-097 §3a: correct the TRecord layout (no separate value_len prefix; the value is encoded by its declared TypeDef:* kind).
260 lines
9.6 KiB
Markdown
260 lines
9.6 KiB
Markdown
# ADR-097: Schema Annotations — Endianness, Alignment, Encoding, and TUnion Discriminators
|
|
|
|
## Status
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
The typedef engine needs concrete JSON shapes for schema-level
|
|
annotations that control binary layout behavior. The POCs validated the
|
|
semantics; this ADR pins the shapes.
|
|
|
|
Four annotation categories need concrete shapes:
|
|
|
|
1. **Endianness** — safetensors is little-endian, SFTP is big-endian.
|
|
The engine needs to know which to use.
|
|
2. **Alignment** — different backends have different alignment
|
|
requirements (wgpu: 256-byte, protocols: natural, mmap: page).
|
|
3. **Variable-length encoding** — inline length-prefixing vs offset
|
|
indirection for strings, byte arrays, and other variable-length types.
|
|
4. **TUnion discriminators** — byte-offset (protocol dispatch) vs
|
|
field-name (typedef.ts pattern).
|
|
|
|
## Decision
|
|
|
|
### 1. Endianness
|
|
|
|
**Schema-level annotation with a default of little-endian.**
|
|
|
|
```json
|
|
{
|
|
"TypeDef:Struct": true,
|
|
"endian": "big",
|
|
"properties": { ... }
|
|
}
|
|
```
|
|
|
|
- `"endian": "little"` (default) — read/write in little-endian byte order.
|
|
- `"endian": "big"` — read/write in big-endian byte order.
|
|
- The annotation applies to the entire schema and all nested types.
|
|
- Mixed endianness within one schema is not supported (pathological; no
|
|
known protocol requires it).
|
|
- The default is little-endian, matching safetensors, wgpu, and most
|
|
modern formats. SFTP consumers specify `"endian": "big"`.
|
|
|
|
### 2. Alignment
|
|
|
|
**Both struct-level and field-level, with field-level overriding
|
|
struct-level.**
|
|
|
|
```json
|
|
{
|
|
"TypeDef:Struct": true,
|
|
"align": 256,
|
|
"properties": {
|
|
"header": { "TypeDef:Struct": true, "properties": { ... } },
|
|
"weight": { "TypeDef:Float32": true, "align": 16 }
|
|
}
|
|
}
|
|
```
|
|
|
|
- Struct-level `"align"` sets the default alignment for all fields in
|
|
that struct. The struct's total size is rounded up to this alignment.
|
|
- Field-level `"align"` overrides the struct default for that specific
|
|
field.
|
|
- Default alignment (when no annotation is present): 1 for u8/bool, 2
|
|
for u16/i16, 4 for u32/i32/f32, 8 for u64/i64/f64, max field alignment
|
|
for structs.
|
|
- Alignment is only meaningful in aligned static mode (ADR-096). In
|
|
packed sequential mode, alignment annotations are ignored — fields are
|
|
packed with no padding.
|
|
|
|
### 3. Variable-length encoding
|
|
|
|
**Three strategies for variable-length types, selected by the `encoding`
|
|
annotation and the standard JSON Schema `maxLength` keyword.**
|
|
|
|
```json
|
|
// Strategy 1: Inline length-prefixing (default, shorthand)
|
|
{ "TypeDef:String": true }
|
|
|
|
// Strategy 1: Explicit inline length-prefixing
|
|
{ "TypeDef:String": { "encoding": "length-prefixed" } }
|
|
|
|
// Strategy 2: Fixed-size reservation (uses standard maxLength)
|
|
{ "TypeDef:String": true, "maxLength": 256 }
|
|
|
|
// Strategy 3: Offset indirection (opt-in)
|
|
{ "TypeDef:String": { "encoding": "offset-indirect" } }
|
|
```
|
|
|
|
**Strategy 1: Inline length-prefixing (default).** The field's fixed
|
|
portion is a 4-byte length prefix at a computed offset. The variable data
|
|
follows immediately after. In packed sequential mode, the length prefix
|
|
determines the position of subsequent fields. In aligned static mode, the
|
|
length prefix is at a known offset; the variable data is not included in
|
|
the static layout. This is the universal pattern used by channels, SFTP,
|
|
TTY, and most binary protocols.
|
|
|
|
**Strategy 2: Fixed-size reservation.** When a variable-length field
|
|
declares `maxLength` (a standard JSON Schema keyword), the engine reserves
|
|
`maxLength` bytes at a fixed offset in aligned static mode. Data shorter
|
|
than `maxLength` is zero-padded; data longer than `maxLength` is a
|
|
validation error. This makes the field fixed-size from the layout
|
|
perspective — subsequent fields have known, unchanging offsets. This is
|
|
the database `VARCHAR(N)` pattern and the metatensor struct-tensor
|
|
pattern for fields with known maximum sizes. In packed sequential mode,
|
|
`maxLength` is a validation constraint only — the engine still uses
|
|
inline length-prefixing (strategy 1).
|
|
|
|
**Strategy 3: Offset indirection.** The field is a struct
|
|
`{offset: u32, length: u32}` that points into a separate data region.
|
|
This is the metatensor blob tensor pattern — the index struct lives in
|
|
one region, the blob data lives in another. The consumer provides the
|
|
data region separately. Enables mmap-friendly random access to
|
|
variable-length data without parsing length prefixes and without
|
|
reserving worst-case space.
|
|
|
|
**Default strategy selection:**
|
|
- In packed sequential mode: always strategy 1 (inline length-prefixing).
|
|
`maxLength` is a validation constraint only.
|
|
- In aligned static mode: strategy 2 (fixed-size reservation) if
|
|
`maxLength` is declared; strategy 3 (offset indirection) if
|
|
`"encoding": "offset-indirect"` is declared; strategy 1 (inline
|
|
length-prefixing) otherwise.
|
|
|
|
- `true` is a shorthand for the default (length-prefixed). This keeps
|
|
the common case concise and the override explicit.
|
|
- The `encoding` annotation and `maxLength` apply to all variable-length
|
|
types: `TypeDef:String`, `TypeDef:Bytes`, `TypeDef:Array`,
|
|
`TypeDef:Record`, `TypeDef:Timestamp`.
|
|
|
|
### 3a. TRecord value type
|
|
|
|
`TypeDef:Record` is a string-keyed map. The value type is declared via
|
|
the `"values"` property in the schema:
|
|
|
|
```json
|
|
{
|
|
"TypeDef:Record": true,
|
|
"values": { "TypeDef:Float32": true }
|
|
}
|
|
```
|
|
|
|
- `"values"` is a schema object declaring the `TypeDef:*` kind of all
|
|
values in the record. All values share the same type.
|
|
- The binary layout is a count-prefixed sequence of `(key, value)` pairs:
|
|
`[count: u32][key_len: u32][key_bytes][value]...` repeated `count`
|
|
times. Each key is a length-prefixed UTF-8 string. Each value is
|
|
encoded according to its declared `TypeDef:*` kind — a `Record<Uint32>`
|
|
value is 4 raw bytes; a `Record<String>` value is itself a
|
|
length-prefixed string; a `Record<Struct>` value is the struct's
|
|
fields laid out inline. There is **no separate `value_len` prefix** —
|
|
the value's size is determined by its kind (fixed-size kinds have a
|
|
known size; variable-length kinds carry their own length prefix).
|
|
- The count and key-length prefixes respect the schema's endianness.
|
|
- In aligned static mode with `maxLength`, the entire record is reserved
|
|
at `maxLength` bytes (zero-padded).
|
|
|
|
### 4. TUnion discriminators
|
|
|
|
**Two discriminator kinds: byte-offset (protocol dispatch) and
|
|
field-name (typedef.ts pattern).**
|
|
|
|
#### Kind A: Byte-offset discriminator
|
|
|
|
```json
|
|
{
|
|
"TypeDef:Union": true,
|
|
"discriminator": {
|
|
"kind": "byte",
|
|
"offset": 0,
|
|
"type": "TypeDef:Uint8"
|
|
},
|
|
"mapping": {
|
|
"1": { "$ref": "#/$defs/Init" },
|
|
"3": { "$ref": "#/$defs/Open" },
|
|
"5": { "$ref": "#/$defs/Read" },
|
|
"6": { "$ref": "#/$defs/Write" },
|
|
"101": { "$ref": "#/$defs/Status" }
|
|
}
|
|
}
|
|
```
|
|
|
|
- The discriminator is a fixed-size integer at a known byte offset.
|
|
- `"offset"` is the byte position of the discriminator within the union's
|
|
buffer.
|
|
- `"type"` is the `TypeDef:*` kind of the discriminator (typically
|
|
`TypeDef:Uint8` for protocol type bytes).
|
|
- The mapping keys are stringified integers (`"1"`, `"5"`, `"101"`).
|
|
The engine parses the key to match the discriminator value.
|
|
- The variant struct starts at `offset + discriminator_size`.
|
|
- This is the SFTP `Packet` enum pattern and the call protocol's event
|
|
type dispatch.
|
|
|
|
#### Kind B: Field-name discriminator
|
|
|
|
```json
|
|
{
|
|
"TypeDef:Union": true,
|
|
"discriminator": {
|
|
"kind": "field",
|
|
"name": "type"
|
|
},
|
|
"mapping": {
|
|
"read": { "$ref": "#/$defs/Read" },
|
|
"write": { "$ref": "#/$defs/Write" }
|
|
}
|
|
}
|
|
```
|
|
|
|
- The discriminator is a named field within the struct.
|
|
- `"name"` is the field name that holds the discriminator value.
|
|
- The mapping keys are string values matching the discriminator field's
|
|
value.
|
|
- The discriminator field is just another field in the struct — its
|
|
offset is computed like any other field.
|
|
- This is the typedef.ts `TUnion` pattern.
|
|
|
|
#### Mapping values
|
|
|
|
Mapping values may be either inline schemas or `$ref` pointers. `$ref`
|
|
is cleaner for large unions (29 SFTP variants) but requires a `$defs`
|
|
section. Inline schemas are simpler for small unions (5 call protocol
|
|
event types). Both work.
|
|
|
|
## Consequences
|
|
|
|
### Positive
|
|
|
|
- **Concrete, validated shapes.** All four annotation categories have
|
|
concrete JSON shapes that were validated by the POCs.
|
|
- **Sensible defaults.** Little-endian, natural alignment, inline
|
|
length-prefixing — the common case requires no annotations.
|
|
- **Explicit overrides.** Big-endian, custom alignment, offset
|
|
indirection — the uncommon case is explicit and self-documenting.
|
|
- **TUnion covers both protocol and typedef.ts patterns.** The
|
|
byte-offset discriminator handles SFTP type bytes and call protocol
|
|
event types. The field-name discriminator handles the typedef.ts string
|
|
pattern. No separate union type needed.
|
|
|
|
### Negative
|
|
|
|
- **Keyword value shape change.** `"TypeDef:String": true` (boolean) and
|
|
`"TypeDef:String": { "encoding": "length-prefixed" }` (object) are both
|
|
valid. The engine must handle both shapes. This is a minor parsing
|
|
concern — the POC already handles it.
|
|
- **Alignment annotations are mode-specific.** Alignment is only
|
|
meaningful in aligned static mode. In packed sequential mode, alignment
|
|
annotations are ignored. This is documented, not enforced — a consumer
|
|
that specifies alignment in packed mode gets no error, just no effect.
|
|
|
|
## References
|
|
|
|
- `docs/research/alknet-typedef/findings.md` §"Open Questions" — the
|
|
annotation shape questions this ADR resolves
|
|
- [ADR-095](095-alknet-typedef-purpose-scope-jsonschema-engine.md) —
|
|
purpose and scope
|
|
- [ADR-096](096-two-layout-modes-packed-vs-aligned.md) — the two layout
|
|
modes (alignment only meaningful in aligned static mode)
|