glm-5.2
2c4a4994dc
Copy the binary struct engine (src/, tests/) verbatim from alknet/crates/alknet-typedef and create a standalone Cargo.toml (workspace-inherited fields inlined). Port the architecture docs (specs, ADRs 095-102, OQs 069-071) from alknet's nested multi-crate layout to a flat single-crate layout, fixing relative link paths. Build, 295 tests, and clippy all pass clean.
status, last_updated
| status | last_updated |
|---|---|
| draft | 2026-07-22 |
alknet-typedef
The binary struct engine: a small Rust crate that takes a JSON Schema
with TypeDef:* custom keywords and produces an offset map, read/write
functions, and validation — all driven by the schema. The schema is the
format definition; the engine is generic.
Documents
| Document | Status | Description |
|---|---|---|
| overview.md | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
| schema-layer.md | draft | The 19 TypeDef:* kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations |
| layout-engine.md | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
| data-access.md | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
| validation.md | draft | Custom keyword validators for all 19 TypeDef:* kinds, TypedefError, load-time vs access-time validation, TypedefEngine |
Applicable ADRs
| ADR | Title | Relevance |
|---|---|---|
| 095 | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
| 096 | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; LayoutBuilder/SequentialReader vs OffsetMap |
| 097 | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations |
| 098 | Error Handling and Validation Strategy | TypedefError enum; load-time build, access-time check; field-path-carrying errors |
| 099 | Int64/Uint64 as First-Class Kinds | 64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat |
| 100 | Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode | Prevents silent data corruption (inline variable data clobbering subsequent fields) |
| 101 | Packed-Mode Read API — Engine as SequentialReader Factory | engine.sequential_reader() returns an owned reader, not a reference |
| 102 | Reject TUnion in Aligned Mode for v1 | Unions are the protocol pattern; aligned-mode union semantics were broken |
Relevant Open Questions
| OQ | Title | Status | Relevance |
|---|---|---|---|
| OQ-069 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it |
| OQ-070 | no_std + alloc support |
deferred(scope) | Target std for v1; blocked on an embedded use case |
| OQ-071 | Builder API for schema construction | deferred(scope) | Schemas are authored in TypeBox or hand-written JSON for v1; blocked on a concrete need |
Key Design Principles
-
The schema is the format. A JSON Schema with
TypeDef:*custom keywords is both the validation spec and the layout spec. No separate format definition, no separate parser, no separate validator. One schema, three uses: validate, compute offsets, access data. See overview.md and ADR-095. -
jsonschema is the validation engine, not a custom engine. The
jsonschemacrate (v0.46.5, Draft 2020-12) handles validation with custom keyword support. The novel code is the offset computation, not the validation. This eliminates ~14,000 lines of hand-rolled schema engines (typebox-rs, alktype). See schema-layer.md and ADR-095. -
Two layout modes for two use cases. Packed sequential (
LayoutBuilder/SequentialReader) for protocol wire formats (SFTP, channels, TTY). Aligned static (OffsetMap) for mmap-friendly formats (metatensor). The consumer selects the mode; the schema is the same. See layout-engine.md and ADR-096. -
Variable-length types default to inline length-prefixing.
[length: u32][data]is the universal pattern used by channels, SFTP, TTY, and most binary protocols. Offset indirection (the metatensor blob tensor pattern) is opt-in via theencodingannotation. See layout-engine.md and ADR-097. -
TUnion supports both byte-offset and field-name discriminators. Byte-offset for protocol dispatch (SFTP type bytes, call protocol event types). Field-name for the typedef.ts string pattern. See data-access.md and ADR-097.
-
Endianness is per-schema, default little-endian. The engine reads the
"endian"annotation and byte-swaps accordingly. SFTP consumers specify"endian": "big". See layout-engine.md and ADR-097. -
Validation is opt-in, built once at load time. The jsonschema validator is compiled once at schema load time. Access-time validation is a fast
is_valid()check. High-throughput paths can skip validation; security-sensitive paths can validate every frame. See validation.md and ADR-098. -
Not a serialization framework. The typedef engine is not a general-purpose serde replacement. It operates on raw byte buffers at computed offsets — no intermediate
Valuetree, no reflection, no dynamic dispatch per field. For JSON data, use serde. For binary data with a known schema, use typedef. See overview.md and ADR-095.
References
docs/research/alknet-typedef/findings.md— POC results (26 tests passing, two layout modes, TUnion dispatch, endianness)docs/research/call-channels-unification/findings.md§"alknet-typedef: JSON Schema as the binary struct engine" — the origin of this research thread/workspace/@alkdev/typebox/example/typedef/typedef.ts— the TypeBox schema kinds (619 lines)/workspace/jsonschema/— the jsonschema crate (v0.46.5, Draft 2020-12)/workspace/alknet-typedef-poc/— the POC code (disposable)/workspace/@alkimiadev/typebox-rs/— prior attempt, replaced by typedef/workspace/@alkimiadev/alktype/— prior attempt, replaced by typedef