Documents the metatensor format: a binary data format where a TypeBox/jsonschema
schema describes the layout of binary data at schema-computed offsets. Extends
safetensors (fixed TensorRef schema) to arbitrary schemas, enabling struct tensors
(records), blob tensors (variable-length via indirection), and nested layouts.
Key points:
- TypeBox schemas render to standard JSON Schema; the jsonschema Rust crate
validates them with zero translation. Custom typedef.ts kinds (TFloat32,
TInt32, TStruct) map to jsonschema custom keywords via with_keyword().
- This eliminates typebox-rs as a schema engine — replaced by jsonschema +
a small offset-computation module + ~50 lines of custom keyword impls.
- Three tensor kinds: flat (safetensor today), struct (record of typed fields),
blob (struct tensor as index + flat tensor as data store, for variable-length)
- Memory-mappable: parse header, compute offsets, mmap data, typed views per
schema. No copy, no deserialization.
- QUIC-streamable: header is one small JSON message, each tensor is a separate
stream. Lazy loading, parallel transfer, incremental compute.
- ujsx-authorable: <Tensor>, <Struct>, <Field> as layout components, same
reconciler that diffs UI trees diffs model schemas. Model versioning is
tree diffing.
- Category-theory foundation: ujsx as universal typed-tree IR, HostConfig as
interpreter. <Tensor> is no stranger than <div>.
glm-5.2
b7b5337586