docs(research): split alknet-tensor into alknet-runtime + alknet-compute + alknet-tensor

Extract the shared JS+wgpu substrate (verified by the alknet-desktop POCs)
as alknet-runtime — the generalized QuickJS-NG + wgpu runtime that both
alknet-desktop (render) and alknet-compute (tensor compute) build on. Key
property driving the split: wgpu on llvmpipe is genuinely useful compute
with no physical GPU (WGSL → optimized SIMD beats JS for non-trivial
workloads), so wgpu is unconditional in the runtime rather than a feature
flag.

Reframes the original alknet-tensor architecture-summary as alknet-compute
(builds on alknet-runtime + alknet-tensor) with ShaderGenerator as a trait
(WGSL first impl, SPIR-V/GLSL/naga-IR later per wgpu multi-input-language
support). alknet-tensor/metatensor-format.md is now clearly the pure binary
format crate (no JS or wgpu dep), usable standalone by a pure-Rust model
server.

Layering: alknet-runtime depends on alknet-call (registry authority stays
per ADR-013); alknet-compute and alknet-desktop depend on alknet-runtime;
alknet-tensor is a pure-format sibling.
This commit is contained in:
2026-06-30 12:44:39 +00:00
parent b71db99753
commit 303b9a58e2
4 changed files with 315 additions and 57 deletions

View File

@@ -1,9 +1,11 @@
# Metatensor: Schema-Driven Binary Tensor Format
# alknet-tensor: Schema-Driven Binary Tensor Format (Metatensor)
**Status:** Research / concept design. The schema layer is proven (TypeBox + jsonschema interop confirmed by API inspection); the offset computation and mmap layer are designed but not yet implemented.
**Date:** 2026-06-20
**Date:** 2026-06-20 (original), 2026-06-30 (reframed for crate split)
**Scope:** A schema-driven binary tensor format extending safetensors, using TypeBox (JS) / jsonschema (Rust) as the layout specification language. Supports flat, struct, and blob tensor kinds for fixed-size, record-structured, and variable-length data. Memory-mappable, QUIC-streamable, and authorable via ujsx.
**Crate note (2026-06-30):** This doc was originally titled `alknet-tensor/metatensor-format.md` and paired with `alknet-tensor/architecture-summary.md`. The crate-decomposition session split the original `alknet-tensor` concept into two crates: `alknet-tensor` (this doc — the pure binary format, no JS or wgpu dependency) and `alknet-compute` (the wgpu compute engine — now at `docs/research/alknet-compute/architecture-summary.md`, builds on `alknet-runtime` + `alknet-tensor`). A pure-Rust model server can use `alknet-tensor` for the format alone; `alknet-compute` is what bridges the format to wgpu buffers via `load_model`/`stream_model` ops registered on `alknet-runtime`'s registry.
---
## Executive Summary
@@ -417,4 +419,5 @@ Loading a tensor from mmap into a wgpu buffer: is it a copy (mmap → staging bu
- **safetensor format reference:** `/workspace/research/typebox_research/metatensor/basic.ts` — 8-byte LE header length + JSON header + raw bytes; `TensorRef = {dtype, shape, data_offsets}`
- **ujsx reconciler (verified on quickjs):** `/workspace/@alkdev/ujsx/src/host/{reconcile,config,fiber}.ts` — fiber-based reconciler, the typed-tree diff engine
- **flowgraph (compute graph layer, uses ujsx):** `/workspace/@alkdev/flowgraph/``<Operation>`, `<Sequential>`, `<Parallel>`, `<Conditional>`, `<Map>` components; `GraphologyHostConfig` + `ReactiveHostConfig`
- **alknet-tensor architecture (parent doc):** `/workspace/@alkdev/alknet/docs/research/alknet-tensor/architecture-summary.md` — the tensor compute architecture this format serves
- **alknet-compute architecture (consumer of this format):** `docs/research/alknet-compute/architecture-summary.md` — the wgpu compute engine that builds on `alknet-runtime` and this format; registers the `load_model`/`stream_model` ops that bridge metatensor files to wgpu buffers
- **alknet-runtime (substrate, sibling dependency):** `docs/research/alknet-runtime/summary.md` — the JS+wgpu substrate; this format crate does not depend on the runtime (pure-Rust path), but `alknet-compute` depends on both