diff --git a/docs/research/iroh-blobs-eval.md b/docs/research/iroh-blobs-eval.md index ba767f5..1aa7857 100644 --- a/docs/research/iroh-blobs-eval.md +++ b/docs/research/iroh-blobs-eval.md @@ -69,8 +69,8 @@ temp/.temp staging (must be same-device as data/) all (`fs/import.rs:160-221`, `DESIGN.md:71`). iroh couples "hash overhead" and "storage tier" at the same boundary; under our git-oid derivation there is no outboard to elide (whole-blob digest only) and - the tier boundary is a pure dispatch choice — POC #3's benchmark says - ~64-128 KiB, not 16 KiB (see `poc-largeblob-findings.md`, B-finding). +the tier boundary is a pure dispatch choice — POC #3's benchmark says +~128-256 KiB, not 16 KiB (see `poc-largeblob-findings.md`, finding A4). - **Crash-consistency story, borrowed as a conclusion** (`DESIGN.md: 126-142`, "files are hard"): persist data+outboard+sizes *first*, the checksummed bitfield *last*, with truncate-to-zero-on-read marking a @@ -195,4 +195,4 @@ findings (first-party, measured). 6. Crash-consistency ordering (data first, coverage/index last, with dirty-marking) — only if partial entries ever exist. 7. The inline threshold concept, re-sized by our own benchmark - (~64-128 KiB crossover, git-blob workload) instead of their 16 KiB. \ No newline at end of file + (~128-256 KiB crossover, git-blob workload) instead of their 16 KiB. \ No newline at end of file diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md index 61fb132..7656424 100644 --- a/docs/research/phase-0.md +++ b/docs/research/phase-0.md @@ -1,24 +1,26 @@ --- -status: draft -last_updated: 2026-10-02 (POC #3 executed and passed — see -poc-largeblob-findings.md plus iroh-blobs-eval.md; POC #1 executed and -passed — see poc-trait-dispatch-findings.md; POC #2 absorbed earlier; -register complete; earlier rounds: hash simplification, GC/pooling, -dedup/p2p, setup draft) +status: converged +last_updated: 2026-10-02 (Phase 0 convergence — all POCs passed, OQs +settled into decisions + residuals; phase-0 plan complete) --- # alkblobs — Phase 0 (Exploration) This document captures Phase 0 (Exploration) for the `alkblobs` crate: -vision, guiding principles, prior art, and open questions (OQ-BL-01..NN). -Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding -principles; research options; validate approaches; converge on a -recommended approach.* It is the input to Phase 1 (Architecture), where -the Architect will produce `docs/architecture/` specs, ADRs, and the -open-questions tracker. +vision, guiding principles, prior art, the validated approach, and the +open-question register (OQ-BL-01..06). Phase 0's objective per +`docs/sdd_process.md`: *capture vision and guiding principles; research +options; validate approaches; converge on a recommended approach.* That +objective is now met — all POC register rows passed or were absorbed, +and this document carries the converged recommendation into Phase 1 +(Architecture), where the Architect will produce `docs/architecture/` +specs, ADRs, and the open-questions tracker. -Drafted 2026-09-30, emerging from the initial setup discussion. Nothing -is converged yet — this is the working sketch, not a specification. +Drafted 2026-09-30; converged 2026-10-02 after the POC rounds. The +chronological accretion from the research rounds has been folded into a +single settled posture; the evidence trail lives in the findings +documents (`poc-trait-dispatch-findings.md`, `poc-largeblob-findings.md`, +`iroh-blobs-eval.md`). ## Vision and guiding principles @@ -33,73 +35,63 @@ network operations riding the alkcall seam when they exist. 1. **alkgit** (planning phase) needs an object backend that is *not* its proposed default file-based backend ("kind of gross") — blobs - stored under git's own object hashes, but with iroh-blobs-like - handling for large blobs (where git typically delegates to git-lfs; - we could instead follow an iroh-blobs-shaped large-blob path). The - original framing of this as a "hash conflict" dissolved in the - 2026-10-01 rounds: git's oid derivation was adopted as the store's - canonical hash (§OQ-BL-03), so git objects land in the pool as - first-class entries with zero indirection — the conflict was an - artifact of inheriting iroh-blobs' BLAKE3-only choice, and - once that inheritance was rejected the conflict went with it. + stored under git's own object hashes, with an iroh-blobs-shaped + large-blob path standing in where git typically delegates to + git-lfs. The once-called "hash conflict" does not exist: git's oid + derivation is the store's canonical hash (OQ-BL-03), so git objects + land in the pool as first-class entries with zero indirection. 2. **Workspace/file-shaped storage in the alk* family** (the "appfile" - shape, originally surfaced by old alknet research — historical - lineage only; the consumers below are current): small blobs need a - store faster than the filesystem (now first-party measured, POC #3 - finding A4: sqlite ~9-10× faster at 1-16 KiB), large blobs want a - filesystem fallback, with a filename↔hash mapping above. Dispatching - across backends at the same time is a store-layer problem this crate - should own (OQ-BL-02; POC #1 validated the seam, POC #3 the + shape; old alknet research is historical lineage only): small blobs + need a store faster than the filesystem (first-party measured: + sqlite ~9-10× faster at 1-16 KiB, POC #3 finding A4), large blobs + want a filesystem fallback, with a filename↔hash mapping above. + Dispatching across backends is a store-layer problem this crate + owns (OQ-BL-02; validated by POC #1's trait seam and POC #3's streaming + threshold evidence). 3. **Agent workspaces** — many agents working simultaneously in workspaces that are mostly identical to each other's, each making - small edits in specific areas. Per-agent stores multiply the near- - identical content; a shared content-addressed store dedups it by - construction, and a path→hash manifest per workspace makes the - per-agent delta just its changed hashes (a path→hash manifest is - straightforward by construction; the old alknet probe's manifest - mapping sketch showed the same). + small edits in specific areas. Per-agent stores multiply the + near-identical content; a shared content-addressed store dedups it + by construction, and a path→hash manifest per workspace makes each + agent's delta just its changed hashes. -**The p2p shape and what it pins (added 2026-10-01, from the dedup -discussion).** Two alkgit use cases exist: self-hosted git (where -cross-repo dedup matters little — you don't fork your own repos) and -**p2p git for OSS projects**, the demanding one. The p2p sketch: -ownership/ACL live in a smart contract on a low-fee network; -replicators are push/clone endpoints that watch the contract and cache -its state; repos sync via iroh-style gossip (hashes, not content) and -pull/sync on demand. Neither alkgit approach described so far takes -advantage of dedup — which is exactly a store-layer property. The -implications for this crate: +**The p2p shape (from the dedup discussion).** Two alkgit use cases +exist: self-hosted git (where cross-repo dedup matters little — you +don't fork your own repos) and **p2p git for OSS projects**, the +demanding one. The p2p sketch: ownership/ACL live in a smart contract +on a low-fee network; replicators are push/clone endpoints that watch +the contract and cache its state; repos sync via iroh-style gossip +(hashes, not content) and pull/sync on demand. Neither alkgit approach +takes advantage of dedup — which is exactly a store-layer property. +The implications for this crate: -- Both hard consumers (p2p replicators, agent swarms) reduce to the - same shape: **a pooled CAS per node**, where repos / workspaces are - *sets of hash references* (trees/manifests), not separate object +- Both demanding consumers (p2p replicators, agent swarms) reduce to + the same shape: **a pooled CAS per node**, where repos / workspaces + are *sets of hash references* (trees/manifests), not separate object stores. Cross-repo dedup is then by construction; git's own `alternates`/object-pool mechanism is the same idea done awkwardly. - With the canonical-hash resolution (§OQ-BL-03) the dedup strengthens + With the canonical-hash resolution (OQ-BL-03) the dedup strengthens further: workspaces and git repos collide into one address domain — the same file content has the same address whether it entered via a git oid or a workspace manifest. - Because everything is hash-addressed, the p2p wire story collapses to "announce hashes, diff hash sets, fetch missing blobs verified" — - which is precisely the ops surface OQ-BL-01 left provisional. The - consumer exists now: if p2p git is a confirmed alkgit direction, - the ops surface upgrades from "leaning store-only" to a confirmed - have/need + fetch op family, auth-gated via the alkcall seam. + exactly the ops surface OQ-BL-01 scoped. The consumer exists; the + ops surface is a confirmed have/need + fetch op family, auth-gated + via the alkcall seam, whether it lives in this crate or a sibling. -**The scope line (current posture, revisit as evidence arrives):** -this crate is the *store*, not the *transport*. iroh-blobs welds store -+ provider protocol + tickets + postcard into one crate; the defining -posture here is the opposite split — put/get/verify against a hash is -its own layer, and any provider/protocol/ops surface lives above it or -in feature-gated modules (the alk* inversion-point pattern). The ops -surface (a channels-native "have/need + fetch blob with this hash" op -family, auth-gated via `AccessControl`) is a Phase 0/1 question — the -p2p-git consumer (§The p2p shape) upgrades it from speculative to -likely, but the crate-boundary decision itself is still open -(OQ-BL-01). What stays *above* the crate regardless: repos/workspaces -as reference sets (trees/manifests), path→hash mapping, git semantics, -the contract/gossip/replicator policy layer. +**The scope line:** this crate is the *store*, not the *transport*. +iroh-blobs welds store + provider protocol + tickets + postcard into +one crate; the defining posture here is the opposite split — +put/get/verify against a hash is its own layer, and any +provider/protocol/ops surface lives above it or in feature-gated +modules (the alk* inversion-point pattern). The confirmed ops shape +(a channels-native "have/need + fetch blob with this hash" op family, +auth-gated via `AccessControl`) has its crate-boundary decision +recorded in OQ-BL-01's convergence. What stays *above* the crate +regardless: repos/workspaces as reference sets (trees/manifests), +path→hash mapping, git semantics, the contract/gossip/replicator +policy layer. Guiding principles, inherited from the alk* family: @@ -108,195 +100,206 @@ Guiding principles, inherited from the alk* family: reassembled anywhere particular. Backends behind an injected seam (alktty `TtyBackend` / alktunnels pump-halves precedent). 2. **One canonical hash; the git-family derivation.** The store keys - entries under a single canonical algorithm — the git-blob derivation - (`blob \0` domain-separated input, SHA-256 for new content; - OQ-BL-03). This is the resolution of the earlier "multi-hash" - framing: BLAKE3's presence was an inherited iroh-blobs choice, not - a requirement, and once rejected the hash-conflict motivation - dissolved with it. Legacy SHA-1 (existing git repos) stays - tolerated via the git-blob prefix; the abstraction is still a small - enum/capability set rather than a hardcoded single algorithm — - SHA-1-vs-SHA-256 is already two, and the key encoding is a one-way - door. + entries under a single canonical algorithm — the git-blob + derivation (`blob \0` domain-separated input, SHA-256 for new + content; OQ-BL-03, implementation-validated by POC #1). Legacy + SHA-1 (existing git repos) stays tolerated via the same prefix; the + abstraction is a small enum/capability set, and the key encoding is + a one-way door. 3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets, - postcard serialization, and provider protocol are design-welded - choices we do not inherit. Its *store-shape* lessons (kv + flat - backends, verification flow, chunking) are fair game per prior-art - reading. + postcard serialization, provider protocol, and Command-actor store + abstraction are design-welded choices we do not inherit. Its + *store-shape* conclusions (kv + flat backends, verification flow, + chunking limits, put-commit primitives) are fair game — see + `iroh-blobs-eval.md`. 4. **Producer/consumer vocabulary** for any network-facing surface; authorization via alkcall's `AccessControl`/identity seam, never an in-band invented scheme (convention 6). -5. **Verify where it matters.** iroh-blobs' core virtue is verified - transfer (bao outboard encoding) — but its verification story is - BLAKE3-native, and we rejected the BLAKE3 inheritance. Git objects - self-verify, so the git case needs nothing from us. The remaining - question is the large-blob path: per-blob digest checks (rudolfs' - verify-as-decorator shape), a conditional bao-like chunk tree - (BLAKE3 confined to that encoding layer if ever adopted), or - backend-native — open (OQ-BL-04). +5. **Verify where it matters.** Git objects self-verify under git's + own model — the git case needs nothing from us. Per-range + verification is impossible under the canonical digest (the preamble + hashes the length; POC #3 finding A2), so verification is + whole-blob for stored content; the only genuine consumer of a + chunk-tree encoding is networked ranged fetch (a transfer-layer + conditional — OQ-BL-04). 6. **Pooling is the point.** The dedup wins both demanding consumers want (Forknet-style multi-repo OSS content, agent swarms) require that repos/workspaces are *sets of hash references* over one pooled CAS per node, not walled-off per-repo stores. Pooling forces a GC story (OQ-BL-05) and a multi-backend dispatch story (OQ-BL-02). -7. **Whole-file CAS as the default; chunking earns its keep only +7. **Whole-file CAS is the default; chunking earns its keep only where it must.** For source-code-scale content, whole-file hashing captures the agent/edit delta perfectly and needs no chunk tree; fixed-size chunk trees are actually *hostile* to insertion/deletion edits (byte shifts cascade). Content-defined chunking (CDC — fastcdc/restic-style, shift-resistant) only pays on large binary - files receiving small edits — the git-lfs weakness. iroh-blobs/bao - use fixed-size trees; do not inherit that as the default. Where - chunking lives (store encoding vs manifest layer) is OQ-BL-04. + files receiving small edits — the git-lfs weakness. + iroh-blobs/bao use fixed-size trees; that is not inherited as the + default. Where chunking would live is scoped in OQ-BL-04. -## What is already known (settled, thin) +## The settled approach -Almost nothing is pinned — deliberately. The postures agreed so far: +These are the converged postures. None is a Phase 1 ADR yet; each +becomes one (or is already recorded as resolved here) — the ADR +backlog is in §Convergence. -- **Not a fork of iroh-blobs.** A downstream store crate inspired by - its `src/store` work, diverging deliberately on hashing and wire - surface (see `iroh-blobs-eval` below when written). -- **Canonical hash = the git-family derivation** (2026-10-01 hash - simplification round; was "multi-hash from day one"). git-blob-SHA-256 - as primary, legacy SHA-1 tolerated, BLAKE3 demoted to a conditional - large-blob encoding consideration. Details and the reasoning trail - in OQ-BL-03. -- **Backend pluralism assumed, then designed.** kv/sqlite for small - blobs, filesystem fallback for large — the appfile shape — with - *how* multi-backend dispatch works validated by POC #1 (the lean - trait seam) and POC #3 (streaming + threshold evidence) (OQ-BL-02). -- **Pooled CAS, not per-repo stores** (agreed in the 2026-10-01 - dedup discussion — the load-bearing scope decision so far): repos - and agent workspaces are sets of hash references over one shared +- **Not a fork of iroh-blobs.** A downstream store crate that borrows + its store-shape conclusions (per `iroh-blobs-eval.md`) while + diverging deliberately on hashing and wire surface. +- **Canonical hash = the git-family derivation** (OQ-BL-03, + RESOLVED): `git-blob-sha-256` primary, legacy `git-blob-sha-1` + tolerated, BLAKE3 demoted to a conditional large-blob transfer + encoding. Implementation-validated against the real git CLI, + including byte-exact external-oid interop (POC #1 finding 4/5). +- **Backend pluralism, dispatched by size threshold** (OQ-BL-02, + base settled): the lean `Backend` trait contract (opaque byte keys, + `has`/`get`/`put`/`delete`/`list`/`name`, `list()` complete by + contract) with deterministic size-threshold routing — kv/sqlite for + small blobs, fs fallback for large. Validated by POC #1 (trait seam) + and POC #3 (streaming shapes, sqlite-vs-fs benchmark: ~9-10× at + 1-16 KiB, crossover ~128-256 KiB). Phase 1 default threshold lean: + 128 KiB (tunable constructor parameter). +- **Streaming put seam = `(Option, stream)`** (OQ-BL-02 / + finding A1): known-length one-pass hashing as the encouraged path; + unknown-length staging-then-hash, forced by the git preamble, not a + design choice. Get-side: small tier returns values, large tier + streams; range reads return slice + out-of-band slice digest. +- **Pooled CAS, not per-repo stores** (OQ-BL-05): repos and agent + workspaces are sets of hash references over one shared content-addressed store. This is the property both demanding - consumers actually need; everything else (GC, manifests, git - semantics) lives around it. Details open (OQ-BL-05). -- **Physically flat, logically namespaced** (2026-10-01 GC/pooling - round; the rudolfs inversion): the byte layer is a flat - dedup-by-construction CAS; namespaces are reference tables above it - (GC roots, sweep scoping, ACL boundary). Backends stay - namespace-blind. -- **Whole-file CAS is the default granularity**; chunking is scoped - to the large-binary-blob case (principle 7), not the default path. + consumers need; everything else lives around it. +- **Physically flat, logically namespaced** (OQ-BL-05; the rudolfs + inversion): the byte layer is a flat dedup-by-construction CAS + (`hash → bytes`, backends namespace-blind, `{prefix}/{prefix}/{hash}` + sharding survives); namespaces are reference tables above it — GC + roots, sweep scoping, accounting, ACL boundary. +- **Mark-and-sweep GC with protect-callback + RAII pinning** (OQ-BL-05, + mechanism validated by POC #1 findings 3/7/8): pins protect + not-yet-referenced puts, liveness sources register into a seam, + protection-source failure aborts the sweep (iroh's + `ProtectOutcome::Abort` conclusion), delete-then-recover is a re-put. +- **Pack tension resolved in shape** (OQ-BL-05): git objects enter the + pool as loose-equivalent kv entries (small tier, the common case); + if packfile serving is ever wanted, packs are stored as large blobs + and served through the store's range-read surface (git's own `.idx` + does per-object offset lookup). Object-level dedup lives in the + loose tier; pack-level dedup across unrelated repos was never the + goal. +- **Whole-file CAS is the default granularity**; chunking is scoped to + the networked-large-blob transfer case (principle 7 + OQ-BL-04). ## Prior art ### iroh-blobs — the shape inspiration (evaluated, not the base) -`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference). -Specifically `src/store`: **the full store eval is now written — -`docs/research/iroh-blobs-eval.md` (2026-10-02, against checkout -e82cbdc / v0.103.0)**: no Store trait (message-protocol actor pattern), -fs backend on-disk layout (redb entry-state + inline thresholds + crash -ordering), verification (subtree-local, chunk-tree-driven), streaming -shapes, and the do-not-inherit / keep (conclusions-only) lists. -Previously-read GC mechanics (kept here for the detail): +`/workspace/iroh-blobs` (read-only reference), evaluated against +checkout e82cbdc / v0.103.0 in `docs/research/iroh-blobs-eval.md`: no +Store trait (message-protocol actor pattern), fs backend on-disk +layout (redb entry-state + inline thresholds + crash ordering), +verification (subtree-local, chunk-tree-driven), streaming shapes, and +the do-not-inherit / keep (conclusions-only) lists. -**GC mechanics — verified against the current checkout (2026-10-01, -`src/store/gc.rs`, `src/util/temp_tag.rs`, `src/store/fs/delete_set.rs`). -This is the load-bearing prior art for OQ-BL-05:** +**GC mechanics — verified against that checkout (`src/store/gc.rs`, +`src/util/temp_tag.rs`, `src/store/fs/delete_set.rs`). This is the +load-bearing prior art for OQ-BL-05:** - **Mark-and-sweep, not refcounting, for persistent liveness.** `gc_mark_task` collects roots from three sources: persistent *named - tags* (a flat `tags-0` table, name → `HashAndFormat`), in-memory * - `TempTag`s* (RAII-refcounted, `#[must_use]`, `.leak()` for + tags* (a flat `tags-0` table, name → `HashAndFormat`), in-memory + *`TempTag`s* (RAII-refcounted, `#[must_use]`, `.leak()` for pin-until-exit-of-process), and an injectable `ProtectCb` — a callback GC consults before each run that can add externally-known hashes or **abort the run** (`ProtectOutcome::Abort`; a flaky protection source skips the sweep rather than risking deletion). -- **Format-aware mark traversal** — a non-raw root (HashSeq/collection) - contributes all reachable children to the live set via the bao hash - stream (gc.rs:63-78). Protecting a collection protects everything +- **Format-aware mark traversal** — a non-raw root (HashSeq/ + collection) contributes all reachable children to the live set via + the bao hash stream. Protecting a collection protects everything reachable from it. - **Sweep** lists the whole store and batch-deletes (~100/batch) anything not live. `list()` being complete is therefore load-bearing - for GC — relevant to the backend trait contract (OQ-BL-02). + for GC — carried into the backend trait contract (OQ-BL-02). - **Write-safety against a concurrent sweep** — the fs backend carries a separate `DeleteSet` transaction layer (`ProtectHandle` / - mark-for-delete / protect-cancel / commit); `TempTag` holds a refcount - so a blob put but not yet referenced cannot vanish mid-write. Temp - tags are scoped (batch scope / process scope) with per-scope counters. -- **No namespaces in their model** — flat named tags over a flat - pool. Our namespace concept (OQ-BL-05) lands on their machinery as - policy over what counts as a root: `namespace → root manifest → - (mark-walk) → {oids}` is tags plus traversal, not new storage - concepts. + mark-for-delete / protect-cancel / commit); `TempTag` holds a + refcount so a blob put but not yet referenced cannot vanish + mid-write. Temp tags are scoped (batch scope / process scope). +- **No namespaces in their model** — flat named tags over a flat pool. + Our namespace concept (OQ-BL-05) lands on their machinery as policy + over what counts as a root: `namespace → root manifest → (mark-walk) + → {oids}` is tags plus traversal, not new storage concepts. ### rudolfs — git-lfs server; the composite-key + decorator-storage prior art -`/workspace/rudolfs` (v0.3.8, MIT, read-only reference; the alknet -research doc is -`/workspace/@alkdev/alknet/docs/research/references/gitlfs/ -rudolfs-reference.md`). A git-lfs server whose **storage layer** -(`src/storage/`) is the most direct ancestor of our composite-key + -dispatch shape: +`/workspace/rudolfs` (v0.3.8, MIT, read-only reference; detailed notes: +`/workspace/@alkdev/alknet/docs/research/references/gitlfs/rudolfs-reference.md`). +Its **storage layer** (`src/storage/`) is the most direct ancestor of +the composite-key + dispatch shape: -- **`StorageKey = (Namespace, Oid)`** — a composite key over a flat CAS - (`Namespace = (org, project)` strings, `Oid = SHA-256`). All backend - operations take the composite key. -- **`Storage` trait** — `get`/`put`/`size`/`delete`/`list`/` - total_size`/`max_size`/`public_url`/`upload_url`, with +- **`StorageKey = (Namespace, Oid)`** — a composite key over a flat + CAS (`Namespace = (org, project)` strings, `Oid = SHA-256`). All + backend operations take the composite key. +- **`Storage` trait** — `get`/`put`/`size`/`delete`/`list`/ + `total_size`/`max_size`/`public_url`/`upload_url`, with `LFSObject = (len: u64, ByteStream)` — the streaming-first put/get - shape (pinned boxed async byte stream), the natural large-blob API - and likely the channels ops-surface shape. + shape, the natural large-blob API and the template for the ops + surface's fetch op. - **Decorator composition** — `Verify ↔ Encrypted ↔ Cached(LRU → permanent) ↔ Retrying → s3/disk`. The `Cached` decorator is the appfile shape in LRU form: "fast store in front of fallback" as a composable wrapper, an alternative dispatch policy to size thresholds. `fanout()` duplicates one stream into two lock-step - copies (serve + persist) — precisely "serve the fetch while - persisting on receipt" for networked gets. -- **Verify-as-decorator (OQ-BL-04 input)** — streaming SHA-256 check on - both put and get paths with auto-purge of a corrupted tier. -- **Footnote if at-rest encryption is ever considered** — nonce derived - from the oid: deterministic and correct, but a key rotation - invalidates every object and breaks dedup across keys (the known - encryption-vs-dedup tension). + copies (serve + persist) — rebuilt at our layer and validated as a + broadcast seam (POC #3 finding A3). +- **Verify-as-decorator (OQ-BL-04 input)** — streaming SHA-256 check + on both put and get paths with auto-purge of a corrupted tier. +- **Footnote if at-rest encryption is ever considered** — nonce + derived from the oid: deterministic and correct, but key rotation + invalidates every object and breaks dedup across keys. - **THE ANTI-PATTERN (inverted, not inherited)** — namespaces are *physical*: `s3://{org}/{project}/{sha256}` — identical content in two orgs is stored twice. It buys tenant isolation by paying the cross-tenant dedup that is our whole reason for pooled CAS. The - reconciliation recorded in OQ-BL-05: keep the composite-key API - shape, put the namespace half in metadata (reference tables) above - the byte layer, and keep physical storage flat (`oid → bytes`, - backends namespace-blind). + reconciliation (OQ-BL-05): keep the composite-key API shape, put the + namespace half in reference tables above the byte layer, keep + physical storage flat (`oid → bytes`, backends namespace-blind). - **`list()` is load-bearing** — the S3 backend punts on `list()` - (returns an empty stream) and its `delete()` is a no-op, so it can - *never* GC. A lesson for the backend trait contract: list/sweep - support is a real requirement, not an optional extra. + (empty stream) and its `delete()` is a no-op, so it can *never* GC. + A backend-trait-contract lesson: list/sweep support is a real + requirement, not an optional extra (POC #1 finding 2 sharpens it: + a *malformed* list is as deadly as an incomplete one). ### gix-odb — git's own object database (alkgit's baseline) -`/workspace/git-oxide/gix-odb` (read-only reference). The backend -alkgit currently plans against; the hashing-algorithm baseline git -actually uses (SHA-1/SHA-256), git's loose-object and packfile layout, -and git's already-hash-addressed object model. The alkgit question is -whether this crate can sit *under or beside* gix-odb semantics — git -objects are already content-addressed and verified by git's own model, -so the crate's value there is the large-blob story (git-lfs-shaped) -and a better small-object backend than the proposed default, without -fighting git's hash model. Note for the p2p case: git's `alternates` -/ object-pool mechanism (e.g. GitLab object pools for fork networks) -is the prior art for cross-repo dedup among related repos — the -pooled-CAS posture (OQ-BL-05) generalizes it to unrelated repos and -to p2p replication. +`/workspace/git-oxide/gix-odb` (v0.84; read-only reference). The +backend alkgit currently plans against; the hashing-algorithm baseline +git actually uses (SHA-1/SHA-256), git's loose-object and packfile +layout, and git's already-hash-addressed object model. This crate can +sit under or beside gix-odb semantics because git objects are already +content-addressed and self-verified by git's own model; the crate's +value there is the large-blob story (git-lfs-shaped) and a better +small-object backend than the proposed default, without fighting git's +hash model. Audit conclusions carried into OQ-BL-04/OQ-BL-05: +alternates/multi-pack-index give shared-read across packs with no +dedup/GC/large-blob/streaming value-add (gix-lfs is an empty +placeholder); gitoxide's pack-ID-stability machinery exists only +because packs are immutable-with-stable-IDs, which a per-object +appending pool avoids entirely. Note for the p2p case: git's +`alternates`/object-pool mechanism (e.g. GitLab object pools for fork +networks) is the prior art for cross-repo dedup among related repos — +the pooled-CAS posture generalizes it to unrelated repos and to p2p +replication. -### alknet's appfile external-store probe — historical reference only +### alknet's appfile external-store probe — historical cross-check only -`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/` -(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written -against an older iroh-blobs. **Demoted 2026-10-02: alknet's code is -not a design input for this crate** (it is old, was poorly planned, -and has been decomposed into the alk* family; the substrate is -alkcall). What remains relevant from it is only the *shape* lineage — -the appfile pattern (small blobs in kv/sqlite, large on fs fallback, -filename↔hash mapping) and the multi-backend dispatch pain it -recorded, both of which this crate now owns by its own validated -design (POC #1, POC #3). Its re-verification ran 2026-10-02 as a -historical cross-check (`poc-largeblob-findings.md`, final section): -architectural claims about iroh-blobs held up, its performance claims -were asserted-never-measured and are superseded by POC #3's -first-party benchmark. +`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/` — written +against an older iroh-blobs. **Not a design input**: alknet's code is +old, was poorly planned, and has been decomposed into the alk* family; +the substrate is alkcall. The re-verification ran 2026-10-02 as part of +POC #3 (`poc-largeblob-findings.md`, final section): its architectural +claims about iroh-blobs held up, its performance claims were +asserted-never-measured and are superseded by POC #3's first-party +benchmark. What remains relevant is only the *shape* lineage — the +appfile pattern this crate now owns by its own validated design. ### alkcall — the substrate @@ -304,362 +307,233 @@ first-party benchmark. authorization seam for network-facing blob ops (`AccessControl`, producer/consumer vocabulary, alkcall ADR-022/037) and, if blob transfer rides channels, the established data-path patterns -(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even -belongs in this crate is itself open (OQ-BL-01); alkcall is the -substrate it composes with *when it does*. +(`BiStream`, two-pump `pump_bidi` ADR-050). The ops surface composes +with alkcall when and where it lands (OQ-BL-01's convergence). ## Open Questions -Numbering is provisional until the register solidifies; promote the -final set into Phase 1's `docs/architecture/open-questions.md`. +Numbering stable (OQ-BL-01..06); the final set is promoted into Phase +1's `docs/architecture/open-questions.md` with these states. -### OQ-BL-01: Crate scope — store-only, or store + ops surface? +### OQ-BL-01: Crate scope — store-only, or store + ops surface? — RESOLVED (shape) -The store (put/get/verify) is clearly in. What about the network ops -layer — a channels-native op family on the alkcall substrate, -auth-gated via `AccessControl`? iroh-blobs has it (provider protocol); -our posture is to separate it. Options: (a) store-only crate, ops in a -sibling crate later; (b) store + optional feature-gated ops module -here; (c) undecided pending the first consumer's shape. +The confirmed ops shape: a channels-native **have/need + verified +fetch** op family on the alkcall substrate, auth-gated via +`AccessControl` — p2p git made it a first-class consumer, not a +speculative one (everything is hash-addressed, so p2p sync *is* +hash-set diff + fetch). The streaming fetch handler shape is the +broadcast-fanout seam POC #3 validated (finding A3); the put/get +shapes ride POC #3's `LFSObject`-shaped seams. -Original lean was (a)/(b) on the inversion-point pattern. **Updated -2026-10-01:** the p2p-git consumer (§The p2p shape) makes the ops -surface likely rather than speculative — because everything is -hash-addressed, p2p sync *is* "announce hashes, diff hash sets, fetch -missing blobs verified," which is an op family on this store. Whether -that family lives here (feature-gated) or in the sibling that owns -replicator/gossip policy is the remaining question; the deciding input -is what alkgit's replicator actually needs and whether the -appfile-shaped consumers ever transfer blobs over the network. Even -so, the *shape* of -the ops (have/need + verified fetch, ACL-gated) is now firm enough to -plan against. +**Remaining decision (Phase 1, first wire consumer in sight):** whether +the op family lives here as a feature-gated module or in a sibling +crate that owns replicator/gossip policy. Deciding input: what +alkgit's replicator needs and whether appfile-shaped consumers ever +transfer blobs over the network. The shape is firm enough to plan +against either way. -### OQ-BL-02: Multi-backend dispatch — the appfile problem +### OQ-BL-02: Multi-backend dispatch — SETTLED (base); residuals named -**Base settled by POC #1 (2026-10-02, `poc-trait-dispatch-findings.md` -findings 1/2/6):** the lean `Backend` trait contract works — opaque -byte keys (typed `Key` at the store-layer boundary only), -`has`/`get`/`put`/`delete`/`list`/`name`, `list()` complete by -contract (a malformed list is as deadly as an incomplete one — the -reedb trap in finding 2), and size-threshold dual dispatch (kv small / -fs large) is a thin layer with deterministic per-digest routing. The -"virgin store" read path must treat a missing table as absent/empty. +**Settled (POC #1 findings 1/2/6, POC #3 findings A1-A5):** -Remaining (above-trait concerns, no further POC gate): migration -between backends policy; whether per-namespace backend config is ever -needed; the sqlite-vs-fs micro-benchmark (POC #3's pull-out) still -anchors the small-blob belief the dispatch leans on; streaming put/ -get shape rides POC #3's findings (`Vec` was the *contract* -question, not the performance question). +- The lean `Backend` trait contract works: opaque byte keys (typed + `Key` converts at the store-layer boundary only), + `has`/`get`/`put`/`delete`/`list`/`name`, and `list()` complete by + contract — a malformed list is as deadly as an incomplete one (the + redb key-vs-value trap; list correctness is only observable through + GC). +- Virgin-store read paths must treat a missing table as absent/empty. +- Size-threshold dual dispatch (kv small / fs large) is a thin layer + with deterministic per-digest routing (same content ⇒ same length ⇒ + same backend). Default Phase 1 threshold: **128 KiB** (tunable + constructor parameter; measured crossover ~128-256 KiB, sqlite + ~9-10× faster at 1-16 KiB — benchmark harness included in the POC + crate, re-runnable on real media). +- Streaming put seam: `(Option, stream)`; known-length one-pass + hashing as the encouraged path, unknown-length stage-then-hash + forced by the git preamble (finding A1). Stage-file hygiene: every + failure path converges on stage-discard, every success path on + commit-rename (finding A5). +- Ops-layer fetch handler: broadcast fanout above the backends + (finding A3), late joiners degrade to get-after-commit. -**Updated 2026-10-02 (POC #3, `poc-largeblob-findings.md`):** the -small-blob belief is now first-party measured — sqlite beats fs ~9-10× -at 1-16 KiB, crossover ~128-256 KiB (finding A4; harness included in -the POC crate, re-runnable on real media). Default Phase 1 threshold -lean: 128 KiB (tunable constructor param per POC #1's contract). -Streaming put/get shape also landed (findings A1/A2/A3): the put seam -is `(Option, stream)` with known-length as the encouraged one-pass -path and unknown-length staging-then-hashing (forced by the git -preamble); the ops layer's fetch handler should be the broadcast -fanout shape. An unknown-length-vs-known-length dispatch asymmetry -(finding A6) is a named Phase 1 decision, options recorded. +**Residuals (Phase 1 ADR inputs, no POC gate):** migration-between- +backends policy; whether per-namespace backend config is ever needed; +the unknown-length-vs-known-length dispatch asymmetry (finding A6: an +unknown-length re-put of a small blob lands on fs — options (a) +tier-correction sweep, (b) require `len` for small tier, (c) accept +duplication, (d) bounded-memory pre-threshold buffering; (d) looks +natural, weigh against batch-scope pinning). ### OQ-BL-03: Hash abstraction — RESOLVED: canonical git-family hash -**RESOLVED 2026-10-01 (the hash simplification round).** The original +**Resolved 2026-10-01; implementation-validated by POC #1 +(byte-exact vs `git hash-object`, 2026-10-02).** The original question — "trait, enum, or per-backend config for tolerating multiple -hash algorithms?" — was based on a premise worth questioning first: -*why did iroh-blobs' BLAKE3 ever enter our requirements?* traced back to -pure inheritance ("iroh-blobs uses it"). With that inheritance -explicitly rejected (convention 7's posture), the reason evaporated, and -the multi-hash framing dissolved with it. It was replaced by asking -which algorithm the *consumers* force: +hash algorithms?" — was based on an inherited premise: iroh-blobs uses +BLAKE3, so we must. With that inheritance rejected (convention 7), the +multi-hash framing dissolved; what the *consumers* force replaced it: -- **alkgit forces git's derivation** — by protocol definition, not - preference. Unavoidable. -- **The vfs/appfile/workspace case forces nothing** — any cryptographically - sound content hash works; the consumer never sees a hash *algorithm* - requirement, only *addresses*. It can use whatever the store keys. -- **Nothing else forces BLAKE3.** Its real virtues were (i) merkle- - nativeness (chunk-tree verification/range reads — iroh's transfer - story, valuable only if/when the conditional large-blob encoding - adopts chunking, OQ-BL-04) and (ii) raw hash throughput (irrelevant - at our scales: small-file paths are I/O/SQLite-bound; large-file - paths hash at disk speed even with SHA-256's hardware-accelerated - ~1-2 GB/s). Neither is a requirement today. +- **alkgit forces git's derivation** — by protocol definition. + Unavoidable. +- **The vfs/appfile/workspace case forces nothing** — any + cryptographically sound content hash works; consumers see + *addresses*, not algorithms. +- **Nothing forces BLAKE3.** Its virtues (merkle-native chunk trees, + raw throughput) are irrelevant at our scales under the canonical + digest. **The resolution, three tiers:** 1. **Canonical: `git-blob-sha-256`** — git's oid derivation (`"blob \0" + content`, SHA-256, uncompressed content) is the store's canonical algorithm. Git objects land in the pool as - first-class entries (the odb addresses the pool directly, no - indirection); and because one algorithm domain = one address space, - non-git consumers (workspaces, appfiles) collide into the same - pool *by construction* — the workspace manifest's path→hash entries - point at git oids without any mapping layer. A stronger dedup - posture than the multi-hash design offered: cross-consumer dedup - for free. + first-class entries (no indirection); one algorithm domain = one + address space, so non-git consumers collide into the same pool *by + construction* — cross-consumer dedup for free. 2. **Tolerated legacy: `git-blob-sha-1`** — existing git repos are - overwhelmingly SHA-1; the store must accept them (protocol - necessity, git's own hardened-SHA-1 threat model inherited verbatim - — a store entry under `git-sha-1` behaves like any git odb entry). - This is what keeps the abstraction an enum/capability set rather - than a hardcoded single algorithm: SHA-1-vs-SHA-256 is already two, - and the key encoding is a one-way door (a hash's length is baked - into physical keyspace and any wire surface). -3. **Conditional: BLAKE3** — demoted from requirement to encoding - consideration. If the large-blob path ever adopts a bao-like - chunk-tree verification encoding, BLAKE3 lives *there* (confined - to the encoding layer); its trees' verified leaves/wholes would be - registered into the pool as ordinary entries, like any other - content. Not designed for now; recorded so the demotion is - deliberate rather than forgotten. + overwhelmingly SHA-1; the store accepts them (protocol necessity; + git's own hardened-SHA-1 threat model inherited verbatim). +3. **Conditional: BLAKE3** — demoted to an encoding consideration: if + a bao-like chunk-tree transfer encoding is ever adopted (OQ-BL-04), + BLAKE3 lives there, confined to the encoding layer; its verified + whole-contents register into the pool as ordinary + git-blob-sha-256 entries. -**Security footnote for the record:** one algorithm across consumers is -safe here — SHA-256 has no practical cross-protocol ambiguity at these -input shapes, and the git-blob prefix domain-separates the derivation. -The SHA-1 caveat is already git's accepted position (collision-hardened -SHA-1); any real git consumer lives with the same. +**Security footnote:** one algorithm across consumers is safe here — +SHA-256 has no practical cross-protocol ambiguity at these input +shapes, and the git-blob prefix domain-separates the derivation. The +SHA-1 caveat is already git's accepted collision-hardened position. -**Residual (Phase 1, now smaller):** abstraction shape is a small enum -over `{git-sha-256 (canonical), git-sha-1 (tolerated), [blake3 -(conditional)]}` with domain-separated input handling (the -`blob \0` preamble is why `fn(content) → digest` alone is -insufficient) — but the *why* of each member is now decided; what -remains is key-encoding mechanics (byte layouts, length prefixes, -wire representation), which is a Phase 1 ADR with the first wire -consumer in sight (OQ-BL-01). The wire-format question reads OQ-BL-03 -first — a wire ADR inherits whatever key-encoding mechanics this -settles. +**Key-encoding mechanics (from POC #1 finding 5) — Phase 1 ADR:** the +tagged key encoding (`[tag][algorithm byte][digest]`) validated; +algorithm identity must remain part of the key itself (a per-store +single-algorithm config would not have survived the external-git-oid +interop test); the single tag byte is probably needless (one key kind +today); fixed-length enum-tagged keys sort cleanly for the sweep's +live-set diffs. This encoding is the one-way door — byte layouts, +length prefixes, wire representation are the ADR's content, and the +wire-format question (OQ-BL-01) inherits whatever it settles. -**Implementation-validated by POC #1 (2026-10-02):** the -domain-separated preamble abstraction carried both algorithms; -`git-sha256` and `git-sha1` coexist in one flat pool via the tagged -key encoding (`[algorithm byte][digest]`); external-git oids address -pool entries with no mapping layer (byte-exact vs `git hash-object`, -including the empty blob). Residual Phase 1 notes from the POC: the -algo-discriminant must remain part of the key itself (a per-store -single-algorithm config would not have survived the interop test); a -single leading tag byte is probably needless (currently one key -kind); fixed-length enum-tagged keys sort cleanly for the sweep's -live-set diffs. +### OQ-BL-04: Verification and chunking — SETTLED (posture); encoding conditional -### OQ-BL-04: Verification and chunking story +**Settled posture:** -iroh-blobs' verification is bao outboard encoding over fixed-size -BLAKE3 chunk trees (BLAKE3-native, and we rejected the BLAKE3 -inheritance — OQ-BL-03); git objects are self-verifying under git's own -model; raw large blobs (git-lfs-shaped, appfile large files) need -*some* verification story from us. Options (per the hash resolution, -none of them BLAKE3-mandatory): per-blob digest checks on put/get -paths (rudolfs' verify-as-decorator, verified prior art), a -bao-like chunk-tree encoding as a *conditional* later addition (BLAKE3 -confined to that encoding layer, OQ-BL-03 tier 3), or backend-native -verification. +- Whole-file CAS is the default granularity; git objects (and any + small-tier content) self-verify by reading whole and checking the + canonical digest — put/get paths hash-check. +- **Per-range verification is impossible under the canonical digest** + (POC #3 finding A2, independently reproduced in + `iroh-blobs-eval.md`): the preamble hashes the length, so no slice + has any hash relationship to the whole. iroh gets range + verification from the bao *chunk tree*, not from BLAKE3 the + algorithm. Consequences: local range reads need no verification + beyond a transport-integrity slice digest the caller carries + out-of-band (the store's `read_range` returns one); git objects + self-verify under git's model when read whole; small-tier range + reads slice whole values in memory. +- **The chunk-tree encoding is a transfer-layer conditional, not a + storage design.** Its only genuine consumer is networked ranged + fetch of large blobs (p2p git + LFS-shaped artifacts). If adopted + (BLAKE3-confined per OQ-BL-03 tier 3), its verified whole-contents + register as git-blob-sha-256 pool entries, so chunked and whole-file + paths dedup in the one pool. Content-defined chunking (fastcdc/ + restic-style) would be the shift-resistant choice for small-edit + large binaries; fixed-size trees (iroh/bao) are not inherited as the + default. Git objects never want chunking — they are unit blobs. +- If chunks were ever materialized as *themselves* small blobs (the + manifest-layer option), the store stays whole-file-only and chunks + pool like any other content. Not designed; parked behind the same + transfer-layer condition. -**Chunking is scoped (2026-10-01, principle 7), not open-ended:** the -default is whole-file CAS — content-defined chunking (fastcdc/ -restic-style; shift-resistant, unlike bao's fixed trees) is an -encoding consideration *only* for large binary blobs receiving small -edits, and git objects are unit blobs that never want chunking. The -remaining question is *where* variable chunking lives if adopted: in -the store's encoding layer (a blob is stored as chunk tree + the -store serves ranges) or in the manifest layer above (chunks are -themselves small blobs; the store stays whole-file-only — the dedup- -friendlier option, since chunks are then pooled and cross-file shared -like any other content). Decision input: how alkgit handles -git-lfs-shaped large files vs how agent workspaces transfer large -artifacts, and whether range reads are a store API or a -reassemble-above concern. With the canonical-hash resolution -(OQ-BL-03), a chunk-tree encoding that does get adopted would register -its verified whole-contents as git-blob-sha-256 pool entries (a -`git-blob-sha-256` digest over the reassembled content), so chunked -and whole-file paths dedup against each other in the one pool. +**Phase 1 surface addendum:** a cheap `stat(digest)` (length/type +probe) — gix's header-only read is git's cheapest primitive, and the +POC's `LfsObject.len` covers only the get side. +`stat(digest)` + `read_range` + `stream` form the large-blob serving +surface (also what packfile serving rides, OQ-BL-05). -**Sharpened by POC #3 (2026-10-02, `poc-largeblob-findings.md` -finding A2):** per-range verification is *impossible* under the -canonical git-blob digest — the preamble hashes the length, so no -slice has any hash relationship to the whole (iroh gets range -verification from the bao *chunk tree*, not from BLAKE3 the -algorithm). Consequences carried: (i) local range reads need no -verification beyond transport-integrity slice digests the caller -carries out-of-band (the store's `read_range` returns one); (ii) the -only genuine consumer of a chunk-tree encoding is *networked ranged -fetch of large blobs* — making the bao-like encoding a transfer-layer -conditional, not a storage design; (iii) git blobs self-verify under -git's own model when read whole, and small-tier range reads slice -whole values in memory. The whole-file CAS posture is strengthened, -not changed. Also added to Phase 1 surface notes: a cheap -`stat(digest)` (length/type probe; gix's header-only read is git's -cheapest primitive and the POC's `LfsObject.len` already covers the -get side). +### OQ-BL-05: Pooling and GC — settled in shape; residual mechanics named -### OQ-BL-05: Pooling and GC — namespaces as reference sets over a flat CAS - -**The decisions from the discussions (2026-10-01, principles 6; the -GC/pooling round; the rudolfs collision):** +**Settled decisions (2026-10-01 rounds; POC #1 findings 3/7/8; +POC #3 pack analysis):** 1. **One pooled CAS per node**; repos/workspaces are sets of hash references (manifests/trees) rather than isolated stores — cross-repo dedup by construction. 2. **Physically flat, logically namespaced** (the rudolfs inversion). - Physical layer: `hash → bytes`, flat, backends never see the - namespace (`{hex-prefix}/{hex-prefix}/{hash}` sharding survives — - fan-out, not partitioning). Logical layer: `namespace → {hashes}` - reference tables, which simultaneously give GC roots, per-namespace - sweeps, accounting, and the ACL boundary (alkcall `AccessControl` - gates the namespace for network ops). rudolfs' composite - `StorageKey(ns, oid)` API shape is kept; only its physical - namespacing is inverted. + Physical layer: `hash → bytes`, flat, `{hex-prefix}/{hex-prefix}/{hash}` + sharding survives, backends never see the namespace. Logical layer: + `namespace → {hashes}` reference tables giving GC roots, per- + namespace sweeps, accounting, and the ACL boundary (alkcall + `AccessControl` gates the namespace for network ops). 3. **Tags are the root table; references stay above the crate.** - iroh-blobs' named `Tag` (flat name → hash+format table) is the - "an external consumer cares about this hash" concept — namespaces - map onto it as `namespace → root manifest tag → mark-walk → {oids}`. - Git-flavored: refs (`refs/heads/*`) *are* the names; the consumer - registers them as tags. LFS pointer files are the same shape in - another hat — the pointer (oid + size) in the tree *is* the - reference; no extra tag needed because the consumer's own - structure holds it. **Store stays structure-blind**: it does not - walk consumer-side manifests. + iroh-blobs' named `Tag` is the "an external consumer cares about + this hash" concept — namespaces map onto it as `namespace → root + manifest tag → {oids}`. Git refs *are* the names; LFS pointer files + are the same shape (the pointer in the tree *is* the reference). + **Store stays structure-blind**: it does not walk consumer-side + manifests. +4. **GC mechanism: mark-and-sweep from registered roots** — validated + by POC #1 (exact sweep counts, abort semantics, delete-then-recover + as re-put byte-identical under the same oid). Roots = registered + namespace/root tags + put-path RAII pins + a protect callback with + abort semantics (`ProtectOutcome::Abort` — protection-source errors + skip the sweep rather than risk deletion). Liveness computation + beyond "these roots exist" is the consumer's job, via the callback + seam. +5. **Traversal ownership: lean (a) generalized protect callback + + temp-tag pinning** — the layer above computes liveness and hands + hash sets to GC; the store stays structure-blind. Option (b) + (store-native manifest traversal) stays parked unless a real + consumer's live-set computation proves too heavy; option (c) + (children-of callback walkthrough) superseded by the lean. +6. **Pack tension resolved in shape** — git objects enter the pool as + loose-equivalent kv entries (small tier, the common case); if + packfile serving is ever wanted, packs are stored as large blobs + and served via the store's range-read surface (git's own `.idx` + does per-object offset lookup; local range serving is fine per + finding A2). Dedup at pack granularity is weak, but pack-level + dedup across unrelated repos was never the goal — object-level + dedup lives in the loose tier. A per-object appending pool also + avoids gitoxide's pack-ID-stability machinery entirely (there are + no IDs to rebind; the address is the content). -**The GC picture, assembled from verified prior art:** +**Residuals (Phase 1 ADR inputs):** -- **Mechanism: mark-and-sweep from registered roots** (iroh-blobs' - model, verified — see §Prior art). Roots = registered namespace/root - tags + put-path RAII pins (TempTag-shaped) + a protect callback with - abort semantics (`ProtectOutcome` — protection-source errors skip the - sweep rather than risk deletion). The store does sweep; liveness - beyond "these roots exist" is the consumer's job (it walks its own - manifest format) — the callback seam. -- **Traversal ownership is THE open sub-question.** Iroh can walk - children in-mark because HashSeq is store-native; our - manifests/trees live above the crate, so either **(a) generalized - protect callback** — the layer above computes liveness and hands - hash sets to GC (store stays structure-blind; simplest; liveness - computation is the consumer's), **(b) store-native manifest format** - (a hashseq-like encoding so mark can traverse — cheap GC, but drags - structure into the store layer and creates a wire-ish format), or - **(c) reference-tracker seam** — roots registered with a - children-of(hash) callback; the store does mark/sweep through it. - Tentative lean: **(a) + temp-tag pinning**, revisiting (b) only if - traversal-per-sweep proves expensive (a large multi-tenant node - could pass large hash sets each run; incremental sweeps are an - in-(a) option since the callback is injectable). The three options - are a Phase 1 ADR, decided with the first consumer in sight. -- **Temp-tag pinning is needed regardless** of which option wins: a - blob put but not yet referenced by any namespace must survive - mid-write before any sweep sees it (iroh's `TempTag`/batch-scope - pattern). -- **Per-namespace sweeps** become possible with the reference tables - (delete only within a reclamation scope) — the multi-tenant - refinement the flat iroh model can't express; not needed at minimum. -- **Delete-then-recover semantics** — CAS append-only by nature; a - swept-but-somehow-referenced blob is a re-put, not corruption. The - dangerous direction is only deleting liveness (data loss), which the - abort-callback + pinning layers exist to prevent. +- Namespace registry shape (namespaced root tags vs separate reference + tables — POC #1 built in-memory `NamespaceTables`; table persistence + is mechanical Phase 1 work). +- GC scheduling (interval + injectable protection vs consumer-driven + sweeps; incremental sweeps are an in-(a) option since the callback + is injectable). +- **Concurrency — the sweep-vs-put race is a named Phase 1 + requirement** (POC #1 finding 3: the pin list read under a lock + leaves a real race window between "list" and "delete"). Prior art + named: iroh's DeleteSet/ProtectHandle transactions; the per-hash + serialized-actor + snapshot-observation pattern + (`iroh-blobs-eval.md` "re-borrowed conclusion"). Batch-scope pins + (iroh's `Batch::temp_tag`) for multi-put writes (manifest writes are + exactly this). +- API-shape lesson to carry: the liveness seam must be live-shared — + name it `register_liveness_source` (or have the source hold and + register itself), never `install_into` (POC #1 finding 7: the + "install" verb implied copy semantics and broke liveness). -**Still open (the residual mechanics):** namespace registry shape -(namespaced root tags vs separate reference tables — the POC built -in-memory reference tables; persistence of the tables is Phase 1 -mechanical work), GC scheduling (interval + injectable protect vs -consumer-driven sweeps), **concurrency** — the sweep-vs-put race is a -named Phase 1 requirement with iroh's DeleteSet/ProtectHandle -transactions as verified prior art (POC #1 finding 3; the POC worked -single-threaded), batch-scope pins for multi-put writes (finding 3), -and the callback-must-be-live-shared API lesson (finding 7 — prefer a -`register_liveness_source`-shaped seam over any install-copies-state -verb). Mark-and-sweep-with-protect-callback (lean (a) + pinning) is -now empirically the right mechanism; option (b) (store-native -manifest traversal) stays parked unless a real consumer's live-set -computation proves too heavy. - -**The pack tension (alkgit-shaped, constrains the store surface)** — -git packfiles are pack-efficient but blob one object-store-per-pack -(bad for cross-repo dedup); loose-per-object in the pooled store dedups -but is fs-inefficient at git's object volumes. Small git -objects in the kv backend largely dissolve this for the common -case; the residual question is what the store must support for -large/packed content — per-object granularity, range reads into -packed blobs, or both (interacts with OQ-BL-04's range-read -question — likely read them together). - -**Resolved in shape by POC #3 (2026-10-02, gix-odb analysis in -`poc-largeblob-findings.md`):** gix-odb's own model confirms the two -halves — alternates/multi-pack-index give shared-read across packs -with no dedup/GC/large-blob/streaming value-add (gix-lfs is an empty -placeholder), and gitoxide's whole pack-ID-stability machinery exists -only because packs are immutable-with-stable-IDs; a per-object -appending pool has no such IDs and avoids that machinery entirely. -The evidence-supported resolution: **git objects enter the pool as -loose-equivalent kv entries (small tier, the common case); if packfile -serving is ever wanted, packs are stored as large blobs and served by -the store's range-read surface** (git's own .idx does per-object -offset lookup — the store is a flat byte server for the pack; finding -A2 says local range serving is fine). Dedup at pack granularity is -weak, but pack-level dedup across unrelated repos was never the goal; -object-level dedup lives in the loose tier. Phase 1 ADR should record -this as the default resolution, revisit only with a real consumer -demand. - -### OQ-BL-06: POC register (draft) - -Numbered POCs, opened as research reaches them (findings land in -`docs/research/`; worktree placement per the SDD process). The GC round -reshaped the register — see the discussion at the end for what each POC -is actually deciding: +### OQ-BL-06: POC register — COMPLETE | # | What | Status | Where | |---|------|--------|-------| -| 1 | Backend-trait + dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path; the hash-enum abstraction with git-blob-sha-256's domain-separated preamble + git-sha-1 tolerance case | **Passed 2026-10-02** — `poc-trait-dispatch-findings.md` (8 findings; code: `/workspace/alkblobs-trait-poc`) | findings landed | -| 2 | ~~Multi-hash store~~ **Absorbed into #1** (2026-10-01 hash round): the canonical-hash resolution removed the "two families coexisting" question; what remains (the preamble abstraction + SHA-1 tolerance) is POC #1's trait work | **Absorbed** (and validated by #1) | — | -| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads; streaming `LFSObject`-shaped put/get + `fanout` seam) + the small-object sqlite-vs-fs micro-benchmark (the dual-belief anchor) | **Passed 2026-10-02** — `poc-largeblob-findings.md` (6 findings A1-A6 + pack-tension analysis + historical alknet probe re-check; code: `/workspace/alkblobs-largeblob-poc`) + `iroh-blobs-eval.md` | findings landed | -| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Covered in miniature by #1** (2026-10-02): namespace tables + sweep + recover all validated single-threaded; the concurrency half (sweep-vs-put race, batch-scope pins) is Phase 1 implementation work, not a POC gate | `poc-trait-dispatch-findings.md` findings 3/7/8 | +| 1 | Backend-trait + dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path; the hash-enum abstraction with git-blob-sha-256's domain-separated preamble + git-sha-1 tolerance case | **Passed 2026-10-02** (8 findings) | `poc-trait-dispatch-findings.md`; code: `/workspace/alkblobs-trait-poc` | +| 2 | ~~Multi-hash store~~ **Absorbed into #1** (2026-10-01 hash round) — the canonical-hash resolution removed the "two families coexisting" question; the residual (preamble abstraction + SHA-1 tolerance) was POC #1's trait work | **Absorbed**, validated by #1 | — | +| 3 | Large-blob path (streaming `LFSObject`-shaped put/get, fanout seam, fs range reads) + the small-object sqlite-vs-fs micro-benchmark (the dual-belief anchor) | **Passed 2026-10-02** (6 findings A1-A6 + pack-tension analysis + alknet probe re-check) | `poc-largeblob-findings.md` + `iroh-blobs-eval.md`; code: `/workspace/alkblobs-largeblob-poc` | +| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Covered in miniature by #1** — namespace tables + sweep + recover validated single-threaded; the concurrency half is Phase 1 implementation work, not a POC gate | `poc-trait-dispatch-findings.md` findings 3/7/8 | -Sequencing note: #1 and #3 are done; #2 absorbed/validated under #1; -#4's single-threaded core is covered by #1 (see the register row) and -the residual is implementation work. **The POC register is complete — -Phase 0's remaining step is convergence → Phase 1.** - -**What each POC is deciding (updated 2026-10-02, post-POC-#3):** - -- **POC #1 — DONE.** Decided the backend trait's minimum contract - (opaque byte keys, `list()` complete-by-contract, put-path - pinning) and validated the canonical-hash key encoding (git's - domain-separated preamble for both SHA-256 and SHA-1; external-git - oid interop byte-exact). Findings summary in the table row; full - detail in `poc-trait-dispatch-findings.md` (findings 1-8: opaque- - keys-at-the-boundary, the redb list/read-path traps, the pinning - shape + sweep-race caveat, the CLI-validated digest, the tagged key - format, dispatch simplicity, the callback-must-be-live-shared - lesson, GC semantics). -- **POC #2 is no more** — the "SHA-256 + BLAKE3 coexisting" question - evaporated when BLAKE3's only reason-for-being (iroh-blobs - inheritance) was rejected, and its residual content validated under - #1. -- **POC #3 is DONE** — the last POC. Streaming large-blob path - validated (`LFSObject` put/get; known-length one-pass vs unknown- - length stage-then-hash, forced by the git preamble — finding A1); - the fanout seam (serve-while-persisting) works as a broadcast above - the backends (A3); range reads work with the verification gap - documented honestly (A2 — impossible under the canonical digest, - sharpening the chunk-tree question to "transfer-layer conditional"); - the sqlite-vs-fs benchmark anchored the small-blob belief first-party - (~9-10× at 1-16 KiB, crossover ~128-256 KiB — A4); stage-file hygiene - tests caught a real leak (A5). The pack tension analyzed from - gix-odb and resolved in shape (packs-as-large-blobs served by range - reads). The iroh-blobs store eval landed as `iroh-blobs-eval.md`; - alknet's probe re-check is recorded as historical cross-check only - (per the 2026-10-02 clarification, alknet's code is not a design - input — the substrate is alkcall). -- **POC #4's single-threaded core is covered by #1** (namespace - tables + mark-and-sweep + protect-callback abort + delete-then- - recover all validated; `poc-trait-dispatch-findings.md` findings - 3/7/8). The concurrent half (sweep-vs-put race, batch-scope pins, - incremental scheduling) is Phase 1 implementation work — the - mechanism choice it was gating is settled, and races are - construction details with iroh's DeleteSet as the named prior art. +Sequencing outcome: #1 and #3 passed; #2 absorbed/validated under #1; +#4's single-threaded core covered by #1 (the mechanism choice it +gated — mark-and-sweep-with-protect-callback — is settled empirically; +races are construction details with iroh's DeleteSet as named prior +art). **The POC register is complete — Phase 0 ended here; Phase 1 +begins with the ADR backlog below.** POC placement conventions (inherited from alksocks/alktunnels): a POC that needs code from this repo runs in a worktree/branch @@ -668,23 +542,71 @@ self-contained POC runs as a standalone crate in the global workspace with findings written into `docs/research/` here. Findings always land in `docs/research/` regardless of where the code lives. -## Phase 0 plan (next steps) +## Convergence -1. ~~Write `iroh-blobs-eval.md`~~ — **done 2026-10-02** (POC #3's - reading against the current checkout; `docs/research/ - iroh-blobs-eval.md` — store traits/API, fs backend layout, bao/- - verification conclusions, small-blob handling, do-not-inherit and - keep-lists). -2. ~~Re-verify alknet's probe~~ — **done 2026-10-02**, recorded as a - historical cross-check only in `poc-largeblob-findings.md` - (alknet's code is not a design input; substrate is alkcall). The - one load-bearing output: the probe's small-blob performance claim - was asserted-never-measured; POC #3's benchmark is the anchor. -3. ~~Read gix-odb for the alkgit baseline~~ — **done 2026-10-02** - (POC #3's pack-tension analysis; conclusions folded into OQ-BL-05 - and the findings doc; gix-lfs is a placeholder crate, the - large-blob lane is open as assumed). -4. ~~POC #3~~ — **done 2026-10-02**, `poc-largeblob-findings.md`. -5. **Converge**: recommended approach + final OQ register → Phase 1. - All gates passed; the register rows above carry the empirical - inputs (thresholds, seams, lessons) into the SDD's Phase 1. \ No newline at end of file +Phase 0's final step per `docs/sdd_process.md`: *converge on a +recommended approach.* This section is that convergence — the +recommended approach, the evidence it stands on, and the ADR backlog +Phase 1 opens with. + +### Recommended approach (one paragraph) + +Build a content-addressed blob store keyed by the +git-blob-sha-256 derivation (git-blob-sha-1 tolerated, BLAKE3 +conditionally available only as a transfer-encoding leaf function): +a lean `Backend` trait (opaque byte keys, complete-by-contract +`list()`) with size-threshold dual dispatch — sqlite-like kv for +small blobs, flat sharded fs for large (default threshold 128 KiB, +constructor-tunable) — fronted by a store layer that owns typed keys, +streaming put seam `(Option, stream)` with stage-then-rename +commit discipline, get/range-read surface (`get`, `stat`, `read_range` +with out-of-band slice digests), RAII put-path pinning, and +mark-and-sweep GC whose liveness sources register via a live-shared +seam (protect callback with abort semantics) over a physically flat, +logically namespaced pool. Everything structure-aware — manifests, +refs, path→hash mapping, git semantics, replicator/gossip policy — +lives above the crate; any network ops surface (have/need + verified +fetch, broadcast fanout) is feature-gated here or in a sibling on the +alkcall substrate, decided with the first wire consumer. + +### Decision table (what was decided, on what evidence, what remains) + +| Area | Decision | Evidence | +|------|----------|----------| +| Canonical hash | git-blob-sha-256 canonical; git-blob-sha-1 tolerated; BLAKE3 conditional (encoding-layer only) | OQ-BL-03 resolution; POC #1 findings 4/5 (byte-exact vs git CLI, tagged-key coexistence) | +| Key encoding direction | Algorithm identity inside the key; fixed-length enum-tagged keys; tag byte likely dropped | POC #1 finding 5 (interop test survives per-store config) | +| Backend contract | Lean `Backend` trait: opaque byte keys, `has/get/put/delete/list/name`, `list()` complete by contract, virgin-store reads are no-ops | POC #1 findings 1/2 (redb traps); rudolfs S3 anti-lesson | +| Dispatch | Size-threshold dual routing, deterministic per digest; default 128 KiB | POC #1 finding 6; POC #3 A4 (sqlite ~9-10× at 1-16 KiB, crossover ~128-256 KiB) | +| Streaming | Put seam `(Option, stream)`; known-length one-pass encouraged, unknown-length stage-then-hash; stage-hygiene invariants tested | POC #3 A1/A5; iroh-blobs-eval (same shape, different reasons) | +| Ops fetch shape | Broadcast fanout (one reader, store arm + subscriber arms), late-join = get-after-commit | POC #3 A3; iroh `get_blob_ranges_impl`; rudolfs `fanout()` | +| Verification | Whole-blob via canonical digest; range reads carry out-of-band slice digests; chunk-tree encoding is a transfer-layer conditional only | POC #3 A2; iroh-blobs-eval §Verification | +| Pooling | One pooled CAS per node; physically flat, logically namespaced (reference tables above the byte layer) | Dedup discussion; rudolfs inversion; POC #1 namespace tests | +| GC | Mark-and-sweep; roots = tags + RAII pins + protect callback (abort semantics); traversal ownership lean (a), consumer computes liveness | iroh gc.rs verified read; POC #1 findings 3/7/8 (exact counts, abort, recover) | +| Git packs | Loose-equivalent kv entries are the default storage; packs, if stored, are large blobs served by range reads | POC #3 gix-odb analysis; finding A2 confirms local range serving is sound | +| Naming/API lessons | `register_liveness_source` (live-shared), not `install_into`; `stat(digest)` in the surface | POC #1 finding 7; POC #3 gix analysis | + +### Phase 1 ADR backlog (open with these) + +| ADR candidate | Scope | +|---------------|-------| +| Key encoding (OQ-BL-03 residual) | Byte layouts, length prefixes, algorithm discriminant, wire representation — the one-way door; must precede any wire consumer (OQ-BL-01) | +| Crate boundary & ops surface (OQ-BL-01 residual) | Store-only vs feature-gated ops module vs sibling crate; have/need + fetch op family shape; alkcall `AccessControl` integration | +| GC concurrency & scheduling (OQ-BL-05 residual) | Sweep-vs-put race solution (DeleteSet-shaped transactions / per-hash actors), batch-scope pins, scheduling, namespace-table persistence | +| Dispatch policy residuals (OQ-BL-02 residual) | Migration-between-backends; per-namespace backend config; unknown-length put asymmetry (finding A6 options) | +| Transfer encoding (OQ-BL-04 conditional) | Whether/when to adopt a chunk-tree (BLAKE3-confined; CDC if chunking for edit-resistance) encoding for networked ranged fetch — deferred until that consumer exists | +| Phase 1 open-questions register | Promote OQ-BL-01..06 (with the states above: OQ-BL-03 fully resolved; OQ-BL-01/02/04/05 partially resolved with named residuals; OQ-BL-06 complete) into `docs/architecture/open-questions.md` | + +## Phase 0 plan (retrospective — complete) + +1. ~~Write `iroh-blobs-eval.md`~~ — done 2026-10-02 (POC #3's reading + against checkout e82cbdc / v0.103.0). +2. ~~Re-verify alknet's probe~~ — done 2026-10-02, historical + cross-check only (`poc-largeblob-findings.md` final section; + alknet's code is not a design input — the substrate is alkcall). +3. ~~Read gix-odb for the alkgit baseline~~ — done 2026-10-02 (pack + tension folded into OQ-BL-05; gix-lfs is a placeholder crate). +4. ~~POC #3~~ — done 2026-10-02, `poc-largeblob-findings.md`. +5. ~~**Converge**~~ — done 2026-10-02: see §Convergence above. + Inputs to Phase 1: the settled-approach list, the decision table, + and the ADR backlog. Phase 1 opens with `docs/architecture/` and + the promoted open-questions register. \ No newline at end of file diff --git a/docs/research/poc-largeblob-findings.md b/docs/research/poc-largeblob-findings.md index 1d42639..b363d44 100644 --- a/docs/research/poc-largeblob-findings.md +++ b/docs/research/poc-largeblob-findings.md @@ -48,7 +48,7 @@ rides is alkcall (like alkhttp/alksocks/alktty/alktunnels). | Streaming get without whole-blob load | **Yes** — fs arm streams a file handle; kv arm one bounded read | | Fanout (serve while persisting) | **Yes** — one reader task, broadcast channel, store arm + N subscriber arms (finding A3) | | Range reads on fs | **Yes** — pread-window only; **verification has a real gap** (finding A2) | -| sqlite faster than fs for small blobs | **Confirmed and measured** — ~10× below 16 KiB; crossover ~128-256 KiB (B-finding) | +| sqlite faster than fs for small blobs | **Confirmed and measured** — ~9-10× at 1-16 KiB; crossover ~128-256 KiB (finding A4) | | Digest = git oid on streaming paths | **Byte-exact vs `git hash-object`** for both put paths | | Stage-file hygiene | **Verified** — failures (stream error, overflow, early-EOF) leave zero stage files (finding A5) |