alkblobs/docs/research/poc-trait-dispatch-findings.md

POC: trait contract + digest abstraction + dispatch — findings

POC register #1 (absorbing #2), per docs/research/phase-0.md OQ-BL-06. Code: standalone crate /workspace/alkblobs-trait-poc (findings land here regardless of where the code lives, per the established convention). Date: 2026-10-02. Status: passed, with findings that reshape OQ-BL-02, OQ-BL-03 (key mechanics), and OQ-BL-05 (GC seam).

What this POC set out to decide

From the phase-0 register (as updated by the GC and hash rounds):

1. Backend trait's minimum contract — can a lean Backend trait (has/get/put/delete/list) carry everything the store layer needs, with:
   • list() complete by contract (the rudolfs anti-lesson — incomplete listing is why its S3 backend can never GC);
   • not-yet-referenced pinning on the put path (the TempTag/RAII shape)?
2. The digest abstraction's hardest member — git's oid derivation ("blob <len>\0" + content, per OQ-BL-03's canonical resolution): does "algorithm as domain-separated derivation" work, can git-sha256 (canonical) and git-sha1 (tolerated) coexist in one flat pool, and does the pinned key encoding hold up?
3. Dual-backend dispatch — does the kv-small/fs-large appfile dispatch hold together empirically, including the get-fall-through case?
4. GC mechanism — namespace reference tables + mark-and-sweep with a protect callback + RAII pinning, exercised to the delete- then-recover semantics (POC #4's core, pulled forward because it needed only pinning, which the trait work required anyway).

Result summary

Question	Verdict
Lean Backend trait sufficient?	Yes — with Key→bytes at the boundary (finding 1)
list() complete by contract	Yes — dual-backend list() union passed; redb key-vs-value trap found (finding 2)
RAII pinning shape	Yes — Pinned guard with refcount-on-pins; sweep honors + callback can abort (finding 3)
git-blob derivation as digest	Validated end to end — byte-exact vs git hash-object (finding 4)
sha256 + sha1 coexisting via tagged keys	Yes (finding 5)
Size-threshold dual dispatch	Yes — routing, fall-through get, miss path (finding 6)
Namespace tables + sweep liveness	Yes with a callback-share lesson (finding 7)
Delete-then-recover (re-put)	Yes — re-put lands byte-identical under the same oid

All 17 tests pass (cargo test), clippy -D warnings and rustfmt clean. Digest validations run against the real git hash-object CLI (git 2.43), not just hardcoded vectors.

Findings

Finding 1: the Backend trait should speak opaque bytes, not typed keys — Key→bytes at the boundary

The Backend trait uses &[u8] keys everywhere; Key (the typed digest wrapper) converts at the store-layer boundary via Key::as_bytes(). This kept backends dumb (they cannot know what a digest is) and the digest layer free to evolve. For the real crate this suggests the seam as built:

• Backend = opaque byte keys + byte values (+ list, name), exactly what was built in the POC (src/backend.rs).
• typed keys live in the store/digest layer only.

The alternative — making Backend generic over K: Key — was not tried; with only one key type in play, generics would have bought nothing and the opaque boundary is also the wire/persistence boundary (backend files/rows must survive digest-layer evolution: a redb row or fs filename has no schema migration story). Opaque bytes with a self-describing tag byte (finding 5) is the durable choice.

Finding 2: redb's list() returns what you iterate — read the key, not the value; missing-table startup; and the "list is load-bearing" proof

Three concrete redb behaviors worth recording for the real kv backend:

1. Key-vs-value trap (caught by the tests, not by review): table.range(..) yields (key, value) pairs; iterating with the wrong destructure produced a list() of values (the content bytes!). The failure showed up as a sweep that deleted 2 blobs when 1 was expected (the live-check compared garbage keys, so the pinned blob's "key" was found in the computed live set... of content values). Lesson recorded: list() correctness is only observable through GC — a wrong list() is silent until a sweep deletes the wrong things, i.e. the rudolfs "incomplete list = broken GC" lesson generalize: a malformed list is equally deadly and equally silent. The test caught it only because deletion counts asserted exactly.
2. Fresh-database read path: a redb database opened but never written has no tables — open_table errors with TableDoesNotExist on the first read. Read paths must tolerate the missing-table error (treat as empty/absent) or the first has() on a fresh store fails. In the POC, open_read_table() maps TableDoesNotExist → None (and list() → empty). The real crate will hit the same thing no matter the kv engine; the general shape is "read paths on a virgin store must be no-ops, not errors."
3. redb 2.x has no in-memory database (iroh-blobs' previous-versions kv memory store notwithstanding — that is a different engine). The POC used open_temp() (temp dir file) for kv tests. For the real crate: an in-process ephemeral kv would be either MemBackend (the POC's BTreeMap one) or sqlite.

Finding 3: pinning (TempTag equivalent) works at the store layer, above backends

The Pinned guard (src/pin.rs): put-path RAII that holds a per-key refcount in the store's pin map. Sweep treats pin-map keys as part of the live set. This proves the iroh-blobs TempTag conclusion without their mechanism (no weak-pointer tag-drop plumbing): a boxed closure dropped on guard release. What the POC did NOT yet build (and the real crate must):

• Batch-scope pins (iroh's Batch::temp_tag): a scoped lifetime that protects a set of in-flight blobs and drops them together. The POC's per-blob guard sufficed for single-put flows; multi-put batches (manifest writes are exactly this) will want the scope form.
• The sweep-vs-put race: the POC's pins are a parking_lot::Mutex map, and sweep reads it under the lock before listing — for a real concurrent sweep (tokio task) the race window (a pin added between "list" and "delete") is real; iroh solves it with their DeleteSet transactions (ProtectHandle/protect-cancel). Recorded as an open implementation requirement for Phase 1, not solved by this POC.

Finding 4: git-blob-sha-256 works — byte-exact against the git CLI

Digest::git_sha256(content) (preamble blob <len>\0 + content, SHA-256) matched git hash-object --stdin --object-format=sha256 for every test input: empty, short text, binary-with-NULs, 75 KB. Same for git-sha1 (git 2.43's default) at up to 130 KB. The domain-separated preamble abstraction (an algorithm defines its own input preprocessing) was sufficient — no fn(content) → digest bare-func limitation was felt anywhere. Note the empirical gotcha: --object-format=sha256 is not a git hash-object flag in git 2.43 — it must go to git init --object-format=sha256 and the hash-object then runs inside that repo (test harness detail, but cost an iteration to learn).

Corollary validated (store_addresses_git_oids_directly test): an oid produced by external git addresses the same pool entry our store puts — Digest::from_git_oid_hex(...) round-trips into the store, the put dedups against it, and the get returns the content. This is the OQ-BL-03 "no mapping layer" resolution confirmed at the implementation level, not just on paper.

Finding 5: tagged key format — one byte of tag + algorithm discriminant

The POC's key encoding: [tag=0x01][algorithm byte][digest bytes], total 1+1+len. Notes for the real design:

• The single tag byte is probably needless in the real crate (there is currently exactly one key kind); the algorithm byte is the load-bearing part — it makes git-sha256 and git-sha1 keys disjoint in the same flat pool with zero mapping. Test key_format_is_tagged_and_length_checked validated length-checking rejects malformed keys (truncated, unknown algorithm).
• Alternative considered but not built: variable-length length-prefixed keys. With only two algorithms, the enum-tagged fixed-length form is simpler and total-order stable (both key kinds sort cleanly in a BTreeMap — required for the sweep's live-set diffs).
• This encoding is inside the one-way door OQ-BL-03 flags; whatever the real crate picks, algorithm identity must be part of the key itself (a per-store single-algorithm config would not have survived the finding-4 corollary: external-git oids and pooled workspaces live in one table).

Finding 6: dual-dispatch works as simply as hoped

DualDispatch (kv small / fs large, size threshold): routing by value.len(), fall-through get (small miss → large), the wrong-backend-miss case, and per-backend backend_of(). The appfile dispatch is genuinely a thin layer over the trait — no surprises. One structural note: since put routes by size, a re-put of the same content will always route to the same backend (same length), so dispatch is deterministic for a given digest; the "blob migrates between backends" question (phase-0 OQ-BL-02) remains open but is independent of the trait shape — nothing in the built contract prevents a migrate-between-backends sweep later.

The list() union across backends (dedup on key) passed with both backends disjoint — completing the contract requirement that GC's sweep sees the whole pool.

Finding 7: namespace tables + protect callback — the snapshot bug that proves the seam

The NamespaceTables (src/namespace.rs): reference sets keyed by namespace name, installed into a store as a protect callback. The first implementation cloned the tables into the callback (a snapshot) — registering roots after install was invisible to GC, and liveness broke. The fix (share the Arc<RwLock<...>> handle, not a snapshot) is trivial but the failure mode is the important finding: the protect callback must be live-shared, and consumers will trip on this if the real crate's API implies "install" copies state. API-shape lesson for Phase 1: call the seam register_liveness_source (or have NamespaceTables hold the store reference and register itself) rather than install_into — the "install" verb implied copy semantics that broke exactly like the test showed.

With shared handles, the namespace tests pass cleanly: dropping a namespace leaves the other namespace's blobs alive, the shared blob survives both drops, and delete-then-recover re-puts byte-identically — the OQ-BL-05 mechanics, validated in miniature.

Finding 8: GC semantics — sweep counts are exact, and liveness composition works

The sweep composes liveness from three sources (pins, protect callbacks, and the "nothing live" case) and the tests verify exact deletion counts. The protect_callback_can_abort_sweep test validates the iroh-blobs ProtectOutcome::Abort conclusion — a protection- source failure aborts the run (typed error), nothing deleted.

What the POC did not test (honest scope limits, all Phase 1 work): concurrent sweep-vs-put (see finding 3), incremental/interval scheduling (the POC sweeps are explicit calls), and cross-hash registration (deferred by OQ-BL-03 pending any consumer asking).

Consequences for the phase-0 OQ register

• OQ-BL-02 (dispatch): the lean Backend trait contract with complete list() + opaque keys is validated as the base. Remaining: migration-between-backends policy and (maybe) per-namespace backend config — both above-trait concerns, no further POC gate.
• OQ-BL-03 (hash): the canonical git-family resolution is now implementation-validated, including external-git oid interop. Residual: only key-encoding mechanics for Phase 1 (finding 5 notes).
• OQ-BL-05 (pooling/GC): mark-and-sweep-with-protect-callback validated as the mechanism; namespace-as-root-tables works. Phase 1 ADR inputs now include: the callback-must-be-live-shared lesson (finding 7), the sweep-vs-put race requirement from finding 3, and the pack tension (unchanged — rides POC #3's reading).
• POC #4's GC worked example is effectively done in miniature by this POC (namespace tables + sweep honored + recover semantics). The register should mark #4 as covered by #1 with the concurrency half still open as Phase 1 implementation work (not a POC gate).

POC quality notes

• 17 tests total (5 lib-level, 12 integration), all passing; digest vectors cross-checked against the real git CLI (2.43.0), not just hardcoded — though the hardcoded sha256 vectors were themselves initially wrong (transcription) and the CLI check is what actually proved the implementation; a good reminder that known- answer tests are only as trustworthy as their source of truth.
• clippy -D warnings clean, rustfmt clean.
• Deliberate scope limits: no async I/O in backends (sync under block_in_place for the kv; fs is sync std), no streaming put/get (the POC's Vec<u8> API is the contract question, not the performance question — streaming shape rides POC #3's findings), no persistence tests across process restarts (redb/fs both durable engines; nothing store-layer-specific to prove).