alkblobs/docs/research/poc-trait-dispatch-findings.md

# POC: trait contract + digest abstraction + dispatch — findings

> **POC register #1** (absorbing #2), per `docs/research/phase-0.md` OQ-BL-06.
> Code: standalone crate `/workspace/alkblobs-trait-poc` (findings land here
> regardless of where the code lives, per the established convention).
> Date: 2026-10-02. Status: **passed**, with findings that reshape
> OQ-BL-02, OQ-BL-03 (key mechanics), and OQ-BL-05 (GC seam).

## What this POC set out to decide

From the phase-0 register (as updated by the GC and hash rounds):

1. **Backend trait's minimum contract** — can a lean `Backend` trait
   (`has`/`get`/`put`/`delete`/`list`) carry everything the store layer
   needs, with:
   - `list()` complete by contract (the rudolfs anti-lesson — incomplete
     listing is why its S3 backend can never GC);
   - not-yet-referenced pinning on the put path (the TempTag/RAII shape)?
2. **The digest abstraction's hardest member** — git's oid derivation
   (`"blob <len>\0" + content`, per OQ-BL-03's canonical resolution):
   does "algorithm as domain-separated derivation" work, can
   `git-sha256` (canonical) and `git-sha1` (tolerated) coexist in one
   flat pool, and does the pinned key encoding hold up?
3. **Dual-backend dispatch** — does the kv-small/fs-large appfile
   dispatch hold together empirically, including the get-fall-through
   case?
4. **GC mechanism** — namespace reference tables + mark-and-sweep with
   a protect callback + RAII pinning, exercised to the delete-
   then-recover semantics (POC #4's core, pulled forward because it
   needed only pinning, which the trait work required anyway).

## Result summary

| Question | Verdict |
|----------|---------|
| Lean `Backend` trait sufficient? | **Yes** — with `Key→bytes` at the boundary (finding 1) |
| `list()` complete by contract | **Yes** — dual-backend `list()` union passed; redb key-vs-value trap found (finding 2) |
| RAII pinning shape | **Yes** — `Pinned` guard with refcount-on-pins; sweep honors + callback can abort (finding 3) |
| git-blob derivation as digest | **Validated end to end** — byte-exact vs `git hash-object` (finding 4) |
| sha256 + sha1 coexisting via tagged keys | **Yes** (finding 5) |
| Size-threshold dual dispatch | **Yes** — routing, fall-through get, miss path (finding 6) |
| Namespace tables + sweep liveness | **Yes with a callback-share lesson** (finding 7) |
| Delete-then-recover (re-put) | **Yes** — re-put lands byte-identical under the same oid |

All 17 tests pass (`cargo test`), clippy `-D warnings` and `rustfmt`
clean. Digest validations run against the real `git hash-object` CLI
(git 2.43), not just hardcoded vectors.

## Findings

### Finding 1: the `Backend` trait should speak opaque bytes, not typed keys — Key→bytes at the boundary

The `Backend` trait uses `&[u8]` keys everywhere; `Key` (the typed
digest wrapper) converts at the store-layer boundary via
`Key::as_bytes()`. This kept backends dumb (they cannot know what a
digest is) and the digest layer free to evolve. For the real crate this
suggests the seam as built:

- `Backend` = opaque byte keys + byte values (+ `list`, `name`),
  exactly what was built in the POC (`src/backend.rs`).
- typed keys live in the store/digest layer only.

The alternative — making `Backend` generic over `K: Key` — was not
tried; with only one key type in play, generics would have bought
nothing and the opaque boundary is also the wire/persistence boundary
(backend files/rows must survive digest-layer evolution: a redb row or
fs filename has no schema migration story). Opaque bytes with a
self-describing tag byte (finding 5) is the durable choice.

### Finding 2: redb's `list()` returns what you iterate — read the key, not the value; missing-table startup; and the "list is load-bearing" proof

Three concrete redb behaviors worth recording for the real kv backend:

1. **Key-vs-value trap (caught by the tests, not by review):**
   `table.range(..)` yields `(key, value)` pairs; iterating with the
   wrong destructure produced a `list()` of *values* (the content
   bytes!). The failure showed up as a sweep that deleted 2 blobs when
   1 was expected (the live-check compared garbage keys, so the pinned
   blob's "key" was found in the computed live set... of content
   values). Lesson recorded: **`list()` correctness is only observable
   through GC** — a wrong list() is silent until a sweep deletes the
   wrong things, i.e. the rudolfs "incomplete list = broken GC" lesson
   generalize: **a *malformed* list is equally deadly and equally
   silent.** The test caught it only because deletion counts asserted
   exactly.
2. **Fresh-database read path:** a redb database opened but never
   written has *no tables* — `open_table` errors with
   `TableDoesNotExist` on the first read. Read paths must tolerate the
   missing-table error (treat as empty/absent) or the first `has()`
   on a fresh store fails. In the POC, `open_read_table()` maps
   `TableDoesNotExist → None` (and `list() → empty`). The real crate
   will hit the same thing no matter the kv engine; the general shape
   is "read paths on a virgin store must be no-ops, not errors."
3. **redb 2.x has no in-memory database** (iroh-blobs'
   previous-versions kv memory store notwithstanding — that is a
   different engine). The POC used `open_temp()` (temp dir file) for
   kv tests. For the real crate: an in-process ephemeral kv would be
   either `MemBackend` (the POC's `BTreeMap` one) or sqlite.

### Finding 3: pinning (TempTag equivalent) works at the store layer, above backends

The `Pinned` guard (`src/pin.rs`): put-path RAII that holds a per-key
refcount in the store's pin map. Sweep treats pin-map keys as part of
the live set. This proves the iroh-blobs `TempTag` *conclusion* without
their mechanism (no weak-pointer tag-drop plumbing): a boxed closure
dropped on guard release. What the POC did NOT yet build (and the real
crate must):

- Batch-scope pins (iroh's `Batch::temp_tag`): a scoped lifetime that
  protects a set of in-flight blobs and drops them together. The POC's
  per-blob guard sufficed for single-put flows; multi-put batches
  (manifest writes are exactly this) will want the scope form.
- The sweep-vs-put race: the POC's pins are a `parking_lot::Mutex`
  map, and sweep reads it under the lock before listing — for a real
  concurrent sweep (`tokio` task) the race window (a pin added between
  "list" and "delete") is real; iroh solves it with their DeleteSet
  transactions (`ProtectHandle`/protect-cancel). **Recorded as an open
  implementation requirement for Phase 1**, not solved by this POC.

### Finding 4: git-blob-sha-256 works — byte-exact against the git CLI

`Digest::git_sha256(content)` (preamble `blob <len>\0` + content,
SHA-256) matched `git hash-object --stdin --object-format=sha256` for
every test input: empty, short text, binary-with-NULs, 75 KB. Same for
`git-sha1` (git 2.43's default) at up to 130 KB. The domain-separated
preamble abstraction (an algorithm defines its own input preprocessing)
was sufficient — no `fn(content) → digest` bare-func limitation was
felt anywhere. Note the empirical gotcha: `--object-format=sha256` is
*not* a `git hash-object` flag in git 2.43 — it must go to
`git init --object-format=sha256` and the hash-object then runs inside
that repo (test harness detail, but cost an iteration to learn).

**Corollary validated (`store_addresses_git_oids_directly` test):** an
oid produced by *external git* addresses the same pool entry our store
puts — `Digest::from_git_oid_hex(...)` round-trips into the store, the
put dedups against it, and the get returns the content. This is the
OQ-BL-03 "no mapping layer" resolution confirmed at the implementation
level, not just on paper.

### Finding 5: tagged key format — one byte of tag + algorithm discriminant

The POC's key encoding: `[tag=0x01][algorithm byte][digest bytes]`,
total 1+1+len. Notes for the real design:

- The single tag byte is probably needless in the real crate (there is
  currently exactly one key *kind*); the **algorithm byte is the
  load-bearing part** — it makes `git-sha256` and `git-sha1` keys
  disjoint in the same flat pool with zero mapping. Test
  `key_format_is_tagged_and_length_checked` validated
  length-checking rejects malformed keys (truncated, unknown
  algorithm).
- Alternative considered but not built: variable-length
  length-prefixed keys. With only two algorithms, the enum-tagged
  fixed-length form is simpler and total-order stable (both key kinds
  sort cleanly in a BTreeMap — required for the sweep's live-set
  diffs).
- This encoding is inside the one-way door OQ-BL-03 flags; whatever
  the real crate picks, **algorithm identity must be part of the key
  itself** (a per-store single-algorithm config would not have
  survived the finding-4 corollary: external-git oids and pooled
  workspaces live in one table).

### Finding 6: dual-dispatch works as simply as hoped

`DualDispatch` (kv small / fs large, size threshold): routing by
`value.len()`, fall-through get (small miss → large), the
wrong-backend-miss case, and per-backend `backend_of()`. The appfile
dispatch is genuinely a thin layer over the trait — no surprises. One
structural note: since put routes by size, a *re-put* of the same
content will always route to the same backend (same length), so
dispatch is deterministic for a given digest; the "blob migrates
between backends" question (phase-0 OQ-BL-02) remains open but is
independent of the trait shape — nothing in the built contract
prevents a migrate-between-backends sweep later.

The `list()` union across backends (dedup on key) passed with both
backends disjoint — completing the contract requirement that GC's
sweep sees the whole pool.

### Finding 7: namespace tables + protect callback — the snapshot bug that proves the seam

The `NamespaceTables` (`src/namespace.rs`): reference sets keyed by
namespace name, installed into a store as a protect callback. The first
implementation **cloned the tables into the callback** (a snapshot) —
registering roots *after* install was invisible to GC, and liveness
broke. The fix (share the `Arc<RwLock<...>>` handle, not a snapshot) is
trivial but the failure mode is the important finding: **the protect
callback must be live-shared, and consumers will trip on this if the
real crate's API implies "install" copies state.** API-shape lesson
for Phase 1: call the seam `register_liveness_source` (or have
`NamespaceTables` *hold* the store reference and register itself)
rather than `install_into` — the "install" verb implied copy
semantics that broke exactly like the test showed.

With shared handles, the namespace tests pass cleanly: dropping a
namespace leaves the other namespace's blobs alive, the shared blob
survives both drops, and delete-then-recover re-puts
byte-identically — the OQ-BL-05 mechanics, validated in miniature.

### Finding 8: GC semantics — sweep counts are exact, and liveness composition works

The sweep composes liveness from three sources (pins, protect
callbacks, and the "nothing live" case) and the tests verify exact
deletion counts. The `protect_callback_can_abort_sweep` test validates
the iroh-blobs `ProtectOutcome::Abort` conclusion — a protection-
source failure aborts the run (typed error), nothing deleted.

What the POC did not test (honest scope limits, all Phase 1 work):
concurrent sweep-vs-put (see finding 3), incremental/interval
scheduling (the POC sweeps are explicit calls), and cross-hash
registration (deferred by OQ-BL-03 pending any consumer asking).

## Consequences for the phase-0 OQ register

- **OQ-BL-02 (dispatch):** the lean `Backend` trait contract with
  complete `list()` + opaque keys is validated as the base. Remaining:
  migration-between-backends policy and (maybe) per-namespace backend
  config — both above-trait concerns, no further POC gate.
- **OQ-BL-03 (hash):** the canonical git-family resolution is now
  implementation-validated, including external-git oid interop.
  Residual: only key-encoding mechanics for Phase 1 (finding 5 notes).
- **OQ-BL-05 (pooling/GC):** mark-and-sweep-with-protect-callback
  validated as the mechanism; namespace-as-root-tables works. Phase 1
  ADR inputs now include: the callback-must-be-live-shared lesson
  (finding 7), the sweep-vs-put race requirement from finding 3, and
  the pack tension (unchanged — rides POC #3's reading).
- **POC #4's GC worked example is effectively done in miniature** by
  this POC (namespace tables + sweep honored + recover semantics).
  The register should mark #4 as covered by #1 with the concurrency
  half still open as Phase 1 implementation work (not a POC gate).

## POC quality notes

- 17 tests total (5 lib-level, 12 integration), all passing; digest
  vectors cross-checked against the real `git` CLI (2.43.0),
  **not just hardcoded** — though the hardcoded sha256 vectors were
  themselves initially wrong (transcription) and the CLI check is what
  actually proved the implementation; a good reminder that known-
  answer tests are only as trustworthy as their source of truth.
- clippy `-D warnings` clean, `rustfmt` clean.
- Deliberate scope limits: no async I/O in backends (sync under
  `block_in_place` for the kv; fs is sync std), no streaming put/get
  (the POC's `Vec<u8>` API is the *contract* question, not the
  performance question — streaming shape rides POC #3's findings),
  no persistence tests across process restarts (redb/fs both durable
  engines; nothing store-layer-specific to prove).