docs(research): POC #1 passed — trait contract + digest + dispatch
POC code: /workspace/alkblobs-trait-poc (17 tests passing, clippy -D warnings + fmt clean; digests validated byte-exact against the real git hash-object CLI, both object formats) Findings (poc-trait-dispatch-findings.md, 8): - lean Backend trait (opaque byte keys, complete list(), put-path pinning) is sufficient; Key typed wrapper lives at the boundary - redb traps: list() key-vs-value destructure (silent until GC catches it — malformed list is as deadly as incomplete), virgin-db TableDoesNotExist on first read, no in-memory db (temp file used) - pinning: Pinned RAII guard + refcount map works; batch-scope pins and the sweep-vs-put race are named Phase 1 requirements (iroh DeleteSet = prior art) - git-blob preamble abstraction validated; external-git oids address pool entries with zero indirection (the OQ-BL-03 resolution, implementation-confirmed) - tagged key format ([algo byte][digest]) makes git-sha256 + git-sha1 coexist; sort-stable for sweep diffs - dual dispatch works as simply as hoped; routing deterministic - namespace tables: protect callback MUST share live state (the snapshot-install bug proves the API lesson: register_liveness_source shape, not install_into) - GC semantics verified: exact deletion counts, abort-callback, delete-then-recover byte-identical re-put phase-0.md: POC #1 marked passed, #2 absorbed (validated), #4 covered in miniature (concurrency half deferred to Phase 1 impl work); OQ-BL-02 base settled; OQ-BL-03 implementation-validated; OQ-BL-05 mechanism settled with findings folded in; plan updated (POC #3 is the last open POC)
This commit is contained in:
glm-5.3-flash committed
1 parent
cde279b76d
commit
493516424a
2 files changed
+340
-65
No files matched your search
+94
-65
@@ -1,10 +1,9 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-01 (hash simplification round: canonical git-family
|
||||
hash resolution — BLAKE3 demoted from a requirement to a conditional;
|
||||
OQ-BL-03 rewritten, POC #2 absorbed into #1 and #3 given the
|
||||
micro-benchmark pull-out; earlier rounds: GC/pooling round,
|
||||
dedup/p2p consumer round, setup draft)
|
||||
last_updated: 2026-10-02 (POC #1 executed and passed — see
|
||||
poc-trait-dispatch-findings.md; POC #2 absorbed earlier; register
|
||||
updated; earlier rounds: hash simplification, GC/pooling, dedup/p2p,
|
||||
setup draft)
|
||||
---
|
||||
|
||||
# alkblobs — Phase 0 (Exploration)
|
||||
@@ -329,16 +328,21 @@ plan against.
|
||||
|
||||
### OQ-BL-02: Multi-backend dispatch — the appfile problem
|
||||
|
||||
Small blobs in kv/sqlite, large on fs fallback — the shape is agreed,
|
||||
the mechanics are not: how does put/get pick a backend (size
|
||||
thresholds? per-algorithm routing? per-namespace config?), how does it
|
||||
work when a blob should migrate between backends, and what happens on a
|
||||
get when the "wrong" backend was probed? alknet's probe hit exactly
|
||||
this ("dealing with more than one backend at a time"); its findings
|
||||
need re-verification against the current iroh-blobs before relying on
|
||||
them. Expected shape: a backend trait + a dispatch layer, but the trait
|
||||
shape is a one-way door once written — deserves a POC before
|
||||
committing.
|
||||
**Base settled by POC #1 (2026-10-02, `poc-trait-dispatch-findings.md`
|
||||
findings 1/2/6):** the lean `Backend` trait contract works — opaque
|
||||
byte keys (typed `Key` at the store-layer boundary only),
|
||||
`has`/`get`/`put`/`delete`/`list`/`name`, `list()` complete by
|
||||
contract (a malformed list is as deadly as an incomplete one — the
|
||||
reedb trap in finding 2), and size-threshold dual dispatch (kv small /
|
||||
fs large) is a thin layer with deterministic per-digest routing. The
|
||||
"virgin store" read path must treat a missing table as absent/empty.
|
||||
|
||||
Remaining (above-trait concerns, no further POC gate): migration
|
||||
between backends policy; whether per-namespace backend config is ever
|
||||
needed; the sqlite-vs-fs micro-benchmark (POC #3's pull-out) still
|
||||
anchors the small-blob belief the dispatch leans on; streaming put/
|
||||
get shape rides POC #3's findings (`Vec<u8>` was the *contract*
|
||||
question, not the performance question).
|
||||
|
||||
### OQ-BL-03: Hash abstraction — RESOLVED: canonical git-family hash
|
||||
|
||||
@@ -409,6 +413,18 @@ consumer in sight (OQ-BL-01). The wire-format question reads OQ-BL-03
|
||||
first — a wire ADR inherits whatever key-encoding mechanics this
|
||||
settles.
|
||||
|
||||
**Implementation-validated by POC #1 (2026-10-02):** the
|
||||
domain-separated preamble abstraction carried both algorithms;
|
||||
`git-sha256` and `git-sha1` coexist in one flat pool via the tagged
|
||||
key encoding (`[algorithm byte][digest]`); external-git oids address
|
||||
pool entries with no mapping layer (byte-exact vs `git hash-object`,
|
||||
including the empty blob). Residual Phase 1 notes from the POC: the
|
||||
algo-discriminant must remain part of the key itself (a per-store
|
||||
single-algorithm config would not have survived the interop test); a
|
||||
single leading tag byte is probably needless (currently one key
|
||||
kind); fixed-length enum-tagged keys sort cleanly for the sweep's
|
||||
live-set diffs.
|
||||
|
||||
### OQ-BL-04: Verification and chunking story
|
||||
|
||||
iroh-blobs' verification is bao outboard encoding over fixed-size
|
||||
@@ -506,19 +522,29 @@ GC/pooling round; the rudolfs collision):**
|
||||
abort-callback + pinning layers exist to prevent.
|
||||
|
||||
**Still open (the residual mechanics):** namespace registry shape
|
||||
(namespaced root tags vs separate reference tables), GC scheduling
|
||||
(interval + injectable protect vs consumer-driven sweeps), and the pack
|
||||
tension below.
|
||||
(namespaced root tags vs separate reference tables — the POC built
|
||||
in-memory reference tables; persistence of the tables is Phase 1
|
||||
mechanical work), GC scheduling (interval + injectable protect vs
|
||||
consumer-driven sweeps), **concurrency** — the sweep-vs-put race is a
|
||||
named Phase 1 requirement with iroh's DeleteSet/ProtectHandle
|
||||
transactions as verified prior art (POC #1 finding 3; the POC worked
|
||||
single-threaded), batch-scope pins for multi-put writes (finding 3),
|
||||
and the callback-must-be-live-shared API lesson (finding 7 — prefer a
|
||||
`register_liveness_source`-shaped seam over any install-copies-state
|
||||
verb). Mark-and-sweep-with-protect-callback (lean (a) + pinning) is
|
||||
now empirically the right mechanism; option (b) (store-native
|
||||
manifest traversal) stays parked unless a real consumer's live-set
|
||||
computation proves too heavy.
|
||||
|
||||
**The pack tension (alkgit-shaped, constrains the store surface)** —
|
||||
git packfiles are pack-efficient but blob one object-store-per-pack
|
||||
(bad for cross-repo dedup); loose-per-object in the pooled store dedups
|
||||
but is fs-inefficient at git's object volumes. Small git objects in the
|
||||
kv backend largely dissolve this for the common case; the residual
|
||||
question is what the store must support for large/packed content —
|
||||
per-object granularity, range reads into packed blobs, or both
|
||||
(interacts with OQ-BL-04's range-read question — likely read them
|
||||
together).
|
||||
but is fs-inefficient at git's object volumes. Small git
|
||||
objects in the kv backend largely dissolve this for the common
|
||||
case; the residual question is what the store must support for
|
||||
large/packed content — per-object granularity, range reads into
|
||||
packed blobs, or both (interacts with OQ-BL-04's range-read
|
||||
question — likely read them together).
|
||||
|
||||
### OQ-BL-06: POC register (draft)
|
||||
|
||||
@@ -529,50 +555,50 @@ is actually deciding:
|
||||
|
||||
| # | What | Status | Where |
|
||||
|---|------|--------|-------|
|
||||
| 1 | Backend-trait + dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path; the hash-enum abstraction with git-blob-sha-256's domain-separated preamble + git-sha-1 tolerance case | **Pending** — first POC | findings file TBD |
|
||||
| 2 | ~~Multi-hash store~~ **Absorbed into #1** (2026-10-01 hash round): the canonical-hash resolution removed the "two families coexisting" question; what remains (the preamble abstraction + SHA-1 tolerance) is POC #1's trait work | **Absorbed** | — |
|
||||
| 1 | Backend-trait + dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path; the hash-enum abstraction with git-blob-sha-256's domain-separated preamble + git-sha-1 tolerance case | **Passed 2026-10-02** — `poc-trait-dispatch-findings.md` (8 findings; code: `/workspace/alkblobs-trait-poc`) | findings landed |
|
||||
| 2 | ~~Multi-hash store~~ **Absorbed into #1** (2026-10-01 hash round): the canonical-hash resolution removed the "two families coexisting" question; what remains (the preamble abstraction + SHA-1 tolerance) is POC #1's trait work | **Absorbed** (and validated by #1) | — |
|
||||
| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads; streaming `LFSObject`-shaped put/get + `fanout` seam) + the small-object sqlite-vs-fs micro-benchmark (the dual-belief anchor) | **Pending** — reading-and-design POC + benchmark | findings file TBD |
|
||||
| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Pending** — after #1 (needs a backend to pool over) | findings file TBD |
|
||||
| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Covered in miniature by #1** (2026-10-02): namespace tables + sweep + recover all validated single-threaded; the concurrency half (sweep-vs-put race, batch-scope pins) is Phase 1 implementation work, not a POC gate | `poc-trait-dispatch-findings.md` findings 3/7/8 |
|
||||
|
||||
Sequencing note: #1 is the single opening worktree now (multi-hash
|
||||
subsumed; the git-preamble hashing is part of its trait prove-out);
|
||||
Sequencing note: #1 is done (single opening worktree; multi-hash
|
||||
subsumed; git-preamble hashing validated byte-exact vs the git CLI);
|
||||
#3 is a reading-and-design POC against the current iroh-blobs checkout
|
||||
plus the alknet probe re-verification, with its micro-benchmark
|
||||
pull-out; #4's GC half matters most — it is a correctness surface and
|
||||
should be a worked example at minimum. The pack tension (OQ-BL-05)
|
||||
rides #3's reading rather than earning its own POC yet.
|
||||
pull-out; #4's single-threaded core is covered by #1 (see the
|
||||
register row) and the residual is implementation work. The pack
|
||||
tension (OQ-BL-05) rides #3's reading rather than earning its own POC
|
||||
yet.
|
||||
|
||||
**What each POC is deciding (updated 2026-10-01, post-hash round):**
|
||||
**What each POC is deciding (updated 2026-10-02, post-POC-#1):**
|
||||
|
||||
- **POC #1 decides the backend trait's minimum contract and the
|
||||
canonical-hash key encoding** — and both are already sharpened by
|
||||
the prior rounds: `list()` must be complete (rudolfs' S3 backend
|
||||
punting on list is *why* it can never GC), the put path must
|
||||
support not-yet-referenced pinning (TempTag-shaped RAII, iroh-blobs'
|
||||
batch-scope pattern), and the digest/key abstraction must handle
|
||||
git's domain-separated preamble (`blob <len>\0`) for both SHA-256
|
||||
(canonical) and SHA-1 (tolerated) — i.e. `fn(content) → digest`
|
||||
alone is insufficient, per OQ-BL-03's resolution. The trait and key
|
||||
shapes are one-way doors; this POC is where they get evidence
|
||||
rather than instinct.
|
||||
- **POC #1 — DONE.** Decided the backend trait's minimum contract
|
||||
(opaque byte keys, `list()` complete-by-contract, put-path
|
||||
pinning) and validated the canonical-hash key encoding (git's
|
||||
domain-separated preamble for both SHA-256 and SHA-1; external-git
|
||||
oid interop byte-exact). Findings summary in the table row; full
|
||||
detail in `poc-trait-dispatch-findings.md` (findings 1-8: opaque-
|
||||
keys-at-the-boundary, the redb list/read-path traps, the pinning
|
||||
shape + sweep-race caveat, the CLI-validated digest, the tagged key
|
||||
format, dispatch simplicity, the callback-must-be-live-shared
|
||||
lesson, GC semantics).
|
||||
- **POC #2 is no more** — the "SHA-256 + BLAKE3 coexisting" question
|
||||
evaporated when BLAKE3's only reason-for-being (iroh-blobs
|
||||
inheritance) was rejected. Its residual content (the preamble
|
||||
abstraction, the SHA-1 tolerance case) is POC #1's hardest part now.
|
||||
- **POC #3 is the reading-and-design leg** — against the *current*
|
||||
iroh-blobs checkout (the store eval) + re-verifying alknet's probe.
|
||||
The **micro-benchmark pull-out** (2026-10-01, this round): the
|
||||
"kv/sqlite is faster for small blobs" belief underlies the dual-
|
||||
backend design and traces to *old* alknet research — worth anchoring
|
||||
empirically (small objects, 1KB-64KB, both backends, random +
|
||||
sequential) before the dispatch policy leans on it. The pack
|
||||
tension (OQ-BL-05) is analyzed here without its own POC.
|
||||
- **POC #4 is the GC worked example** — the correctness surface of
|
||||
OQ-BL-05: namespace reference tables + mark-and-sweep with
|
||||
protect-callback + TempTag pinning + delete-then-recover, exercised
|
||||
until the failure modes (pin missed, sweep racing a write, live-set
|
||||
miscomputed) are named rather than theoretical. Expected to be the
|
||||
POC that actually constrains the trait in return.
|
||||
inheritance) was rejected, and its residual content validated under
|
||||
#1.
|
||||
- **POC #3 is the next (and last) POC** — reading-and-design against
|
||||
the *current* iroh-blobs checkout (the store eval) + re-verifying
|
||||
alknet's probe + the **micro-benchmark pull-out** (the "kv/sqlite
|
||||
is faster for small blobs" belief underlies the dual-backend design
|
||||
and traces to *old* alknet research — anchor it: small objects,
|
||||
1KB-64KB, both backends, random + sequential). The pack tension
|
||||
(OQ-BL-05) is analyzed here without its own POC.
|
||||
- **POC #4's single-threaded core is covered by #1** (namespace
|
||||
tables + mark-and-sweep + protect-callback abort + delete-then-
|
||||
recover all validated; `poc-trait-dispatch-findings.md` findings
|
||||
3/7/8). The concurrent half (sweep-vs-put race, batch-scope pins,
|
||||
incremental scheduling) is Phase 1 implementation work — the
|
||||
mechanism choice it was gating is settled, and races are
|
||||
construction details with iroh's DeleteSet as the named prior art.
|
||||
|
||||
POC placement conventions (inherited from alksocks/alktunnels): a POC
|
||||
that needs code from this repo runs in a worktree/branch
|
||||
@@ -583,14 +609,17 @@ in `docs/research/` regardless of where the code lives.
|
||||
|
||||
## Phase 0 plan (next steps)
|
||||
|
||||
1. **Write `iroh-blobs-eval.md`** against the *current* checkout,
|
||||
focused on `src/store` (kv + flat backends, bao, chunking) — the
|
||||
conclusions inventory we may borrow, and the weld points we won't.
|
||||
1. ~~Write `iroh-blobs-eval.md`~~ — still open, rides POC #3's
|
||||
reading against the *current* checkout (kv + flat backends, bao,
|
||||
chunking; the GC mechanics eval is already done — see
|
||||
`poc-trait-dispatch-findings.md`'s context and phase-0's verified
|
||||
GC section).
|
||||
2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`,
|
||||
`poc-summary.md`) against the current upstream; mark what carried
|
||||
over.
|
||||
over — rides POC #3 too.
|
||||
3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit
|
||||
baseline: its storage layout, object model, and where a blob-store
|
||||
crate under/beside it earns its keep.
|
||||
4. **Open POCs** per OQ-BL-06, in the order above.
|
||||
4. **POC #3** (the last open POC): reading-and-design + the
|
||||
sqlite-vs-fs micro-benchmark per OQ-BL-06.
|
||||
5. Converge: recommended approach + final OQ register → Phase 1.
|
||||
@@ -0,0 +1,246 @@
|
||||
# POC: trait contract + digest abstraction + dispatch — findings
|
||||
|
||||
> **POC register #1** (absorbing #2), per `docs/research/phase-0.md` OQ-BL-06.
|
||||
> Code: standalone crate `/workspace/alkblobs-trait-poc` (findings land here
|
||||
> regardless of where the code lives, per the established convention).
|
||||
> Date: 2026-10-02. Status: **passed**, with findings that reshape
|
||||
> OQ-BL-02, OQ-BL-03 (key mechanics), and OQ-BL-05 (GC seam).
|
||||
|
||||
## What this POC set out to decide
|
||||
|
||||
From the phase-0 register (as updated by the GC and hash rounds):
|
||||
|
||||
1. **Backend trait's minimum contract** — can a lean `Backend` trait
|
||||
(`has`/`get`/`put`/`delete`/`list`) carry everything the store layer
|
||||
needs, with:
|
||||
- `list()` complete by contract (the rudolfs anti-lesson — incomplete
|
||||
listing is why its S3 backend can never GC);
|
||||
- not-yet-referenced pinning on the put path (the TempTag/RAII shape)?
|
||||
2. **The digest abstraction's hardest member** — git's oid derivation
|
||||
(`"blob <len>\0" + content`, per OQ-BL-03's canonical resolution):
|
||||
does "algorithm as domain-separated derivation" work, can
|
||||
`git-sha256` (canonical) and `git-sha1` (tolerated) coexist in one
|
||||
flat pool, and does the pinned key encoding hold up?
|
||||
3. **Dual-backend dispatch** — does the kv-small/fs-large appfile
|
||||
dispatch hold together empirically, including the get-fall-through
|
||||
case?
|
||||
4. **GC mechanism** — namespace reference tables + mark-and-sweep with
|
||||
a protect callback + RAII pinning, exercised to the delete-
|
||||
then-recover semantics (POC #4's core, pulled forward because it
|
||||
needed only pinning, which the trait work required anyway).
|
||||
|
||||
## Result summary
|
||||
|
||||
| Question | Verdict |
|
||||
|----------|---------|
|
||||
| Lean `Backend` trait sufficient? | **Yes** — with `Key→bytes` at the boundary (finding 1) |
|
||||
| `list()` complete by contract | **Yes** — dual-backend `list()` union passed; redb key-vs-value trap found (finding 2) |
|
||||
| RAII pinning shape | **Yes** — `Pinned` guard with refcount-on-pins; sweep honors + callback can abort (finding 3) |
|
||||
| git-blob derivation as digest | **Validated end to end** — byte-exact vs `git hash-object` (finding 4) |
|
||||
| sha256 + sha1 coexisting via tagged keys | **Yes** (finding 5) |
|
||||
| Size-threshold dual dispatch | **Yes** — routing, fall-through get, miss path (finding 6) |
|
||||
| Namespace tables + sweep liveness | **Yes with a callback-share lesson** (finding 7) |
|
||||
| Delete-then-recover (re-put) | **Yes** — re-put lands byte-identical under the same oid |
|
||||
|
||||
All 17 tests pass (`cargo test`), clippy `-D warnings` and `rustfmt`
|
||||
clean. Digest validations run against the real `git hash-object` CLI
|
||||
(git 2.43), not just hardcoded vectors.
|
||||
|
||||
## Findings
|
||||
|
||||
### Finding 1: the `Backend` trait should speak opaque bytes, not typed keys — Key→bytes at the boundary
|
||||
|
||||
The `Backend` trait uses `&[u8]` keys everywhere; `Key` (the typed
|
||||
digest wrapper) converts at the store-layer boundary via
|
||||
`Key::as_bytes()`. This kept backends dumb (they cannot know what a
|
||||
digest is) and the digest layer free to evolve. For the real crate this
|
||||
suggests the seam as built:
|
||||
|
||||
- `Backend` = opaque byte keys + byte values (+ `list`, `name`),
|
||||
exactly what was built in the POC (`src/backend.rs`).
|
||||
- typed keys live in the store/digest layer only.
|
||||
|
||||
The alternative — making `Backend` generic over `K: Key` — was not
|
||||
tried; with only one key type in play, generics would have bought
|
||||
nothing and the opaque boundary is also the wire/persistence boundary
|
||||
(backend files/rows must survive digest-layer evolution: a redb row or
|
||||
fs filename has no schema migration story). Opaque bytes with a
|
||||
self-describing tag byte (finding 5) is the durable choice.
|
||||
|
||||
### Finding 2: redb's `list()` returns what you iterate — read the key, not the value; missing-table startup; and the "list is load-bearing" proof
|
||||
|
||||
Three concrete redb behaviors worth recording for the real kv backend:
|
||||
|
||||
1. **Key-vs-value trap (caught by the tests, not by review):**
|
||||
`table.range(..)` yields `(key, value)` pairs; iterating with the
|
||||
wrong destructure produced a `list()` of *values* (the content
|
||||
bytes!). The failure showed up as a sweep that deleted 2 blobs when
|
||||
1 was expected (the live-check compared garbage keys, so the pinned
|
||||
blob's "key" was found in the computed live set... of content
|
||||
values). Lesson recorded: **`list()` correctness is only observable
|
||||
through GC** — a wrong list() is silent until a sweep deletes the
|
||||
wrong things, i.e. the rudolfs "incomplete list = broken GC" lesson
|
||||
generalize: **a *malformed* list is equally deadly and equally
|
||||
silent.** The test caught it only because deletion counts asserted
|
||||
exactly.
|
||||
2. **Fresh-database read path:** a redb database opened but never
|
||||
written has *no tables* — `open_table` errors with
|
||||
`TableDoesNotExist` on the first read. Read paths must tolerate the
|
||||
missing-table error (treat as empty/absent) or the first `has()`
|
||||
on a fresh store fails. In the POC, `open_read_table()` maps
|
||||
`TableDoesNotExist → None` (and `list() → empty`). The real crate
|
||||
will hit the same thing no matter the kv engine; the general shape
|
||||
is "read paths on a virgin store must be no-ops, not errors."
|
||||
3. **redb 2.x has no in-memory database** (iroh-blobs'
|
||||
previous-versions kv memory store notwithstanding — that is a
|
||||
different engine). The POC used `open_temp()` (temp dir file) for
|
||||
kv tests. For the real crate: an in-process ephemeral kv would be
|
||||
either `MemBackend` (the POC's `BTreeMap` one) or sqlite.
|
||||
|
||||
### Finding 3: pinning (TempTag equivalent) works at the store layer, above backends
|
||||
|
||||
The `Pinned` guard (`src/pin.rs`): put-path RAII that holds a per-key
|
||||
refcount in the store's pin map. Sweep treats pin-map keys as part of
|
||||
the live set. This proves the iroh-blobs `TempTag` *conclusion* without
|
||||
their mechanism (no weak-pointer tag-drop plumbing): a boxed closure
|
||||
dropped on guard release. What the POC did NOT yet build (and the real
|
||||
crate must):
|
||||
|
||||
- Batch-scope pins (iroh's `Batch::temp_tag`): a scoped lifetime that
|
||||
protects a set of in-flight blobs and drops them together. The POC's
|
||||
per-blob guard sufficed for single-put flows; multi-put batches
|
||||
(manifest writes are exactly this) will want the scope form.
|
||||
- The sweep-vs-put race: the POC's pins are a `parking_lot::Mutex`
|
||||
map, and sweep reads it under the lock before listing — for a real
|
||||
concurrent sweep (`tokio` task) the race window (a pin added between
|
||||
"list" and "delete") is real; iroh solves it with their DeleteSet
|
||||
transactions (`ProtectHandle`/protect-cancel). **Recorded as an open
|
||||
implementation requirement for Phase 1**, not solved by this POC.
|
||||
|
||||
### Finding 4: git-blob-sha-256 works — byte-exact against the git CLI
|
||||
|
||||
`Digest::git_sha256(content)` (preamble `blob <len>\0` + content,
|
||||
SHA-256) matched `git hash-object --stdin --object-format=sha256` for
|
||||
every test input: empty, short text, binary-with-NULs, 75 KB. Same for
|
||||
`git-sha1` (git 2.43's default) at up to 130 KB. The domain-separated
|
||||
preamble abstraction (an algorithm defines its own input preprocessing)
|
||||
was sufficient — no `fn(content) → digest` bare-func limitation was
|
||||
felt anywhere. Note the empirical gotcha: `--object-format=sha256` is
|
||||
*not* a `git hash-object` flag in git 2.43 — it must go to
|
||||
`git init --object-format=sha256` and the hash-object then runs inside
|
||||
that repo (test harness detail, but cost an iteration to learn).
|
||||
|
||||
**Corollary validated (`store_addresses_git_oids_directly` test):** an
|
||||
oid produced by *external git* addresses the same pool entry our store
|
||||
puts — `Digest::from_git_oid_hex(...)` round-trips into the store, the
|
||||
put dedups against it, and the get returns the content. This is the
|
||||
OQ-BL-03 "no mapping layer" resolution confirmed at the implementation
|
||||
level, not just on paper.
|
||||
|
||||
### Finding 5: tagged key format — one byte of tag + algorithm discriminant
|
||||
|
||||
The POC's key encoding: `[tag=0x01][algorithm byte][digest bytes]`,
|
||||
total 1+1+len. Notes for the real design:
|
||||
|
||||
- The single tag byte is probably needless in the real crate (there is
|
||||
currently exactly one key *kind*); the **algorithm byte is the
|
||||
load-bearing part** — it makes `git-sha256` and `git-sha1` keys
|
||||
disjoint in the same flat pool with zero mapping. Test
|
||||
`key_format_is_tagged_and_length_checked` validated
|
||||
length-checking rejects malformed keys (truncated, unknown
|
||||
algorithm).
|
||||
- Alternative considered but not built: variable-length
|
||||
length-prefixed keys. With only two algorithms, the enum-tagged
|
||||
fixed-length form is simpler and total-order stable (both key kinds
|
||||
sort cleanly in a BTreeMap — required for the sweep's live-set
|
||||
diffs).
|
||||
- This encoding is inside the one-way door OQ-BL-03 flags; whatever
|
||||
the real crate picks, **algorithm identity must be part of the key
|
||||
itself** (a per-store single-algorithm config would not have
|
||||
survived the finding-4 corollary: external-git oids and pooled
|
||||
workspaces live in one table).
|
||||
|
||||
### Finding 6: dual-dispatch works as simply as hoped
|
||||
|
||||
`DualDispatch` (kv small / fs large, size threshold): routing by
|
||||
`value.len()`, fall-through get (small miss → large), the
|
||||
wrong-backend-miss case, and per-backend `backend_of()`. The appfile
|
||||
dispatch is genuinely a thin layer over the trait — no surprises. One
|
||||
structural note: since put routes by size, a *re-put* of the same
|
||||
content will always route to the same backend (same length), so
|
||||
dispatch is deterministic for a given digest; the "blob migrates
|
||||
between backends" question (phase-0 OQ-BL-02) remains open but is
|
||||
independent of the trait shape — nothing in the built contract
|
||||
prevents a migrate-between-backends sweep later.
|
||||
|
||||
The `list()` union across backends (dedup on key) passed with both
|
||||
backends disjoint — completing the contract requirement that GC's
|
||||
sweep sees the whole pool.
|
||||
|
||||
### Finding 7: namespace tables + protect callback — the snapshot bug that proves the seam
|
||||
|
||||
The `NamespaceTables` (`src/namespace.rs`): reference sets keyed by
|
||||
namespace name, installed into a store as a protect callback. The first
|
||||
implementation **cloned the tables into the callback** (a snapshot) —
|
||||
registering roots *after* install was invisible to GC, and liveness
|
||||
broke. The fix (share the `Arc<RwLock<...>>` handle, not a snapshot) is
|
||||
trivial but the failure mode is the important finding: **the protect
|
||||
callback must be live-shared, and consumers will trip on this if the
|
||||
real crate's API implies "install" copies state.** API-shape lesson
|
||||
for Phase 1: call the seam `register_liveness_source` (or have
|
||||
`NamespaceTables` *hold* the store reference and register itself)
|
||||
rather than `install_into` — the "install" verb implied copy
|
||||
semantics that broke exactly like the test showed.
|
||||
|
||||
With shared handles, the namespace tests pass cleanly: dropping a
|
||||
namespace leaves the other namespace's blobs alive, the shared blob
|
||||
survives both drops, and delete-then-recover re-puts
|
||||
byte-identically — the OQ-BL-05 mechanics, validated in miniature.
|
||||
|
||||
### Finding 8: GC semantics — sweep counts are exact, and liveness composition works
|
||||
|
||||
The sweep composes liveness from three sources (pins, protect
|
||||
callbacks, and the "nothing live" case) and the tests verify exact
|
||||
deletion counts. The `protect_callback_can_abort_sweep` test validates
|
||||
the iroh-blobs `ProtectOutcome::Abort` conclusion — a protection-
|
||||
source failure aborts the run (typed error), nothing deleted.
|
||||
|
||||
What the POC did not test (honest scope limits, all Phase 1 work):
|
||||
concurrent sweep-vs-put (see finding 3), incremental/interval
|
||||
scheduling (the POC sweeps are explicit calls), and cross-hash
|
||||
registration (deferred by OQ-BL-03 pending any consumer asking).
|
||||
|
||||
## Consequences for the phase-0 OQ register
|
||||
|
||||
- **OQ-BL-02 (dispatch):** the lean `Backend` trait contract with
|
||||
complete `list()` + opaque keys is validated as the base. Remaining:
|
||||
migration-between-backends policy and (maybe) per-namespace backend
|
||||
config — both above-trait concerns, no further POC gate.
|
||||
- **OQ-BL-03 (hash):** the canonical git-family resolution is now
|
||||
implementation-validated, including external-git oid interop.
|
||||
Residual: only key-encoding mechanics for Phase 1 (finding 5 notes).
|
||||
- **OQ-BL-05 (pooling/GC):** mark-and-sweep-with-protect-callback
|
||||
validated as the mechanism; namespace-as-root-tables works. Phase 1
|
||||
ADR inputs now include: the callback-must-be-live-shared lesson
|
||||
(finding 7), the sweep-vs-put race requirement from finding 3, and
|
||||
the pack tension (unchanged — rides POC #3's reading).
|
||||
- **POC #4's GC worked example is effectively done in miniature** by
|
||||
this POC (namespace tables + sweep honored + recover semantics).
|
||||
The register should mark #4 as covered by #1 with the concurrency
|
||||
half still open as Phase 1 implementation work (not a POC gate).
|
||||
|
||||
## POC quality notes
|
||||
|
||||
- 17 tests total (5 lib-level, 12 integration), all passing; digest
|
||||
vectors cross-checked against the real `git` CLI (2.43.0),
|
||||
**not just hardcoded** — though the hardcoded sha256 vectors were
|
||||
themselves initially wrong (transcription) and the CLI check is what
|
||||
actually proved the implementation; a good reminder that known-
|
||||
answer tests are only as trustworthy as their source of truth.
|
||||
- clippy `-D warnings` clean, `rustfmt` clean.
|
||||
- Deliberate scope limits: no async I/O in backends (sync under
|
||||
`block_in_place` for the kv; fs is sync std), no streaming put/get
|
||||
(the POC's `Vec<u8>` API is the *contract* question, not the
|
||||
performance question — streaming shape rides POC #3's findings),
|
||||
no persistence tests across process restarts (redb/fs both durable
|
||||
engines; nothing store-layer-specific to prove).
|
||||
Reference in new issue
Block a user