Files
alkblobs/docs/research/phase-0.md
T
glm-5.3-flash 55450d6bf0 docs(research): consolidate GC/pooling round into phase-0
- iroh-blobs GC mechanics verified against current checkout:
  mark-and-sweep (named tags + TempTag RAII + ProtectCb with abort),
  format-aware mark traversal, DeleteSet write-safety, no namespaces
  (flat tags) — our namespace concept maps onto their machinery
- rudolfs prior art section: StorageKey(ns, oid) composite key +
  decorator storage stack; THE ANTI-PATTERN inverted (physically
  namespaced storage defeats cross-tenant dedup); list() is
  load-bearing; verify-as-decorator; streaming LFSObject + fanout
- OQ-BL-03: oid<->hash mapping DISSOLVED — git's oid derivation is
  itself a hash algorithm (domain-separated preamble); per-algorithm
  namespaces; mapping only reappears for transfer-vs-storage hashes
- OQ-BL-05 consolidated: physically flat/logically namespaced; tags
  are the root table; references stay above the crate; mark-and-sweep
  with protect-callback lean (a) + TempTag pinning; traversal
  ownership (a/b/c) is the Phase 1 ADR
- POC register updated: #1 adds list() completeness + pinning
  contract, #2 reframed as validating oid-dissolution, #4 as the GC
  worked example; 'what each POC decides' section added
2026-10-01 06:23:32 +00:00

534 lines
30 KiB
Markdown

---
status: draft
last_updated: 2026-10-01 (GC/pooling round: iroh-blobs mark/sweep verified
against the current checkout, rudolfs prior art added, oid↔hash mapping
dissolved into OQ-BL-03 as per-algorithm namespaces, OQ-BL-05 consolidated
around tags-as-roots; earlier rounds: dedup/p2p consumer round, setup draft)
---
# alkblobs — Phase 0 (Exploration)
This document captures Phase 0 (Exploration) for the `alkblobs` crate:
vision, guiding principles, prior art, and open questions (OQ-BL-01..NN).
Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding
principles; research options; validate approaches; converge on a
recommended approach.* It is the input to Phase 1 (Architecture), where
the Architect will produce `docs/architecture/` specs, ADRs, and the
open-questions tracker.
Drafted 2026-09-30, emerging from the initial setup discussion. Nothing
is converged yet — this is the working sketch, not a specification.
## Vision and guiding principles
**One sentence:** content-addressed blob storage in the alk* family —
a multi-hash put/get/verify store with pluggable backends (small blobs
in a kv/sqlite-ish store, large blobs on a filesystem fallback), split
from any transport/protocol layer and tolerant of hash-algorithm
conflicts (the alkgit case), with auth-gated network operations riding
the alkcall seam when they exist.
**Why this crate exists (three converging consumers):**
1. **alkgit** (planning phase) needs an object backend that is *not*
its proposed default file-based backend ("kind of gross") — blobs
stored under git's own object hashes, but with iroh-blobs-like
handling for large blobs (where git typically delegates to git-lfs;
we could instead follow an iroh-blobs-shaped large-blob path). The
conflict: iroh-blobs is BLAKE3-only and content-hashes with bao
verification, while git's object database is SHA-1/SHA-256 with
git's own object format. A store that insists on one hash algorithm
cannot serve git objects directly.
2. **The alknet rewrite** (the original alk* project, being decomposed
and improved) needs the "appfile" external-store shape: small blobs
in a kv/sqlite store (faster than the filesystem for small items —
true beyond sqlite: it applies to the kv store iroh-blobs uses
too), large blobs on a filesystem fallback, with filename↔hash
mapping. alknet's own research hit the core awkwardness of
dispatching across more than one backend at a time — which is
exactly a store-layer problem this crate should own.
3. **Agent workspaces** — many agents working simultaneously in
workspaces that are mostly identical to each other's, each making
small edits in specific areas. Per-agent stores multiply the near-
identical content; a shared content-addressed store dedups it by
construction, and a path→hash manifest per workspace makes the
per-agent delta just its changed hashes (the alknet-filesystem
probe shows the manifest mapping is straightforward).
**The p2p shape and what it pins (added 2026-10-01, from the dedup
discussion).** Two alkgit use cases exist: self-hosted git (where
cross-repo dedup matters little — you don't fork your own repos) and
**p2p git for OSS projects**, the demanding one. The p2p sketch:
ownership/ACL live in a smart contract on a low-fee network;
replicators are push/clone endpoints that watch the contract and cache
its state; repos sync via iroh-style gossip (hashes, not content) and
pull/sync on demand. Neither alkgit approach described so far takes
advantage of dedup — which is exactly a store-layer property. The
implications for this crate:
- Both hard consumers (p2p replicators, agent swarms) reduce to the
same shape: **a pooled CAS per node**, where repos / workspaces are
*sets of hash references* (trees/manifests), not separate object
stores. Cross-repo dedup is then by construction; git's own
`alternates`/object-pool mechanism is the same idea done awkwardly.
- Because everything is hash-addressed, the p2p wire story collapses
to "announce hashes, diff hash sets, fetch missing blobs verified" —
which is precisely the ops surface OQ-BL-01 left provisional. The
consumer exists now: if p2p git is a confirmed alkgit direction,
the ops surface upgrades from "leaning store-only" to a confirmed
have/need + fetch op family, auth-gated via the alkcall seam.
**The scope line (current posture, revisit as evidence arrives):**
this crate is the *store*, not the *transport*. iroh-blobs welds store
+ provider protocol + tickets + postcard into one crate; the defining
posture here is the opposite split — put/get/verify against a hash is
its own layer, and any provider/protocol/ops surface lives above it or
in feature-gated modules (the alk* inversion-point pattern). The ops
surface (a channels-native "have/need + fetch blob with this hash" op
family, auth-gated via `AccessControl`) is a Phase 0/1 question — the
p2p-git consumer (§The p2p shape) upgrades it from speculative to
likely, but the crate-boundary decision itself is still open
(OQ-BL-01). What stays *above* the crate regardless: repos/workspaces
as reference sets (trees/manifests), path→hash mapping, git semantics,
the contract/gossip/replicator policy layer.
Guiding principles, inherited from the alk* family:
1. **Substrate-agnostic by construction.** The store must not know
whether bytes arrive over the network, from a local writer, or get
reassembled anywhere particular. Backends behind an injected seam
(alktty `TtyBackend` / alktunnels pump-halves precedent).
2. **Hashes are data, not transport identity.** Multiple hash
algorithms must coexist (git SHA-1/SHA-256, BLAKE3), each with its
own domain-separated derivation ("git-blob-sha256" hashes
`blob <len>\0 + content`, not bare content — OQ-BL-03's resolution);
consumers address the pool by the algorithm their format demands,
no mapping layer. The abstraction shape (trait, enum, per-backend
config) is an open question (OQ-BL-03).
3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets,
postcard serialization, and provider protocol are design-welded
choices we do not inherit. Its *store-shape* lessons (kv + flat
backends, verification flow, chunking) are fair game per prior-art
reading.
4. **Producer/consumer vocabulary** for any network-facing surface;
authorization via alkcall's `AccessControl`/identity seam, never an
in-band invented scheme (convention 6).
5. **Verify where it matters.** iroh-blobs' core virtue is verified
transfer (bao outboard encoding). The alkgit case already has
verified content (git objects are hash-addressed by git itself).
Which verification story the crate owns — bao trees, per-blob
digests, backend-native, or the rudolfs verify-as-decorator shape —
is open (OQ-BL-04).
6. **Pooling is the point.** The dedup wins both demanding consumers
want (Forknet-style multi-repo OSS content, agent swarms) require
that repos/workspaces are *sets of hash references* over one pooled
CAS per node, not walled-off per-repo stores. Pooling forces a GC
story (OQ-BL-05) and a multi-backend dispatch story (OQ-BL-02).
7. **Whole-file CAS as the default; chunking earns its keep only
where it must.** For source-code-scale content, whole-file hashing
captures the agent/edit delta perfectly and needs no chunk tree;
fixed-size chunk trees are actually *hostile* to insertion/deletion
edits (byte shifts cascade). Content-defined chunking (CDC —
fastcdc/restic-style, shift-resistant) only pays on large binary
files receiving small edits — the git-lfs weakness. iroh-blobs/bao
use fixed-size trees; do not inherit that as the default. Where
chunking lives (store encoding vs manifest layer) is OQ-BL-04.
## What is already known (settled, thin)
Almost nothing is pinned — deliberately. The only postures agreed at
setup:
- **Not a fork of iroh-blobs.** A downstream store crate inspired by
its `src/store` work, diverging deliberately on hashing and wire
surface (see `iroh-blobs-eval` below when written).
- **Multi-hash from day one** (hard requirement from alkgit — do not
hardcode BLAKE3).
- **Backend pluralism assumed, not designed.** kv/sqlite for small
blobs, filesystem fallback for large — the appfile shape — but
*how* multi-backend dispatch works is exactly what alknet's research
ran into, so it earns research and probably POCs (OQ-BL-02).
- **Pooled CAS, not per-repo stores** (agreed in the 2026-10-01
dedup discussion — the load-bearing scope decision so far): repos
and agent workspaces are sets of hash references over one shared
content-addressed store. This is the property both demanding
consumers actually need; everything else (GC, manifests, git
semantics) lives around it. Details open (OQ-BL-05).
- **Physically flat, logically namespaced** (2026-10-01 GC/pooling
round; the rudolfs inversion): the byte layer is a flat
dedup-by-construction CAS; namespaces are reference tables above it
(GC roots, sweep scoping, ACL boundary). Backends stay
namespace-blind.
- **Whole-file CAS is the default granularity**; chunking is scoped
to the large-binary-blob case (principle 7), not the default path.
## Prior art
### iroh-blobs — the shape inspiration (evaluated, not the base)
`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference).
Specifically `src/store`: the kv backend (small blobs, in-process) and
flat file backend (large blobs) split; bao outboard encoding and
verification flow; chunking. Its BLAKE3-only hashing, tickets, postcard
serialization, and provider protocol are the design-welded choices we
diverge from. **The store eval should be written against the current
checkout** — alknet's older research refers to an older iroh-blobs and
its conclusions must be re-verified rather than inherited.
**GC mechanics — verified against the current checkout (2026-10-01,
`src/store/gc.rs`, `src/util/temp_tag.rs`, `src/store/fs/delete_set.rs`).
This is the load-bearing prior art for OQ-BL-05:**
- **Mark-and-sweep, not refcounting, for persistent liveness.**
`gc_mark_task` collects roots from three sources: persistent *named
tags* (a flat `tags-0` table, name → `HashAndFormat`), in-memory *
`TempTag`s* (RAII-refcounted, `#[must_use]`, `.leak()` for
pin-until-exit-of-process), and an injectable `ProtectCb` — a
callback GC consults before each run that can add externally-known
hashes or **abort the run** (`ProtectOutcome::Abort`; a flaky
protection source skips the sweep rather than risking deletion).
- **Format-aware mark traversal** — a non-raw root (HashSeq/collection)
contributes all reachable children to the live set via the bao hash
stream (gc.rs:63-78). Protecting a collection protects everything
reachable from it.
- **Sweep** lists the whole store and batch-deletes (~100/batch)
anything not live. `list()` being complete is therefore load-bearing
for GC — relevant to the backend trait contract (OQ-BL-02).
- **Write-safety against a concurrent sweep** — the fs backend carries
a separate `DeleteSet` transaction layer (`ProtectHandle` /
mark-for-delete / protect-cancel / commit); `TempTag` holds a refcount
so a blob put but not yet referenced cannot vanish mid-write. Temp
tags are scoped (batch scope / process scope) with per-scope counters.
- **No namespaces in their model** — flat named tags over a flat
pool. Our namespace concept (OQ-BL-05) lands on their machinery as
policy over what counts as a root: `namespace → root manifest →
(mark-walk) → {oids}` is tags plus traversal, not new storage
concepts.
### rudolfs — git-lfs server; the composite-key + decorator-storage prior art
`/workspace/rudolfs` (v0.3.8, MIT, read-only reference; the alknet
research doc is
`/workspace/@alkdev/alknet/docs/research/references/gitlfs/
rudolfs-reference.md`). A git-lfs server whose **storage layer**
(`src/storage/`) is the most direct ancestor of our composite-key +
dispatch shape:
- **`StorageKey = (Namespace, Oid)`** — a composite key over a flat CAS
(`Namespace = (org, project)` strings, `Oid = SHA-256`). All backend
operations take the composite key.
- **`Storage` trait** — `get`/`put`/`size`/`delete`/`list`/`
total_size`/`max_size`/`public_url`/`upload_url`, with
`LFSObject = (len: u64, ByteStream)` — the streaming-first put/get
shape (pinned boxed async byte stream), the natural large-blob API
and likely the channels ops-surface shape.
- **Decorator composition** — `Verify ↔ Encrypted ↔ Cached(LRU →
permanent) ↔ Retrying → s3/disk`. The `Cached` decorator is the
appfile shape in LRU form: "fast store in front of fallback" as a
composable wrapper, an alternative dispatch policy to size
thresholds. `fanout()` duplicates one stream into two lock-step
copies (serve + persist) — precisely "serve the fetch while
persisting on receipt" for networked gets.
- **Verify-as-decorator (OQ-BL-04 input)** — streaming SHA-256 check on
both put and get paths with auto-purge of a corrupted tier.
- **Footnote if at-rest encryption is ever considered** — nonce derived
from the oid: deterministic and correct, but a key rotation
invalidates every object and breaks dedup across keys (the known
encryption-vs-dedup tension).
- **THE ANTI-PATTERN (inverted, not inherited)** — namespaces are
*physical*: `s3://{org}/{project}/{sha256}` — identical content in
two orgs is stored twice. It buys tenant isolation by paying the
cross-tenant dedup that is our whole reason for pooled CAS. The
reconciliation recorded in OQ-BL-05: keep the composite-key API
shape, put the namespace half in metadata (reference tables) above
the byte layer, and keep physical storage flat (`oid → bytes`,
backends namespace-blind).
- **`list()` is load-bearing** — the S3 backend punts on `list()`
(returns an empty stream) and its `delete()` is a no-op, so it can
*never* GC. A lesson for the backend trait contract: list/sweep
support is a real requirement, not an optional extra.
### gix-odb — git's own object database (alkgit's baseline)
`/workspace/git-oxide/gix-odb` (read-only reference). The backend
alkgit currently plans against; the hashing-algorithm baseline git
actually uses (SHA-1/SHA-256), git's loose-object and packfile layout,
and git's already-hash-addressed object model. The alkgit question is
whether this crate can sit *under or beside* gix-odb semantics — git
objects are already content-addressed and verified by git's own model,
so the crate's value there is the large-blob story (git-lfs-shaped)
and a better small-object backend than the proposed default, without
fighting git's hash model. Note for the p2p case: git's `alternates`
/ object-pool mechanism (e.g. GitLab object pools for fork networks)
is the prior art for cross-repo dedup among related repos — the
pooled-CAS posture (OQ-BL-05) generalizes it to unrelated repos and
to p2p replication.
### alknet's appfile external-store probe — the direct ancestor
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written
against an older iroh-blobs; conclusions re-verify in this Phase 0.
The load-bearing bits: the appfile shape (small blobs in kv/sqlite,
large on fs fallback), the filename↔hash mapping problem, and the
multi-backend dispatch pain the probe hit (which is this crate's
reason to own that dispatch). Old-data warning: any API or behavior
claims there describe an older upstream; re-check against
`/workspace/iroh-blobs` as checked out today.
### alkcall — the substrate
`/workspace/@alkdev/alkcall` (the call + channels RPC crate). The
authorization seam for network-facing blob ops (`AccessControl`,
producer/consumer vocabulary, alkcall ADR-022/037) and, if blob
transfer rides channels, the established data-path patterns
(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even
belongs in this crate is itself open (OQ-BL-01); alkcall is the
substrate it composes with *when it does*.
## Open Questions
Numbering is provisional until the register solidifies; promote the
final set into Phase 1's `docs/architecture/open-questions.md`.
### OQ-BL-01: Crate scope — store-only, or store + ops surface?
The store (put/get/verify) is clearly in. What about the network ops
layer — a channels-native op family on the alkcall substrate,
auth-gated via `AccessControl`? iroh-blobs has it (provider protocol);
our posture is to separate it. Options: (a) store-only crate, ops in a
sibling crate later; (b) store + optional feature-gated ops module
here; (c) undecided pending the first consumer's shape.
Original lean was (a)/(b) on the inversion-point pattern. **Updated
2026-10-01:** the p2p-git consumer (§The p2p shape) makes the ops
surface likely rather than speculative — because everything is
hash-addressed, p2p sync *is* "announce hashes, diff hash sets, fetch
missing blobs verified," which is an op family on this store. Whether
that family lives here (feature-gated) or in the sibling that owns
replicator/gossip policy is the remaining question; the deciding input
is what alkgit's replicator actually needs and whether alknet's
appfile case ever transfers over the network. Even so, the *shape* of
the ops (have/need + verified fetch, ACL-gated) is now firm enough to
plan against.
### OQ-BL-02: Multi-backend dispatch — the appfile problem
Small blobs in kv/sqlite, large on fs fallback — the shape is agreed,
the mechanics are not: how does put/get pick a backend (size
thresholds? per-algorithm routing? per-namespace config?), how does it
work when a blob should migrate between backends, and what happens on a
get when the "wrong" backend was probed? alknet's probe hit exactly
this ("dealing with more than one backend at a time"); its findings
need re-verification against the current iroh-blobs before relying on
them. Expected shape: a backend trait + a dispatch layer, but the trait
shape is a one-way door once written — deserves a POC before
committing.
### OQ-BL-03: Hash abstraction — trait, enum, or per-backend config?
The hard requirement: git SHA-1/SHA-256 and BLAKE3 must coexist (alkgit
conflict).
**Major resolution (2026-10-01, from the GC/pooling round): the
oid↔hash *mapping* problem dissolves rather than gets solved.** The
assumed shape was "two identifiers per blob (consumer oid, store hash)
+ a mapping table." That table is only necessary if the store insists
on one canonical hash. Since multi-hash is already committed, the
resolution is: **"git's oid derivation" is itself one of the store's
hash algorithms.** A store entry keyed by `git-blob-sha256(content)` *is*
the oid; the consumer (alkgit's odb) addresses the pool directly, zero
indirection. Two consequences:
- **Algorithms must allow domain separation in their input** — git's
oid hashes `blob <len>\0 + content` (a preamble, and uncompressed
content), so the hash-algorithm abstraction is not a bare `fn(content)
→ digest`; an algorithm may define its own preamble/preprocessing.
Per-algorithm namespace tags (`git-sha1`, `git-sha256`, `blake3`)
then coexist in one flat pool collision-free.
- **The mapping problem only reappears for transfer vs storage** — if
content arrives over the network keyed by one hash (e.g. BLAKE3 for
bao verification) and is stored under another (a git oid). The git
case doesn't need this (git objects are self-verifying, so no
transfer-verification layer); whether any consumer ever needs a
cross-hash registration (verify-under-A, store-under-B) is deferred
until a consumer asks.
Remaining sub-questions: is the hash algorithm a *store* parameter
(one algorithm per store instance, chosen by the consumer) or
*per-blob* data (multi-algorithm within one store — the resolution
above implies per-blob, or rather per-algorithm-namespace within one
store)? Does verification (bao trees) get per-algorithm treatment or
does bao stay BLAKE3-bound (git objects don't need bao verification
anyway)? This interacts with OQ-BL-01 and the wire-format question — a
wire ADR must follow whatever this settles.
### OQ-BL-04: Verification and chunking story
iroh-blobs' verification is bao outboard encoding over fixed-size
BLAKE3 chunk trees; git objects are self-verifying under git's own
model; raw large blobs (git-lfs-shaped, appfile large files) need
*some* verification story from us. Options: bao (borrow the
*conclusion*, dependency posture per convention 7), our own digest
scheme, or pluggable verification per blob-kind.
**Chunking is scoped (2026-10-01, principle 7), not open-ended:** the
default is whole-file CAS — content-defined chunking (fastcdc/
restic-style; shift-resistant, unlike bao's fixed trees) is an
encoding consideration *only* for large binary blobs receiving small
edits, and git objects are unit blobs that never want chunking. The
remaining question is *where* variable chunking lives if adopted: in
the store's encoding layer (a blob is stored as chunk tree + the
store serves ranges) or in the manifest layer above (chunks are
themselves small blobs; the store stays whole-file-only — the dedup-
friendlier option, since chunks are then pooled and cross-file shared
like any other content). Decision input: how alkgit handles
git-lfs-shaped large files vs how agent workspaces transfer large
artifacts, and whether range reads are a store API or a
reassemble-above concern.
### OQ-BL-05: Pooling and GC — namespaces as reference sets over a flat CAS
**The decisions from the discussions (2026-10-01, principles 6; the
GC/pooling round; the rudolfs collision):**
1. **One pooled CAS per node**; repos/workspaces are sets of hash
references (manifests/trees) rather than isolated stores —
cross-repo dedup by construction.
2. **Physically flat, logically namespaced** (the rudolfs inversion).
Physical layer: `hash → bytes`, flat, backends never see the
namespace (`{hex-prefix}/{hex-prefix}/{hash}` sharding survives —
fan-out, not partitioning). Logical layer: `namespace → {hashes}`
reference tables, which simultaneously give GC roots, per-namespace
sweeps, accounting, and the ACL boundary (alkcall `AccessControl`
gates the namespace for network ops). rudolfs' composite
`StorageKey(ns, oid)` API shape is kept; only its physical
namespacing is inverted.
3. **Tags are the root table; references stay above the crate.**
iroh-blobs' named `Tag` (flat name → hash+format table) is the
"an external consumer cares about this hash" concept — namespaces
map onto it as `namespace → root manifest tag → mark-walk → {oids}`.
Git-flavored: refs (`refs/heads/*`) *are* the names; the consumer
registers them as tags. LFS pointer files are the same shape in
another hat — the pointer (oid + size) in the tree *is* the
reference; no extra tag needed because the consumer's own
structure holds it. **Store stays structure-blind**: it does not
walk consumer-side manifests.
**The GC picture, assembled from verified prior art:**
- **Mechanism: mark-and-sweep from registered roots** (iroh-blobs'
model, verified — see §Prior art). Roots = registered namespace/root
tags + put-path RAII pins (TempTag-shaped) + a protect callback with
abort semantics (`ProtectOutcome` — protection-source errors skip the
sweep rather than risk deletion). The store does sweep; liveness
beyond "these roots exist" is the consumer's job (it walks its own
manifest format) — the callback seam.
- **Traversal ownership is THE open sub-question.** Iroh can walk
children in-mark because HashSeq is store-native; our
manifests/trees live above the crate, so either **(a) generalized
protect callback** — the layer above computes liveness and hands
hash sets to GC (store stays structure-blind; simplest; liveness
computation is the consumer's), **(b) store-native manifest format**
(a hashseq-like encoding so mark can traverse — cheap GC, but drags
structure into the store layer and creates a wire-ish format), or
**(c) reference-tracker seam** — roots registered with a
children-of(hash) callback; the store does mark/sweep through it.
Tentative lean: **(a) + temp-tag pinning**, revisiting (b) only if
traversal-per-sweep proves expensive (a large multi-tenant node
could pass large hash sets each run; incremental sweeps are an
in-(a) option since the callback is injectable). The three options
are a Phase 1 ADR, decided with the first consumer in sight.
- **Temp-tag pinning is needed regardless** of which option wins: a
blob put but not yet referenced by any namespace must survive
mid-write before any sweep sees it (iroh's `TempTag`/batch-scope
pattern).
- **Per-namespace sweeps** become possible with the reference tables
(delete only within a reclamation scope) — the multi-tenant
refinement the flat iroh model can't express; not needed at minimum.
- **Delete-then-recover semantics** — CAS append-only by nature; a
swept-but-somehow-referenced blob is a re-put, not corruption. The
dangerous direction is only deleting liveness (data loss), which the
abort-callback + pinning layers exist to prevent.
**Still open (the residual mechanics):** namespace registry shape
(namespaced root tags vs separate reference tables), GC scheduling
(interval + injectable protect vs consumer-driven sweeps), and the pack
tension below.
**The pack tension (alkgit-shaped, constrains the store surface)** —
git packfiles are pack-efficient but blob one object-store-per-pack
(bad for cross-repo dedup); loose-per-object in the pooled store dedups
but is fs-inefficient at git's object volumes. Small git objects in the
kv backend largely dissolve this for the common case; the residual
question is what the store must support for large/packed content —
per-object granularity, range reads into packed blobs, or both
(interacts with OQ-BL-04's range-read question — likely read them
together).
### OQ-BL-06: POC register (draft)
Numbered POCs, opened as research reaches them (findings land in
`docs/research/`; worktree placement per the SDD process). The GC round
reshaped the register — see the discussion at the end for what each POC
is actually deciding:
| # | What | Status | Where |
|---|------|--------|-------|
| 1 | Backend-trait + dual-dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path | **Pending** — first POC | findings file TBD |
| 2 | Multi-hash store (git-sha256 + BLAKE3 coexisting in one flat pool, per-algorithm namespaces; oid = a store entry keyed by git's derivation) | **Pending** — rides #1's data | findings file TBD |
| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads; streaming `LFSObject`-shaped put/get + `fanout` seam) | **Pending** — reading-and-design POC | findings file TBD |
| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Pending** — after #1/#2 (needs a backend to pool over) | findings file TBD |
Sequencing note: #1 and #2 are probably one worktree (the dispatch POC
naturally exercises two hash algorithms); #3 is a reading-and-design
POC against the current iroh-blobs checkout plus the alknet probe
re-verification; #4's GC half matters most — it is a correctness
surface and should be a worked example at minimum. The pack tension
(OQ-BL-05) rides #3's reading rather than earning its own POC yet.
**What each POC is deciding (2026-10-01, post-GC round):**
- **POC #1 decides the backend trait's minimum contract** — and the GC
round already sharpened it: `list()` must be complete (rudolfs'
S3 backend punting on list is *why* it can never GC), and the put
path must support not-yet-referenced pinning (TempTag-shaped RAII,
iroh-blobs' batch-scope pattern). The trait is a one-way door; this
POC is where the shape gets evidence rather than instinct.
- **POC #2 validates "oid = a store entry keyed by git's own
derivation"** — the multi-hash resolution in OQ-BL-03 (git-blob
prefix as an algorithm's domain-separated input, per-algorithm
namespaces in one flat pool). If it works, the alkgit odb binds
directly with no mapping layer; if it fights back (e.g. the
preamble/preprocessing abstraction gets ugly), we learn that before
the trait is written, not after.
- **POC #3 is the reading-and-design leg** — against the *current*
iroh-blobs checkout (the store eval) + re-verifying alknet's probe.
Also where the pack tension is analyzed (no independent POC yet).
- **POC #4 is the GC worked example** — the correctness surface of
OQ-BL-05: namespace reference tables + mark-and-sweep with
protect-callback + TempTag pinning + delete-then-recover, exercised
until the failure modes (pin missed, sweep racing a write, live-set
miscomputed) are named rather than theoretical. Expected to be the
POC that actually constrains the trait in return.
POC placement conventions (inherited from alksocks/alktunnels): a POC
that needs code from this repo runs in a worktree/branch
(`.worktrees/research/<task-id>/` per the SDD process); a
self-contained POC runs as a standalone crate in the global workspace
with findings written into `docs/research/` here. Findings always land
in `docs/research/` regardless of where the code lives.
## Phase 0 plan (next steps)
1. **Write `iroh-blobs-eval.md`** against the *current* checkout,
focused on `src/store` (kv + flat backends, bao, chunking) — the
conclusions inventory we may borrow, and the weld points we won't.
2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`,
`poc-summary.md`) against the current upstream; mark what carried
over.
3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit
baseline: its storage layout, object model, and where a blob-store
crate under/beside it earns its keep.
4. **Open POCs** per OQ-BL-06, in the order above.
5. Converge: recommended approach + final OQ register → Phase 1.