- iroh-blobs GC mechanics verified against current checkout: mark-and-sweep (named tags + TempTag RAII + ProtectCb with abort), format-aware mark traversal, DeleteSet write-safety, no namespaces (flat tags) — our namespace concept maps onto their machinery - rudolfs prior art section: StorageKey(ns, oid) composite key + decorator storage stack; THE ANTI-PATTERN inverted (physically namespaced storage defeats cross-tenant dedup); list() is load-bearing; verify-as-decorator; streaming LFSObject + fanout - OQ-BL-03: oid<->hash mapping DISSOLVED — git's oid derivation is itself a hash algorithm (domain-separated preamble); per-algorithm namespaces; mapping only reappears for transfer-vs-storage hashes - OQ-BL-05 consolidated: physically flat/logically namespaced; tags are the root table; references stay above the crate; mark-and-sweep with protect-callback lean (a) + TempTag pinning; traversal ownership (a/b/c) is the Phase 1 ADR - POC register updated: #1 adds list() completeness + pinning contract, #2 reframed as validating oid-dissolution, #4 as the GC worked example; 'what each POC decides' section added
534 lines
30 KiB
Markdown
534 lines
30 KiB
Markdown
---
|
|
status: draft
|
|
last_updated: 2026-10-01 (GC/pooling round: iroh-blobs mark/sweep verified
|
|
against the current checkout, rudolfs prior art added, oid↔hash mapping
|
|
dissolved into OQ-BL-03 as per-algorithm namespaces, OQ-BL-05 consolidated
|
|
around tags-as-roots; earlier rounds: dedup/p2p consumer round, setup draft)
|
|
---
|
|
|
|
# alkblobs — Phase 0 (Exploration)
|
|
|
|
This document captures Phase 0 (Exploration) for the `alkblobs` crate:
|
|
vision, guiding principles, prior art, and open questions (OQ-BL-01..NN).
|
|
Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding
|
|
principles; research options; validate approaches; converge on a
|
|
recommended approach.* It is the input to Phase 1 (Architecture), where
|
|
the Architect will produce `docs/architecture/` specs, ADRs, and the
|
|
open-questions tracker.
|
|
|
|
Drafted 2026-09-30, emerging from the initial setup discussion. Nothing
|
|
is converged yet — this is the working sketch, not a specification.
|
|
|
|
## Vision and guiding principles
|
|
|
|
**One sentence:** content-addressed blob storage in the alk* family —
|
|
a multi-hash put/get/verify store with pluggable backends (small blobs
|
|
in a kv/sqlite-ish store, large blobs on a filesystem fallback), split
|
|
from any transport/protocol layer and tolerant of hash-algorithm
|
|
conflicts (the alkgit case), with auth-gated network operations riding
|
|
the alkcall seam when they exist.
|
|
|
|
**Why this crate exists (three converging consumers):**
|
|
|
|
1. **alkgit** (planning phase) needs an object backend that is *not*
|
|
its proposed default file-based backend ("kind of gross") — blobs
|
|
stored under git's own object hashes, but with iroh-blobs-like
|
|
handling for large blobs (where git typically delegates to git-lfs;
|
|
we could instead follow an iroh-blobs-shaped large-blob path). The
|
|
conflict: iroh-blobs is BLAKE3-only and content-hashes with bao
|
|
verification, while git's object database is SHA-1/SHA-256 with
|
|
git's own object format. A store that insists on one hash algorithm
|
|
cannot serve git objects directly.
|
|
2. **The alknet rewrite** (the original alk* project, being decomposed
|
|
and improved) needs the "appfile" external-store shape: small blobs
|
|
in a kv/sqlite store (faster than the filesystem for small items —
|
|
true beyond sqlite: it applies to the kv store iroh-blobs uses
|
|
too), large blobs on a filesystem fallback, with filename↔hash
|
|
mapping. alknet's own research hit the core awkwardness of
|
|
dispatching across more than one backend at a time — which is
|
|
exactly a store-layer problem this crate should own.
|
|
3. **Agent workspaces** — many agents working simultaneously in
|
|
workspaces that are mostly identical to each other's, each making
|
|
small edits in specific areas. Per-agent stores multiply the near-
|
|
identical content; a shared content-addressed store dedups it by
|
|
construction, and a path→hash manifest per workspace makes the
|
|
per-agent delta just its changed hashes (the alknet-filesystem
|
|
probe shows the manifest mapping is straightforward).
|
|
|
|
**The p2p shape and what it pins (added 2026-10-01, from the dedup
|
|
discussion).** Two alkgit use cases exist: self-hosted git (where
|
|
cross-repo dedup matters little — you don't fork your own repos) and
|
|
**p2p git for OSS projects**, the demanding one. The p2p sketch:
|
|
ownership/ACL live in a smart contract on a low-fee network;
|
|
replicators are push/clone endpoints that watch the contract and cache
|
|
its state; repos sync via iroh-style gossip (hashes, not content) and
|
|
pull/sync on demand. Neither alkgit approach described so far takes
|
|
advantage of dedup — which is exactly a store-layer property. The
|
|
implications for this crate:
|
|
|
|
- Both hard consumers (p2p replicators, agent swarms) reduce to the
|
|
same shape: **a pooled CAS per node**, where repos / workspaces are
|
|
*sets of hash references* (trees/manifests), not separate object
|
|
stores. Cross-repo dedup is then by construction; git's own
|
|
`alternates`/object-pool mechanism is the same idea done awkwardly.
|
|
- Because everything is hash-addressed, the p2p wire story collapses
|
|
to "announce hashes, diff hash sets, fetch missing blobs verified" —
|
|
which is precisely the ops surface OQ-BL-01 left provisional. The
|
|
consumer exists now: if p2p git is a confirmed alkgit direction,
|
|
the ops surface upgrades from "leaning store-only" to a confirmed
|
|
have/need + fetch op family, auth-gated via the alkcall seam.
|
|
|
|
**The scope line (current posture, revisit as evidence arrives):**
|
|
this crate is the *store*, not the *transport*. iroh-blobs welds store
|
|
+ provider protocol + tickets + postcard into one crate; the defining
|
|
posture here is the opposite split — put/get/verify against a hash is
|
|
its own layer, and any provider/protocol/ops surface lives above it or
|
|
in feature-gated modules (the alk* inversion-point pattern). The ops
|
|
surface (a channels-native "have/need + fetch blob with this hash" op
|
|
family, auth-gated via `AccessControl`) is a Phase 0/1 question — the
|
|
p2p-git consumer (§The p2p shape) upgrades it from speculative to
|
|
likely, but the crate-boundary decision itself is still open
|
|
(OQ-BL-01). What stays *above* the crate regardless: repos/workspaces
|
|
as reference sets (trees/manifests), path→hash mapping, git semantics,
|
|
the contract/gossip/replicator policy layer.
|
|
|
|
Guiding principles, inherited from the alk* family:
|
|
|
|
1. **Substrate-agnostic by construction.** The store must not know
|
|
whether bytes arrive over the network, from a local writer, or get
|
|
reassembled anywhere particular. Backends behind an injected seam
|
|
(alktty `TtyBackend` / alktunnels pump-halves precedent).
|
|
2. **Hashes are data, not transport identity.** Multiple hash
|
|
algorithms must coexist (git SHA-1/SHA-256, BLAKE3), each with its
|
|
own domain-separated derivation ("git-blob-sha256" hashes
|
|
`blob <len>\0 + content`, not bare content — OQ-BL-03's resolution);
|
|
consumers address the pool by the algorithm their format demands,
|
|
no mapping layer. The abstraction shape (trait, enum, per-backend
|
|
config) is an open question (OQ-BL-03).
|
|
3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets,
|
|
postcard serialization, and provider protocol are design-welded
|
|
choices we do not inherit. Its *store-shape* lessons (kv + flat
|
|
backends, verification flow, chunking) are fair game per prior-art
|
|
reading.
|
|
4. **Producer/consumer vocabulary** for any network-facing surface;
|
|
authorization via alkcall's `AccessControl`/identity seam, never an
|
|
in-band invented scheme (convention 6).
|
|
5. **Verify where it matters.** iroh-blobs' core virtue is verified
|
|
transfer (bao outboard encoding). The alkgit case already has
|
|
verified content (git objects are hash-addressed by git itself).
|
|
Which verification story the crate owns — bao trees, per-blob
|
|
digests, backend-native, or the rudolfs verify-as-decorator shape —
|
|
is open (OQ-BL-04).
|
|
6. **Pooling is the point.** The dedup wins both demanding consumers
|
|
want (Forknet-style multi-repo OSS content, agent swarms) require
|
|
that repos/workspaces are *sets of hash references* over one pooled
|
|
CAS per node, not walled-off per-repo stores. Pooling forces a GC
|
|
story (OQ-BL-05) and a multi-backend dispatch story (OQ-BL-02).
|
|
7. **Whole-file CAS as the default; chunking earns its keep only
|
|
where it must.** For source-code-scale content, whole-file hashing
|
|
captures the agent/edit delta perfectly and needs no chunk tree;
|
|
fixed-size chunk trees are actually *hostile* to insertion/deletion
|
|
edits (byte shifts cascade). Content-defined chunking (CDC —
|
|
fastcdc/restic-style, shift-resistant) only pays on large binary
|
|
files receiving small edits — the git-lfs weakness. iroh-blobs/bao
|
|
use fixed-size trees; do not inherit that as the default. Where
|
|
chunking lives (store encoding vs manifest layer) is OQ-BL-04.
|
|
|
|
## What is already known (settled, thin)
|
|
|
|
Almost nothing is pinned — deliberately. The only postures agreed at
|
|
setup:
|
|
|
|
- **Not a fork of iroh-blobs.** A downstream store crate inspired by
|
|
its `src/store` work, diverging deliberately on hashing and wire
|
|
surface (see `iroh-blobs-eval` below when written).
|
|
- **Multi-hash from day one** (hard requirement from alkgit — do not
|
|
hardcode BLAKE3).
|
|
- **Backend pluralism assumed, not designed.** kv/sqlite for small
|
|
blobs, filesystem fallback for large — the appfile shape — but
|
|
*how* multi-backend dispatch works is exactly what alknet's research
|
|
ran into, so it earns research and probably POCs (OQ-BL-02).
|
|
- **Pooled CAS, not per-repo stores** (agreed in the 2026-10-01
|
|
dedup discussion — the load-bearing scope decision so far): repos
|
|
and agent workspaces are sets of hash references over one shared
|
|
content-addressed store. This is the property both demanding
|
|
consumers actually need; everything else (GC, manifests, git
|
|
semantics) lives around it. Details open (OQ-BL-05).
|
|
- **Physically flat, logically namespaced** (2026-10-01 GC/pooling
|
|
round; the rudolfs inversion): the byte layer is a flat
|
|
dedup-by-construction CAS; namespaces are reference tables above it
|
|
(GC roots, sweep scoping, ACL boundary). Backends stay
|
|
namespace-blind.
|
|
- **Whole-file CAS is the default granularity**; chunking is scoped
|
|
to the large-binary-blob case (principle 7), not the default path.
|
|
|
|
## Prior art
|
|
|
|
### iroh-blobs — the shape inspiration (evaluated, not the base)
|
|
|
|
`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference).
|
|
Specifically `src/store`: the kv backend (small blobs, in-process) and
|
|
flat file backend (large blobs) split; bao outboard encoding and
|
|
verification flow; chunking. Its BLAKE3-only hashing, tickets, postcard
|
|
serialization, and provider protocol are the design-welded choices we
|
|
diverge from. **The store eval should be written against the current
|
|
checkout** — alknet's older research refers to an older iroh-blobs and
|
|
its conclusions must be re-verified rather than inherited.
|
|
|
|
**GC mechanics — verified against the current checkout (2026-10-01,
|
|
`src/store/gc.rs`, `src/util/temp_tag.rs`, `src/store/fs/delete_set.rs`).
|
|
This is the load-bearing prior art for OQ-BL-05:**
|
|
|
|
- **Mark-and-sweep, not refcounting, for persistent liveness.**
|
|
`gc_mark_task` collects roots from three sources: persistent *named
|
|
tags* (a flat `tags-0` table, name → `HashAndFormat`), in-memory *
|
|
`TempTag`s* (RAII-refcounted, `#[must_use]`, `.leak()` for
|
|
pin-until-exit-of-process), and an injectable `ProtectCb` — a
|
|
callback GC consults before each run that can add externally-known
|
|
hashes or **abort the run** (`ProtectOutcome::Abort`; a flaky
|
|
protection source skips the sweep rather than risking deletion).
|
|
- **Format-aware mark traversal** — a non-raw root (HashSeq/collection)
|
|
contributes all reachable children to the live set via the bao hash
|
|
stream (gc.rs:63-78). Protecting a collection protects everything
|
|
reachable from it.
|
|
- **Sweep** lists the whole store and batch-deletes (~100/batch)
|
|
anything not live. `list()` being complete is therefore load-bearing
|
|
for GC — relevant to the backend trait contract (OQ-BL-02).
|
|
- **Write-safety against a concurrent sweep** — the fs backend carries
|
|
a separate `DeleteSet` transaction layer (`ProtectHandle` /
|
|
mark-for-delete / protect-cancel / commit); `TempTag` holds a refcount
|
|
so a blob put but not yet referenced cannot vanish mid-write. Temp
|
|
tags are scoped (batch scope / process scope) with per-scope counters.
|
|
- **No namespaces in their model** — flat named tags over a flat
|
|
pool. Our namespace concept (OQ-BL-05) lands on their machinery as
|
|
policy over what counts as a root: `namespace → root manifest →
|
|
(mark-walk) → {oids}` is tags plus traversal, not new storage
|
|
concepts.
|
|
|
|
### rudolfs — git-lfs server; the composite-key + decorator-storage prior art
|
|
|
|
`/workspace/rudolfs` (v0.3.8, MIT, read-only reference; the alknet
|
|
research doc is
|
|
`/workspace/@alkdev/alknet/docs/research/references/gitlfs/
|
|
rudolfs-reference.md`). A git-lfs server whose **storage layer**
|
|
(`src/storage/`) is the most direct ancestor of our composite-key +
|
|
dispatch shape:
|
|
|
|
- **`StorageKey = (Namespace, Oid)`** — a composite key over a flat CAS
|
|
(`Namespace = (org, project)` strings, `Oid = SHA-256`). All backend
|
|
operations take the composite key.
|
|
- **`Storage` trait** — `get`/`put`/`size`/`delete`/`list`/`
|
|
total_size`/`max_size`/`public_url`/`upload_url`, with
|
|
`LFSObject = (len: u64, ByteStream)` — the streaming-first put/get
|
|
shape (pinned boxed async byte stream), the natural large-blob API
|
|
and likely the channels ops-surface shape.
|
|
- **Decorator composition** — `Verify ↔ Encrypted ↔ Cached(LRU →
|
|
permanent) ↔ Retrying → s3/disk`. The `Cached` decorator is the
|
|
appfile shape in LRU form: "fast store in front of fallback" as a
|
|
composable wrapper, an alternative dispatch policy to size
|
|
thresholds. `fanout()` duplicates one stream into two lock-step
|
|
copies (serve + persist) — precisely "serve the fetch while
|
|
persisting on receipt" for networked gets.
|
|
- **Verify-as-decorator (OQ-BL-04 input)** — streaming SHA-256 check on
|
|
both put and get paths with auto-purge of a corrupted tier.
|
|
- **Footnote if at-rest encryption is ever considered** — nonce derived
|
|
from the oid: deterministic and correct, but a key rotation
|
|
invalidates every object and breaks dedup across keys (the known
|
|
encryption-vs-dedup tension).
|
|
- **THE ANTI-PATTERN (inverted, not inherited)** — namespaces are
|
|
*physical*: `s3://{org}/{project}/{sha256}` — identical content in
|
|
two orgs is stored twice. It buys tenant isolation by paying the
|
|
cross-tenant dedup that is our whole reason for pooled CAS. The
|
|
reconciliation recorded in OQ-BL-05: keep the composite-key API
|
|
shape, put the namespace half in metadata (reference tables) above
|
|
the byte layer, and keep physical storage flat (`oid → bytes`,
|
|
backends namespace-blind).
|
|
- **`list()` is load-bearing** — the S3 backend punts on `list()`
|
|
(returns an empty stream) and its `delete()` is a no-op, so it can
|
|
*never* GC. A lesson for the backend trait contract: list/sweep
|
|
support is a real requirement, not an optional extra.
|
|
|
|
### gix-odb — git's own object database (alkgit's baseline)
|
|
|
|
`/workspace/git-oxide/gix-odb` (read-only reference). The backend
|
|
alkgit currently plans against; the hashing-algorithm baseline git
|
|
actually uses (SHA-1/SHA-256), git's loose-object and packfile layout,
|
|
and git's already-hash-addressed object model. The alkgit question is
|
|
whether this crate can sit *under or beside* gix-odb semantics — git
|
|
objects are already content-addressed and verified by git's own model,
|
|
so the crate's value there is the large-blob story (git-lfs-shaped)
|
|
and a better small-object backend than the proposed default, without
|
|
fighting git's hash model. Note for the p2p case: git's `alternates`
|
|
/ object-pool mechanism (e.g. GitLab object pools for fork networks)
|
|
is the prior art for cross-repo dedup among related repos — the
|
|
pooled-CAS posture (OQ-BL-05) generalizes it to unrelated repos and
|
|
to p2p replication.
|
|
|
|
### alknet's appfile external-store probe — the direct ancestor
|
|
|
|
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
|
|
(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written
|
|
against an older iroh-blobs; conclusions re-verify in this Phase 0.
|
|
The load-bearing bits: the appfile shape (small blobs in kv/sqlite,
|
|
large on fs fallback), the filename↔hash mapping problem, and the
|
|
multi-backend dispatch pain the probe hit (which is this crate's
|
|
reason to own that dispatch). Old-data warning: any API or behavior
|
|
claims there describe an older upstream; re-check against
|
|
`/workspace/iroh-blobs` as checked out today.
|
|
|
|
### alkcall — the substrate
|
|
|
|
`/workspace/@alkdev/alkcall` (the call + channels RPC crate). The
|
|
authorization seam for network-facing blob ops (`AccessControl`,
|
|
producer/consumer vocabulary, alkcall ADR-022/037) and, if blob
|
|
transfer rides channels, the established data-path patterns
|
|
(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even
|
|
belongs in this crate is itself open (OQ-BL-01); alkcall is the
|
|
substrate it composes with *when it does*.
|
|
|
|
## Open Questions
|
|
|
|
Numbering is provisional until the register solidifies; promote the
|
|
final set into Phase 1's `docs/architecture/open-questions.md`.
|
|
|
|
### OQ-BL-01: Crate scope — store-only, or store + ops surface?
|
|
|
|
The store (put/get/verify) is clearly in. What about the network ops
|
|
layer — a channels-native op family on the alkcall substrate,
|
|
auth-gated via `AccessControl`? iroh-blobs has it (provider protocol);
|
|
our posture is to separate it. Options: (a) store-only crate, ops in a
|
|
sibling crate later; (b) store + optional feature-gated ops module
|
|
here; (c) undecided pending the first consumer's shape.
|
|
|
|
Original lean was (a)/(b) on the inversion-point pattern. **Updated
|
|
2026-10-01:** the p2p-git consumer (§The p2p shape) makes the ops
|
|
surface likely rather than speculative — because everything is
|
|
hash-addressed, p2p sync *is* "announce hashes, diff hash sets, fetch
|
|
missing blobs verified," which is an op family on this store. Whether
|
|
that family lives here (feature-gated) or in the sibling that owns
|
|
replicator/gossip policy is the remaining question; the deciding input
|
|
is what alkgit's replicator actually needs and whether alknet's
|
|
appfile case ever transfers over the network. Even so, the *shape* of
|
|
the ops (have/need + verified fetch, ACL-gated) is now firm enough to
|
|
plan against.
|
|
|
|
### OQ-BL-02: Multi-backend dispatch — the appfile problem
|
|
|
|
Small blobs in kv/sqlite, large on fs fallback — the shape is agreed,
|
|
the mechanics are not: how does put/get pick a backend (size
|
|
thresholds? per-algorithm routing? per-namespace config?), how does it
|
|
work when a blob should migrate between backends, and what happens on a
|
|
get when the "wrong" backend was probed? alknet's probe hit exactly
|
|
this ("dealing with more than one backend at a time"); its findings
|
|
need re-verification against the current iroh-blobs before relying on
|
|
them. Expected shape: a backend trait + a dispatch layer, but the trait
|
|
shape is a one-way door once written — deserves a POC before
|
|
committing.
|
|
|
|
### OQ-BL-03: Hash abstraction — trait, enum, or per-backend config?
|
|
|
|
The hard requirement: git SHA-1/SHA-256 and BLAKE3 must coexist (alkgit
|
|
conflict).
|
|
|
|
**Major resolution (2026-10-01, from the GC/pooling round): the
|
|
oid↔hash *mapping* problem dissolves rather than gets solved.** The
|
|
assumed shape was "two identifiers per blob (consumer oid, store hash)
|
|
+ a mapping table." That table is only necessary if the store insists
|
|
on one canonical hash. Since multi-hash is already committed, the
|
|
resolution is: **"git's oid derivation" is itself one of the store's
|
|
hash algorithms.** A store entry keyed by `git-blob-sha256(content)` *is*
|
|
the oid; the consumer (alkgit's odb) addresses the pool directly, zero
|
|
indirection. Two consequences:
|
|
|
|
- **Algorithms must allow domain separation in their input** — git's
|
|
oid hashes `blob <len>\0 + content` (a preamble, and uncompressed
|
|
content), so the hash-algorithm abstraction is not a bare `fn(content)
|
|
→ digest`; an algorithm may define its own preamble/preprocessing.
|
|
Per-algorithm namespace tags (`git-sha1`, `git-sha256`, `blake3`)
|
|
then coexist in one flat pool collision-free.
|
|
- **The mapping problem only reappears for transfer vs storage** — if
|
|
content arrives over the network keyed by one hash (e.g. BLAKE3 for
|
|
bao verification) and is stored under another (a git oid). The git
|
|
case doesn't need this (git objects are self-verifying, so no
|
|
transfer-verification layer); whether any consumer ever needs a
|
|
cross-hash registration (verify-under-A, store-under-B) is deferred
|
|
until a consumer asks.
|
|
|
|
Remaining sub-questions: is the hash algorithm a *store* parameter
|
|
(one algorithm per store instance, chosen by the consumer) or
|
|
*per-blob* data (multi-algorithm within one store — the resolution
|
|
above implies per-blob, or rather per-algorithm-namespace within one
|
|
store)? Does verification (bao trees) get per-algorithm treatment or
|
|
does bao stay BLAKE3-bound (git objects don't need bao verification
|
|
anyway)? This interacts with OQ-BL-01 and the wire-format question — a
|
|
wire ADR must follow whatever this settles.
|
|
|
|
### OQ-BL-04: Verification and chunking story
|
|
|
|
iroh-blobs' verification is bao outboard encoding over fixed-size
|
|
BLAKE3 chunk trees; git objects are self-verifying under git's own
|
|
model; raw large blobs (git-lfs-shaped, appfile large files) need
|
|
*some* verification story from us. Options: bao (borrow the
|
|
*conclusion*, dependency posture per convention 7), our own digest
|
|
scheme, or pluggable verification per blob-kind.
|
|
|
|
**Chunking is scoped (2026-10-01, principle 7), not open-ended:** the
|
|
default is whole-file CAS — content-defined chunking (fastcdc/
|
|
restic-style; shift-resistant, unlike bao's fixed trees) is an
|
|
encoding consideration *only* for large binary blobs receiving small
|
|
edits, and git objects are unit blobs that never want chunking. The
|
|
remaining question is *where* variable chunking lives if adopted: in
|
|
the store's encoding layer (a blob is stored as chunk tree + the
|
|
store serves ranges) or in the manifest layer above (chunks are
|
|
themselves small blobs; the store stays whole-file-only — the dedup-
|
|
friendlier option, since chunks are then pooled and cross-file shared
|
|
like any other content). Decision input: how alkgit handles
|
|
git-lfs-shaped large files vs how agent workspaces transfer large
|
|
artifacts, and whether range reads are a store API or a
|
|
reassemble-above concern.
|
|
|
|
### OQ-BL-05: Pooling and GC — namespaces as reference sets over a flat CAS
|
|
|
|
**The decisions from the discussions (2026-10-01, principles 6; the
|
|
GC/pooling round; the rudolfs collision):**
|
|
|
|
1. **One pooled CAS per node**; repos/workspaces are sets of hash
|
|
references (manifests/trees) rather than isolated stores —
|
|
cross-repo dedup by construction.
|
|
2. **Physically flat, logically namespaced** (the rudolfs inversion).
|
|
Physical layer: `hash → bytes`, flat, backends never see the
|
|
namespace (`{hex-prefix}/{hex-prefix}/{hash}` sharding survives —
|
|
fan-out, not partitioning). Logical layer: `namespace → {hashes}`
|
|
reference tables, which simultaneously give GC roots, per-namespace
|
|
sweeps, accounting, and the ACL boundary (alkcall `AccessControl`
|
|
gates the namespace for network ops). rudolfs' composite
|
|
`StorageKey(ns, oid)` API shape is kept; only its physical
|
|
namespacing is inverted.
|
|
3. **Tags are the root table; references stay above the crate.**
|
|
iroh-blobs' named `Tag` (flat name → hash+format table) is the
|
|
"an external consumer cares about this hash" concept — namespaces
|
|
map onto it as `namespace → root manifest tag → mark-walk → {oids}`.
|
|
Git-flavored: refs (`refs/heads/*`) *are* the names; the consumer
|
|
registers them as tags. LFS pointer files are the same shape in
|
|
another hat — the pointer (oid + size) in the tree *is* the
|
|
reference; no extra tag needed because the consumer's own
|
|
structure holds it. **Store stays structure-blind**: it does not
|
|
walk consumer-side manifests.
|
|
|
|
**The GC picture, assembled from verified prior art:**
|
|
|
|
- **Mechanism: mark-and-sweep from registered roots** (iroh-blobs'
|
|
model, verified — see §Prior art). Roots = registered namespace/root
|
|
tags + put-path RAII pins (TempTag-shaped) + a protect callback with
|
|
abort semantics (`ProtectOutcome` — protection-source errors skip the
|
|
sweep rather than risk deletion). The store does sweep; liveness
|
|
beyond "these roots exist" is the consumer's job (it walks its own
|
|
manifest format) — the callback seam.
|
|
- **Traversal ownership is THE open sub-question.** Iroh can walk
|
|
children in-mark because HashSeq is store-native; our
|
|
manifests/trees live above the crate, so either **(a) generalized
|
|
protect callback** — the layer above computes liveness and hands
|
|
hash sets to GC (store stays structure-blind; simplest; liveness
|
|
computation is the consumer's), **(b) store-native manifest format**
|
|
(a hashseq-like encoding so mark can traverse — cheap GC, but drags
|
|
structure into the store layer and creates a wire-ish format), or
|
|
**(c) reference-tracker seam** — roots registered with a
|
|
children-of(hash) callback; the store does mark/sweep through it.
|
|
Tentative lean: **(a) + temp-tag pinning**, revisiting (b) only if
|
|
traversal-per-sweep proves expensive (a large multi-tenant node
|
|
could pass large hash sets each run; incremental sweeps are an
|
|
in-(a) option since the callback is injectable). The three options
|
|
are a Phase 1 ADR, decided with the first consumer in sight.
|
|
- **Temp-tag pinning is needed regardless** of which option wins: a
|
|
blob put but not yet referenced by any namespace must survive
|
|
mid-write before any sweep sees it (iroh's `TempTag`/batch-scope
|
|
pattern).
|
|
- **Per-namespace sweeps** become possible with the reference tables
|
|
(delete only within a reclamation scope) — the multi-tenant
|
|
refinement the flat iroh model can't express; not needed at minimum.
|
|
- **Delete-then-recover semantics** — CAS append-only by nature; a
|
|
swept-but-somehow-referenced blob is a re-put, not corruption. The
|
|
dangerous direction is only deleting liveness (data loss), which the
|
|
abort-callback + pinning layers exist to prevent.
|
|
|
|
**Still open (the residual mechanics):** namespace registry shape
|
|
(namespaced root tags vs separate reference tables), GC scheduling
|
|
(interval + injectable protect vs consumer-driven sweeps), and the pack
|
|
tension below.
|
|
|
|
**The pack tension (alkgit-shaped, constrains the store surface)** —
|
|
git packfiles are pack-efficient but blob one object-store-per-pack
|
|
(bad for cross-repo dedup); loose-per-object in the pooled store dedups
|
|
but is fs-inefficient at git's object volumes. Small git objects in the
|
|
kv backend largely dissolve this for the common case; the residual
|
|
question is what the store must support for large/packed content —
|
|
per-object granularity, range reads into packed blobs, or both
|
|
(interacts with OQ-BL-04's range-read question — likely read them
|
|
together).
|
|
|
|
### OQ-BL-06: POC register (draft)
|
|
|
|
Numbered POCs, opened as research reaches them (findings land in
|
|
`docs/research/`; worktree placement per the SDD process). The GC round
|
|
reshaped the register — see the discussion at the end for what each POC
|
|
is actually deciding:
|
|
|
|
| # | What | Status | Where |
|
|
|---|------|--------|-------|
|
|
| 1 | Backend-trait + dual-dispatch shape (kv small / fs large); trait must include `list()` complete by contract (rudolfs anti-lesson) + temp-tag/pinning on the put path | **Pending** — first POC | findings file TBD |
|
|
| 2 | Multi-hash store (git-sha256 + BLAKE3 coexisting in one flat pool, per-algorithm namespaces; oid = a store entry keyed by git's derivation) | **Pending** — rides #1's data | findings file TBD |
|
|
| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads; streaming `LFSObject`-shaped put/get + `fanout` seam) | **Pending** — reading-and-design POC | findings file TBD |
|
|
| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Pending** — after #1/#2 (needs a backend to pool over) | findings file TBD |
|
|
|
|
Sequencing note: #1 and #2 are probably one worktree (the dispatch POC
|
|
naturally exercises two hash algorithms); #3 is a reading-and-design
|
|
POC against the current iroh-blobs checkout plus the alknet probe
|
|
re-verification; #4's GC half matters most — it is a correctness
|
|
surface and should be a worked example at minimum. The pack tension
|
|
(OQ-BL-05) rides #3's reading rather than earning its own POC yet.
|
|
|
|
**What each POC is deciding (2026-10-01, post-GC round):**
|
|
|
|
- **POC #1 decides the backend trait's minimum contract** — and the GC
|
|
round already sharpened it: `list()` must be complete (rudolfs'
|
|
S3 backend punting on list is *why* it can never GC), and the put
|
|
path must support not-yet-referenced pinning (TempTag-shaped RAII,
|
|
iroh-blobs' batch-scope pattern). The trait is a one-way door; this
|
|
POC is where the shape gets evidence rather than instinct.
|
|
- **POC #2 validates "oid = a store entry keyed by git's own
|
|
derivation"** — the multi-hash resolution in OQ-BL-03 (git-blob
|
|
prefix as an algorithm's domain-separated input, per-algorithm
|
|
namespaces in one flat pool). If it works, the alkgit odb binds
|
|
directly with no mapping layer; if it fights back (e.g. the
|
|
preamble/preprocessing abstraction gets ugly), we learn that before
|
|
the trait is written, not after.
|
|
- **POC #3 is the reading-and-design leg** — against the *current*
|
|
iroh-blobs checkout (the store eval) + re-verifying alknet's probe.
|
|
Also where the pack tension is analyzed (no independent POC yet).
|
|
- **POC #4 is the GC worked example** — the correctness surface of
|
|
OQ-BL-05: namespace reference tables + mark-and-sweep with
|
|
protect-callback + TempTag pinning + delete-then-recover, exercised
|
|
until the failure modes (pin missed, sweep racing a write, live-set
|
|
miscomputed) are named rather than theoretical. Expected to be the
|
|
POC that actually constrains the trait in return.
|
|
|
|
POC placement conventions (inherited from alksocks/alktunnels): a POC
|
|
that needs code from this repo runs in a worktree/branch
|
|
(`.worktrees/research/<task-id>/` per the SDD process); a
|
|
self-contained POC runs as a standalone crate in the global workspace
|
|
with findings written into `docs/research/` here. Findings always land
|
|
in `docs/research/` regardless of where the code lives.
|
|
|
|
## Phase 0 plan (next steps)
|
|
|
|
1. **Write `iroh-blobs-eval.md`** against the *current* checkout,
|
|
focused on `src/store` (kv + flat backends, bao, chunking) — the
|
|
conclusions inventory we may borrow, and the weld points we won't.
|
|
2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`,
|
|
`poc-summary.md`) against the current upstream; mark what carried
|
|
over.
|
|
3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit
|
|
baseline: its storage layout, object model, and where a blob-store
|
|
crate under/beside it earns its keep.
|
|
4. **Open POCs** per OQ-BL-06, in the order above.
|
|
5. Converge: recommended approach + final OQ register → Phase 1. |