alkdev/alkblobs
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(architecture): Phase 1 spec tree — 7 specs, ADRs 001-006, OQ tracker

Browse Source 
- overview/hashing/store-api/backends/gc/ops specs + README index,
  all draft status with YAML frontmatter and cross-referenced ADR/OQ
- ADR-001: substrate posture (store layer alkcall-free = family
  conformance, not deviation) + ops placement decided (feature-gated
  module, no consumer-waited deferral)
- ADR-002: canonical git-blob-sha-256 + key encoding — the declared
  wire-format ADR (one-way door)
- ADR-003: backend contract, sqlite/fs/mem tiers, pure-function
  dispatch, no migration, unknown-length buffering semantics pinned
- ADR-004: exactly two production backends; manifest/path-tree layers
  are consumer-side (the corrected two-vs-three-backends framing)
- ADR-005: pooled CAS, three liveness sources, delete windows with
  pin-before-publish + delete-time arbitration (race closed), no
  ambient scheduling
- ADR-006: whole-blob verification; chunk-tree encodings excluded by
  scoping, not deferred on a paused consumer
- open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07
  /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task
- AGENTS.md: status updated to Phase 1, gix-odb path corrected

Verification: cargo test / clippy -D warnings / fmt --check clean
This commit is contained in:
glm-5.3-flash committed 2026-10-01 14:45:32 +00:00
1 parent 7d9ac8a6ec
commit 4b5009d85a
16 files changed
+1953 -8
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/decisions/001-substrate-posture-and-ops-placement.md
+134
View File
@@ -0,0 +1,134 @@
# ADR-001: Substrate posture and ops-surface placement
## Status
Accepted
## Context
Phase 0 settled that a network ops surface for the blob store exists
and rides alkcall (OQ-BL-01). Two residuals were left:
1. **Placement** — feature-gated `ops` module in this crate vs a sibling
crate. Phase 0's lean for the module was gated on "deciding input:
what alkgit's replicator wires first" — a deferral pattern a review
of the convergence has rejected as Schrödinger's code: alkgit is
paused *because* this crate is being built; "what the paused consumer
does first" never becomes a deciding input. The deferral would never
collapse; the decision must stand on evidence in hand.
2. **Deviation framing** — the user-level concern that a crate whose
store layer carries no alkcall dependency *deviates from the family
pattern* (alktty, alktunnels, alksocks all depend on
`alkcall = 0.8.x`). If this is a deviation, its rationale must be
explicit rather than assumed; if it is not, the conformance claim
must also be explicit, or every future session re-litigates it.
Also loaded here: the user-level concern that the family pattern must
be stated so the two *coming network consumers* (alkgit, alkfs) — which
will certainly use alkcall — see this crate fitting under it, not
beside it.
## What is actually decided
### 1. The family pattern is "nothing invents wire/ops/ACL", not "everyone depends on alkcall"
The family pattern (conventions 4–6) is: wire framing, op dispatch, and
authorization are invented at most once in the family — by alkcall — and
consumed everywhere else. Whether a crate *depends* on alkcall is then a
function of whether it faces the network, not of family membership:
- alkcall itself: zero sibling deps (it *is* the substrate; the pattern's
home).
- alktty / alktunnels / alksocks: depend on alkcall because they define
network-facing protocol layers.
- alkblobs' **store layer**: does not face the network. It is a library
seam — the same posture as alkcall's own `Connection` (handed in, not
dialed) and alktty's `TtyBackend` (backend-injected). A store cannot
carry alkcall without violating convention 4 (substrate-agnostic by
construction): the store must not know whether bytes arrive over a
network.
- alkblobs' **ops surface**: faces the network; rides alkcall like every
other network-facing family member.
The store layer's alkcall-freeness is therefore **conformance, not
deviation**: it is the pattern applied to a layer that has no network
side. The deviation framing dissolves once layers are named — a crate
may contain both a substrate-free core and a substrate-backed shell, as
this one does. What would be an actual deviation — an ops surface that
invents its own framing/ACL to avoid the alkcall dependency — is
rejected.
### 2. Why alkcall for the ops surface (kept from Phase 0, now recorded)
Both op shapes the blob family needs exist there with validation: JSON
ops with schema validation + typed error schemas (alkcall ADR-016), and
binary streaming via `channel_open` (alkcall ADR-047) in both
directions (Sub ADR-021 = verified fetch; Pub/Sink ADR-046 = put). ACL
maps onto `AccessControl` (alkcall ADR-011) + the ADR-017 privilege
model; an in-band scheme would be a second authorization story
(convention 6). Not adopting iroh-blobs' wire surface ("tickets",
postcard, provider protocol) remains deliberate (convention 7) — the
choice is not "iroh's protocol vs nothing", it is "the family substrate
vs inventing one".
### 3. Placement: feature-gated ops module, decided (no deferral)
The ops surface ships as an `ops` module gated behind feature `ops`
(default-off; the alkcall dep is feature-gated so the base crate stays
lean — convention 8). Decided on evidence in hand:
- The op family is **store-shaped** (hashes in, streams out), not
policy-shaped; a sibling crate buys nothing until a second
store-adjacent op family appears.
- Convention 8 is established family precedent (alktunnels/alksocks
feature-gate their alkcall-facing layers the same way).
- Module→sibling promotion, if ever needed, is mechanically additive —
the types move, the consumers re-point; no stored bytes change. The
cost of being wrong is bounded and small.
Promotion criteria (documented, not a hedge): if a second store-adjacent
op family appears outside this crate's scope (e.g., an alkfs sync op
family that wants blob ops but not this crate's registration), the ops
module may be promoted to a sibling crate. That would be a new ADR — the
decision recorded here is where it ships *now*.
### 4. The store layer never grows wire concerns — even in ops code
The ops module is a consumer of the store facade, not a side door: all
ops go through put/get/stat/read_range/pin (ADR-005, ADR-006). There is
no ops-internal path touching backends directly. This is what makes the
claim "if the ops surface were re-homed, the store loses nothing" true
rather than aspirational.
## Consequences
**Positive**
- Family conformance is explicit and traceable; no future session
re-litigates "why doesn't the store use alkcall".
- The base crate builds lean and substrate-free; the ops story exists
without forcing an alkcall dependency onto non-network consumers.
- alkgit/alkfs integrate the store core and the ops shell à la carte,
with the authorization story unified in alkcall.
**Negative**
- The feature-gate is a configuration surface to preserve in CI
(default + all-features builds; see AGENTS conventions).
- Two-layer naming ("alkblobs core" vs "alkblobs ops") must be kept
tight in docs so consumers don't mistake the shell for the core.
**Neutral**
- alkgit/alkfs will still carry *their own* direct alkcall deps for
their protocol layers; nothing about ADR-001 constrains them.
## References
- `docs/research/phase-0.md` OQ-BL-01 (substrate settlement, promoted
into this ADR)
- `docs/sdd_process.md` (deferral policy — Schrödinger's-code rule)
- [overview.md](../overview.md); [ops-surface.md](../ops-surface.md)
- alkcall ADRs 011/012/016/017/021/046/047/050
- ADR-003 (backends stay substrate-free too); ADR-005 (facade-only ops
access); ADR-006 (verified fetch)
docs/architecture/decisions/002-canonical-hash-and-key-encoding.md
+125
View File
@@ -0,0 +1,125 @@
# ADR-002: Canonical hash and key encoding
## Status
Accepted
## Context
The store keys every entry under a content hash. Phase 0 (OQ-BL-03)
resolved the hash *policy* — canonical git-blob-sha-256, tolerated
git-blob-sha-1, BLAKE3 demoted — after the inherited premise "iroh uses
BLAKE3, so we must" was rejected (convention 7: borrow conclusions, not
welds; nothing in the consumer set forces BLAKE3: alkgit forces git's
derivation by protocol definition, and every other consumer sees
addresses, not algorithms).
The residual was the **key encoding mechanics** — the byte layout of
keys inside backends. This is the crate's declared one-way door
(AGENTS convention 5): key bytes persist in kv rows and fs filenames,
with no schema-migration story; and this is the wire-format ADR that
must exist before the first consumer (convention 7's gate on publishing
wire-relevant decisions). POC #1 validated the mechanics empirically:
byte-exact `git hash-object` interop (finding 4, both algorithms,
external-oid round-trip into the pool), and tagged-key coexistence of
both algorithms in one flat pool (finding 5).
## Decision
### Hash policy (three tiers)
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
SHA-256. The store's canonical algorithm; git objects are first-class
entries with zero indirection; one algorithm domain = one address
space, so non-git consumers dedup into the same pool by construction
(ADR-005's premise).
2. **Tolerated legacy — `git-blob-sha-1`:** same derivation, SHA-1 —
accepted for existing-repo interop (a protocol necessity; git's own
collision-hardened SHA-1 threat model inherited verbatim).
3. **Excluded — BLAKE3:** not present in this crate. It was an
iroh-blobs inheritance, not a requirement (its virtues — merkle-native
trees, raw throughput — are irrelevant here under the canonical
digest at our scales). If a chunk-tree transfer encoding is ever
adopted it is a *new* decision at a *new* layer (ADR-006 excludes it
from this crate's scope by scoping, not by hedge).
Security footnote (carried from OQ-BL-03): one algorithm across
consumers is safe here — SHA-256 has no practical cross-protocol
ambiguity at these input shapes; the preamble domain-separates the
derivation. SHA-1's caveat is git's accepted position, inherited.
### Key encoding (the one-way door's bytes)
Physical key layout inside all backends:
```
key := <algorithm byte> <digest bytes>
algorithm byte: 0x01 = git-blob-sha-256 (32-byte digest)
0x02 = git-blob-sha-1 (20-byte digest)
```
- **Algorithm identity is inside the key.** A per-store
single-algorithm configuration would not have survived POC #1's
external-oid interop test — external git oids and pooled workspace
content must be addressable from the same table.
- **The tag byte is dropped** (POC finding 5: exactly one key kind
exists; the algorithm byte is the load-bearing part). Honesty note:
the POC validated the *tagged* layout `[tag][algorithm][digest]`;
dropping the single tag byte is the decided simplification of that
layout (the finding's own reasoning — one key kind — is the evidence
the simplification rests on; the algorithm byte, the load-bearing
half, is the tested part).
- **Fixed length per algorithm** (32 or 20 bytes); total key lengths
33/21. Fixed-length enum-tagged keys sort cleanly in total order —
required for the sweep's live-set diffs (BTreeMap/range-scan walks).
- **Unknown-algorithm and truncated keys are rejects**, not guesses
(POC's length-checking test is the standard).
- **Backends are opaque to all of this:** they store `&[u8]` keys; only
the store core constructs/interprets typed keys (ADR-003).
Algorithm bytes are assigned from this ADR's registry; new algorithms
require a new ADR (another one-way-door byte).
### The preamble as abstraction
An algorithm = domain-separated derivation (its own input
preprocessing) + digest. The preamble `"blob <len>\0"` is not a
special case — POC #1's domain-separated-premise abstraction held with
no special-casing anywhere. Note the structural consequence: the length
is *inside* the hashed input, which is what forces the two put paths
(ADR-003) and caps verification at whole-blob (ADR-006) — both accepted
rather than worked around.
## Consequences
**Positive**
- Git oids (either format) address pool entries directly from external
producers — no mapping layer, byte-exact (tested against the git CLI,
not just vectors).
- One address space across all consumers; dedup is structural.
- Key bytes are self-describing; backend data survives digest-layer
evolution (opaque at the boundary).
**Negative**
- The encoding is frozen once a backend ships data (the door closes);
algorithm byte 0x00 is reserved and never assigned (a future encoding
revision would need a different key universe or a migration ADR).
- SHA-1 acceptance inherits git's threat model — documented, not
re-litigated here.
**Neutral**
- Variable-length length-prefixed keys were considered (POC finding 5)
and rejected for now: two algorithms make fixed-length simpler and
sort-stable. Extension to a third algorithm re-opens byte layout —
new ADR.
## References
- `docs/research/phase-0.md` OQ-BL-03 (resolution + encoding notes)
- `docs/research/poc-trait-dispatch-findings.md` findings 4/5
- [hashing-and-keys.md](../hashing-and-keys.md)
- ADR-003 (opaque byte keys; two put paths the preamble forces);
ADR-005 (one address space); ADR-006 (whole-blob verification)
docs/architecture/decisions/003-backend-contract-and-dispatch.md
+148
View File
@@ -0,0 +1,148 @@
# ADR-003: Backend contract, shipped backends, and dispatch policy
## Status
Accepted
## Context
OQ-BL-02 settled the *base* contract in Phase 0 (lean `Backend` trait,
opaque keys, complete `list()`, size-threshold dual dispatch) with
named residuals: migration-between-backends policy, per-namespace
backend config, and the unknown-length put asymmetry (POC #3 finding
A6: an unknown-length re-put of a small blob lands on fs, duplicating
content the same digest's known-length put would put in kv).
One framing correction also lands here: the "two backends is ugly" /
"three backends" framing that influenced early rounds — corrected to
its real content as ADR-004 (two backends are *required*, and the
manifest layer is downstream, not a backend). This ADR records the
physical layer; ADR-004 records the boundary.
Evidence in hand (no further POC or consumer-waiting needed):
- POC #1 findings 1/2/6 (trait seam, list traps, dispatch thinness),
- POC #3 findings A1/A4/A5/A6 (put shapes, benchmark, stage hygiene,
the asymmetry),
- rudolfs anti-lesson (S3 backend with no list can never GC).
## Decision
### Backend trait contract
- **Opaque byte keys (`&[u8]`), opaque byte values.** Typed `Key`
converts at the store boundary only (ADR-002). Backends must survive
digest-layer evolution — a kv row or fs filename has no migration
story, so opaque, self-describing bytes are the durable choice. (The
generic-over-`K` alternative was considered and rejected in POC #1
finding 1: with one key type, generics buy nothing and break the
persistence boundary.)
- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`. Streaming and
range shapes ride the store layer above (fs `get` yields a file
handle, not a loaded buffer).
- **`list()` complete by contract** — and *well-formed*: a malformed
list (wrong destructure, values-as-keys) is as deadly as an
incomplete one, and silent until a sweep deletes the wrong things
(POC #1 finding 2). Consequence: backend implementations are accepted
only with sweep-outcome tests proving list correctness.
- **Virgin-store reads are no-ops** (missing table = absent/empty).
- **Namespace-blind** (ADR-005): backends see hashes only.
### Shipped backends (two, default-on; plus a testing tier)
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
the regime holding most git objects, workspace files, and manifests.
WAL + `synchronous=NORMAL` as shipped durability; prepared
statements; no-tables-yet tolerance.
- **`fs` (default-on)** for large blobs — flat
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
conclusion); stage-then-commit-rename; pread range reads (sound
locally per ADR-006).
- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and
in-process ephemerality. Never a production story.
Both production backends are default-on (batteries-included, the
alkgit feature-model precedent); `mem` is not.
### Dispatch policy
- **Size-threshold routing; default 128 KiB, constructor-tunable**
(midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4).
- **Unknown-length puts buffer in memory up to the threshold; the
buffer's fate has three pinned semantics** (finding A6's option (d)):
1. **Threshold exceeded mid-stream** (non-restartable sources are
normal — network streams can't be replayed): the buffered prefix
flushes *into a fresh stage file*, and the stream continues
appending into it. There is no "restart the stream" requirement
anywhere.
2. **Verification runs over the eventual stored artifact as a
whole**: the memory buffer for the sub-threshold kv commit; the
complete staged bytes for the fs commit. "Two passes over staged
bytes" and "hash-check against the canonical derivation" are both
literally true under this — there is no stitched-input case left
ambiguous (the buffer is *in* the stage file by the time the
second pass runs).
3. **Stream error before commit**: no entry lands and no stage file
remains (the stage-hygiene invariant extends to the buffered
case: buffer discard on the failure path, exactly like stage
discard).
This is finding A6's option (d), decided over the alternatives:
- (a) tier-correction sweeps are moving machinery that re-creates
migration, and are unnecessary if routing never misplaces a blob;
- (b) rejecting unknown-length small puts is hostile to streaming
producers for no safety gain (dedup still holds under (d));
- (c) accepting duplicated content across tiers makes `list()` lie
about the pool (two entries for one digest) and complicates GC's
live-set diffs for no benefit.
Option (d) makes **routing a pure function of content length** — the
property that eliminates the asymmetry *and* the migration question
with it. Batch-scope pinning (POC #1 finding 3) is orthogonal — it
protects in-flight puts, it does not route them — and ships anyway
(ADR-005).
- **No migration between backends.** Content is immutable; its length
never changes; routing is pure. The migration question existed only
to patch the asymmetry and is *resolved by elimination*: there is
nothing to migrate. A future backend needing migration writes a new
ADR.
- **No per-namespace backend configuration.** It would re-weld
namespacing into the physical layer — the exact rudolfs anti-pattern
ADR-005 inverts. Tenancy differences that ever require physical
separation (a different pool per tenant) are a deployment/topology
concern (two stores), not a backend-config concern.
## Consequences
**Positive**
- The dispatch is deterministic and test-invariant: same content ⇒ same
backend, always — including across mixed put paths.
- GC's enumerate-everything substrate (`list()` unions) is contract-
enforced upfront rather than discovered against the rudolfs failure.
- The shipped default matches the measured economics on the git/
workspace regime, re-tunable per deployment without API change.
**Negative**
- Unknown-length puts of large content pay two passes (forced by the
preamble — ADR-002 §Consequences) plus staging I/O; bounded-memory
buffering adds a copy for sub-threshold unknown-length puts (cheap,
per the benchmark).
- sqlite's on-disk format enters the crate's surface durability story
(file compatibility across rusqlite/sqlite versions is upstream
sqlite's guarantee, not ours).
**Neutral**
- A network-backed backend (S3-like) remains possible via the same
trait but ships nothing here; its list/GC obligations are the
acceptance gate.
## References
- `docs/research/poc-trait-dispatch-findings.md` findings 1/2/6
- `docs/research/poc-largeblob-findings.md` findings A1/A4/A5/A6
- `docs/research/phase-0.md` OQ-BL-02
- [backends-and-dispatch.md](../backends-and-dispatch.md);
[store-api.md](../store-api.md)
- ADR-002 (key bytes), ADR-004 (boundary vs manifest layer), ADR-005
(namespace-blindness, pins)
docs/architecture/decisions/004-two-backends-no-third.md
+101
View File
@@ -0,0 +1,101 @@
# ADR-004: Exactly two backends; manifest layers stay above the crate
## Status
Accepted
## Context
An early Phase 0 framing — "using two backends is ugly" — shaped several
rounds before being edited away, but the correction it was reaching for
arrived only in the Phase 1 review, and needs recording so no future
round reabsorbs the concern incorrectly:
- There are **always** two physical backends in this problem: storage
for small content (fast, kv/sqlite) and the filesystem for large
content. This is not ugly, it is the measured shape of the problem
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
- The "two backends is ugly" sentiment was actually about a **third
layer** in the old alknet-filesystem research — the *virtual
filesystem / appfile* stack: path trees, branches, tombstones,
filename↔hash maps living *alongside* the blob store (in that research,
literally a co-equal SQLite store beside iroh-blobs).
If left implicit, this misframing has two failure modes: (a) this crate
absorbs a path-tree/manifest layer ("three backends"), growing tables,
schemas, and branching semantics that belong to consumers; (b) this
crate shapes itself such that a future vfs builder must hack around it.
alkfs is a pending consumer precisely of that manifest layer — this ADR
is the boundary it will build against.
## Decision
**This crate ships exactly two production backends and nothing more;
every manifest/namespace/path-tree layer is a consumer, by construction
and by contract.** (A non-production ephemeral `mem` tier exists for
tests — ADR-003; it is not a storage story and does not extend this
boundary.)
1. **Two production backends is the complete set.** kv/sqlite (small)
+ fs (large) cover the scale economics with measured evidence
(ADR-003); no other physical tier is needed by any current or
planned consumer's *blob* needs.
2. **The "third backend" is not this crate's layer.** The vfs/appfile
shape — path→hash mapping, branches/tombstones/deltas per workspace,
the alknet-filesystem lineage — is storage *above* the pool:
reference sets over hashes (ADR-005's namespace machinery), not a
Backend implementation. Consumers build it on their own storage
(alkfs's path tree, alkgit's refs).
3. **The store provides the seams the manifest layer needs, and nothing
that forces it to hack:**
- hash-addressed put/get with a verified, git-interop-compatible
addressing scheme (ADR-002) — manifests reference entries by the
same canonical addresses git uses;
- liveness registration so manifest-holds maps to GC roots without
the store learning manifest formats (ADR-005);
- `stat`/`read_range` — enough for serving path-resolved content and
large-file ranges;
- pinned batch puts — manifest-atomic multi-file writes are
protected in flight (ADR-005).
4. **Namespacing at the store level is logical only** (ADR-005):
reference tables above, flat bytes below. If a consumer wants
physical isolation per tenant/repo, that is two pool deployments (a
topology choice), never a namespace flag inside one.
## Consequences
**Positive**
- The crate's scope stays testable and small: two backends, one pool,
one hash domain. No schema/versioning surface for manifest formats —
consumers iterate on theirs freely.
- alkgit and alkfs build their (very different) mapping layers on one
shared pool without either bending the store; their manifests interop
at the address level by construction.
- The alknet-filesystem lineage is honored as *shape guidance* for
consumers (branches, write sessions, chain walks), with zero of its
storage-layer mechanics (SQLite path tables, honker wiring, CRDT
sync) leaking in here.
**Negative**
- Consumers each re-build the small amount of manifest plumbing they
need (a reference table and a root registration). This is deliberate:
the manifest needs of alkgit (git refs) and alkfs (path trees) are
different enough that a shared one would fit neither.
**Neutral**
- If a future consumer *did* want a first-party manifest layer, it
would be a sibling crate over this one (e.g. `alkfs` itself), not an
extension of this crate.
## References
- `docs/research/phase-0.md` (settled approach: pooled CAS; OQ-BL-05)
- `/workspace/@alkdev/alknet/docs/research/alknet-filesystem/poc-summary.md`
— the historical lineage this ADR explicitly bounds (not a design
input)
- ADR-003 (the two backends), ADR-005 (namespaces as reference sets)
- [overview.md](../overview.md) (consumer map: alkgit, alkfs);
[backends-and-dispatch.md](../backends-and-dispatch.md)
docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md
+201
View File
@@ -0,0 +1,201 @@
# ADR-005: Pooled CAS, liveness seams, and mark-and-sweep GC
## Status
Accepted
## Context
OQ-BL-05 settled the pooling/GC *shape* in Phase 0 (one pooled CAS;
mark-and-sweep from registered roots; lean traversal ownership) with
residual mechanics named: namespace registry shape, GC scheduling,
and the sweep-vs-put race. All were resolvable on evidence in hand —
the prior art was read and verified (iroh `gc.rs`, `delete_set.rs`), the
mechanism was POC-validated (POC #1 findings 3/7/8: exact sweep counts,
abort semantics, delete-then-recover as byte-identical re-put, live-
shared callback seam), and the residuals turned on which of the
validated shapes to adopt — not on unknown facts (the
Schrödinger's-code rule of the deferral policy applies: none of these
may wait on paused consumers).
Also loaded: the "physically flat, logically namespaced" inversion of
rudolfs (its `s3://{org}/{project}/{sha256}` physical namespaces pay
cross-tenant dedup — the whole reason for pooling — for tenant
isolation), and the packfile tension (resolved in Phase 0's shape:
loose-equivalent kv entries; packs, if ever stored, are large blobs
served by range reads).
## Decision
### One pooled CAS; namespaces are reference sets above it
- One pool per node; repos/workspaces/all consumers are sets of hash
references over it. Cross-repo dedup by construction (the property
both demanding consumers need; git `alternates` pools as the awkward
prior).
- **Physically flat** — backends see `hash → bytes`, never namespaces
(ties to ADR-003's no-per-namespace-config).
- **Logically namespaced** — a namespace is a consumer-held reference
set; the store-side counterpart is liveness registration only.
- **The store persists no root table.** Namespaced root tags vs separate
reference-table persistence (a Phase 0 residual) resolves as: *the
store holds no durable roots*; consumers keep durable references
(git refs, alkfs path trees) and hand liveness over per sweep via the
live-shared seam. In-memory namespace tables remain available for
ephemeral consumers. This also dissolves the hidden wrinkle of a
store-internal root table under fs-only deployments (where would its
durable bytes live?). Ephemeral consumers are covered by pins; a
deployment that needs store-durable roots registers them as
consumer-owned durable state and sweeps through the seam.
- **ACL boundary:** the namespace is the alkcall `AccessControl`
resource for network ops (resource_id_path; ADR-001/ops-surface.md).
### Marks: three liveness sources
1. **Registered liveness sources** — live-shared consumers (the seam
named `register_liveness_source`; the POC's finding 7 copy-semantics
failure is why the verb is pinned — "install" implies snapshot, and
snapshotting breaks consumer registration that happens after).
2. **Put-path pins (RAII)** — a `Pin` guard holds a refcount per key
until dropped or converted into a consumer reference; batch scope
available for multi-put writes (manifest writes are exactly this;
POC finding 3's named Phase 1 requirement, satisfied here). The
conversion ordering is pinned: a pin's drop and its replacement
reference's registration into a liveness source must be one
ordered step under the same arbitration lock the sweep consults —
a consumer may not release a pin before its registered source
observes the replacement (the local analog of the ops-layer pin
token contract).
3. **Protect callback** — consulted pre-sweep; may add known-live keys
or **abort the run** (typed `GcAborted`, nothing deleted). iroh's
`ProtectOutcome::Abort` conclusion adopted: a flaky protection
source skips the sweep rather than risk deletion.
### Sweep: mark → delete window → commit
- Enumerate the pool via backends' complete `list()` (ADR-003's
contract; its reason to exist).
- Compute the live set from all three sources; batch-delete the dead
(~100/batch; iroh's proven shape).
- **Delete windows** close the sweep-vs-put race (POC finding 3's named
Phase 1 requirement — the pin-map lock alone leaves a real window
between "list" and "delete"):
1. **mark:** enumerate the pool; compute the live set from all three
liveness sources; stage the candidate-dead set;
2. **delete window opens:** for each candidate, at *deletion time*
(not once upfront), arbitrate under the pin/liveness lock: if the
key is pinned or re-observed live, cancel it from the window;
otherwise proceed to delete;
3. **commit:** remaining candidates deleted in batches; window
closed.
The invariant holds because both halves of the race are now
ordered under the same lock:
- **Put commits pin before publish.** A put's `Pin` is acquired
*before* the entry becomes visible in its backend; an entry cannot
appear in a later sweep's `list()` without already carrying its
pin (or its liveness registration, whichever the consumer
converted the pin into) — publish and protection are one step,
and a put for an already-pool-existing entry (dedup) acquires no
*new pool entry* — the returned `Pin` refcounts the existing entry
under the same arbitration lock (a dedup call's returned Pin
protects exactly as a fresh put's does).
- **Deletes arbitrate under the pin lock at delete time.** A
deletion of key *k* cannot proceed while *k* carries a pin; since
any visible entry is pinned (invariant above), any deletion of a
visible entry happens only with no pin held — and a *concurrent*
re-put of *k* is either (a) blocked on the arbitration lock until
the deletion commits, after which the re-put lands as a fresh
pinned put (delete-then-recover semantics, validated), or (b)
ordered after, in which case its pin already protects it.
Net property, test-asserted: **a visible pool entry is never
deleted while liveness (pin or registered source) protects it, and
an in-flight put is never deleted by a sweep started before it
committed.** The arbitration can be cheap: the pin check is one
map lookup per candidate key; the delete window exists so the
reconciliation happens precisely at the delete boundary rather
than wholesale.
The iroh DeleteSet/ProtectHandle transactions and per-hash
serialized-actor pattern are the re-borrowed prior art this shape
generalizes (their serialized actor is exactly "arbitrate at
delete time"; our window batches the arbitration).
- `has`/`get` during a window see no torn state (immutable entries;
existence flips atomically per key).
- **Delete-then-recover is a re-put** (validated: byte-identical under
the same key); no tombstone layer exists.
- **Direct `delete(key)`:** refuses pinned keys with a typed error
(same arbitration); permitted for embedder correction flows on
unpinned entries. Outside-of-sweep deletion is unusual by posture —
most deletion should flow through sweeps.
### Traversal and scheduling
- **Lean (a) confirmed:** consumers compute liveness beyond "these
roots exist" and hand hash sets over; the store stays structure-blind
(what Phase 0 called "lean (a)": consumer-side traversal, vs option
(b) store-native traversal of consumer manifests). Store-native
traversal re-opens only on a *measured* consumer cost (an external
fact, not a parked hedge).
- **Explicit `sweep()`; no ambient timers.** The embedder owns cadence
(idle sweeps, interval sweeps, consumer-driven prunes — all one API).
- **Accounting:** pool-level totals via `list`+`stat`; per-namespace
accounting is computed by consumers from their reference sets.
### Packfiles (the Phase 0 pack tension, carried forward)
Git objects enter the pool as loose-equivalent kv entries (small tier
— the common case). If packfile serving is ever wanted by alkgit, packs
are stored as large blobs and served through `stat`/`read_range` (git's
`.idx` does per-object offset lookup; local range serving is sound —
ADR-006). This avoids gitoxide's pack-ID-stability machinery entirely
(there are no IDs to rebind; the address is the content). This restate-
as-decision carries no new open question; whether alkgit wants it is
OQ-07's.
## Consequences
**Positive**
- The dedup property the consumers exist for is structural, not
configured; pooling semantics can't be "off".
- Sweep safety is testable to exact counts (POC-proven) at the
architecture level: the invariant (delete-window re-observation) is
the test gate, independent of mechanism choice.
- No GC coupling in the store: a consumer without registered liveness
sources cannot have its content deleted (sweep aborts) — safe-by-
default.
**Negative**
- GC requires complete, well-formed `list()` from every backend
(enforced upstream, ADR-003) and delete-window machinery in the
sweep path — the most intricate state machine in the crate.
- Explicit sweep means an embedder that never schedules one accumulates
garbage (their choice; documented posture, not a defect).
- Pin/liveness misuse by consumers is runtime-observable (a sweep may
delete unreferenced content if a consumer deregisters early); the
seam's correctness contract is on consumers — documented in
store-api.md's invariants.
**Neutral**
- Cross-node GC coordination (p2p replicators) lives in the replicator
policy layer, above; the seam is all it needs.
## References
- `docs/research/phase-0.md` OQ-BL-05 (settled decisions + residuals,
all promoted here)
- `docs/research/poc-trait-dispatch-findings.md` findings 3/7/8
- `docs/research/iroh-blobs-eval.md` (gc.rs / delete_set.rs reads)
- rudolfs (the physical-namespace anti-pattern inverted);
gix-odb (alternates/pool prior art)
- ADR-002 (one address space), ADR-003 (list contract, pins), ADR-004
(manifest layers above), ADR-006 (verification limits inform recover
semantics)
- [gc-and-namespaces.md](../gc-and-namespaces.md);
[store-api.md](../store-api.md)
docs/architecture/decisions/006-verification-posture-and-transfer-encoding.md
+106
View File
@@ -0,0 +1,106 @@
# ADR-006: Verification posture — whole-blob checks; transfer encodings excluded by scoping
## Status
Accepted
## Context
OQ-BL-04's settled Phase 0 posture: whole-file CAS as the default
granularity; per-range verification impossible under the canonical
digest (the preamble hashes the length — POC #3 finding A2,
independently reproduced in the iroh-blobs eval; no slice has any hash
relationship to the whole); the chunk-tree encoding framed as "a
transfer-layer conditional — deferred until a consumer exists".
That last framing is a deferral this crate cannot make under its own
deferral policy: "the consumer that would want networked verified
ranged fetch" (p2p git's replicator) is paused *because* this crate is
being built. A condition gated on it is Schrödinger's code — it never
collapses. The decision must stand on the evidence in hand, which is
complete for the *scoping* question even though nothing measures a
future consumer:
- What the consumers need today is verifiable **whole-blob** transfer:
fetch bytes, hash-check against the canonical digest the caller
carries. That is exactly verified-fetch (the `Sub` op with the digest
in the offer), works at any size, and needs no encoding — POC #3's
streaming paths are validated byte-exact against git.
- What would need a chunk tree is *per-chunk integrity during ranged
transfer of one large blob* — a property nothing in the current
consumer set requires, and which cannot be retrofitted onto the
canonical digest anyway (impossible under the preamble, finding A2;
iroh gets it from the bao *tree*, not BLAKE3 the algorithm).
- Storing chunk trees would add a second stored artifact per blob
(outboard encoding), a second write path, new GC surface, and a
second hash domain — speculative cost before any consumer names the
need.
Also settled here: local range reads need only transport-integrity
slice digests (out-of-band, carried by the caller — the store returns
one from `read_range`); git objects self-verify under git's own model
when read whole; small-tier range reads slice whole values in memory.
## Decision
1. **Verification is whole-blob.** Put and get paths hash-check against
the canonical derivation (ADR-002). `stat` probes length without
reading content.
2. **Range reads return `(slice, slice_digest)`** — the slice digest is
a transport-integrity check carried out-of-band by the caller (e.g.
the offering side of a fetch). The store performs no per-range
verification *because none is derivable* under the canonical digest;
local serving (packfiles, large-file reads) is sound on honest media
(finding A2).
3. **No chunk-tree, CDC, or otherwise partial-verification encoding is
adopted in this crate — by scoping, not by deferral.** The crate's
domain is the store; a transfer encoding is a network-layer artifact
and has no owner here. If a future networked consumer (a p2p
replicator above an un-paused alkgit, alkfs sync, a future ops
extension) names a requirement for verified ranged fetch of one
large blob, that requirement is met by a *new decision at a new
layer*: an optional encoding module (BLAKE3-confined per ADR-002's
tier 3 if a tree encoding is chosen; content-defined chunking if
edit-resistance is the goal) whose verified whole-contents register
back into the pool as ordinary canonical entries — dedup between
chunked and whole-file paths is preserved by registration, not by
shared structure.
4. **The verified-fetch op (ADR-001) carries whole-blob verification**
as its integrity story: digest in the offer, hash-check on receipt,
fanout above the store for concurrent subscribers (POC #3 finding
A3); late joiners degrade to post-commit verified `get`.
## Consequences
**Positive**
- The store stays single-granularity, single-hash-domain, single
write-path family — the simplest shape consistent with all
consumers' *current* needs; no speculative outboard/GC surface.
- Verified transfer exists at every blob size with zero extra machinery
(whole-blob digest check), which is what the confirmed ops family
serves.
**Negative**
- A future large-blob verified-ranged-fetch consumer pays a new module
and a second decision — deliberately deferred *cost*, not unmade
decision (the distinction this ADR records).
- Callers wanting mid-transfer corruption localization on very large
blobs get error-at-EOF, not error-at-chunk, without an encoding layer
(acceptable; named above).
**Neutral**
- BLAKE3's "large-blob encoding consideration" stays conditional and
out-of-crate; nothing here allocates an API slot for it.
## References
- `docs/research/phase-0.md` OQ-BL-04 (posture + surface addendum
promoted here and to ADR-003/store-api)
- `docs/research/poc-largeblob-findings.md` findings A2/A3
- `docs/research/iroh-blobs-eval.md` §Verification
- ADR-001 (ops surface; verified-fetch op), ADR-002 (digest + preamble
consequence), ADR-005 (range reads in packfile serving)
- [store-api.md](../store-api.md); [ops-surface.md](../ops-surface.md)