docs(architecture): Phase 1 spec tree — 7 specs, ADRs 001-006, OQ tracker

- overview/hashing/store-api/backends/gc/ops specs + README index,
  all draft status with YAML frontmatter and cross-referenced ADR/OQ
- ADR-001: substrate posture (store layer alkcall-free = family
  conformance, not deviation) + ops placement decided (feature-gated
  module, no consumer-waited deferral)
- ADR-002: canonical git-blob-sha-256 + key encoding — the declared
  wire-format ADR (one-way door)
- ADR-003: backend contract, sqlite/fs/mem tiers, pure-function
  dispatch, no migration, unknown-length buffering semantics pinned
- ADR-004: exactly two production backends; manifest/path-tree layers
  are consumer-side (the corrected two-vs-three-backends framing)
- ADR-005: pooled CAS, three liveness sources, delete windows with
  pin-before-publish + delete-time arbitration (race closed), no
  ambient scheduling
- ADR-006: whole-blob verification; chunk-tree encodings excluded by
  scoping, not deferred on a paused consumer
- open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07
  /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task
- AGENTS.md: status updated to Phase 1, gix-odb path corrected

Verification: cargo test / clippy -D warnings / fmt --check clean
This commit is contained in:
glm-5.3-flash committed 2026-10-01 14:45:32 +00:00
1 parent 7d9ac8a6ec
commit 4b5009d85a
16 files changed
+1953 -8

No files matched your search

+136
View File
@@ -0,0 +1,136 @@
---
status: draft
last_updated: 2026-10-01
---
# Pooling, namespaces, and GC
## What this is
The layer that makes the store *pooled*: one content-addressed pool per
node, dedup across every consumer by construction, with garbage
collection that is safe against concurrent writes. The pool property is
the reason both demanding consumers exist (overview.md); GC is its
price, and this document specifies the mechanism the POCs validated
(ADR-005).
## The pooled CAS
- **One pool per node.** Repos, workspaces, and appfile stores are all
*sets of hash references* over the same CAS — never walled-off
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
object pools) is the same idea done awkwardly; the pool generalizes it
to unrelated repos and to p2p replication.
- **One address space.** Under the canonical hash (ADR-002), the same
file content has the same address whether it entered via a git oid or
a workspace manifest — cross-consumer dedup is free by construction,
not a feature.
- **Physically flat, logically namespaced** (the rudolfs inversion).
The byte layer is `hash → bytes`, backends namespace-blind; the
namespace is a *reference set* above the crate:
## Namespaces (logical, above the pool)
- A namespace is `namespace → set of root hashes` held by the consumer
(git refs *are* the names; an LFS pointer file in a tree *is* the
reference; an alkfs path tree *is* the manifest). The store never
walks consumer-side manifests — it stays structure-blind.
- The store-side counterpart is the **liveness seam**: consumers
register live-shared liveness sources (ADR-005, §Decision —
`register_liveness_source`; the POC's copy-semantics failure finding 7
is the pinning reason). Namespace tables may also exist purely
in-memory for ephemeral consumers; nothing in the store persists
namespaces — where roots live when a deployment runs fs-only is a
non-question by design: roots live with the consumer, whose references
are as durable as they need to be.
- ACL mapping for network ops: the namespace is the resource
(`resource_id_path` selects it) — see ops-surface.md.
## Mark-and-sweep GC (ADR-005)
Validated mechanism (POC #1 findings 3/7/8 — exact sweep counts, abort
semantics, delete-then-recover as byte-identical re-put):
- **Liveness sources (roots):**
1. consumer-registered liveness sources (live-shared, above),
2. put-path **pins** (RAII; batch scope available for multi-put
writes — manifest writes are exactly this),
3. the **protect callback** consulted before each sweep: it may add
externally-known hashes or **abort the run** (`GcAborted`; a
flaky protection source skips the sweep rather than risk
deletion — iroh's `ProtectOutcome::Abort` conclusion, adopted).
- **Sweep:** enumerate the whole pool (backends' complete `list()`,
ADR-003), compute the live set, batch-delete the dead
(batch-sized, ~100/batch, iroh's proven shape). Delete-then-recover
is a re-put — byte-identical under the same key (validated).
- **Traversal ownership: lean.** Liveness computation beyond "these
roots exist" is the consumer's job, handed over via the seam. The
store never learns manifest formats. If a real consumer's live-set
computation proves too heavy, store-native traversal is a *new* ADR
(documented constraint, not a parked hedge — the un-pause condition
is an external fact: a measured consumer cost).
## Delete windows (the sweep-vs-put race)
The race (POC #1 finding 3): a pin added between sweep-"list" and
sweep-"delete" is a lost pin — a blob deleted while in flight. The
architecture resolves it as a **delete window** with three phases —
mark (enumerate + live-set), per-key arbitration under the
pin/liveness lock *at delete time*, batched commit — resting on two
ordering invariants: **put commits pin before publish**, and **deletes
arbitrate under the pin lock at delete time** (full protocol + proof:
ADR-005 §Sweep; iroh `delete_set.rs`'s ProtectHandle/protect-cancel
shape and its serialized-actor pattern are the re-borrowed prior art —
their serialized actor is exactly "arbitrate at delete time"; the
window batches the arbitration).
Test-asserted property: **a visible pool entry is never deleted while
liveness (pin or registered source) protects it, and an in-flight put
is never deleted by a sweep started before it committed.** Direct
`delete(key)` refuses pinned keys (typed error; same arbitration).
`has()`/`get()` during a window observe either the old or the new
state; there is no torn observation (the pool's entries are immutable;
existence flips atomically per key).
## GC scheduling and scope (ADR-005)
- **Explicit `sweep()`; embedder owns cadence.** No background timers
in the crate. An embedder that wants interval sweeps adds them above
(one call). Consumer-driven sweeps (after a manifest prune, say) are
the same API.
- **Sweep scoping:** whole-pool is the base. Namespace-scoped sweeps
run by computing a narrower live set from that namespace's liveness
source; the pool remains shared (scoping is a liveness computation
choice, not a physical partition).
- **Accounting** (per-namespace sizes): computed from reference sets by
consumers; the store reports pool-level totals only (`list` +
`stat`). Physical-layer accounting would re-introduce namespace
physics — rejected with the per-namespace backend config
(ADR-003 §Consequences).
## Design Decisions
| ADR | Decision | Summary |
|---|---|---|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
## Open Questions
None owned by this document. The p2p replicator's cross-node GC
coordination is a property of the replicator policy layer (above the
crate), governed by the same seam — it does not open a question here
until a consumer names a requirement (OQ-07/OQ-08 intake).
## References
- `docs/research/phase-0.md` OQ-BL-05 (settled decisions + residuals)
- `docs/research/poc-trait-dispatch-findings.md` findings 3/7/8
- `docs/research/iroh-blobs-eval.md` — gc.rs / delete_set.rs reads
(the re-borrowed conclusions)
- rudolfs — the physical-namespace anti-pattern (inverted)
- gix-odb — alternates/pool prior art for the p2p case
- ADR-005; [store-api.md](store-api.md) (Pin semantics, sweep API);
[ops-surface.md](ops-surface.md) (namespace-as-resource ACL)