docs(architecture): Phase 1 spec tree — 7 specs, ADRs 001-006, OQ tracker
- overview/hashing/store-api/backends/gc/ops specs + README index, all draft status with YAML frontmatter and cross-referenced ADR/OQ - ADR-001: substrate posture (store layer alkcall-free = family conformance, not deviation) + ops placement decided (feature-gated module, no consumer-waited deferral) - ADR-002: canonical git-blob-sha-256 + key encoding — the declared wire-format ADR (one-way door) - ADR-003: backend contract, sqlite/fs/mem tiers, pure-function dispatch, no migration, unknown-length buffering semantics pinned - ADR-004: exactly two production backends; manifest/path-tree layers are consumer-side (the corrected two-vs-three-backends framing) - ADR-005: pooled CAS, three liveness sources, delete windows with pin-before-publish + delete-time arbitration (race closed), no ambient scheduling - ADR-006: whole-blob verification; chunk-tree encodings excluded by scoping, not deferred on a paused consumer - open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07 /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task - AGENTS.md: status updated to Phase 1, gix-odb path corrected Verification: cargo test / clippy -D warnings / fmt --check clean
This commit is contained in:
1 parent
7d9ac8a6ec
commit
4b5009d85a
16 files changed
+1953
-8
No files matched your search
@@ -0,0 +1,136 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-01
|
||||
---
|
||||
|
||||
# Pooling, namespaces, and GC
|
||||
|
||||
## What this is
|
||||
|
||||
The layer that makes the store *pooled*: one content-addressed pool per
|
||||
node, dedup across every consumer by construction, with garbage
|
||||
collection that is safe against concurrent writes. The pool property is
|
||||
the reason both demanding consumers exist (overview.md); GC is its
|
||||
price, and this document specifies the mechanism the POCs validated
|
||||
(ADR-005).
|
||||
|
||||
## The pooled CAS
|
||||
|
||||
- **One pool per node.** Repos, workspaces, and appfile stores are all
|
||||
*sets of hash references* over the same CAS — never walled-off
|
||||
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
|
||||
object pools) is the same idea done awkwardly; the pool generalizes it
|
||||
to unrelated repos and to p2p replication.
|
||||
- **One address space.** Under the canonical hash (ADR-002), the same
|
||||
file content has the same address whether it entered via a git oid or
|
||||
a workspace manifest — cross-consumer dedup is free by construction,
|
||||
not a feature.
|
||||
- **Physically flat, logically namespaced** (the rudolfs inversion).
|
||||
The byte layer is `hash → bytes`, backends namespace-blind; the
|
||||
namespace is a *reference set* above the crate:
|
||||
|
||||
## Namespaces (logical, above the pool)
|
||||
|
||||
- A namespace is `namespace → set of root hashes` held by the consumer
|
||||
(git refs *are* the names; an LFS pointer file in a tree *is* the
|
||||
reference; an alkfs path tree *is* the manifest). The store never
|
||||
walks consumer-side manifests — it stays structure-blind.
|
||||
- The store-side counterpart is the **liveness seam**: consumers
|
||||
register live-shared liveness sources (ADR-005, §Decision —
|
||||
`register_liveness_source`; the POC's copy-semantics failure finding 7
|
||||
is the pinning reason). Namespace tables may also exist purely
|
||||
in-memory for ephemeral consumers; nothing in the store persists
|
||||
namespaces — where roots live when a deployment runs fs-only is a
|
||||
non-question by design: roots live with the consumer, whose references
|
||||
are as durable as they need to be.
|
||||
- ACL mapping for network ops: the namespace is the resource
|
||||
(`resource_id_path` selects it) — see ops-surface.md.
|
||||
|
||||
## Mark-and-sweep GC (ADR-005)
|
||||
|
||||
Validated mechanism (POC #1 findings 3/7/8 — exact sweep counts, abort
|
||||
semantics, delete-then-recover as byte-identical re-put):
|
||||
|
||||
- **Liveness sources (roots):**
|
||||
1. consumer-registered liveness sources (live-shared, above),
|
||||
2. put-path **pins** (RAII; batch scope available for multi-put
|
||||
writes — manifest writes are exactly this),
|
||||
3. the **protect callback** consulted before each sweep: it may add
|
||||
externally-known hashes or **abort the run** (`GcAborted`; a
|
||||
flaky protection source skips the sweep rather than risk
|
||||
deletion — iroh's `ProtectOutcome::Abort` conclusion, adopted).
|
||||
- **Sweep:** enumerate the whole pool (backends' complete `list()`,
|
||||
ADR-003), compute the live set, batch-delete the dead
|
||||
(batch-sized, ~100/batch, iroh's proven shape). Delete-then-recover
|
||||
is a re-put — byte-identical under the same key (validated).
|
||||
- **Traversal ownership: lean.** Liveness computation beyond "these
|
||||
roots exist" is the consumer's job, handed over via the seam. The
|
||||
store never learns manifest formats. If a real consumer's live-set
|
||||
computation proves too heavy, store-native traversal is a *new* ADR
|
||||
(documented constraint, not a parked hedge — the un-pause condition
|
||||
is an external fact: a measured consumer cost).
|
||||
|
||||
## Delete windows (the sweep-vs-put race)
|
||||
|
||||
The race (POC #1 finding 3): a pin added between sweep-"list" and
|
||||
sweep-"delete" is a lost pin — a blob deleted while in flight. The
|
||||
architecture resolves it as a **delete window** with three phases —
|
||||
mark (enumerate + live-set), per-key arbitration under the
|
||||
pin/liveness lock *at delete time*, batched commit — resting on two
|
||||
ordering invariants: **put commits pin before publish**, and **deletes
|
||||
arbitrate under the pin lock at delete time** (full protocol + proof:
|
||||
ADR-005 §Sweep; iroh `delete_set.rs`'s ProtectHandle/protect-cancel
|
||||
shape and its serialized-actor pattern are the re-borrowed prior art —
|
||||
their serialized actor is exactly "arbitrate at delete time"; the
|
||||
window batches the arbitration).
|
||||
|
||||
Test-asserted property: **a visible pool entry is never deleted while
|
||||
liveness (pin or registered source) protects it, and an in-flight put
|
||||
is never deleted by a sweep started before it committed.** Direct
|
||||
`delete(key)` refuses pinned keys (typed error; same arbitration).
|
||||
|
||||
`has()`/`get()` during a window observe either the old or the new
|
||||
state; there is no torn observation (the pool's entries are immutable;
|
||||
existence flips atomically per key).
|
||||
|
||||
## GC scheduling and scope (ADR-005)
|
||||
|
||||
- **Explicit `sweep()`; embedder owns cadence.** No background timers
|
||||
in the crate. An embedder that wants interval sweeps adds them above
|
||||
(one call). Consumer-driven sweeps (after a manifest prune, say) are
|
||||
the same API.
|
||||
- **Sweep scoping:** whole-pool is the base. Namespace-scoped sweeps
|
||||
run by computing a narrower live set from that namespace's liveness
|
||||
source; the pool remains shared (scoping is a liveness computation
|
||||
choice, not a physical partition).
|
||||
- **Accounting** (per-namespace sizes): computed from reference sets by
|
||||
consumers; the store reports pool-level totals only (`list` +
|
||||
`stat`). Physical-layer accounting would re-introduce namespace
|
||||
physics — rejected with the per-namespace backend config
|
||||
(ADR-003 §Consequences).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| ADR | Decision | Summary |
|
||||
|---|---|---|
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
|
||||
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
|
||||
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
|
||||
|
||||
## Open Questions
|
||||
|
||||
None owned by this document. The p2p replicator's cross-node GC
|
||||
coordination is a property of the replicator policy layer (above the
|
||||
crate), governed by the same seam — it does not open a question here
|
||||
until a consumer names a requirement (OQ-07/OQ-08 intake).
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/phase-0.md` OQ-BL-05 (settled decisions + residuals)
|
||||
- `docs/research/poc-trait-dispatch-findings.md` findings 3/7/8
|
||||
- `docs/research/iroh-blobs-eval.md` — gc.rs / delete_set.rs reads
|
||||
(the re-borrowed conclusions)
|
||||
- rudolfs — the physical-namespace anti-pattern (inverted)
|
||||
- gix-odb — alternates/pool prior art for the p2p case
|
||||
- ADR-005; [store-api.md](store-api.md) (Pin semantics, sweep API);
|
||||
[ops-surface.md](ops-surface.md) (namespace-as-resource ACL)
|
||||
Reference in new issue
Block a user