# ADR-005: Pooled CAS, liveness seams, and mark-and-sweep GC ## Status Accepted ## Context OQ-BL-05 settled the pooling/GC *shape* in Phase 0 (one pooled CAS; mark-and-sweep from registered roots; lean traversal ownership) with residual mechanics named: namespace registry shape, GC scheduling, and the sweep-vs-put race. All were resolvable on evidence in hand — the prior art was read and verified (iroh `gc.rs`, `delete_set.rs`), the mechanism was POC-validated (POC #1 findings 3/7/8: exact sweep counts, abort semantics, delete-then-recover as byte-identical re-put, live- shared callback seam), and the residuals turned on which of the validated shapes to adopt — not on unknown facts (the Schrödinger's-code rule of the deferral policy applies: none of these may wait on paused consumers). Also loaded: the "physically flat, logically namespaced" inversion of rudolfs (its `s3://{org}/{project}/{sha256}` physical namespaces pay cross-tenant dedup — the whole reason for pooling — for tenant isolation), and the packfile tension (resolved in Phase 0's shape: loose-equivalent kv entries; packs, if ever stored, are large blobs served by range reads). ## Decision ### One pooled CAS; namespaces are reference sets above it - One pool per node; repos/workspaces/all consumers are sets of hash references over it. Cross-repo dedup by construction (the property both demanding consumers need; git `alternates` pools as the awkward prior). - **Physically flat** — backends see `hash → bytes`, never namespaces (ties to ADR-003's no-per-namespace-config). - **Logically namespaced** — a namespace is a consumer-held reference set; the store-side counterpart is liveness registration only. - **The store persists no root table.** Namespaced root tags vs separate reference-table persistence (a Phase 0 residual) resolves as: *the store holds no durable roots*; consumers keep durable references (git refs, alkfs path trees) and hand liveness over per sweep via the live-shared seam. In-memory namespace tables remain available for ephemeral consumers. This also dissolves the hidden wrinkle of a store-internal root table under fs-only deployments (where would its durable bytes live?). Ephemeral consumers are covered by pins; a deployment that needs store-durable roots registers them as consumer-owned durable state and sweeps through the seam. - **ACL boundary:** the namespace is the alkcall `AccessControl` resource for network ops (resource_id_path; ADR-001/ops-surface.md). ### Marks: three liveness sources 1. **Registered liveness sources** — live-shared consumers (the seam named `register_liveness_source`; the POC's finding 7 copy-semantics failure is why the verb is pinned — "install" implies snapshot, and snapshotting breaks consumer registration that happens after). 2. **Put-path pins (RAII)** — a `Pin` guard holds a refcount per key until dropped or converted into a consumer reference; batch scope available for multi-put writes (manifest writes are exactly this; POC finding 3's named Phase 1 requirement, satisfied here). The conversion ordering is pinned: a pin's drop and its replacement reference's registration into a liveness source must be one ordered step under the same arbitration lock the sweep consults — a consumer may not release a pin before its registered source observes the replacement (the local analog of the ops-layer pin token contract). 3. **Protect callback** — consulted pre-sweep; may add known-live keys or **abort the run** (typed `GcAborted`, nothing deleted). iroh's `ProtectOutcome::Abort` conclusion adopted: a flaky protection source skips the sweep rather than risk deletion. ### Sweep: mark → delete window → commit - Enumerate the pool via backends' complete `list()` (ADR-003's contract; its reason to exist). - Compute the live set from all three sources; batch-delete the dead (~100/batch; iroh's proven shape). - **Delete windows** close the sweep-vs-put race (POC finding 3's named Phase 1 requirement — the pin-map lock alone leaves a real window between "list" and "delete"): 1. **mark:** enumerate the pool; compute the live set from all three liveness sources; stage the candidate-dead set; 2. **delete window opens:** for each candidate, at *deletion time* (not once upfront), arbitrate under the pin/liveness lock: if the key is pinned or re-observed live, cancel it from the window; otherwise proceed to delete; 3. **commit:** remaining candidates deleted in batches; window closed. The invariant holds because both halves of the race are now ordered under the same lock: - **Put commits pin before publish.** A put's `Pin` is acquired *before* the entry becomes visible in its backend; an entry cannot appear in a later sweep's `list()` without already carrying its pin (or its liveness registration, whichever the consumer converted the pin into) — publish and protection are one step, and a put for an already-pool-existing entry (dedup) acquires no *new pool entry* — the returned `Pin` refcounts the existing entry under the same arbitration lock (a dedup call's returned Pin protects exactly as a fresh put's does). - **Deletes arbitrate under the pin lock at delete time.** A deletion of key *k* cannot proceed while *k* carries a pin; since any visible entry is pinned (invariant above), any deletion of a visible entry happens only with no pin held — and a *concurrent* re-put of *k* is either (a) blocked on the arbitration lock until the deletion commits, after which the re-put lands as a fresh pinned put (delete-then-recover semantics, validated), or (b) ordered after, in which case its pin already protects it. Net property, test-asserted: **a visible pool entry is never deleted while liveness (pin or registered source) protects it, and an in-flight put is never deleted by a sweep started before it committed.** The arbitration can be cheap: the pin check is one map lookup per candidate key; the delete window exists so the reconciliation happens precisely at the delete boundary rather than wholesale. The iroh DeleteSet/ProtectHandle transactions and per-hash serialized-actor pattern are the re-borrowed prior art this shape generalizes (their serialized actor is exactly "arbitrate at delete time"; our window batches the arbitration). - `has`/`get` during a window see no torn state (immutable entries; existence flips atomically per key). - **Delete-then-recover is a re-put** (validated: byte-identical under the same key); no tombstone layer exists. - **Direct `delete(key)`:** refuses pinned keys with a typed error (same arbitration); permitted for embedder correction flows on unpinned entries. Outside-of-sweep deletion is unusual by posture — most deletion should flow through sweeps. ### Traversal and scheduling - **Lean (a) confirmed:** consumers compute liveness beyond "these roots exist" and hand hash sets over; the store stays structure-blind (what Phase 0 called "lean (a)": consumer-side traversal, vs option (b) store-native traversal of consumer manifests). Store-native traversal re-opens only on a *measured* consumer cost (an external fact, not a parked hedge). - **Explicit `sweep()`; no ambient timers.** The embedder owns cadence (idle sweeps, interval sweeps, consumer-driven prunes — all one API). - **Accounting:** pool-level totals via `list`+`stat`; per-namespace accounting is computed by consumers from their reference sets. ### Packfiles (the Phase 0 pack tension, carried forward) Git objects enter the pool as loose-equivalent kv entries (small tier — the common case). If packfile serving is ever wanted by alkgit, packs are stored as large blobs and served through `stat`/`read_range` (git's `.idx` does per-object offset lookup; local range serving is sound — ADR-006). This avoids gitoxide's pack-ID-stability machinery entirely (there are no IDs to rebind; the address is the content). This restate- as-decision carries no new open question; whether alkgit wants it is OQ-07's. ## Consequences **Positive** - The dedup property the consumers exist for is structural, not configured; pooling semantics can't be "off". - Sweep safety is testable to exact counts (POC-proven) at the architecture level: the invariant (delete-window re-observation) is the test gate, independent of mechanism choice. - No GC coupling in the store: a consumer without registered liveness sources cannot have its content deleted (sweep aborts) — safe-by- default. **Negative** - GC requires complete, well-formed `list()` from every backend (enforced upstream, ADR-003) and delete-window machinery in the sweep path — the most intricate state machine in the crate. - Explicit sweep means an embedder that never schedules one accumulates garbage (their choice; documented posture, not a defect). - Pin/liveness misuse by consumers is runtime-observable (a sweep may delete unreferenced content if a consumer deregisters early); the seam's correctness contract is on consumers — documented in store-api.md's invariants. **Neutral** - Cross-node GC coordination (p2p replicators) lives in the replicator policy layer, above; the seam is all it needs. ## References - `docs/research/phase-0.md` OQ-BL-05 (settled decisions + residuals, all promoted here) - `docs/research/poc-trait-dispatch-findings.md` findings 3/7/8 - `docs/research/iroh-blobs-eval.md` (gc.rs / delete_set.rs reads) - rudolfs (the physical-namespace anti-pattern inverted); gix-odb (alternates/pool prior art) - ADR-002 (one address space), ADR-003 (list contract, pins), ADR-004 (manifest layers above), ADR-006 (verification limits inform recover semantics) - [gc-and-namespaces.md](../gc-and-namespaces.md); [store-api.md](../store-api.md)