alkblobs/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md

# ADR-005: Pooled CAS, liveness seams, and mark-and-sweep GC

## Status

Accepted (the sweep-safety protocol here is the single-node posture;
ADR-008 extends the mechanism to fleet topologies — DB-backed pins,
advisory-locked single sweeper, SQL-level delete arbitration under
shared-pool engines (REQ-2) — while keeping this ADR's invariants
and the in-process protocol unchanged everywhere else. The "one pool
per node" phrasing below reads as "per deployment" for fleets; the
vocabulary is pinned in requirements.md)

## Context

OQ-BL-05 settled the pooling/GC *shape* in Phase 0 (one pooled CAS;
mark-and-sweep from registered roots; lean traversal ownership) with
residual mechanics named: namespace registry shape, GC scheduling,
and the sweep-vs-put race. All were resolvable on evidence in hand —
the prior art was read and verified (iroh `gc.rs`, `delete_set.rs`), the
mechanism was POC-validated (POC #1 findings 3/7/8: exact sweep counts,
abort semantics, delete-then-recover as byte-identical re-put, live-
shared callback seam), and the residuals turned on which of the
validated shapes to adopt — not on unknown facts (the
Schrödinger's-code rule of the deferral policy applies: none of these
may wait on paused consumers).

Also loaded: the "physically flat, logically namespaced" inversion of
rudolfs (its `s3://{org}/{project}/{sha256}` physical namespaces pay
cross-tenant dedup — the whole reason for pooling — for tenant
isolation), and the packfile tension (resolved in Phase 0's shape:
loose-equivalent kv entries; packs, if ever stored, are large blobs
served by range reads).

## Decision

### One pooled CAS; namespaces are reference sets above it

- One pool per node; repos/workspaces/all consumers are sets of hash
  references over it. Cross-repo dedup by construction (the property
  both demanding consumers need; git `alternates` pools as the awkward
  prior).
- **Physically flat** — backends see `hash → bytes`, never namespaces
  (ties to ADR-003's no-per-namespace-config).
- **Logically namespaced** — a namespace is a consumer-held reference
  set; the store-side counterpart is liveness registration only.
- **The store persists no root table.** Namespaced root tags vs separate
  reference-table persistence (a Phase 0 residual) resolves as: *the
  store holds no durable roots*; consumers keep durable references
  (git refs, alkfs path trees) and hand liveness over per sweep via the
  live-shared seam. In-memory namespace tables remain available for
  ephemeral consumers. This also dissolves the hidden wrinkle of a
  store-internal root table under fs-only deployments (where would its
  durable bytes live?). Ephemeral consumers are covered by pins; a
  deployment that needs store-durable roots registers them as
  consumer-owned durable state and sweeps through the seam.
- **ACL boundary:** the namespace is the alkcall `AccessControl`
  resource for network ops (resource_id_path; ADR-001/ops-surface.md).

### Marks: three liveness sources

1. **Registered liveness sources** — live-shared consumers (the seam
   named `register_liveness_source`; the POC's finding 7 copy-semantics
   failure is why the verb is pinned — "install" implies snapshot, and
   snapshotting breaks consumer registration that happens after).
2. **Put-path pins (RAII)** — a `Pin` guard holds a refcount per key
   until dropped or converted into a consumer reference; batch scope
   available for multi-put writes (manifest writes are exactly this;
   POC finding 3's named Phase 1 requirement, satisfied here). The
   conversion ordering is pinned: a pin's drop and its replacement
   reference's registration into a liveness source must be one
   ordered step under the same arbitration lock the sweep consults —
   a consumer may not release a pin before its registered source
   observes the replacement (the local analog of the ops-layer pin
   token contract).
3. **Protect callback** — consulted pre-sweep; may add known-live keys
   or **abort the run** (typed `GcAborted`, nothing deleted). iroh's
   `ProtectOutcome::Abort` conclusion adopted: a flaky protection
   source skips the sweep rather than risk deletion.

### Sweep: mark → delete window → commit

- Enumerate the pool via backends' complete `list()` (ADR-003's
  contract; its reason to exist).
- Compute the live set from all three sources; batch-delete the dead
  (~100/batch; iroh's proven shape).
- **Delete windows** close the sweep-vs-put race (POC finding 3's named
  Phase 1 requirement — the pin-map lock alone leaves a real window
  between "list" and "delete"):

  1. **mark:** enumerate the pool; compute the live set from all three
     liveness sources; stage the candidate-dead set;
  2. **delete window opens:** for each candidate, at *deletion time*
     (not once upfront), arbitrate under the pin/liveness lock: if the
     key is pinned or re-observed live, cancel it from the window;
     otherwise proceed to delete;
  3. **commit:** remaining candidates deleted in batches; window
     closed.

  The invariant holds because both halves of the race are now
  ordered under the same lock:

  - **Put commits pin before publish.** A put's `Pin` is acquired
    *before* the entry becomes visible in its backend; an entry cannot
    appear in a later sweep's `list()` without already carrying its
    pin (or its liveness registration, whichever the consumer
    converted the pin into) — publish and protection are one step,
    and a put for an already-pool-existing entry (dedup) acquires no
    *new pool entry* — the returned `Pin` refcounts the existing entry
    under the same arbitration lock (a dedup call's returned Pin
    protects exactly as a fresh put's does).
  - **Deletes arbitrate under the pin lock at delete time.** A
    deletion of key *k* cannot proceed while *k* carries a pin; since
    any visible entry is pinned (invariant above), any deletion of a
    visible entry happens only with no pin held — and a *concurrent*
    re-put of *k* is either (a) blocked on the arbitration lock until
    the deletion commits, after which the re-put lands as a fresh
    pinned put (delete-then-recover semantics, validated), or (b)
    ordered after, in which case its pin already protects it.

  Net property, test-asserted: **a visible pool entry is never
  deleted while liveness (pin or registered source) protects it, and
  an in-flight put is never deleted by a sweep started before it
  committed.** The arbitration can be cheap: the pin check is one
  map lookup per candidate key; the delete window exists so the
  reconciliation happens precisely at the delete boundary rather
  than wholesale.

  The iroh DeleteSet/ProtectHandle transactions and per-hash
  serialized-actor pattern are the re-borrowed prior art this shape
  generalizes (their serialized actor is exactly "arbitrate at
  delete time"; our window batches the arbitration).
- `has`/`get` during a window see no torn state (immutable entries;
  existence flips atomically per key).
- **Delete-then-recover is a re-put** (validated: byte-identical under
  the same key); no tombstone layer exists.
- **Direct `delete(key)`:** refuses pinned keys with a typed error
  (same arbitration); permitted for embedder correction flows on
  unpinned entries. Outside-of-sweep deletion is unusual by posture —
  most deletion should flow through sweeps.

### Traversal and scheduling

- **Lean (a) confirmed:** consumers compute liveness beyond "these
  roots exist" and hand hash sets over; the store stays structure-blind
  (what Phase 0 called "lean (a)": consumer-side traversal, vs option
  (b) store-native traversal of consumer manifests). Store-native
  traversal re-opens only on a *measured* consumer cost (an external
  fact, not a parked hedge).
- **Explicit `sweep()`; no ambient timers.** The embedder owns cadence
  (idle sweeps, interval sweeps, consumer-driven prunes — all one API).
- **Accounting:** pool-level totals via `list`+`stat`; per-namespace
  accounting is computed by consumers from their reference sets.

### Packfiles (the Phase 0 pack tension, carried forward)

Git objects enter the pool as loose-equivalent kv entries (small tier
— the common case). If packfile serving is ever wanted by alkgit, packs
are stored as large blobs and served through `stat`/`read_range` (git's
`.idx` does per-object offset lookup; local range serving is sound —
ADR-006). This avoids gitoxide's pack-ID-stability machinery entirely
(there are no IDs to rebind; the address is the content). This restate-
as-decision carries no new open question; whether alkgit wants it is
OQ-07's.

## Consequences

**Positive**

- The dedup property the consumers exist for is structural, not
  configured; pooling semantics can't be "off".
- Sweep safety is testable to exact counts (POC-proven) at the
  architecture level: the invariant (delete-window re-observation) is
  the test gate, independent of mechanism choice.
- No GC coupling in the store: a consumer without registered liveness
  sources cannot have its content deleted (sweep aborts) — safe-by-
  default.

**Negative**

- GC requires complete, well-formed `list()` from every backend
  (enforced upstream, ADR-003) and delete-window machinery in the
  sweep path — the most intricate state machine in the crate.
- Explicit sweep means an embedder that never schedules one accumulates
  garbage (their choice; documented posture, not a defect).
- Pin/liveness misuse by consumers is runtime-observable (a sweep may
  delete unreferenced content if a consumer deregisters early); the
  seam's correctness contract is on consumers — documented in
  store-api.md's invariants.

**Neutral**

- Cross-node GC coordination (p2p replicators) lives in the replicator
  policy layer, above; the seam is all it needs.

## References

- `docs/research/phase-0.md` OQ-BL-05 (settled decisions + residuals,
  all promoted here)
- `docs/research/poc-trait-dispatch-findings.md` findings 3/7/8
- `docs/research/iroh-blobs-eval.md` (gc.rs / delete_set.rs reads)
- rudolfs (the physical-namespace anti-pattern inverted);
  gix-odb (alternates/pool prior art)
- ADR-002 (one address space), ADR-003 (list contract, pins), ADR-004
  (manifest layers above), ADR-006 (verification limits inform recover
  semantics)
- [gc-and-namespaces.md](../gc-and-namespaces.md);
  [store-api.md](../store-api.md)