alkblobs/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md

ADR-005: Pooled CAS, liveness seams, and mark-and-sweep GC

Status

Accepted (the sweep-safety protocol here is the single-node posture; ADR-008 extends the mechanism to fleet topologies — DB-backed pins, advisory-locked single sweeper, SQL-level delete arbitration under shared-pool engines (REQ-2) — while keeping this ADR's invariants and the in-process protocol unchanged everywhere else. The "one pool per node" phrasing below reads as "per deployment" for fleets; the vocabulary is pinned in requirements.md)

Context

OQ-BL-05 settled the pooling/GC shape in Phase 0 (one pooled CAS; mark-and-sweep from registered roots; lean traversal ownership) with residual mechanics named: namespace registry shape, GC scheduling, and the sweep-vs-put race. All were resolvable on evidence in hand — the prior art was read and verified (iroh gc.rs, delete_set.rs), the mechanism was POC-validated (POC #1 findings 3/7/8: exact sweep counts, abort semantics, delete-then-recover as byte-identical re-put, live- shared callback seam), and the residuals turned on which of the validated shapes to adopt — not on unknown facts (the Schrödinger's-code rule of the deferral policy applies: none of these may wait on paused consumers).

Also loaded: the "physically flat, logically namespaced" inversion of rudolfs (its s3://{org}/{project}/{sha256} physical namespaces pay cross-tenant dedup — the whole reason for pooling — for tenant isolation), and the packfile tension (resolved in Phase 0's shape: loose-equivalent kv entries; packs, if ever stored, are large blobs served by range reads).

Decision

One pooled CAS; namespaces are reference sets above it

• One pool per node; repos/workspaces/all consumers are sets of hash references over it. Cross-repo dedup by construction (the property both demanding consumers need; git alternates pools as the awkward prior).
• Physically flat — backends see hash → bytes, never namespaces (ties to ADR-003's no-per-namespace-config).
• Logically namespaced — a namespace is a consumer-held reference set; the store-side counterpart is liveness registration only.
• The store persists no root table. Namespaced root tags vs separate reference-table persistence (a Phase 0 residual) resolves as: the store holds no durable roots; consumers keep durable references (git refs, alkfs path trees) and hand liveness over per sweep via the live-shared seam. In-memory namespace tables remain available for ephemeral consumers. This also dissolves the hidden wrinkle of a store-internal root table under fs-only deployments (where would its durable bytes live?). Ephemeral consumers are covered by pins; a deployment that needs store-durable roots registers them as consumer-owned durable state and sweeps through the seam.
• ACL boundary: the namespace is the alkcall AccessControl resource for network ops (resource_id_path; ADR-001/ops-surface.md).

Marks: three liveness sources

1. Registered liveness sources — live-shared consumers (the seam named register_liveness_source; the POC's finding 7 copy-semantics failure is why the verb is pinned — "install" implies snapshot, and snapshotting breaks consumer registration that happens after).
2. Put-path pins (RAII) — a Pin guard holds a refcount per key until dropped or converted into a consumer reference; batch scope available for multi-put writes (manifest writes are exactly this; POC finding 3's named Phase 1 requirement, satisfied here). The conversion ordering is pinned: a pin's drop and its replacement reference's registration into a liveness source must be one ordered step under the same arbitration lock the sweep consults — a consumer may not release a pin before its registered source observes the replacement (the local analog of the ops-layer pin token contract).
3. Protect callback — consulted pre-sweep; may add known-live keys or abort the run (typed GcAborted, nothing deleted). iroh's ProtectOutcome::Abort conclusion adopted: a flaky protection source skips the sweep rather than risk deletion.

Sweep: mark → delete window → commit

• Enumerate the pool via backends' complete list() (ADR-003's contract; its reason to exist).

• Compute the live set from all three sources; batch-delete the dead (~100/batch; iroh's proven shape).

• Delete windows close the sweep-vs-put race (POC finding 3's named Phase 1 requirement — the pin-map lock alone leaves a real window between "list" and "delete"):

  1. mark: enumerate the pool; compute the live set from all three liveness sources; stage the candidate-dead set;
  2. delete window opens: for each candidate, at deletion time (not once upfront), arbitrate under the pin/liveness lock: if the key is pinned or re-observed live, cancel it from the window; otherwise proceed to delete;
  3. commit: remaining candidates deleted in batches; window closed.

  The invariant holds because both halves of the race are now ordered under the same lock:

  • Put commits pin before publish. A put's Pin is acquired before the entry becomes visible in its backend; an entry cannot appear in a later sweep's list() without already carrying its pin (or its liveness registration, whichever the consumer converted the pin into) — publish and protection are one step, and a put for an already-pool-existing entry (dedup) acquires no new pool entry — the returned Pin refcounts the existing entry under the same arbitration lock (a dedup call's returned Pin protects exactly as a fresh put's does).
  • Deletes arbitrate under the pin lock at delete time. A deletion of key k cannot proceed while k carries a pin; since any visible entry is pinned (invariant above), any deletion of a visible entry happens only with no pin held — and a concurrent re-put of k is either (a) blocked on the arbitration lock until the deletion commits, after which the re-put lands as a fresh pinned put (delete-then-recover semantics, validated), or (b) ordered after, in which case its pin already protects it.

  Net property, test-asserted: a visible pool entry is never deleted while liveness (pin or registered source) protects it, and an in-flight put is never deleted by a sweep started before it committed. The arbitration can be cheap: the pin check is one map lookup per candidate key; the delete window exists so the reconciliation happens precisely at the delete boundary rather than wholesale.

  The iroh DeleteSet/ProtectHandle transactions and per-hash serialized-actor pattern are the re-borrowed prior art this shape generalizes (their serialized actor is exactly "arbitrate at delete time"; our window batches the arbitration).

• has/get during a window see no torn state (immutable entries; existence flips atomically per key).

• Delete-then-recover is a re-put (validated: byte-identical under the same key); no tombstone layer exists.

• Direct delete(key): refuses pinned keys with a typed error (same arbitration); permitted for embedder correction flows on unpinned entries. Outside-of-sweep deletion is unusual by posture — most deletion should flow through sweeps.

Traversal and scheduling

• Lean (a) confirmed: consumers compute liveness beyond "these roots exist" and hand hash sets over; the store stays structure-blind (what Phase 0 called "lean (a)": consumer-side traversal, vs option (b) store-native traversal of consumer manifests). Store-native traversal re-opens only on a measured consumer cost (an external fact, not a parked hedge).
• Explicit sweep(); no ambient timers. The embedder owns cadence (idle sweeps, interval sweeps, consumer-driven prunes — all one API).
• Accounting: pool-level totals via list+stat; per-namespace accounting is computed by consumers from their reference sets.

Packfiles (the Phase 0 pack tension, carried forward)

Git objects enter the pool as loose-equivalent kv entries (small tier — the common case). If packfile serving is ever wanted by alkgit, packs are stored as large blobs and served through stat/read_range (git's .idx does per-object offset lookup; local range serving is sound — ADR-006). This avoids gitoxide's pack-ID-stability machinery entirely (there are no IDs to rebind; the address is the content). This restate- as-decision carries no new open question; whether alkgit wants it is OQ-07's.

Consequences

Positive

• The dedup property the consumers exist for is structural, not configured; pooling semantics can't be "off".
• Sweep safety is testable to exact counts (POC-proven) at the architecture level: the invariant (delete-window re-observation) is the test gate, independent of mechanism choice.
• No GC coupling in the store: a consumer without registered liveness sources cannot have its content deleted (sweep aborts) — safe-by- default.

Negative

• GC requires complete, well-formed list() from every backend (enforced upstream, ADR-003) and delete-window machinery in the sweep path — the most intricate state machine in the crate.
• Explicit sweep means an embedder that never schedules one accumulates garbage (their choice; documented posture, not a defect).
• Pin/liveness misuse by consumers is runtime-observable (a sweep may delete unreferenced content if a consumer deregisters early); the seam's correctness contract is on consumers — documented in store-api.md's invariants.

Neutral

• Cross-node GC coordination (p2p replicators) lives in the replicator policy layer, above; the seam is all it needs.

References

• docs/research/phase-0.md OQ-BL-05 (settled decisions + residuals, all promoted here)
• docs/research/poc-trait-dispatch-findings.md findings 3/7/8
• docs/research/iroh-blobs-eval.md (gc.rs / delete_set.rs reads)
• rudolfs (the physical-namespace anti-pattern inverted); gix-odb (alternates/pool prior art)
• ADR-002 (one address space), ADR-003 (list contract, pins), ADR-004 (manifest layers above), ADR-006 (verification limits inform recover semantics)
• gc-and-namespaces.md; store-api.md