alkblobs/tasks/gc-protocol.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
gc-protocol	In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal	completed	store-construction	broad	high	phase	implementation	wave-2 gc facade

Description

Implement the in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1; non-SQL window staging per ADR-012 §5) on the Store type — the architecture's most intricate state machine, in its single-node form. The fleet extension (DB-backed pins, sweep lease, SQL arbitration) is wave 6; this task is the protocol the fleet form extends without changing its invariants.

• Pin registry (ADR-005 §Marks 2): per-key refcounts under the arbitration lock the sweep consults. Pin guards minted by the put path (store-put-paths wires the call sites; the registry and minting land here); a dedup re-put refcounts the existing entry — a dedup put's guard protects exactly as a fresh put's does. Pin drop decrements. Named design point: the guard's back-reference shape (how the public Pin/Batch types reach the registry — the wave-1 types carry only the key today) and the lock granularity (parking_lot short-held per AGENTS convention 3, poisoning via unwrap_or_else(|e| e.into_inner())).
• Batch machinery: a batch's pins drop together on batch commit or batch drop, in one ordered step under the arbitration lock; no early single-pin release. The conversion ordering rule (ADR-005, pinned in facade.rs docs): a pin's drop and its replacement's registration into a liveness source are one ordered step under the same lock — the registry needs the API shape that makes the ordered step expressible.
• Registered liveness sources (register_liveness_source, the live-shared seam — ADR-005 §Decision): named design point — the source trait/callback shape. Live-shared means consulted at mark time and at delete-time arbitration, never snapshotted at registration (POC #1 finding 7's install-implies-copy failure is the pinning reason).
• Protect callback: consulted pre-sweep; may add externally-known hashes or abort the run — abort is report data (GcAbortCause::ProtectFailed), never an error (ADR-012 §1).
• sweep() → Result<SweepReport, Error> — the delete-window protocol (ADR-005 §Sweep):
  1. mark: enumerate the whole pool (both tiers' complete list() unioned); compute the live set from all three sources (registered sources, pins, protect callback); stage the candidate-dead set.
  2. delete window: per candidate, arbitrate under the pin/ liveness lock at delete time (not once upfront) — pinned or re-observed-live cancels the candidate; otherwise proceed.
  3. commit: remaining candidates deleted in batches (~100/batch, iroh's proven shape); window closed.
  • Ok(SweepReport) always — sweeps are never errors; every abort is data in aborted: Option<GcAbortCause> (exactly three causes; NoLivenessSources is the safe default when zero sources are registered — nothing deleted). Exact counts in the report fields.
  • Non-SQL window staging (ADR-012 §5): local/mem stage candidates in-process and execute the window protocol inline in the same sweep — there is no durable window table on non-SQL engines, and the "executor is the next sweep" clause (ADR-012 §6.7) does not apply to them.
  • The test-asserted property (ADR-005): a visible pool entry is never deleted while liveness protects it, and an in-flight put is never deleted by a sweep started before it committed — concurrency tests (sweep racing a pinned put) assert it.
• Direct delete(key) (wired through the fall-through dispatch — arbitration-then-delete where the entry lives): refuses any protected key — pinned, registered-source-live, or protect-callback-named (the full protection set, ADR-012 §6.4) — with the typed Error::GcRefuse, nothing deleted; unprotected keys delete; absent keys carry Err(Missing). The refusal is decided at the facade (the altitude discipline: GcRefuse never fires below the facade).
• has/get during a window observe no torn state — existence flips atomically per key (immutable entries); document the posture the engines' CAS semantics already give.

Acceptance Criteria

• Pin registry with per-key refcounts under the arbitration lock; dedup re-puts refcount; Pin/Batch drops release through it (batch drops in one ordered step)
• register_liveness_source live-shared (consulted at mark and at delete-time arbitration, never snapshotted); source shape decided and recorded
• Protect callback with abort-as-data (ProtectFailed); NoLivenessSources safe default; sweep() returns Ok(SweepReport) on every abort — never an abort error
• Delete-window protocol: mark → arbitrate-at-delete-time → batched commit (~100/batch), with exact-count SweepReport fields (candidates_staged/deleted/cancelled_by_arbitration)
• Direct delete refuses the full protection set with GcRefuse and deletes unprotected keys through the fall-through dispatch
• Concurrency tests: sweep vs in-flight pinned put (the pin-before-publish property from the sweep side), exact-count outcomes, abort shapes
• cargo test green; cargo test --features mem and cargo test --all-features green; clippy/fmt gates green

References

• docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (the protocol + invariants)
• docs/architecture/gc-and-namespaces.md (§Mark-and-sweep GC; §Delete windows)
• docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (non-SQL staging), §6.4 (full protection set)
• docs/architecture/store-api.md (§Lifecycle — sweep shape; §Invariants 4/5)
• src/facade.rs (the pinned Pin/Batch/SweepReport lifecycle contracts this wires)
• docs/research/poc-trait-dispatch-findings.md (findings 3/7/8)

Notes

Named design points decided here (recorded in src/gc.rs's module doc and on the traits in src/facade.rs):

• Guard back-reference shape: Pin holds Arc<RegistryCore> (the GC home's shared core) alongside its key — so a drop reaches the per-key refcounts wherever the guard travels, and the registry outlives the Store instance when guards still reference it. No weak-pointer tag-drop plumbing (iroh's TempTag conclusion without their mechanism). Batch holds its members' keys; contribution defuses the member guard (armed flag), and the batch's release decrements every member refcount inside one arbitration-lock section, keys in total order — no early single-pin release.
• Lock granularity: one parking_lot::Mutex<RegistryState> — the pin refcounts, the in-flight delete markers, and the registered source lists under one lock, so a delete-time decision (pins ∨ markers ∨ sources) and its marker insert are one atomic section against pin mints. parking_lot does not poison (no into_inner posture to apply). Consumer consults (contains/protect) run under that lock at delete-time arbitration and outside it at mark time; lock order is one-directional (arbitration lock → consumer locks; the ordered-step conversion is the only nesting, same direction).
• Liveness source shape: a sync (non-async) trait, live_keys() (mark-time enumeration, consulted outside the lock)
  • contains() (delete-time probe, consulted under the lock; default derives it linearly from live_keys — indexed sources override). Sync because pin drops are sync RAII and arbitration consults run under the short-held lock. Consumers register Arc<dyn LivenessSource> shared handles — the POC finding-7 shape; the store consults the current contents every time and never snapshots.
• Protect callback shape: one sync verb, protect() → Option<Vec<Key>>; None = failed to answer → sweep aborts with ProtectFailed report data, and a direct delete refuses conservatively (GcRefuse, nothing deleted). Callbacks compose (each registered set unions into the protection).
• The sweep-vs-put race closes with in-flight delete markers, not by holding the arbitration lock across engine I/O: the arbitrate section marks a Proceed-ing key deleting; the engine delete runs outside the lock; marker release notifies blocked pin mints. A re-put's mint landing while the marker is held blocks (async, tokio::sync::Notify with the enable-before-check recipe) until the deletion commits, then lands fresh — ADR-005's case (a) delete-then-recover verbatim; a mint landing before arbitration is case (b), protection composed at the arbitration itself. Holding the lock across the delete was rejected: a parking_lot guard is not Send, the sweep future must be, and AGENTS convention 3 forbids blocking I/O under internal locks.
• The sweep is single-flight in-process: an atomic test-and-set flag is the single-node analog of ADR-013 §2's named exclusive sweep lease (ADR-012 §5's "single-flighter is the in-process lock" posture); a contended second sweeper aborts with SweeperLock report data — nothing deleted, the next run takes the lease.
• Vanished candidates count as arbitration cancels: a key deleted between mark and its window turn (e.g. a concurrent direct delete) settles into cancelled_by_arbitration, keeping the completed sweep's accounting exact: candidates_staged == deleted + cancelled_by_arbitration.
• Registration vocabulary: a public opaque Registration token per registration; unregister_liveness_source / unregister_protect_callback consume it (the correction-flow prerequisite of ADR-012 §6.4). The protect-callback registration verbs are this layer's addition — the direct-delete full-protection set requires consultable protect state on the store, which store-api §Lifecycle implies but does not spell out.
• The ordered-step conversion primitive: GcHome::convert(pin, register) (crate-internal until a consumer-facing surface exists): pin release + registration closure inside one arbitration-lock section; tested sequentially and raced (64-round direct-delete race: no schedule leaves the key unprotected — every delete refuses).

Scope notes: the pin minting API (GcHome::acquire, async, waits on in-flight deletes) is wired; the put path's call sites remain for store-put-paths, which now only needs store.gc().acquire(key) / new_batch() / convert at the publish points. Batch commit semantics (the seam's terminal step) also rides that task; Batch::release_members is its reuse point. Sweep/delete/registration live on Store (src/store.rs), with one new KvEngine/LargeEngine dispatch method (delete). A dev-dependency addition (tokio's rt-multi-thread, test-only) supports the single-flight concurrency test's multi-thread runtime flavor; the base crate's dependency set is unchanged.

Summary

The in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1; non-SQL inline window staging per ADR-012 §5) is wired on Store:

• src/gc.rs — GcHome + RegistryCore: one arbitration lock over pin refcounts, delete markers, and registration lists; async pin minting (acquire) that blocks on in-flight deletes (the delete-window mechanism); the ordered-step convert primitive; batch one-step release; source/protect registration bookkeeping; the single-node sweep lease.
• src/facade.rs — Pin/Batch now carry the registry back-reference and own their drop-release behavior (the wave-1 contracts are now wired behavior); new public vocabulary LivenessSource, ProtectCallback, Registration (re-exported from lib.rs).
• src/store.rs — sweep() (single-flight → zero-source safe default → protect consult → mark over both tiers' list() union → arbitrate-at-delete-time → ~100/batch commit → exact-count report or abort-as-data), direct delete() (existence probe → full protection set arbitration → GcRefuse/Missing/delete through the fall-through), and the four registration/unregistration verbs.
• 16 new tests + the facade pin/batch shape test reworked on the wired machinery: zero-source abort, empty-pool zero report, exact counts across both tiers, dedup refcounts, batch one-step drops, protect add + abort-as-data, live-shared (non-snapshotted) sources, token unregistration, the full direct-delete refusal set (+ Missing/KeyInvalid), the ordered-step conversion sequential and raced (64 rounds, always refused), the sweep-vs-pinned-put race (64 rounds: entry survives in every schedule, byte-identical re-put, exact counts, pin keeps protecting after the race), the publish-after-mark arm, no-torn has/get reads under a window, single-flight contention aborting as data, and malformed-list sweeps as operational errors.

Verification: cargo test (default) 46 passed; cargo test --features mem and --all-features 89 passed each; cargo clippy --all-targets -- -D warnings and --all-features clean; cargo fmt --check clean; cargo doc --no-deps at the pre-existing baseline (24 warnings carried from earlier waves, none new).