--- id: gc-protocol name: In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal status: completed depends_on: [store-construction] scope: broad risk: high impact: phase level: implementation tags: [wave-2, gc, facade] --- ## Description Implement the in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1; non-SQL window staging per ADR-012 §5) on the `Store` type — the architecture's most intricate state machine, in its single-node form. The fleet extension (DB-backed pins, sweep lease, SQL arbitration) is wave 6; this task is the protocol the fleet form extends without changing its invariants. - **Pin registry** (ADR-005 §Marks 2): per-key refcounts under the arbitration lock the sweep consults. `Pin` guards minted by the put path (`store-put-paths` wires the call sites; the registry and minting land here); a dedup re-put refcounts the existing entry — a dedup put's guard protects exactly as a fresh put's does. `Pin` drop decrements. **Named design point**: the guard's back-reference shape (how the public `Pin`/`Batch` types reach the registry — the wave-1 types carry only the key today) and the lock granularity (`parking_lot` short-held per AGENTS convention 3, poisoning via `unwrap_or_else(|e| e.into_inner())`). - **`Batch` machinery**: a batch's pins drop together on batch commit or batch drop, in one ordered step under the arbitration lock; no early single-pin release. The **conversion ordering rule** (ADR-005, pinned in facade.rs docs): a pin's drop and its replacement's registration into a liveness source are one ordered step under the same lock — the registry needs the API shape that makes the ordered step expressible. - **Registered liveness sources** (`register_liveness_source`, the live-shared seam — ADR-005 §Decision): **named design point** — the source trait/callback shape. Live-shared means consulted at mark time and at delete-time arbitration, never snapshotted at registration (POC #1 finding 7's install-implies-copy failure is the pinning reason). - **Protect callback**: consulted pre-sweep; may add externally-known hashes or abort the run — abort is report data (`GcAbortCause::ProtectFailed`), never an error (ADR-012 §1). - **`sweep() → Result`** — the delete-window protocol (ADR-005 §Sweep): 1. **mark**: enumerate the whole pool (both tiers' complete `list()` unioned); compute the live set from all three sources (registered sources, pins, protect callback); stage the candidate-dead set. 2. **delete window**: per candidate, **arbitrate under the pin/ liveness lock at delete time** (not once upfront) — pinned or re-observed-live cancels the candidate; otherwise proceed. 3. **commit**: remaining candidates deleted in batches (~100/batch, iroh's proven shape); window closed. - **`Ok(SweepReport)` always** — sweeps are never errors; every abort is data in `aborted: Option` (exactly three causes; `NoLivenessSources` is the safe default when zero sources are registered — nothing deleted). Exact counts in the report fields. - **Non-SQL window staging** (ADR-012 §5): `local`/`mem` stage candidates in-process and execute the window protocol **inline in the same sweep** — there is no durable window table on non-SQL engines, and the "executor is the next sweep" clause (ADR-012 §6.7) does not apply to them. - The test-asserted property (ADR-005): a visible pool entry is never deleted while liveness protects it, and an in-flight put is never deleted by a sweep started before it committed — concurrency tests (sweep racing a pinned put) assert it. - **Direct `delete(key)`** (wired through the fall-through dispatch — arbitration-then-delete where the entry lives): refuses **any protected key** — pinned, registered-source-live, or protect-callback-named (the full protection set, ADR-012 §6.4) — with the typed `Error::GcRefuse`, nothing deleted; unprotected keys delete; absent keys carry `Err(Missing)`. The refusal is decided at the facade (the altitude discipline: `GcRefuse` never fires below the facade). - **`has`/`get` during a window observe no torn state** — existence flips atomically per key (immutable entries); document the posture the engines' CAS semantics already give. ## Acceptance Criteria - [x] Pin registry with per-key refcounts under the arbitration lock; dedup re-puts refcount; `Pin`/`Batch` drops release through it (batch drops in one ordered step) - [x] `register_liveness_source` live-shared (consulted at mark and at delete-time arbitration, never snapshotted); source shape decided and recorded - [x] Protect callback with abort-as-data (`ProtectFailed`); `NoLivenessSources` safe default; `sweep()` returns `Ok(SweepReport)` on every abort — never an abort error - [x] Delete-window protocol: mark → arbitrate-at-delete-time → batched commit (~100/batch), with exact-count `SweepReport` fields (`candidates_staged`/`deleted`/`cancelled_by_arbitration`) - [x] Direct `delete` refuses the full protection set with `GcRefuse` and deletes unprotected keys through the fall-through dispatch - [x] Concurrency tests: sweep vs in-flight pinned put (the pin-before-publish property from the sweep side), exact-count outcomes, abort shapes - [x] `cargo test` green; `cargo test --features mem` and `cargo test --all-features` green; clippy/fmt gates green ## References - docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (the protocol + invariants) - docs/architecture/gc-and-namespaces.md (§Mark-and-sweep GC; §Delete windows) - docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (non-SQL staging), §6.4 (full protection set) - docs/architecture/store-api.md (§Lifecycle — sweep shape; §Invariants 4/5) - src/facade.rs (the pinned Pin/Batch/SweepReport lifecycle contracts this wires) - docs/research/poc-trait-dispatch-findings.md (findings 3/7/8) ## Notes **Named design points decided here** (recorded in `src/gc.rs`'s module doc and on the traits in `src/facade.rs`): - **Guard back-reference shape**: `Pin` holds `Arc` (the GC home's shared core) alongside its key — so a drop reaches the per-key refcounts wherever the guard travels, and the registry outlives the `Store` instance when guards still reference it. No weak-pointer tag-drop plumbing (iroh's TempTag conclusion without their mechanism). `Batch` holds its members' keys; contribution defuses the member guard (`armed` flag), and the batch's release decrements every member refcount inside **one** arbitration-lock section, keys in total order — no early single-pin release. - **Lock granularity**: one `parking_lot::Mutex` — the pin refcounts, the in-flight delete markers, and the registered source lists under one lock, so a delete-time decision (pins ∨ markers ∨ sources) and its marker insert are one atomic section against pin mints. parking_lot does not poison (no `into_inner` posture to apply). Consumer consults (`contains`/`protect`) run under that lock at delete-time arbitration and outside it at mark time; lock order is one-directional (arbitration lock → consumer locks; the ordered-step conversion is the only nesting, same direction). - **Liveness source shape**: a sync (non-async) trait, `live_keys()` (mark-time enumeration, consulted outside the lock) + `contains()` (delete-time probe, consulted under the lock; default derives it linearly from `live_keys` — indexed sources override). Sync because pin drops are sync RAII and arbitration consults run under the short-held lock. Consumers register `Arc` shared handles — the POC finding-7 shape; the store consults the current contents every time and never snapshots. - **Protect callback shape**: one sync verb, `protect() → Option>`; `None` = failed to answer → sweep aborts with `ProtectFailed` report data, and a direct delete refuses conservatively (`GcRefuse`, nothing deleted). Callbacks compose (each registered set unions into the protection). - **The sweep-vs-put race closes with in-flight delete markers**, not by holding the arbitration lock across engine I/O: the arbitrate section marks a Proceed-ing key `deleting`; the engine delete runs outside the lock; marker release notifies blocked pin mints. A re-put's mint landing while the marker is held blocks (async, `tokio::sync::Notify` with the enable-before-check recipe) until the deletion commits, then lands fresh — ADR-005's case (a) delete-then-recover verbatim; a mint landing before arbitration is case (b), protection composed at the arbitration itself. Holding the lock across the delete was rejected: a `parking_lot` guard is not `Send`, the sweep future must be, and AGENTS convention 3 forbids blocking I/O under internal locks. - **The sweep is single-flight in-process**: an atomic test-and-set flag is the single-node analog of ADR-013 §2's named exclusive sweep lease (ADR-012 §5's "single-flighter is the in-process lock" posture); a contended second sweeper aborts with `SweeperLock` report data — nothing deleted, the next run takes the lease. - **Vanished candidates count as arbitration cancels**: a key deleted between mark and its window turn (e.g. a concurrent direct delete) settles into `cancelled_by_arbitration`, keeping the completed sweep's accounting exact: `candidates_staged == deleted + cancelled_by_arbitration`. - **Registration vocabulary**: a public opaque `Registration` token per registration; `unregister_liveness_source` / `unregister_protect_callback` consume it (the correction-flow prerequisite of ADR-012 §6.4). The protect-callback registration verbs are this layer's addition — the direct-delete full-protection set requires consultable protect state on the store, which store-api §Lifecycle implies but does not spell out. - **The ordered-step conversion primitive**: `GcHome::convert(pin, register)` (crate-internal until a consumer-facing surface exists): pin release + registration closure inside one arbitration-lock section; tested sequentially and raced (64-round direct-delete race: no schedule leaves the key unprotected — every delete refuses). **Scope notes**: the pin minting API (`GcHome::acquire`, async, waits on in-flight deletes) is wired; the put path's call sites remain for `store-put-paths`, which now only needs `store.gc().acquire(key)` / `new_batch()` / `convert` at the publish points. Batch commit semantics (the seam's terminal step) also rides that task; `Batch::release_members` is its reuse point. Sweep/`delete`/registration live on `Store` (`src/store.rs`), with one new `KvEngine`/`LargeEngine` dispatch method (`delete`). A dev-dependency addition (`tokio`'s `rt-multi-thread`, test-only) supports the single-flight concurrency test's multi-thread runtime flavor; the base crate's dependency set is unchanged. ## Summary The in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1; non-SQL inline window staging per ADR-012 §5) is wired on `Store`: - `src/gc.rs` — `GcHome` + `RegistryCore`: one arbitration lock over pin refcounts, delete markers, and registration lists; async pin minting (`acquire`) that blocks on in-flight deletes (the delete-window mechanism); the ordered-step `convert` primitive; batch one-step release; source/protect registration bookkeeping; the single-node sweep lease. - `src/facade.rs` — `Pin`/`Batch` now carry the registry back-reference and own their drop-release behavior (the wave-1 contracts are now wired behavior); new public vocabulary `LivenessSource`, `ProtectCallback`, `Registration` (re-exported from `lib.rs`). - `src/store.rs` — `sweep()` (single-flight → zero-source safe default → protect consult → mark over both tiers' `list()` union → arbitrate-at-delete-time → ~100/batch commit → exact-count report or abort-as-data), direct `delete()` (existence probe → full protection set arbitration → `GcRefuse`/`Missing`/delete through the fall-through), and the four registration/unregistration verbs. - 16 new tests + the facade pin/batch shape test reworked on the wired machinery: zero-source abort, empty-pool zero report, exact counts across both tiers, dedup refcounts, batch one-step drops, protect add + abort-as-data, live-shared (non-snapshotted) sources, token unregistration, the full direct-delete refusal set (+ `Missing`/`KeyInvalid`), the ordered-step conversion sequential and raced (64 rounds, always refused), the sweep-vs-pinned-put race (64 rounds: entry survives in every schedule, byte-identical re-put, exact counts, pin keeps protecting after the race), the publish-after-mark arm, no-torn `has`/`get` reads under a window, single-flight contention aborting as data, and malformed-list sweeps as operational errors. Verification: `cargo test` (default) 46 passed; `cargo test --features mem` and `--all-features` 89 passed each; `cargo clippy --all-targets -- -D warnings` and `--all-features` clean; `cargo fmt --check` clean; `cargo doc --no-deps` at the pre-existing baseline (24 warnings carried from earlier waves, none new).