glm-5.3-flash
79cf1f0220
Read the wave-2 implementation against the pinned ADR text line by line (ADR-003/005/006/010/011/012/013 §2, store-api.md): the three numbered unknown-length rules each test-backed, the delete-window ordering invariants (arbitrate-at-delete-time, pin-before-publish on both put paths and both tiers, raced conversion ordering, delete-then-recover), abort-as-data with all three causes, dispatch purity + fall-through + constructor modes + kv-only's two put-time rejection points, stage-hygiene exact counts on both engines (incl. the Drop-cancellation story), both suite columns green/ADR-stamped/ traceable with the factory binding recorded, the dispatch-coverage shape exercised, all eight named design points recorded and consistent, and the staged-put key-at-commit amendment audited accepted. No wave-3 amendments arise. Inline fixes applied by the gate: - facade.rs: stale Batch::add dead-code allow + stale task-pointer comment removed (put_in_batch's call sites landed) - dispatch.rs: module-level allow(dead_code) removed (route/Tier now driven; clippy clean without it), scope note updated - backend.rs: the allow kept but its scope note corrected to what is actually dead (trait name, kv put until wave 3) - mem.rs: stale constructor doc line updated to the landed state - tasks/gc-protocol.md: stray placeholder lines removed Verification: cargo test (42 lib + 2 conformance + 3 git-interop), cargo test --features mem and --all-features (97 lib + 37 conformance + 4 git-interop), clippy --all-targets and --all-features --all-targets (-D warnings), fmt --check — all green post-fix. Wave 3 decomposition unblocked.
237 lines
13 KiB
Markdown
237 lines
13 KiB
Markdown
---
|
||
id: gc-protocol
|
||
name: In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal
|
||
status: completed
|
||
depends_on: [store-construction]
|
||
scope: broad
|
||
risk: high
|
||
impact: phase
|
||
level: implementation
|
||
tags: [wave-2, gc, facade]
|
||
---
|
||
|
||
## Description
|
||
|
||
Implement the in-process GC protocol (ADR-005; abort-as-data per
|
||
ADR-012 §1; non-SQL window staging per ADR-012 §5) on the `Store` type
|
||
— the architecture's most intricate state machine, in its single-node
|
||
form. The fleet extension (DB-backed pins, sweep lease, SQL
|
||
arbitration) is wave 6; this task is the protocol the fleet form
|
||
extends without changing its invariants.
|
||
|
||
- **Pin registry** (ADR-005 §Marks 2): per-key refcounts under the
|
||
arbitration lock the sweep consults. `Pin` guards minted by the put
|
||
path (`store-put-paths` wires the call sites; the registry and
|
||
minting land here); a dedup re-put refcounts the existing entry — a
|
||
dedup put's guard protects exactly as a fresh put's does. `Pin` drop
|
||
decrements. **Named design point**: the guard's back-reference shape
|
||
(how the public `Pin`/`Batch` types reach the registry — the wave-1
|
||
types carry only the key today) and the lock granularity
|
||
(`parking_lot` short-held per AGENTS convention 3, poisoning via
|
||
`unwrap_or_else(|e| e.into_inner())`).
|
||
- **`Batch` machinery**: a batch's pins drop together on batch commit
|
||
or batch drop, in one ordered step under the arbitration lock; no
|
||
early single-pin release. The **conversion ordering rule** (ADR-005,
|
||
pinned in facade.rs docs): a pin's drop and its replacement's
|
||
registration into a liveness source are one ordered step under the
|
||
same lock — the registry needs the API shape that makes the ordered
|
||
step expressible.
|
||
- **Registered liveness sources** (`register_liveness_source`, the
|
||
live-shared seam — ADR-005 §Decision): **named design point** — the
|
||
source trait/callback shape. Live-shared means consulted at mark
|
||
time and at delete-time arbitration, never snapshotted at
|
||
registration (POC #1 finding 7's install-implies-copy failure is the
|
||
pinning reason).
|
||
- **Protect callback**: consulted pre-sweep; may add externally-known
|
||
hashes or abort the run — abort is report data
|
||
(`GcAbortCause::ProtectFailed`), never an error (ADR-012 §1).
|
||
- **`sweep() → Result<SweepReport, Error>`** — the delete-window
|
||
protocol (ADR-005 §Sweep):
|
||
1. **mark**: enumerate the whole pool (both tiers' complete `list()`
|
||
unioned); compute the live set from all three sources (registered
|
||
sources, pins, protect callback); stage the candidate-dead set.
|
||
2. **delete window**: per candidate, **arbitrate under the pin/
|
||
liveness lock at delete time** (not once upfront) — pinned or
|
||
re-observed-live cancels the candidate; otherwise proceed.
|
||
3. **commit**: remaining candidates deleted in batches (~100/batch,
|
||
iroh's proven shape); window closed.
|
||
- **`Ok(SweepReport)` always** — sweeps are never errors; every abort
|
||
is data in `aborted: Option<GcAbortCause>` (exactly three causes;
|
||
`NoLivenessSources` is the safe default when zero sources are
|
||
registered — nothing deleted). Exact counts in the report fields.
|
||
- **Non-SQL window staging** (ADR-012 §5): `local`/`mem` stage
|
||
candidates in-process and execute the window protocol **inline in
|
||
the same sweep** — there is no durable window table on non-SQL
|
||
engines, and the "executor is the next sweep" clause (ADR-012
|
||
§6.7) does not apply to them.
|
||
- The test-asserted property (ADR-005): a visible pool entry is never
|
||
deleted while liveness protects it, and an in-flight put is never
|
||
deleted by a sweep started before it committed — concurrency tests
|
||
(sweep racing a pinned put) assert it.
|
||
- **Direct `delete(key)`** (wired through the fall-through dispatch —
|
||
arbitration-then-delete where the entry lives): refuses **any
|
||
protected key** — pinned, registered-source-live, or
|
||
protect-callback-named (the full protection set, ADR-012 §6.4) —
|
||
with the typed `Error::GcRefuse`, nothing deleted; unprotected keys
|
||
delete; absent keys carry `Err(Missing)`. The refusal is decided at
|
||
the facade (the altitude discipline: `GcRefuse` never fires below
|
||
the facade).
|
||
- **`has`/`get` during a window observe no torn state** — existence
|
||
flips atomically per key (immutable entries); document the posture
|
||
the engines' CAS semantics already give.
|
||
|
||
## Acceptance Criteria
|
||
|
||
- [x] Pin registry with per-key refcounts under the arbitration lock;
|
||
dedup re-puts refcount; `Pin`/`Batch` drops release through it
|
||
(batch drops in one ordered step)
|
||
- [x] `register_liveness_source` live-shared (consulted at mark and at
|
||
delete-time arbitration, never snapshotted); source shape decided
|
||
and recorded
|
||
- [x] Protect callback with abort-as-data (`ProtectFailed`);
|
||
`NoLivenessSources` safe default; `sweep()` returns
|
||
`Ok(SweepReport)` on every abort — never an abort error
|
||
- [x] Delete-window protocol: mark → arbitrate-at-delete-time →
|
||
batched commit (~100/batch), with exact-count `SweepReport`
|
||
fields (`candidates_staged`/`deleted`/`cancelled_by_arbitration`)
|
||
- [x] Direct `delete` refuses the full protection set with `GcRefuse`
|
||
and deletes unprotected keys through the fall-through dispatch
|
||
- [x] Concurrency tests: sweep vs in-flight pinned put (the
|
||
pin-before-publish property from the sweep side), exact-count
|
||
outcomes, abort shapes
|
||
- [x] `cargo test` green; `cargo test --features mem` and `cargo test
|
||
--all-features` green; clippy/fmt gates green
|
||
|
||
## References
|
||
|
||
- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (the protocol + invariants)
|
||
- docs/architecture/gc-and-namespaces.md (§Mark-and-sweep GC; §Delete windows)
|
||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (non-SQL staging), §6.4 (full protection set)
|
||
- docs/architecture/store-api.md (§Lifecycle — sweep shape; §Invariants 4/5)
|
||
- src/facade.rs (the pinned Pin/Batch/SweepReport lifecycle contracts this wires)
|
||
- docs/research/poc-trait-dispatch-findings.md (findings 3/7/8)
|
||
|
||
## Notes
|
||
|
||
**Named design points decided here** (recorded in `src/gc.rs`'s module
|
||
doc and on the traits in `src/facade.rs`):
|
||
|
||
- **Guard back-reference shape**: `Pin` holds `Arc<RegistryCore>` (the
|
||
GC home's shared core) alongside its key — so a drop reaches the
|
||
per-key refcounts wherever the guard travels, and the registry
|
||
outlives the `Store` instance when guards still reference it. No
|
||
weak-pointer tag-drop plumbing (iroh's TempTag conclusion without
|
||
their mechanism). `Batch` holds its members' keys; contribution
|
||
defuses the member guard (`armed` flag), and the batch's release
|
||
decrements every member refcount inside **one** arbitration-lock
|
||
section, keys in total order — no early single-pin release.
|
||
- **Lock granularity**: one `parking_lot::Mutex<RegistryState>` — the
|
||
pin refcounts, the in-flight delete markers, and the registered
|
||
source lists under one lock, so a delete-time decision (pins ∨
|
||
markers ∨ sources) and its marker insert are one atomic section
|
||
against pin mints. parking_lot does not poison (no `into_inner`
|
||
posture to apply). Consumer consults (`contains`/`protect`) run
|
||
under that lock at delete-time arbitration and outside it at mark
|
||
time; lock order is one-directional (arbitration lock → consumer
|
||
locks; the ordered-step conversion is the only nesting, same
|
||
direction).
|
||
- **Liveness source shape**: a sync (non-async) trait,
|
||
`live_keys()` (mark-time enumeration, consulted outside the lock)
|
||
+ `contains()` (delete-time probe, consulted under the lock; default
|
||
derives it linearly from `live_keys` — indexed sources override).
|
||
Sync because pin drops are sync RAII and arbitration consults run
|
||
under the short-held lock. Consumers register `Arc<dyn
|
||
LivenessSource>` shared handles — the POC finding-7 shape; the store
|
||
consults the current contents every time and never snapshots.
|
||
- **Protect callback shape**: one sync verb, `protect() →
|
||
Option<Vec<Key>>`; `None` = failed to answer → sweep aborts with
|
||
`ProtectFailed` report data, and a direct delete refuses
|
||
conservatively (`GcRefuse`, nothing deleted). Callbacks compose
|
||
(each registered set unions into the protection).
|
||
- **The sweep-vs-put race closes with in-flight delete markers**, not
|
||
by holding the arbitration lock across engine I/O: the arbitrate
|
||
section marks a Proceed-ing key `deleting`; the engine delete runs
|
||
outside the lock; marker release notifies blocked pin mints. A
|
||
re-put's mint landing while the marker is held blocks (async,
|
||
`tokio::sync::Notify` with the enable-before-check recipe) until the
|
||
deletion commits, then lands fresh — ADR-005's case (a)
|
||
delete-then-recover verbatim; a mint landing before arbitration is
|
||
case (b), protection composed at the arbitration itself. Holding the
|
||
lock across the delete was rejected: a `parking_lot` guard is not
|
||
`Send`, the sweep future must be, and AGENTS convention 3 forbids
|
||
blocking I/O under internal locks.
|
||
- **The sweep is single-flight in-process**: an atomic test-and-set
|
||
flag is the single-node analog of ADR-013 §2's named exclusive sweep
|
||
lease (ADR-012 §5's "single-flighter is the in-process lock"
|
||
posture); a contended second sweeper aborts with `SweeperLock`
|
||
report data — nothing deleted, the next run takes the lease.
|
||
- **Vanished candidates count as arbitration cancels**: a key deleted
|
||
between mark and its window turn (e.g. a concurrent direct delete)
|
||
settles into `cancelled_by_arbitration`, keeping the completed
|
||
sweep's accounting exact: `candidates_staged == deleted +
|
||
cancelled_by_arbitration`.
|
||
- **Registration vocabulary**: a public opaque `Registration` token
|
||
per registration; `unregister_liveness_source` /
|
||
`unregister_protect_callback` consume it (the correction-flow
|
||
prerequisite of ADR-012 §6.4). The protect-callback registration
|
||
verbs are this layer's addition — the direct-delete full-protection
|
||
set requires consultable protect state on the store, which store-api
|
||
§Lifecycle implies but does not spell out.
|
||
- **The ordered-step conversion primitive**: `GcHome::convert(pin,
|
||
register)` (crate-internal until a consumer-facing surface exists):
|
||
pin release + registration closure inside one arbitration-lock
|
||
section; tested sequentially and raced (64-round direct-delete
|
||
race: no schedule leaves the key unprotected — every delete
|
||
refuses).
|
||
|
||
**Scope notes**: the pin minting API (`GcHome::acquire`, async, waits
|
||
on in-flight deletes) is wired; the put path's call sites remain for
|
||
`store-put-paths`, which now only needs `store.gc().acquire(key)` /
|
||
`new_batch()` / `convert` at the publish points. Batch commit semantics
|
||
(the seam's terminal step) also rides that task; `Batch::release_members`
|
||
is its reuse point. Sweep/`delete`/registration live on `Store`
|
||
(`src/store.rs`), with one new `KvEngine`/`LargeEngine` dispatch method
|
||
(`delete`). A dev-dependency addition (`tokio`'s `rt-multi-thread`,
|
||
test-only) supports the single-flight concurrency test's multi-thread
|
||
runtime flavor; the base crate's dependency set is unchanged.
|
||
|
||
## Summary
|
||
|
||
The in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1;
|
||
non-SQL inline window staging per ADR-012 §5) is wired on `Store`:
|
||
|
||
- `src/gc.rs` — `GcHome` + `RegistryCore`: one arbitration lock over
|
||
pin refcounts, delete markers, and registration lists; async pin
|
||
minting (`acquire`) that blocks on in-flight deletes (the
|
||
delete-window mechanism); the ordered-step `convert` primitive;
|
||
batch one-step release; source/protect registration bookkeeping;
|
||
the single-node sweep lease.
|
||
- `src/facade.rs` — `Pin`/`Batch` now carry the registry
|
||
back-reference and own their drop-release behavior (the wave-1
|
||
contracts are now wired behavior); new public vocabulary
|
||
`LivenessSource`, `ProtectCallback`, `Registration` (re-exported
|
||
from `lib.rs`).
|
||
- `src/store.rs` — `sweep()` (single-flight → zero-source safe
|
||
default → protect consult → mark over both tiers' `list()` union →
|
||
arbitrate-at-delete-time → ~100/batch commit → exact-count report
|
||
or abort-as-data), direct `delete()` (existence probe → full
|
||
protection set arbitration → `GcRefuse`/`Missing`/delete through
|
||
the fall-through), and the four registration/unregistration verbs.
|
||
- 16 new tests + the facade pin/batch shape test reworked on the wired
|
||
machinery: zero-source abort, empty-pool zero report, exact
|
||
counts across both tiers, dedup refcounts, batch one-step drops,
|
||
protect add + abort-as-data, live-shared (non-snapshotted) sources,
|
||
token unregistration, the full direct-delete refusal set (+
|
||
`Missing`/`KeyInvalid`), the ordered-step conversion sequential and
|
||
raced (64 rounds, always refused), the sweep-vs-pinned-put race
|
||
(64 rounds: entry survives in every schedule, byte-identical
|
||
re-put, exact counts, pin keeps protecting after the race), the
|
||
publish-after-mark arm, no-torn `has`/`get` reads under a window,
|
||
single-flight contention aborting as data, and malformed-list
|
||
sweeps as operational errors.
|
||
|
||
Verification: `cargo test` (default) 46 passed; `cargo test
|
||
--features mem` and `--all-features` 89 passed each; `cargo clippy
|
||
--all-targets -- -D warnings` and `--all-features` clean; `cargo fmt
|
||
--check` clean; `cargo doc --no-deps` at the pre-existing baseline
|
||
(24 warnings carried from earlier waves, none new). |