feat(gc): the in-process GC protocol — pins, liveness, delete windows, sweep, direct delete
The gc-protocol task (ADR-005; abort-as-data per ADR-012 §1; non-SQL inline window staging per ADR-012 §5) wired on the Store facade: - src/gc.rs — GcHome + RegistryCore: one arbitration lock over per-key pin refcounts, in-flight delete markers, and the registered source lists; async pin minting (acquire) that blocks on in-flight deletes (ADR-005's re-put case (a) delete-then-recover without engine I/O under the lock); the ordered-step convert primitive (pin release + registration in one section); batch one-step release; the single-node sweep lease (the in-process analog of ADR-013 §2). - src/facade.rs — Pin/Batch carry the registry back-reference and own their drop release (batch drop decrements every member in one ordered step); new public vocabulary LivenessSource, ProtectCallback, Registration, re-exported from lib.rs. - src/store.rs — sweep() (single-flight → zero-source safe default → protect consult → mark over both tiers' list() union → arbitrate at delete time → ~100/batch commit → exact-count report or abort-as-data), direct delete() (existence probe → full protection set per ADR-012 §6.4 → GcRefuse/Missing/delete through the fall-through), and the registration/unregistration verbs. - 16 new tests + the facade pin/batch shape test reworked on the wired machinery: exact counts across both tiers, abort shapes, dedup refcounts, batch drops, live-shared sources, the conversion ordered step sequentially and raced, sweep-vs-pinned-put (64 rounds, every schedule, byte-identical delete-then-recover), publish-after-mark, no-torn reads under a window, single-flight contention, and malformed-list sweeps as operational errors. - tasks/gc-protocol.md — status completed; named design points recorded in Notes. Verification: cargo test (default) 46 passed; --features mem and --all-features 89 passed each; cargo clippy --all-targets -- -D warnings (default and --all-features) clean; cargo fmt --check clean; cargo doc --no-deps at the pre-existing baseline (no new warnings).
This commit is contained in:
1 parent
f1f505a910
commit
889be4cada
6 files changed
+1872
-78
No files matched your search
+130
-9
@@ -1,7 +1,7 @@
|
||||
---
|
||||
id: gc-protocol
|
||||
name: In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal
|
||||
status: pending
|
||||
status: completed
|
||||
depends_on: [store-construction]
|
||||
scope: broad
|
||||
risk: high
|
||||
@@ -82,24 +82,24 @@ extends without changing its invariants.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Pin registry with per-key refcounts under the arbitration lock;
|
||||
- [x] Pin registry with per-key refcounts under the arbitration lock;
|
||||
dedup re-puts refcount; `Pin`/`Batch` drops release through it
|
||||
(batch drops in one ordered step)
|
||||
- [ ] `register_liveness_source` live-shared (consulted at mark and at
|
||||
- [x] `register_liveness_source` live-shared (consulted at mark and at
|
||||
delete-time arbitration, never snapshotted); source shape decided
|
||||
and recorded
|
||||
- [ ] Protect callback with abort-as-data (`ProtectFailed`);
|
||||
- [x] Protect callback with abort-as-data (`ProtectFailed`);
|
||||
`NoLivenessSources` safe default; `sweep()` returns
|
||||
`Ok(SweepReport)` on every abort — never an abort error
|
||||
- [ ] Delete-window protocol: mark → arbitrate-at-delete-time →
|
||||
- [x] Delete-window protocol: mark → arbitrate-at-delete-time →
|
||||
batched commit (~100/batch), with exact-count `SweepReport`
|
||||
fields (`candidates_staged`/`deleted`/`cancelled_by_arbitration`)
|
||||
- [ ] Direct `delete` refuses the full protection set with `GcRefuse`
|
||||
- [x] Direct `delete` refuses the full protection set with `GcRefuse`
|
||||
and deletes unprotected keys through the fall-through dispatch
|
||||
- [ ] Concurrency tests: sweep vs in-flight pinned put (the
|
||||
- [x] Concurrency tests: sweep vs in-flight pinned put (the
|
||||
pin-before-publish property from the sweep side), exact-count
|
||||
outcomes, abort shapes
|
||||
- [ ] `cargo test` green; `cargo test --features mem` and `cargo test
|
||||
- [x] `cargo test` green; `cargo test --features mem` and `cargo test
|
||||
--all-features` green; clippy/fmt gates green
|
||||
|
||||
## References
|
||||
@@ -115,6 +115,127 @@ extends without changing its invariants.
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
**Named design points decided here** (recorded in `src/gc.rs`'s module
|
||||
doc and on the traits in `src/facade.rs`):
|
||||
|
||||
- **Guard back-reference shape**: `Pin` holds `Arc<RegistryCore>` (the
|
||||
GC home's shared core) alongside its key — so a drop reaches the
|
||||
per-key refcounts wherever the guard travels, and the registry
|
||||
outlives the `Store` instance when guards still reference it. No
|
||||
weak-pointer tag-drop plumbing (iroh's TempTag conclusion without
|
||||
their mechanism). `Batch` holds its members' keys; contribution
|
||||
defuses the member guard (`armed` flag), and the batch's release
|
||||
decrements every member refcount inside **one** arbitration-lock
|
||||
section, keys in total order — no early single-pin release.
|
||||
- **Lock granularity**: one `parking_lot::Mutex<RegistryState>` — the
|
||||
pin refcounts, the in-flight delete markers, and the registered
|
||||
source lists under one lock, so a delete-time decision (pins ∨
|
||||
markers ∨ sources) and its marker insert are one atomic section
|
||||
against pin mints. parking_lot does not poison (no `into_inner`
|
||||
posture to apply). Consumer consults (`contains`/`protect`) run
|
||||
under that lock at delete-time arbitration and outside it at mark
|
||||
time; lock order is one-directional (arbitration lock → consumer
|
||||
locks; the ordered-step conversion is the only nesting, same
|
||||
direction).
|
||||
- **Liveness source shape**: a sync (non-async) trait,
|
||||
`live_keys()` (mark-time enumeration, consulted outside the lock)
|
||||
+ `contains()` (delete-time probe, consulted under the lock; default
|
||||
derives it linearly from `live_keys` — indexed sources override).
|
||||
Sync because pin drops are sync RAII and arbitration consults run
|
||||
under the short-held lock. Consumers register `Arc<dyn
|
||||
LivenessSource>` shared handles — the POC finding-7 shape; the store
|
||||
consults the current contents every time and never snapshots.
|
||||
- **Protect callback shape**: one sync verb, `protect() →
|
||||
Option<Vec<Key>>`; `None` = failed to answer → sweep aborts with
|
||||
`ProtectFailed` report data, and a direct delete refuses
|
||||
conservatively (`GcRefuse`, nothing deleted). Callbacks compose
|
||||
(each registered set unions into the protection).
|
||||
- **The sweep-vs-put race closes with in-flight delete markers**, not
|
||||
by holding the arbitration lock across engine I/O: the arbitrate
|
||||
section marks a Proceed-ing key `deleting`; the engine delete runs
|
||||
outside the lock; marker release notifies blocked pin mints. A
|
||||
re-put's mint landing while the marker is held blocks (async,
|
||||
`tokio::sync::Notify` with the enable-before-check recipe) until the
|
||||
deletion commits, then lands fresh — ADR-005's case (a)
|
||||
delete-then-recover verbatim; a mint landing before arbitration is
|
||||
case (b), protection composed at the arbitration itself. Holding the
|
||||
lock across the delete was rejected: a `parking_lot` guard is not
|
||||
`Send`, the sweep future must be, and AGENTS convention 3 forbids
|
||||
blocking I/O under internal locks.
|
||||
- **The sweep is single-flight in-process**: an atomic test-and-set
|
||||
flag is the single-node analog of ADR-013 §2's named exclusive sweep
|
||||
lease (ADR-012 §5's "single-flighter is the in-process lock"
|
||||
posture); a contended second sweeper aborts with `SweeperLock`
|
||||
report data — nothing deleted, the next run takes the lease.
|
||||
- **Vanished candidates count as arbitration cancels**: a key deleted
|
||||
between mark and its window turn (e.g. a concurrent direct delete)
|
||||
settles into `cancelled_by_arbitration`, keeping the completed
|
||||
sweep's accounting exact: `candidates_staged == deleted +
|
||||
cancelled_by_arbitration`.
|
||||
- **Registration vocabulary**: a public opaque `Registration` token
|
||||
per registration; `unregister_liveness_source` /
|
||||
`unregister_protect_callback` consume it (the correction-flow
|
||||
prerequisite of ADR-012 §6.4). The protect-callback registration
|
||||
verbs are this layer's addition — the direct-delete full-protection
|
||||
set requires consultable protect state on the store, which store-api
|
||||
§Lifecycle implies but does not spell out.
|
||||
- **The ordered-step conversion primitive**: `GcHome::convert(pin,
|
||||
register)` (crate-internal until a consumer-facing surface exists):
|
||||
pin release + registration closure inside one arbitration-lock
|
||||
section; tested sequentially and raced (64-round direct-delete
|
||||
race: no schedule leaves the key unprotected — every delete
|
||||
refuses).
|
||||
|
||||
**Scope notes**: the pin minting API (`GcHome::acquire`, async, waits
|
||||
on in-flight deletes) is wired; the put path's call sites remain for
|
||||
`store-put-paths`, which now only needs `store.gc().acquire(key)` /
|
||||
`new_batch()` / `convert` at the publish points. Batch commit semantics
|
||||
(the seam's terminal step) also rides that task; `Batch::release_members`
|
||||
is its reuse point. Sweep/`delete`/registration live on `Store`
|
||||
(`src/store.rs`), with one new `KvEngine`/`LargeEngine` dispatch method
|
||||
(`delete`). A dev-dependency addition (`tokio`'s `rt-multi-thread`,
|
||||
test-only) supports the single-flight concurrency test's multi-thread
|
||||
runtime flavor; the base crate's dependency set is unchanged.
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
> To be filled on completion
|
||||
|
||||
The in-process GC protocol (ADR-005; abort-as-data per ADR-012 §1;
|
||||
non-SQL inline window staging per ADR-012 §5) is wired on `Store`:
|
||||
|
||||
- `src/gc.rs` — `GcHome` + `RegistryCore`: one arbitration lock over
|
||||
pin refcounts, delete markers, and registration lists; async pin
|
||||
minting (`acquire`) that blocks on in-flight deletes (the
|
||||
delete-window mechanism); the ordered-step `convert` primitive;
|
||||
batch one-step release; source/protect registration bookkeeping;
|
||||
the single-node sweep lease.
|
||||
- `src/facade.rs` — `Pin`/`Batch` now carry the registry
|
||||
back-reference and own their drop-release behavior (the wave-1
|
||||
contracts are now wired behavior); new public vocabulary
|
||||
`LivenessSource`, `ProtectCallback`, `Registration` (re-exported
|
||||
from `lib.rs`).
|
||||
- `src/store.rs` — `sweep()` (single-flight → zero-source safe
|
||||
default → protect consult → mark over both tiers' `list()` union →
|
||||
arbitrate-at-delete-time → ~100/batch commit → exact-count report
|
||||
or abort-as-data), direct `delete()` (existence probe → full
|
||||
protection set arbitration → `GcRefuse`/`Missing`/delete through
|
||||
the fall-through), and the four registration/unregistration verbs.
|
||||
- 16 new tests + the facade pin/batch shape test reworked on the wired
|
||||
machinery: zero-source abort, empty-pool zero report, exact
|
||||
counts across both tiers, dedup refcounts, batch one-step drops,
|
||||
protect add + abort-as-data, live-shared (non-snapshotted) sources,
|
||||
token unregistration, the full direct-delete refusal set (+
|
||||
`Missing`/`KeyInvalid`), the ordered-step conversion sequential and
|
||||
raced (64 rounds, always refused), the sweep-vs-pinned-put race
|
||||
(64 rounds: entry survives in every schedule, byte-identical
|
||||
re-put, exact counts, pin keeps protecting after the race), the
|
||||
publish-after-mark arm, no-torn `has`/`get` reads under a window,
|
||||
single-flight contention aborting as data, and malformed-list
|
||||
sweeps as operational errors.
|
||||
|
||||
Verification: `cargo test` (default) 46 passed; `cargo test
|
||||
--features mem` and `--all-features` 89 passed each; `cargo clippy
|
||||
--all-targets -- -D warnings` and `--all-features` clean; `cargo fmt
|
||||
--check` clean; `cargo doc --no-deps` at the pre-existing baseline
|
||||
(24 warnings carried from earlier waves, none new).
|
||||
Reference in new issue
Block a user