--- id: store-fleet-put name: the fleet put path — the joint entry+pin tx driver and the pin guard's fleet form status: completed depends_on: [store-fleet-construction, engine-pg-lo-joint] scope: broad risk: high impact: phase level: implementation tags: [wave-6, store, gc, fleet, put] --- ## Description Route the put paths through the **joint entry+pin transaction** under `fleet: true` (ADR-008 §2; ADR-010 §2; ADR-012 §3/§4) — the store core holds the tx from `begin_joint_tx` to `pin_commit`, engines contribute mechanics, and publish + protection land in one step — plus the pin guard's fleet form: the **TTL/3 renewal duty** on the guard. The non-fleet put paths are byte-unchanged (the ADR-005 in-process protocol verbatim; every wave-2/3/4/5 put test stays green). - **The joint-tx driver** (the store core, `src/store.rs`): under fleet mode, a put whose entry lands on the kv engine (postgres) runs `begin_joint_tx` → `joint_put` (the engine's CAS row write inside the held tx) → `pin_commit(tx, [PinRow { key, owner, expiry }])` — the pin row carries the store-instance id (`store-fleet-construction`'s owner id) and `expiry = now + TTL` (the store resolves the expiry; ADR-012 §2's rename — the TTL lives with the store's renewal schedule). The public `put` signature is unchanged; the routing branches on the fleet flag, never on re-deriving from the engine set. - **Amendment A1 — the joint-tx drop discipline** (the wave-4 gate's owner lands here): the driver owes tx termination on **every** path from `begin_joint_tx` onward — the natural `?` propagation between the driver's steps must never return a mid-flight `PgJointTx` to the pool (deadpool's `Fast` recycle never resets transaction state — PG-2's hazard). **Named design point — the termination design**: catch-propagate-terminate (every error path between begin and commit terminates the tx before propagating) per A1's recorded options — an explicit abort verb would need a new ADR (the seam shape is pinned; `pin_commit`'s terminal-consume reading stays). Record the design; the zero-residue atomicity is already the engine clause's contract (wave 4) — the driver's duty is never abandoning the tx mid-flight. - **The pin guard's fleet form** (ADR-008 §2: "the pin guard renews its row every TTL/3"): the `Pin` guard returned by a fleet put carries the **renewal duty** — while the guard lives, its pin row's expiry moves forward every TTL/3 (via the engine's `pin_renew(key, owner)` — the never-resurrect semantics: an expired row is never revived; the guard's renewal failing after expiry is the delete-then-recover backstop's territory, ADR-008's expiry semantics verbatim). **Named design points — the renewal lifecycle**: the duty's mechanism (a per-guard tokio task on a TTL/3 interval vs a shared renewal sweep keyed off the guard registry — decide, record), the abort-on-drop discipline (a dropped/converted/batch-contributed guard stops its renewal — no leaked tasks, no renewal past the guard's life), and the failure posture (a failed renewal is logged-and-continued or silent — the row's expiry is the protection, a missed renewal degrades to the expiry semantics, never an error on the put path). The guard's in-process refcount machinery continues to function (instance-local bookkeeping — the batch's one-step release, the ordered-step conversion); the row is the cross-instance protection. Batch puts (`put_in_batch`): each member's joint tx commits its row; the batch's one-step release ends the members' renewals (the rows expire and the sweeper's maintenance step reaps them — `store-fleet-sweep`). - **The dedup re-put under fleet**: CAS dedup means no new entry — the joint tx commits the pin row alone (an empty `joint_put` contribution is the engine's recorded CAS form) and the row upsert-renews `(key, owner)` (the wave-4 recorded shape). A different instance's re-put is a different pin row — protection composes by row (ADR-010 §2's refcount-in-rows reading). - **`large = local` on declared shared media — pin-first, publish-second** (ADR-012 §4): the joint tx is unachievable across media, so the ordering is pinned: the pin row commits on the kv engine (a joint tx carrying only the pin row — `begin_joint_tx` → `pin_commit` without `joint_put`) **before** the stage file's commit-rename publishes the entry. The invariant survives (an entry becomes visible only after its pin row is durable — sweep arbitration can never observe a visible-unpinned entry); the crash window moves to the harmless side (an orphan pin row, reaped by the expiry machinery — no new recovery mechanism). The reverse ordering is rejected (ADR-012 §4) — do not implement it. - **The pg-lo consolidation put** (kv = postgres + large = pg-lo, same instance): the large entry's content reaches the store-held tx through the pg-lo engine's joint contribution (`engine-pg-lo-joint`) — LO writes + companion-row insert inside the tx, the pin-row insert + commit via `pin_commit`. One tx over the shared instance; the dispatch threshold routes to the joint form exactly as the non-fleet path routes to the staged form. - **Unit tests** (env-driven, skipping cleanly server-less; the harness server `pglo-poc`, :15432, db `blobs`, `postgres`/`poc` — verified live 2026-10-12): the kv joint put publishes entry + pin row atomically (an independent connection sees both or neither); the forced-failure zero-residue form (A1's discipline exercised — a mid-driver failure leaves no dirty pooled session); the guard's renewal moves the row's expiry (a short test TTL; the renewal cadence observed); the never-resurrect form (an expired row is not renewed back to life); the batch form (members' rows commit; batch release ends renewals); the dedup re-put's upsert-renewal; the local-shared-media pin-first ordering (the pin row is durable before the committed file exists); the pg-lo consolidation round-trip through the public `put` (both tiers, one tx — the byte-exact large-entry form); the non-fleet paths unchanged (the existing suites green). ## Acceptance Criteria - [x] Under `fleet: true`, kv-tier puts route through the joint tx (entry + pin row in one step; an independent connection never observes one without the other); under `fleet: false` the put paths are byte-unchanged (all existing suites green) - [x] Amendment A1 landed: the driver terminates the tx on every path from `begin_joint_tx` onward (the termination design recorded; a forced mid-driver failure leaves zero residue and returns no dirty session to the pool) - [x] The pin guard's fleet form: renewal every TTL/3 while the guard lives; drop/conversion/batch-contribution stops the renewal with no leaked tasks; a failed renewal never resurrects an expired row (the lifecycle design points decided and recorded) - [x] The dedup re-put commits the pin row alone (CAS; upsert- renewal); a different instance's re-put composes protection by row - [x] `large = local` fleet puts: pin-row-first, publish-second (contract-tested — the pin row is durable before the committed file exists; the orphan-pin crash window is the expiry machinery's) - [x] The pg-lo consolidation put round-trips through the public `put` (one tx over the shared instance; byte-exact large-entry form) - [x] `cargo test` (default) green; `cargo test --features postgres` and `--features "postgres,pg-lo"` green server-less (skipping) and against the harness server at acceptance; `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` green (default + the postgres/pg-lo variants); `cargo doc --no-deps` clean ## References - src/store.rs (the put seam's two internal paths; the fleet declaration's flag from `store-fleet-construction`) - src/gc.rs (`EngineFleetState` — the clauses the driver calls; `PinRow`; the ownership rule) - src/postgres.rs (`PgJointTx` — PG-2's hazard doc; the wave-4 joint-clause bodies) - src/pg_lo.rs (the joint contribution from `engine-pg-lo-joint`) - docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md §2 (pins are rows; the TTL/3 renewal; the expiry semantics) - docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §2 (the store holds the tx; the engine contributes mechanics), §4 (the guard-drops-at-response-time reading — the ops edition; the public-API guard lives as long as the caller holds it) - docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §3 (activation), §4 (the host; pin-first/publish-second for large=local; the one-tx consolidation) - docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §1 (the renewal worker's cadence is embedder composition above the facade — the guard's own TTL/3 duty is the in-crate half) - tasks/review-wave-4.md (amendment A1 — this task is its owner) - tasks/store-fleet-construction.md, tasks/engine-pg-lo-joint.md (the declaration and the contribution this drives) ## Notes Landed 2026-10-12. Design decisions (each recorded in-code as a named design point on the fleet impl block of `src/store.rs`; all keyed off the landed ADR text): - **A1's termination design — catch-propagate-terminate, no abort verb.** The driver's steps between `begin_joint_tx` and `pin_commit` run inside one catch block (an async block's `?` surface); any error terminates the held tx (a `ROLLBACK` through the `PgJointTx::client()` checkout — the wave-5 landed accessor) before propagating; `pin_commit` remains the terminal-consume step (its own failure paths roll back per the wave-4 clause contract). An explicit abort verb was rejected per A1's recorded options (it would need a new ADR; the seam shape is pinned). A ROLLBACK on a session with no active transaction (the clause bodies' having already rolled back) is a completed, warning-level no-op — the blanket termination is always safe. `pin_commit`-consuming txs on failure are the clause's own; the driver never re-uses a consumed tx. - **The renewal duty's mechanism — a per-guard tokio task on a TTL/3 interval**, holding an `Arc` (the kv-tier enum's postgres payload moved to an `Arc` for this: the detached task outlives the `Store` and keeps the backend's pool exactly as long as the pin lives; `PostgresBackend` itself is not `Clone` — its lease checkout must not be). The shared-renewal-sweep keyed off a guard registry was the rejected alternative: it needs precisely the lifecycle notifications (mint/drop/convert/batch-contribute) that an abort handle gives for free. The task is spawned **at the pin row's durability** (after `pin_commit`, never at guard mint — a renewal arriving at a nonexistent row would read the not-yet-pinned window), on the cadence `(ttl / 3).max(1 ms)` (the 1 ms floor is the degenerate-tuning guard: a TTL below 3 ms would floor to a zero poll). The store resolves the expiry (`now + ttl`) at the joint tx's `pin_commit`; successive renewals ride the engine's `pin_renew` (the successor-time body reads the TTL the fleet constructor passed the engine — `store-fleet-construction`'s wiring). - **The failure posture: silent-and-continued on `Err`, ended on `Ok(false)`.** The row's expiry is the protection — a missed renewal degrades to ADR-008's expiry semantics, never an error on the put path (the crate carries no logging dependency; embedder visibility into renewals observes the row). `Ok(false)` — an expired or absent `(key, owner)` row — ends the duty: the engine never resurrects (wave 4's recorded semantics), so continuing would only churn; the duty ends, the delete-then-recover backstop's territory begins. - **The abort-on-drop discipline:** the duty is a `PinRenewal` (a `JoinHandle` wrapper, crate-internal, gated feature `postgres`) attached to the returned `Pin`; it is aborted by the guard's drop, by `Batch`'s one-step release (the duty moves into the batch at `Batch::add` — a member's row keeps renewing while the batch lives), and by the ordered-step conversion (`RegistryCore::convert` pre-stops before its `ManuallyDrop` suppresses the guard's drop). No leaked tasks; no renewal past the guard's life. A task aborted mid-renewal is harmless (the clause's expiry update is idempotent). - **The routing never re-derives from the engine set:** every fleet branch keys off `Store::fleet`'s presence first; the pg-lo tier's joint-vs-staged choice inside fleet mode rides the engine enum (the tier mechanics' selector, the same dispatch as every other fleet branch shape). The public `put`/`put_in_batch` signatures are unchanged; the non-fleet paths are byte-unchanged (all existing suites green untouched). - **The kv joint form's mechanics:** the in-process refcount mints first (unchanged bookkeeping), then `begin_joint_tx` → `joint_put` (the engine's CAS row write; a dedup re-put contributes nothing — the recorded CAS form; the row upsert-renews via `pin_commit`'s `ON CONFLICT (key, owner)` upsert) → `pin_commit(tx, [row])` — publish + protection in one step. The row's `owner` is the store-instance id; `expiry = now + pin_ttl`. - **The consolidated (pg-lo) form:** the tx begins, the tier's joint staging cursor streams the content inside it (the engine-level contract: the 512 KiB knee rides, the unknown-length form's buffer-to-threshold phase is put_unknown's verbatim shape and its derivation pass re-reads the artifact through the cursor's read-your-own-writes seam), the cursor's publish is the CAS contribution (a re-put: the empty-contribution form — the staged LO unlinked in-tx, zero residue), and `pin_commit` lands the row with the entry. - **Test isolation:** every store test takes a fresh engine-owned schema (the LO-free kv tier; `SchemaJanitor`) or a fresh dedicated database (the LO-bearing consolidation tests; `DatabaseJanitor`) with a janitor removing it in every exit path — amendment A2's lesson honored from the start. The A1 dirty-session observable is `pg_stat_activity`'s `idle in transaction%` count asserted to zero (plus the follow-up put's clean round-trip and the zero-residue table assertions). ## Summary The fleet put path landed. Under `fleet: true`, `put`/`put_in_batch` (unchanged signatures) route: **kv tier** — the joint entry+pin tx (`begin_joint_tx` → the engine's CAS `joint_put` → `pin_commit`, publish + protection in one step; a dedup re-put commits the pin row alone, the row upsert-renewing `(key, owner)`; a different instance's re-put mints a different row — protection composes by row); **large = pg-lo (the consolidation posture)** — one tx over the shared instance, the tier's joint staging cursor streaming the content inside the store-held tx (known- and unknown-length; the 512 KiB knee, the CAS publish, the empty-contribution re-put form, read-your-own-writes derivation passes), `pin_commit` closing; **large = local (declared shared media)** — pin-first, publish-second: a row-only joint tx commits the pin row before the staged commit-rename (the forced-publish-failure test proving the ordering: the pin row is durable with no committed file, the orphan reclaimed by the expiry machinery; never the reverse ordering). Amendment A1 landed as its recorded owner — catch-propagate-terminate: every error path between `begin_joint_tx` and `pin_commit` terminates the held tx before propagating (tested: a mid-driver contribution failure leaves zero residue and returns no dirty pooled session — `pg_stat_activity`'s idle-in-transaction count zero — and the pool's follow-up put round-trips clean). The pin guard's fleet form landed: the returned guard carries the row's TTL/3 renewal duty (a per-guard tokio task, spawned at the row's durability, silent on `Err`, ended by the never-resurrect `Ok(false)`, aborted at every guard-life boundary — drop, batch contribution, conversion). The in-process refcount machinery continues to function (instance-local bookkeeping, the batch's one-step release, the ordered-step conversion); the row is the cross-instance protection; nothing expires in non-fleet modes. Non-fleet put paths are byte-unchanged (all existing suites green untouched; a non-fleet test additionally pins that no pin rows are ever committed). Verified: `cargo test` (default) green; `--features postgres` and `--features "postgres,pg-lo"` green server-less (skipping cleanly) and against the harness server (pglo-poc, :15432 — live, all fleet-put tests green); `--all-features` and the pg-lo-pure `--features mem,pg-lo` builds green; clippy `-D warnings` green (default, mem, postgres, all-features, pg-pure, mem+pg-lo, and postgres+pg-lo); `cargo fmt --check` green; `cargo doc --no-deps` zero warnings in the default build (the gated builds' pre-existing warnings counted before and after — identical). The fleet sweep (`store-fleet-sweep`) holds the fleet mechanism's remaining store-core scope (the sweep lease, the SQL arbitration).