feat(store): the pg-lo constructor wiring — the consolidation shape and the window-host completion (wave 5)
- LargeEngine::PgLo slots in behind `pg-lo` + all six large-op dispatch arms; the large cursor/staging bodies land as the LargeCursor/LargeStaging sub-enums behind AnyCursor/AnyStaging's one large arm (the recorded KvCursor pattern's large-tier twin) - dual_tier_postgres_pglo (the consolidation posture; two DSN+opts pairs, implicitly non-fleet, completing the fleet-valid engine set) and dual_tier_mem_pglo (the one-window-host rule's named test topology), both carrying the constructor-tunable delete-window delay - ensure_gc_schema_at_construction: the one-window-host rule's store-core dispatch (kv host on dual-SQL — pg-lo's window clauses dormant; pg-lo host on kv=mem); the sweep's mem arm gains the large-host delegation, the branch keyed off the engine representation never a flag - stale scope notes updated to the landed state (store.rs module doc, LargeEngine, the sweep docs, PUT_CHUNK, pg_lo.rs, gc.rs) - 7 new env-driven constructor tests (dedicated-database janitor discipline; consolidation + mem-pglo both shapes Verified live against the pglo-poc harness server: cargo test default 69/19/3, --features pg-lo 92/19/3, --features "pg-lo,mem" 151/54/4 (server-less 124/37/4), --no-default-features "postgres,pg-lo" 102/19/3 (both live and server-less), --all-features 188/71/4; clippy -D warnings and fmt --check green on all four configurations; cargo doc clean. The literal kv-less pg-lo config re-confirmed structurally uncompilable (d69d9a3's finding; not a gate).
This commit is contained in:
1 parent
c8fc519978
commit
cddcea3cfa
4 files changed
+1305
-115
No files matched your search
@@ -66,11 +66,13 @@
|
||||
//!
|
||||
//! The `#![allow(dead_code)]` below covers the companion-seam scope:
|
||||
//! the window family has its in-tree callers (store construction runs
|
||||
//! `ensure_gc_schema` on every SQL-backed kv engine; the sweep's durable
|
||||
//! form drives `window_stage`/`window_take`), while the fleet family
|
||||
//! waits for wave 6 — its postgres clause bodies landed with wave 4,
|
||||
//! but the store core consumes the family only under the `fleet: true`
|
||||
//! declaration. It shrinks as that lands.
|
||||
//! `ensure_gc_schema` on the window's host — the kv engine when it is
|
||||
//! SQL-backed, pg-lo when the kv engine is not (the one-window-host
|
||||
//! rule, ADR-012 §5); the sweep's durable form drives
|
||||
//! `window_stage`/`window_take` on that same host), while the fleet
|
||||
//! family waits for wave 6 — its postgres clause bodies landed with
|
||||
//! wave 4, but the store core consumes the family only under the
|
||||
//! `fleet: true` declaration. It shrinks as that lands.
|
||||
|
||||
#![allow(dead_code)]
|
||||
|
||||
|
||||
+17
-18
@@ -52,9 +52,9 @@
|
||||
//! 39–43 MB/s from 512 KiB chunks and collapse below (6 MB/s @8 KiB).
|
||||
//! [`LOWRITE_CHUNK`] is the named shipped constant (C2's citation),
|
||||
//! covering the whole-value put directly and the staging cursor's
|
||||
//! append aggregation (the store core's put loop hands
|
||||
//! `crate::dispatch::PUT_CHUNK`-sized 64 KiB windows; the engine
|
||||
//! aggregates appends into 512 KiB `lowrite` statements). [`READ_CHUNK`]
|
||||
//! append aggregation (the store core's put loop hands 64 KiB windows;
|
||||
//! the engine aggregates appends into 512 KiB `lowrite` statements).
|
||||
//! [`READ_CHUNK`]
|
||||
//! bounds the `lo_get` window: larger windows don't beat the
|
||||
//! server-side per-statement ceiling (~65–70 MB/s per in-flight
|
||||
//! statement, C2/C3) and the int4 length parameter caps at 2 GiB.
|
||||
@@ -175,16 +175,15 @@
|
||||
//! Nothing of [`crate::gc::EngineFleetState`] exists on this engine
|
||||
//! (ADR-012 §5's pg-lo row: pin clauses **no** — fleet pins live on
|
||||
//! the kv host; sweep lock **no**). The window-family seam clauses
|
||||
//! ([`crate::gc::EngineWindowState`]) are implemented here (the
|
||||
//! engine-pg-lo-window task): `ensure_gc_schema` creates `gc_window`
|
||||
//! only — never `gc_pins`, never the embedder's liveness table — and
|
||||
//! the window clauses are conditional per the one-window-host rule
|
||||
//! (dormant on a dual-SQL instance — kv=sqlite/postgres + large=pg-lo
|
||||
//! stages on the kv engine; active on the kv=mem + large=pg-lo shape).
|
||||
//! Whether the store core drives them at construction is the
|
||||
//! constructor wiring's business (`store-pg-lo-construction`); this
|
||||
//! module ships the clauses, their engine-level contract tests, and
|
||||
//! the rest of the engine (unit tests only).
|
||||
//! ([`crate::gc::EngineWindowState`]) are implemented here and wired
|
||||
//! by construction (landed with the constructor wiring): the store
|
||||
//! core's construction call follows the one-window-host rule — active
|
||||
//! on the kv=mem + large=pg-lo shape (this module's `ensure_gc_schema`
|
||||
//! runs at construction; its `gc_window` hosts the durable window),
|
||||
//! dormant on a dual-SQL instance (kv=sqlite/postgres + large=pg-lo:
|
||||
//! the kv host's call runs, this schema carries no `gc_window`);
|
||||
//! `ensure_gc_schema` creates `gc_window` only — never `gc_pins`,
|
||||
//! never the embedder's liveness table.
|
||||
//!
|
||||
//! Tests need a live server: connection settings ride the environment
|
||||
//! (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`); tests skip cleanly
|
||||
@@ -455,11 +454,11 @@ impl EngineWindowState for PgLoBackend {
|
||||
/// `gc_pins` is never created here (ADR-012 §5: fleet pins live on
|
||||
/// the kv host) and the embedder's liveness table is never created
|
||||
/// in any case (ADR-008's schema-ownership rule). Idempotent, so
|
||||
/// the twice-at-open call form is a no-op. Whether the store core
|
||||
/// calls this at construction is the one-window-host rule's
|
||||
/// business (`store-pg-lo-construction` wires the active host) —
|
||||
/// the clause body exists and is contract-tested per applicability
|
||||
/// regardless of dormancy.
|
||||
/// the twice-at-open call form is a no-op. The store core calls
|
||||
/// this at construction exactly when this engine hosts the window
|
||||
/// (the one-window-host rule's large-host arm — kv = mem +
|
||||
/// large = pg-lo); on a dual-SQL instance the kv host's call runs
|
||||
/// and this stays dormant, never invoked at construction.
|
||||
async fn ensure_gc_schema(&self) -> Result<(), Error> {
|
||||
let client = self.pool.get().await.map_err(err)?;
|
||||
client
|
||||
|
||||
+1165
-82
File diff suppressed because it is too large.
Load diff
@@ -1,7 +1,7 @@
|
||||
---
|
||||
id: store-pg-lo-construction
|
||||
name: pg-lo constructor wiring — the consolidation shape and the window-host completion
|
||||
status: pending
|
||||
status: completed
|
||||
depends_on: [engine-pg-lo, engine-pg-lo-window]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
@@ -125,27 +125,27 @@ the durable window gains its large-engine host arm.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `LargeEngine::PgLo` slots in; all six large ops dispatch; the
|
||||
- [x] `LargeEngine::PgLo` slots in; all six large ops dispatch; the
|
||||
large cursor/staging bodies slot through the recorded sub-enum
|
||||
design point (the store.rs notes updated to the landed state)
|
||||
- [ ] `dual_tier_postgres_pglo` + `dual_tier_mem_pglo` constructors
|
||||
- [x] `dual_tier_postgres_pglo` + `dual_tier_mem_pglo` constructors
|
||||
exist behind feature `pg-lo`, take connection settings + opts +
|
||||
threshold + window delay, fallible where I/O is fallible; the
|
||||
consolidation constructor's doc records the implicitly-non-fleet
|
||||
posture and the fleet-valid-set completion
|
||||
- [ ] `ensure_gc_schema` follows the one-window-host rule at
|
||||
- [x] `ensure_gc_schema` follows the one-window-host rule at
|
||||
construction (kv host on dual-SQL — pg-lo schema carries no
|
||||
`gc_window`; pg-lo host on kv=mem — contract-tested both shapes)
|
||||
- [ ] The sweep form branch runs the durable form on the pg-lo host
|
||||
- [x] The sweep form branch runs the durable form on the pg-lo host
|
||||
for kv=mem + large=pg-lo and the inline form otherwise; the
|
||||
branch keys off the engine representation, never a flag
|
||||
- [ ] The connection-settings design point (two DSN+opts pairs; the
|
||||
- [x] The connection-settings design point (two DSN+opts pairs; the
|
||||
consolidation corollary) decided and recorded
|
||||
- [ ] The pg-lo pure builds (`--no-default-features --features
|
||||
- [x] The pg-lo pure builds (`--no-default-features --features
|
||||
"pg-lo,mem"` and `--features "postgres,pg-lo"`) compile clean
|
||||
(the literal kv-less config is not a gate — see the Description
|
||||
bullet)
|
||||
- [ ] `cargo test` (default) green; `cargo test --features pg-lo` and
|
||||
- [x] `cargo test` (default) green; `cargo test --features pg-lo` and
|
||||
`--features "pg-lo,mem"` green server-less (skipping);
|
||||
`cargo clippy --all-targets -- -D warnings` and `cargo fmt
|
||||
--check` green (default + the pg-lo feature variants)
|
||||
@@ -174,8 +174,114 @@ the durable window gains its large-engine host arm.
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
> Implemented in `src/store.rs` + `src/pg_lo.rs` + `src/gc.rs` (wave 5,
|
||||
> following `store-postgres-construction` — the structural twin — point
|
||||
> for point). Recorded design points, as resolved:
|
||||
>
|
||||
> - **The large-tier sub-enums landed** — `LargeCursor` and
|
||||
> `LargeStaging` behind `AnyCursor`/`AnyStaging`'s one large arm
|
||||
> (the `KvCursor` pattern's large-tier twin, on the same
|
||||
> bounded-variant argument; recorded for the review gate): local +
|
||||
> pg-lo variants, `LargeStaging` carrying a scoped
|
||||
> `#[allow(clippy::large_enum_variant)]` (the file-descriptor
|
||||
> posture vs the held pooled checkout + tx — payloads structurally
|
||||
> dissimilar, dispatch by reference/consumption; the `KvEngine`
|
||||
> enum's recorded allow posture). No reshaping of `AnyCursor`/
|
||||
> `AnyStaging`/`Blob`.
|
||||
> - **The one-window-host rule's store-core completion** —
|
||||
> `ensure_gc_schema_at_construction(kv, large)`: the store-core
|
||||
> dispatch the constructors call, keyed off the engine
|
||||
> representation (the kv arm on a SQL-backed kv engine; the large
|
||||
> arm consulted only when the kv engine is not SQL-backed; in-
|
||||
> process combinations structurally nothing). The sqlite and
|
||||
> postgres constructors were re-routed through it (`kv_only_*` pass
|
||||
> `None`; the dual-tier locals pass `Some(&large)`) — their
|
||||
> semantics unchanged, and the mem arm's large-host branch is where
|
||||
> the rule completes. The sweep's form branch gained exactly that
|
||||
> arm: `KvEngine::Mem` matches on `self.large()` — `LargeEngine::PgLo`
|
||||
> runs `sweep_window_durable` on the pg-lo host (already generic
|
||||
> over `W: EngineWindowState`, so the arm is a delegation), anything
|
||||
> else stays inline; sqlite/postgres arms untouched.
|
||||
> - **Connection-settings design point** — the consolidation
|
||||
> constructor takes **two DSN + opts pairs** (kv first, large
|
||||
> second): cross-instance is valid non-fleet (ADR-012 §4), and the
|
||||
> consolidation shape passes the same DSN/opts to both. Recorded
|
||||
> with it: the suite factory's corollary (one minted schema passed
|
||||
> to both engines vs two schemas — both engines' table names are
|
||||
> distinct, so either mint is legitimate). The mem-pglo constructor
|
||||
> takes one pair (kv = mem takes none).
|
||||
> - **No `kv_only(pg-lo)`** (pg-lo is a large engine) and **no
|
||||
> sqlite+pg-lo constructor this wave** (valid non-fleet per
|
||||
> ADR-012 §5, not named by the plan) — both recorded in the
|
||||
> store.rs module doc's constructor-modes section.
|
||||
> - **Stale scope notes updated to the landed state** — the
|
||||
> engine-representation doc's wave references, the dual-tier
|
||||
> shapes' wave bullets, the fleet-declaration paragraph (the
|
||||
> fleet-valid engine set is complete in the constructors — wave 6
|
||||
> is the declaration's predicate and parameter alone), the
|
||||
> one-window-host paragraph (dormancy landed; the large-host arm is
|
||||
> the reshaping the rule always named), the `LargeEngine` doc, the
|
||||
> `KvEngine::ensure_gc_schema` doc (now the kv-host arm the helper
|
||||
> consumes), the sweep's docs (two forms' host sets), the
|
||||
> `PUT_CHUNK` note (the engine's `LOWRITE_CHUNK` constant is the
|
||||
> authority — landed), and the pg_lo/gc.rs module docs' forward
|
||||
> references. The batch-delete helper's wave-6 note stays.
|
||||
> - **Clippy scope note** — one `doc list item without indentation`
|
||||
> fixup during the gate: a doc line beginning `+ large=pg-lo)`
|
||||
> re-wrapped (the `+` was a markdown list marker).
|
||||
> - **Tests (7 new, env-driven `ALKBLOBS_PG_*`, skipping cleanly
|
||||
> server-less, janitor discipline from the start — amendment A2)** —
|
||||
> two modules keyed to the constructor feature gates. The mem-pglo
|
||||
> module (4 tests, on per-test dedicated databases — the C7 lesson's
|
||||
> shape — with `DatabaseJanitor`): construction + introspection +
|
||||
> the pg-lo-host schema assertions (`gc_window` present, `gc_pins`
|
||||
> never); the end-to-end round-trip with LO placement asserted (kv
|
||||
> entries never in the pg database, companion row present, size
|
||||
> exact, LO content byte-exact); the window-delay
|
||||
> default/tunings/zero; and the durable-form sweep smoke (zero
|
||||
> delay: first sweep stages into the pg-lo schema's `gc_window` and
|
||||
> executes nothing — the form-branch-is-large-host proof — second
|
||||
> sweep executes the overdue set re-arbitrated at execution time).
|
||||
> The consolidation module (3 tests, same dedicated-database
|
||||
> discipline): construction + the kv-host-wins assertions (both GC
|
||||
> tables in the kv schema, no `gc_window` in the pg-lo schema — two
|
||||
> distinct schemas so the dormancy assertion observes genuine
|
||||
> absence); the round-trip in the recorded corollary form (the same
|
||||
> DSN + one minted schema to both engines, LO placement asserted);
|
||||
> and the window-delay tunings. The test-side `plant_large` helpers
|
||||
> in the existing test modules gained `PgLo` arms (the same
|
||||
> all-features-exhaustiveness pattern as wave 4's `KvEngine` arms).
|
||||
> - **The literal kv-less config (`--no-default-features --features
|
||||
> pg-lo`) remains structurally uncompilable** (E0004 on the empty
|
||||
> kv set, by-reference match arms requiring inhabitated variants)
|
||||
> — the d69d9a3 finding, not a gate; verification used the two
|
||||
> meaningful pure builds, both clean.
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
> Completed 2026-10-12 (wave 5). pg-lo slotted into the facade:
|
||||
> `LargeEngine::PgLo` with all six large-op dispatch arms, the
|
||||
> `LargeCursor`/`LargeStaging` sub-enums behind `AnyCursor`/`AnyStaging`'s
|
||||
> one large arm (the recorded `KvCursor` pattern's large-tier twin),
|
||||
> the one-window-host rule's store-core completion
|
||||
> (`ensure_gc_schema_at_construction` — kv host on dual-SQL, pg-lo
|
||||
> host on kv=mem; the sweep's mem arm gains the large-host delegation,
|
||||
> the branch keyed off the engine representation never a flag), and
|
||||
> the two named constructors (`dual_tier_postgres_pglo` — the
|
||||
> consolidation posture, two DSN+opts pairs, implicitly non-fleet,
|
||||
> completing the fleet-valid engine set — and `dual_tier_mem_pglo` —
|
||||
> the one-window-host rule's named test topology), both carrying the
|
||||
> constructor-tunable delete-window delay (`DEFAULT_DELETE_WINDOW_
|
||||
> DELAY` pinned at 5 min, `0` legal). Stale scope notes updated to the
|
||||
> landed state (store.rs module doc, `LargeEngine`, the sweep docs,
|
||||
> `PUT_CHUNK`, pg_lo.rs, gc.rs). Verification (harness server live
|
||||
> 2026-10-12): `cargo test` 69/19/3 ok (default); `cargo test
|
||||
> --features pg-lo` 92/19/3 ok live and green server-less; `cargo test
|
||||
> --features "pg-lo,mem"` 151/54/4 ok live, 124/37/4 ok server-less;
|
||||
> `cargo test --no-default-features --features "postgres,pg-lo"`
|
||||
> 102/19/3 ok live and server-less; `cargo test --all-features`
|
||||
> 188/71/4 ok (7 new constructor tests green live); clippy `-D
|
||||
> warnings` green on default, all-features, `pg-lo,mem`, and
|
||||
> `postgres,pg-lo`; `cargo fmt --check` green; `cargo doc --no-deps`
|
||||
> clean. The literal kv-less config re-confirmed structurally
|
||||
> uncompilable (d69d9a3's finding; not a gate).
|
||||
Reference in new issue
Block a user