From 5a217f7bc7c7e7b69a6e20fd6c80ccbb3f08dab3 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Mon, 12 Oct 2026 07:13:51 +0000 Subject: [PATCH] =?UTF-8?q?docs(tasks):=20decompose=20wave=205=20=E2=80=94?= =?UTF-8?q?=20the=20pg-lo=20large=20engine,=20the=20window=20host's=20larg?= =?UTF-8?q?e-tier=20arm,=20the=20LO-medium=20SKIP-posture=20column?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - engine-pg-lo: the Backend impl behind feature pg-lo (default-off) — companion-table authority, one-tx staged put at the 512 KiB knee, descriptorless lo_get windows, the orphan-recovery sweep, the posture deltas recorded; the shared-helper/PgOpts/LO-isolation design points named (LOs are database-level — the C7 lesson cited so it is not rediscovered) - engine-pg-lo-window: the EngineWindowState impl per ADR-012 §5's pg-lo row — gc_window only, no fleet family anywhere - store-pg-lo-construction: LargeEngine::PgLo slotting (the large sub-enum design point), the consolidation + mem-pglo constructors (the fleet-valid set completed, implicitly non-fleet), and the one-window-host rule's store-core completion (dormancy + the large-host sweep arm) - suite-column-pg-lo: the second SKIP-posture column, two factories spanning feature pairs, the dormancy row, zero LO residue in teardown - review-wave-5: the engine-wave gate; amendment A2 (janitor adoption) verified here Harness server cited in every server-touching task: dockerized pglo-poc (postgres:16-alpine) on :15432, db blobs, postgres/poc (verified live 2026-10-12); env-driven ALKBLOBS_PG_* settings, clean server-less skips, acceptance requires green against the server. Verification: cargo test 69/19/3 ok, clippy --all-targets -D warnings green, fmt --check green (docs-only change). --- docs/plans/implementation.md | 4 +- tasks/engine-pg-lo-window.md | 114 ++++++++++++++ tasks/engine-pg-lo.md | 237 ++++++++++++++++++++++++++++++ tasks/review-wave-5.md | 120 +++++++++++++++ tasks/store-pg-lo-construction.md | 166 +++++++++++++++++++++ tasks/suite-column-pg-lo.md | 146 ++++++++++++++++++ 6 files changed, 785 insertions(+), 2 deletions(-) create mode 100644 tasks/engine-pg-lo-window.md create mode 100644 tasks/engine-pg-lo.md create mode 100644 tasks/review-wave-5.md create mode 100644 tasks/store-pg-lo-construction.md create mode 100644 tasks/suite-column-pg-lo.md diff --git a/docs/plans/implementation.md b/docs/plans/implementation.md index 5b55298..632eb49 100644 --- a/docs/plans/implementation.md +++ b/docs/plans/implementation.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: "2026-10-12 (wave 4 decomposed into tasks/)" +last_updated: "2026-10-12 (wave 5 decomposed into tasks/)" --- # alkblobs — Implementation plan @@ -52,7 +52,7 @@ ops payload surfaces; release readiness is last. | [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | decomposed (2026-10-11) | | [3](#wave-3--sqlite-engine) | sqlite kv engine (default): WAL posture, prepared statements, engine-state window family, suite column | wave 2 | decomposed (2026-10-11) | | [4](#wave-4--postgres-kv-engine) | postgres kv engine: schema, pool, CAS rows, stable-cursor list, full engine-state clause families, suite column | wave 2 | decomposed (2026-10-12) | -| [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | pending | +| [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | decomposed (2026-10-12) | | [6](#wave-6--fleet-gc) | Fleet GC mechanism: DB-backed pins, joint entry+pin tx, sweep lease, SQL delete arbitration, fleet-validity predicate + constructor declarations | waves 4 + 5 | pending | | [7](#wave-7--ops-surface) | `ops` feature: alkcall-backed op family, ACL mapping, pin tokens + renewal, fanout seam | wave 6 | pending | | [8](#wave-8--fuzzing) | Fuzzing: family pattern targets (key encoding, ops payloads), corpus-replay gate | wave 7 | pending | diff --git a/tasks/engine-pg-lo-window.md b/tasks/engine-pg-lo-window.md new file mode 100644 index 0000000..8679282 --- /dev/null +++ b/tasks/engine-pg-lo-window.md @@ -0,0 +1,114 @@ +--- +id: engine-pg-lo-window +name: pg-lo engine-state window family — the conditional window host +status: pending +depends_on: [engine-pg-lo] +scope: narrow +risk: low +impact: component +level: implementation +tags: [wave-5, engines, pg-lo, gc] +--- + +## Description + +Implement the engine-state companion seam's **window family** on the +pg-lo backend (`src/pg_lo.rs`): the `EngineWindowState` impl per +ADR-012 §5's pg-lo row — `ensure_gc_schema` creates **`gc_window` +only**, the window clauses **conditional** per the one-window-host +rule. **Nothing of `EngineFleetState` may exist on this engine** (the +pg-lo row: pin clauses **no** — fleet pins live on the kv host, §4; +sweep-lock clause **no**). The postgres engine's full-family impl +(`src/postgres.rs`, wave 4) and the sqlite window-only impl +(`src/sqlite.rs`, wave 3) are the reference implementations — same +semantics, different dialect. + +- **`ensure_gc_schema`** — creates **`gc_window` only** + (`gc_window (key bytea PRIMARY KEY, visible_after bigint)` per + ADR-010 §2's schema, in the engine's own schema). Never `gc_pins` + (contract-tested absent); the embedder's liveness table is never + created here (ADR-008's schema-ownership rule — the sweep reads it + and never owns it). **Whether the store core calls this clause at + construction is the one-window-host rule's business** + (`store-pg-lo-construction` wires it): the window hosts on the kv + engine when it is SQL-backed (pg-lo's clauses **dormant** on a + dual-SQL instance — kv=sqlite/postgres + large=pg-lo), and on pg-lo + only when the kv engine is not SQL-backed (the kv=mem + large=pg-lo + shape). The clause body exists and is contract-tested per + applicability regardless of dormancy. +- **`window_stage`** — stage candidates into the durable window, + **idempotent per `(key, deadline)`** (the upsert discipline — a + same-deadline re-stage is idempotent, a new-deadline re-stage updates + the row; the sqlite/postgres engines' recorded semantics). +- **`window_take(now)`** — take overdue candidates + (`visible_after <= now`, the `<=` boundary the seam doc pins): the + rows **leave the window table and return as keys** — one atomic step + (`DELETE … RETURNING`, selection and removal together; a row is + never handed out twice). Empty when nothing is overdue; virgin + store: empty. +- **Timestamp encoding** — the crate-wide encoding: signed i64 + nanosecond count from the Unix epoch, carried as `BIGINT` + (sub-second precision for the test tunings, pre-epoch negatives, + full i64 range); per-engine bodies of the shared helpers (the + shared-helper home decided in `engine-pg-lo`). +- **Virgin-store tolerance split** — the postgres engine's recorded + posture: `window_take` is a read (SQLSTATE 42P01 maps to empty on a + schema-less database); `window_stage` is a write (surfaces the + engine error — never silently creating schema; store construction + always wires `ensure_gc_schema` when the engine hosts). +- **Contract tests** per the applicability row (ADR-012 §5's table is + the contract-test matrix): the schema creates the window table only + (assert `gc_pins` does not exist); stage idempotence per + `(key, deadline)` incl. the re-stage-at-new-deadline form; take + returns exactly the overdue set, removes it, leaves future rows; the + `<=` boundary incl. nanosecond slack; pre-epoch encoding; both + virgin shapes. + +The store core drives these clauses only on the host topology +(`store-pg-lo-construction`); this task ships the clauses and their +engine-level contract tests. + +## Acceptance Criteria + +- [ ] The pg-lo backend implements `EngineWindowState` — the window + family only; no pin-clause or sweep-lock body exists anywhere in + the pg-lo engine (nothing of `EngineFleetState`) +- [ ] `ensure_gc_schema` creates `gc_window` only; `gc_pins` is never + created by pg-lo (contract-tested); the embedder's liveness + table is never created +- [ ] `window_stage` idempotent per `(key, deadline)`; a re-stage at a + new deadline updates the row +- [ ] `window_take` returns exactly the overdue set (`<= now` + boundary), removes it atomically, leaves future rows; empty on + none/virgin +- [ ] Contract tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly + server-less; green against the harness server (`pglo-poc`, + :15432, db `blobs`, `postgres`/`poc`) at acceptance; the janitor + discipline rides from the start (amendment A2) +- [ ] `cargo test` (default) green; `cargo test --features pg-lo` + green server-less (skipping); `cargo clippy --all-targets -- -D + warnings` and `cargo fmt --check` green (default + `--features + pg-lo`); the pg-lo-pure check clean + +## References + +- src/gc.rs (`EngineWindowState` — the seam contract and its clause docs) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md + §5 (the applicability table — the pg-lo row; the one-window-host + rule), §6.7 (the executor) +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md + §2 (the clause bodies; the gc_window schema) +- docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md + §2 (the durable window's semantics) +- src/sqlite.rs, src/postgres.rs (the window family's reference + implementations — same semantics, different dialects) +- tasks/engine-pg-lo.md (the driver posture this rides; the + shared-helper design point) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/engine-pg-lo.md b/tasks/engine-pg-lo.md new file mode 100644 index 0000000..96ea691 --- /dev/null +++ b/tasks/engine-pg-lo.md @@ -0,0 +1,237 @@ +--- +id: engine-pg-lo +name: pg-lo large engine — Large Objects behind the Backend trait +status: pending +depends_on: [] +scope: moderate +risk: medium +impact: component +level: implementation +tags: [wave-5, engines, pg-lo, postgres] +--- + +## Description + +Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off; +ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base) +against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the +2-line module-doc stub behind the pre-existing +`#[cfg(feature = "pg-lo")]` gate). The postgres kv engine +(`src/postgres.rs`, wave 4) is the structural twin for the driver +posture (pool, prepared statements, schema bootstrap, error mapping); +the local engine (`src/local.rs`) is the structural twin for the +large-tier bodies (streaming read cursor, staged put). **`pg-lo` does +not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml +feature carries its own driver deps) — every shared piece must +compile under `--no-default-features --features pg-lo` (the pg-lo-pure +build). + +- **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs + hold the bytes; the companion table + `lo_entries (key bytea PRIMARY KEY, loid oid, size bigint, + committed_at)` is the **contract authority** — `list()`, `size()`, + `has()`, CAS-existence are all table-level. **Never trust or read + `pg_largeobject` for addressing** (the C6 sweep validates against + the table only). One engine-owned PostgreSQL schema (the postgres + engine's bootstrap pattern: idempotent DDL at the constructor, the + quoted-identifier boundary, no migration machinery). The default + schema name and the consolidation shape's schema sharing (both + engines on one pg instance) are **recorded design points**. +- **Driver posture (the postgres engine's, reused)** — deadpool-postgres + over tokio-postgres, pool size configurable with a named shipped + default, **per-connection `prepare_cached` discipline** (statements + die with sessions), natively async — **no `spawn_blocking` seam**, no + blocking I/O on the async path. **Shared-helper design point**: the + postgres engine's crate-internal helpers (`quote_identifier`, the + SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp + helpers) live behind `#[cfg(feature = "postgres")]` today — decide + and record the sharing shape (the natural one: a crate-internal + pg-common module gated `cfg(any(feature = "postgres", + feature = "pg-lo"))`, one module per file, so the pg-lo-pure build + compiles without duplication). The `PgOpts` question rides the same + point: it is `pub` and re-exported behind `postgres` (wave 4's + constructor signature put it on the public surface) — widening its + gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a + pg-lo-local opts type is a recorded design point; whether pg-lo + carries the `synchronous_commit` knob at all (the POC ran the server + with it off; the kv parity knob's posture may or may not carry) is + recorded with it. +- **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` → + chunked `lowrite` → companion-row insert → COMMIT. **512 KiB + statements** (POC #7 C2's measured knee; statement count, not + LOBLKSIZE, is what costs) as a named shipped constant. CAS: + companion-table existence check first — a re-put of an existing key + is a no-op tx, never an overwrite (the C1 shape). The store core's + put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates + appends into 512 KiB `lowrite` statements (recorded design point). +- **Staged put (the engine-level `put_staged` contract)** — the + staging cursor body is a **held tx**: handout opens the checkout + + tx + `lo_create`; `append` rides the chunked `lowrite` aggregation; + `commit` publishes (companion-row insert + COMMIT — the row and the + LO's pages land in one step, zero crash orphans structurally, C6); + `discard` rolls back and returns the checkout. Every failure path + converges on rollback — stage hygiene holds in both forms (discard → + zero residue: no LO pages, no row; failed commit → self-discard), + exact-count tested (the `#[cfg(test)]` fail-commit knob standing in + for media failure, the mem/sqlite/postgres pattern). **Scope + boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet* + joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's + ruling that pg-lo contributes LO writes + companion-row insert into + the store-held tx); its API shape is a wave-6 design point. This + task ships the engine-level staging cursor only. +- **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3, + the C4 posture)** — the read cursor closes over the companion row + (oid + length; content immutable, so the row is authoritative), not + over a tx-scoped descriptor. Range reads are single stateless + statements, pool-friendly. `len()` reads the companion row — + **metadata only, never LO content** (ADR-008's discipline). A range + read beyond content length fails cleanly (`UnexpectedEof`-shaped → + the engine's `Error::Io` mapping; the POC's + `ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*` + 64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any + positioning is used; `lo_get`'s bigint offsets already are — the C7 + int4 trap must not be inherited). The held-descriptor tx posture + exists as the fallback (measured: no advantage even for sequential + full reads) — named in the module doc as not-shipped, never coded. +- **`has`** — companion-table EXISTS probe. **`size`** — the + companion-row `size` probe (metadata-only; the ADR-009 companion-row + probe the suite's INV-6 row names). **`delete`** — one tx + (`lo_unlink` + row delete), rows-affected discipline + (`Err(Error::Missing)` when absent); GC-participating (the delete + window's arbitration is store-core and table-level — nothing + fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`, + complete (the POC's exact-count gate: list == companion table after + mixed ops). +- **Virgin-store posture** — the constructor creates the schema and + table; the tolerance is the postgres engine's recorded read-only + split (reads on a schema-less database map to absent/empty via the + SQLSTATE 42P01 discriminant, writes surface the engine error). The + schema-less test fixture mirrors the postgres engine's + `#[cfg(test)] new_schemaless` pattern. +- **Posture deltas in the module doc, stated not hidden (ADR-009 + §5/§6)** — LO catalog space is reused but never returned + (`pg_total_relation_size('pg_largeobject')` monitoring named; + `VACUUM FULL`/`pg_repack` the only shrink path — an operator + runbook line, never automatic); autovacuum inherited, keep it on; + single-stream cached get 20–50× behind page-cache local fs at + ≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture — + named capacity facts, not solved problems. +- **Orphan-recovery sweep (ADR-009 §5's requirement)** — a + crate-internal `reap_orphans()` for the legacy/bypass class only + (committed LOs without companion rows): enumerate + `SELECT DISTINCT loid FROM pg_largeobject` as a candidate list + **only**, check each against `lo_entries.loid` (the truth), + `lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with + oid count, not content size). Scope-noted `#[allow(dead_code)]` + until a consumer exists (recorded: the ops/maintenance surface is + wave 7+; no store-core verb is scoped this wave), contract-tested + with a planted orphan and a byte-exact survival assertion for + committed content. +- **Unit tests** in-module: the method contract (virgin-store no-ops + incl. the schema-less form, CAS re-put no-op, Missing discipline, + size exactness + metadata-only, list completeness after mixed ops, + ranged reads at window boundaries + beyond-length failure), the + staged-put happy path and both hygiene forms, the orphan sweep. + **Tests need a live server** — the alkstore harness convention: + connection settings ride the environment + (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly + server-less, acceptance requires green against the server. The + harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on + :15432, db `blobs`, user/password `postgres`/`poc` (verified live + 2026-10-12). +- **LO isolation on the shared harness db (named design point — the + C7 lesson, do not rediscover it)**: Large Objects are + **database-level**, not schema-level — the schema-per-test + convention does not isolate them, a `DROP SCHEMA` leaves the LO's + pages in `pg_largeobject`, and shared-LO tests raced each other into + spurious "large object NNNN does not exist" aborts in the POC. Decide + and record the isolation shape: per-test `CREATE DATABASE` (the + POC's proven `open_fresh` shape) or per-instance LOID tracking + + teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor + pattern extended). Either way acceptance includes **zero LO residue + on the shared db** (`pg_largeobject` probed before/after full runs). +- **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the + schema-janitor discipline from the start; the wave-4 lesson (380 + leaked schemas) is not to be re-learned. The `SchemaJanitor` lives + in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared + home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the + shared-helper design point. + +The window-family seam clauses (`EngineWindowState`) are +`engine-pg-lo-window` (separate task); the constructor wiring is +`store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist +on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep +lock **no** — fleet pins live on the kv host). This task ships the +engine and its in-module unit tests only. + +## Acceptance Criteria + +- [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods + + `put_staged`) behind feature `pg-lo` (default-off), with the + descriptorless `lo_get` window cursor body and the held-tx + staging body +- [ ] The companion table is the contract authority (list/size/has/CAS + all table-level); `pg_largeobject` is never read for addressing +- [ ] One engine-owned schema, idempotent bootstrap at the + constructor; the quoted-identifier boundary; the virgin-store + read-only split (42P01 → absent/empty on reads, writes surface) + unit-tested incl. the schema-less form +- [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert → + COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named + constant with the C2 citation +- [ ] Stage hygiene holds in both forms (discard → zero residue; failed + commit → rollback self-discard), exact-count unit-tested +- [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only + from the companion row; beyond-length reads fail cleanly; the + 64-bit-variants discipline recorded (the C7 trap not inherited) +- [ ] `delete` = one tx (lo_unlink + row delete) with the Missing + discipline; `list` complete after mixed ops +- [ ] The orphan-recovery sweep exists (candidates from + `pg_largeobject` only, validated against `lo_entries`), + scope-noted, contract-tested with a planted orphan +- [ ] Posture deltas recorded in the module doc (catalog space never + returned + VACUUM FULL operator line + rel-size monitoring; + autovacuum inherited; the cached-get capacity facts) +- [ ] Shared-helper + PgOpts + LO-isolation design points decided and + recorded; the pg-lo-pure build (`--no-default-features + --features pg-lo`) compiles clean +- [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly + server-less; green against the harness server at acceptance; + zero schema + LO residue on the shared db after runs +- [ ] `cargo test` (default — pg-lo compiles out) green; + `cargo test --features pg-lo` green server-less (skipping); + `cargo clippy --all-targets -- -D warnings` and `cargo fmt + --check` green (default + `--features pg-lo` + the pg-lo-pure + check) + +## References + +- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the + engine ADR — §1–§4 the contract, §5/§6 the posture deltas) +- docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence + base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan + sweep, C7 the harness lessons) +- docs/architecture/decisions/003-backend-contract-and-dispatch.md + (the trait contract) +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md + §1 (the staged-put contract; the large-tier cursor body families) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md + §4/§5 (the consolidation posture; the pg-lo applicability row — + window family only, no fleet state) +- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md + §4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build) +- src/backend.rs (the trait + seam contracts this implements) +- src/postgres.rs (the driver-posture twin: pool, prepare_cached, + schema bootstrap, 42P01 split, test_support janitor) +- src/local.rs (the large-tier body twin: streaming cursor, staged + put hygiene shapes) +- /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line + PgLoBackend this engine productionizes) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/review-wave-5.md b/tasks/review-wave-5.md new file mode 100644 index 0000000..f46bab8 --- /dev/null +++ b/tasks/review-wave-5.md @@ -0,0 +1,120 @@ +--- +id: review-wave-5 +name: Wave 5 review gate — the pg-lo engine vs the contract +status: pending +depends_on: [suite-column-pg-lo] +scope: narrow +risk: low +impact: phase +level: review +tags: [wave-5, review] +--- + +## Description + +The wave-5 review gate — the engine-wave review shape +(docs/plans/implementation.md §Review gates: "engine-vs-contract +conformance"), before wave 6 decomposes. The gate reads the +implementation against the pinned ADR text line by line (the +wave-1/2/3/4 gates' method). + +Gate checks: + +- **Engine-vs-contract conformance**: the suite column green against + dockerized postgres (server-less runs skip cleanly — a skip never + substitutes for acceptance); the engine-state clause families + contract-tested per the applicability row (ADR-012 §5's pg-lo row — + the window family only, conditional per the one-window-host rule); + virgin-store semantics; the `size` metadata-only discipline (the + companion-row probe never loads LO content). +- **The engine shape vs ADR-009 §1–§4**: the companion table is the + contract authority (list/size/has/CAS all table-level); + `pg_largeobject` never read for addressing; put = one tx with the + 512 KiB knee as a named constant (the C2 citation); the staged put's + held-tx body with both hygiene forms exact-count tested; gets ride + descriptorless `lo_get(oid, off, len)` windows (the C4 posture — the + held-descriptor fallback named as not-shipped, never coded); the + 64-bit-variants discipline (the C7 int4 trap not inherited); delete + = one tx under table-level arbitration; transactional LO creation + (zero crash orphans structurally — the C6 reading). +- **The posture deltas vs ADR-009 §5/§6**: carried as requirements, + not hedges — catalog space never returned (the `VACUUM FULL` + operator line + rel-size monitoring named in the module doc); + autovacuum inherited, keep it on; the orphan-recovery sweep present + for the legacy/bypass class (scope-noted, contract-tested with a + planted orphan); the cached-get capacity facts named honestly. +- **The one-window-host rule vs ADR-012 §5**: the store-core + completion — `ensure_gc_schema` follows the host rule at + construction (dormancy on dual-SQL: the consolidation shape's pg-lo + schema carries no `gc_window`; the pg-lo host on kv=mem); the sweep + form branch runs the durable form on the pg-lo host for kv=mem + + large=pg-lo and keys off the engine representation, never a flag; + **no fleet state anywhere in pg-lo** (nothing of + `EngineFleetState`); the fleet joint-put contribution's API shape + explicitly deferred to wave 6 (recorded, not silently absent). +- **The consolidation constructor vs ADR-009/011/012**: the + fleet-valid engine set completed; the implicitly-`fleet: false` + posture recorded verbatim (wave 6's declaration slots in without + reshaping); the connection-settings design point (two DSN+opts + pairs; cross-instance valid non-fleet) recorded. +- **Async/blocking discipline**: tokio-postgres natively async — no + `spawn_blocking` seam; no blocking I/O on the async path; the pool + is the concurrency posture; no `unwrap`/`expect` outside tests. +- **Named design points audited as resolved** (each recorded in its + task's Notes): the schema name + the consolidation shape's schema + sharing; the shared-helper home (quote_identifier, 42P01 + discriminant, pg_err, timestamp helpers) + the `PgOpts` gate + question; the 512 KiB aggregation shape; the LO isolation + + janitor shape (amendment A2 verified — zero schema AND zero + `pg_largeobject` residue on the shared db); the sub-enum slotting + for the large cursor/staging bodies; the connection-settings shape; + the dormancy wiring; the orphan sweep's scope note. +- **Family conventions**: no comments in code (doc comments on public + API fine), thiserror, tokio-only async, no unwrap/expect outside + tests, naming, one-module-per-file re-export shape, the + `#![allow(dead_code)]` scope notes matching their remaining scope. +- **Gates green**: `cargo test` (default), `cargo test --features + pg-lo` (server-less: skipping), `cargo test --features "pg-lo,mem"`, + `cargo test --all-features` (against the server), the pg-lo-pure + checks (`cargo check --no-default-features --features pg-lo` and + `--features "pg-lo,mem"`), `cargo clippy --all-targets -- -D + warnings` (default + the pg-lo variants + `--all-features`), + `cargo fmt --check`, `cargo doc --no-deps`. + +Findings are fixed inline where cheap, or recorded as wave-6 +amendments where they shape the next decomposition (the fleet +mechanism's joint-tx driver already carries amendment A1; any pg-lo +contribution-shape finding lands beside it). The gate's completion is +what unblocks wave 6's decomposition. + +## Acceptance Criteria + +- [ ] Every gate check above read and verified against the cited ADR + text (not from memory of it) +- [ ] Named design points each decided, recorded, and consistent with + the specs +- [ ] Findings fixed inline or recorded as wave-6 amendments with + owners +- [ ] All gates re-run green after any fixes +- [ ] Wave 6 decomposition unblocked (recorded in this task's Summary) + +## References + +- docs/plans/implementation.md (§Wave 5, §Review gates — engine wave + reviews) +- docs/architecture/decisions/009-pg-lo-large-tier-engine.md +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1/§2 +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §3/§4/§5/§6.7 +- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §4 +- docs/architecture/store-api.md +- docs/research/poc-pglo-findings.md (the evidence base the engine + claims) +- tasks/review-wave-4.md (the gate method; the structural twin's gate) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/store-pg-lo-construction.md b/tasks/store-pg-lo-construction.md new file mode 100644 index 0000000..5dd582c --- /dev/null +++ b/tasks/store-pg-lo-construction.md @@ -0,0 +1,166 @@ +--- +id: store-pg-lo-construction +name: pg-lo constructor wiring — the consolidation shape and the window-host completion +status: pending +depends_on: [engine-pg-lo, engine-pg-lo-window] +scope: moderate +risk: medium +impact: component +level: implementation +tags: [wave-5, store, pg-lo, construction] +--- + +## Description + +Slot pg-lo into the facade's engine representation — the slotting-in +pattern `src/store.rs`'s module doc pinned ("adding an enum variant and +a named constructor is slotting in"). This task also completes the +one-window-host rule's store-core wiring, which is more than slotting: +the durable window gains its large-engine host arm. + +- **`LargeEngine::PgLo`** variant behind `#[cfg(feature = "pg-lo")]` + + the six large-op dispatch arms (`put_staged`/`get`/`has`/`size`/ + `delete`/`list`). +- **The large cursor bodies slot in** — `AnyCursor::Large` and + `AnyStaging::Large` are currently **concrete over the local + bodies** (`LocalCursor`/`LocalStaging`). **Named design point (the + recorded `KvCursor` sub-enum pattern's large-tier twin)**: grow a + `LargeCursor`/`LargeStaging` sub-enum behind `AnyCursor`/`AnyStaging`'s + one large arm (local + pg-lo variants) rather than one `AnyCursor` + variant per large engine — the same bounded-memory argument as the + kv side, recorded for the review gate, with the store.rs design-point + notes updated to the landed state. +- **Named constructors**, feature-gated on `pg-lo` (default-off): + - **`dual_tier_postgres_pglo`** — kv = postgres + large = pg-lo: the + **consolidation posture** (ADR-009's "one engine to operate"); its + landing **completes the fleet-valid engine set** (the store.rs + module doc's recorded forward reference — pg-lo's wave-5 + constructor is the last fleet-valid engine). **Implicitly + `fleet: false`** (ADR-012 §3): the ADR-005 in-process protocol + applies verbatim; the `fleet: true` declaration is wave 6 (the + composite fleet-validity predicate + constructor declarations) + slotting in without reshaping — no fleet parameter carried; + recorded in the constructor doc. **Connection-settings design + point**: two DSN + opts parameter pairs (kv and large) — + cross-instance kv=postgres + large=pg-lo is valid non-fleet + (ADR-012 §4), and the consolidation shape passes the same + DSN/opts to both; the suite factory's schema-sharing corollary + (one minted schema passed to both engines vs two schemas) is + recorded with it. + - **`dual_tier_mem_pglo`** — kv = mem + large = pg-lo (features + `mem` + `pg-lo`): **the one-window-host rule's named test + topology** (ADR-012 §5: the durable window hosts on the large + engine when the kv engine is not SQL-backed). Non-fleet (mem is + never fleet-valid). + - **No `kv_only(pg-lo)`** — pg-lo is a large engine; kv-only modes + take kv engines only. **No sqlite+pg-lo constructor this wave** — + valid non-fleet per ADR-012 §5 but not named by the plan; added on + demand (recorded). + - Both take the **delete-window delay** parameter (the + `DEFAULT_DELETE_WINDOW_DELAY` signature shape; `0` the legal + degenerate tuning) — the durable form's cadence knob on either + host. +- **The one-window-host rule's store-core completion** (the substance + of this task): + - **`ensure_gc_schema` at construction** — the call currently + dispatches on the kv-engine enum only. It must follow the host + rule: kv SQL-backed (sqlite/postgres) → the kv host's call (pg-lo's + window clauses dormant — the consolidation shape's pg-lo schema + carries **no `gc_window`**); kv = mem + large = pg-lo → the + **pg-lo host's call** (`gc_window` in the pg-lo schema). The + dispatch consults the large engine only when the kv engine is not + SQL-backed; it keys off the engine representation, never a flag. + - **The sweep form branch** — currently `KvEngine::Mem` runs the + inline form unconditionally. It gains the large-host arm: mem kv + + `LargeEngine::PgLo` runs the **durable form on the pg-lo host** + (`sweep_window_durable` is already generic over + `W: EngineWindowState` — the pg-lo arm slots in); mem kv + + `LargeEngine::Local` (or no large tier) stays inline. The + sqlite/postgres arms are untouched (their hosts win the window per + the rule). The module doc's recorded "pg-lo dormancy slots in + without reshaping" note is updated to the landed state (dormancy + did; the large-host arm is the reshaping the rule always named). +- **Stale scope notes / forward references updated** — the store.rs + module doc's wave-5 mentions (the engine-representation note, the + fleet-valid-set note, the one-window-host paragraph, the + `LargeEngine` doc's "pg-lo slots in at wave 5"), the `PUT_CHUNK` doc + note ("pg-lo's measured 512 KiB `lowrite` knee is that engine wave's + concern" — now landed; the engine's constant is the authority), and + any `#![allow(dead_code)]` scope notes whose scope changes (the + batch-delete helper's wave-6 note stays). +- **Constructor unit tests**: the modes construct and introspect + correctly (`mode()`/`threshold()`); `ensure_gc_schema` ran on the + **right host** in both shapes (consolidation: `gc_pins` + `gc_window` + on the kv schema, **no `gc_window` on the pg-lo schema** — the + dormancy assertion; mem-pglo: `gc_window` on the pg-lo schema); the + window-delay default is the pinned constant, tunable, `0` legal; the + dual-tier shapes round-trip end-to-end (a smoke put/get through both + tiers with LO placement asserted — the companion row + LO present, + kv entries not in pg); a mem-pglo durable-form sweep smoke + (zero-delay: first sweep stages into the pg-lo schema's `gc_window` + and executes nothing, second sweep executes the overdue set + re-arbitrated at execution time — the form-branch-is-large-host + proof). Env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less + (the engine tests' convention); the janitor discipline rides from + the start (amendment A2). +- **The pg-lo-pure builds** — `cargo check --no-default-features + --features pg-lo` (engine + window family compile, constructors + cfg'd out where `mem` is absent) and + `cargo test --no-default-features --features "pg-lo,mem"` (the + mem-pglo constructor constructible): ADR-013 §4's supported + non-default configurations; verify both compile clean. + +## Acceptance Criteria + +- [ ] `LargeEngine::PgLo` slots in; all six large ops dispatch; the + large cursor/staging bodies slot through the recorded sub-enum + design point (the store.rs notes updated to the landed state) +- [ ] `dual_tier_postgres_pglo` + `dual_tier_mem_pglo` constructors + exist behind feature `pg-lo`, take connection settings + opts + + threshold + window delay, fallible where I/O is fallible; the + consolidation constructor's doc records the implicitly-non-fleet + posture and the fleet-valid-set completion +- [ ] `ensure_gc_schema` follows the one-window-host rule at + construction (kv host on dual-SQL — pg-lo schema carries no + `gc_window`; pg-lo host on kv=mem — contract-tested both shapes) +- [ ] The sweep form branch runs the durable form on the pg-lo host + for kv=mem + large=pg-lo and the inline form otherwise; the + branch keys off the engine representation, never a flag +- [ ] The connection-settings design point (two DSN+opts pairs; the + consolidation corollary) decided and recorded +- [ ] The pg-lo-pure builds (`--no-default-features --features pg-lo` + and `--features "pg-lo,mem"`) compile clean +- [ ] `cargo test` (default) green; `cargo test --features pg-lo` and + `--features "pg-lo,mem"` green server-less (skipping); + `cargo clippy --all-targets -- -D warnings` and `cargo fmt + --check` green (default + the pg-lo feature variants) + +## References + +- src/store.rs (the engine-representation design point; the + constructor modes; the slotting-in pattern; the `KvCursor` sub-enum + design point; the sweep form branch; the one-window-host paragraph) +- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md + §4 (the constructor table; the consolidation posture's mode reading) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md + §3 (fleet activation is a declaration — wave 6), §4 (the GC-state + host; cross-instance valid non-fleet), §5 (the one-window-host rule + and the pg-lo dormancy), §6.3 (dual-tier shapes) +- docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md + §2 (the window delay, default 5 min) +- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the + consolidation posture this constructor ships) +- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md + §4 (the feature graph; the pg-lo-pure builds) +- tasks/engine-pg-lo.md, tasks/engine-pg-lo-window.md (the engine + + seam bodies this wires) +- tasks/store-postgres-construction.md (the structural twin's recorded + design points) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/suite-column-pg-lo.md b/tasks/suite-column-pg-lo.md new file mode 100644 index 0000000..1a750d7 --- /dev/null +++ b/tasks/suite-column-pg-lo.md @@ -0,0 +1,146 @@ +--- +id: suite-column-pg-lo +name: Conformance suite — the pg-lo column (the LO-medium SKIP-posture column) +status: pending +depends_on: [store-pg-lo-construction] +scope: moderate +risk: medium +impact: phase +level: implementation +tags: [wave-5, suite, pg-lo] +--- + +## Description + +Grow the **pg-lo column** — the suite's second SKIP-posture column +(README §SKIP posture: SQL-attached factories ride a server; when no +server is reachable the column **skips cleanly** — a skip is not a +failure — and the wave's acceptance requires green **against the +server**). The column runs under feature `pg-lo` (default-off); its +two factories span feature pairs — **the first column whose factories +need different feature sets** (a recorded design point): the mem-pglo +factory gates on `all(feature = "mem", feature = "pg-lo")`, the +postgres-pglo factory on `all(feature = "postgres", +feature = "pg-lo")`; the column's full run (acceptance) is +`--all-features` against the server. + +- **Factories** (`tests/conformance/pg_lo_column.rs`): + - **`DualTierMemPgLoFactory`** — kv = mem + large = pg-lo: the + one-window-host rule's named test topology. The **durable-schedule + sweep rows run with the window hosted on the large engine** + (`gc_window` in the pg-lo schema) — the shape no earlier column + exercises. + - **`DualTierPostgresPgLoFactory`** — kv = postgres + large = pg-lo + on one pg instance: the consolidation posture. Mirrors the + postgres column's inventory with the large medium swapped to LOs, + and carries the **dormancy row** (the pg-lo schema carries no + `gc_window`; the window lives on the kv schema). + Both at column-declared thresholds and the **test tuning of the + window delay (`0` — execute-on-next-sweep)**. **The alkstore + pg-factory pattern, reused** (the postgres column's landed shape): + a fresh engine-owned schema per `open()` — unique per call, tracked + per-instance in a registry, teardown `DROP SCHEMA IF EXISTS … + CASCADE` over exactly the minted schemas, panicking-row-safe. The + **LO-isolation corollary** (the `engine-pg-lo` design point carries): + LOs are database-level — teardown must also reap the instance's LOs + (tracked LOIDs + `lo_unlink`, or the per-test-database shape if that + is what the engine task recorded), and the structural teardown + assertions include **zero `pg_largeobject` residue**. +- **Connection settings ride the environment** — the harness + convention: `ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`, a + reachability probe gating the rows (the SKIP posture's + factory-owner responsibility). **The harness server**: dockerized + `pglo-poc` (`postgres:16-alpine`) on :15432, db `blobs`, + user/password `postgres`/`poc` — verified live 2026-10-12; the + container is long-running in the dev environment. +- **Mirror the form-independent reference rows** (README §The + reference row set): INV-1 round-trips (content matrix × both put + paths, in-body — the LO medium's ranged reads at window boundaries + and the beyond-length clean failure are this column's INV-1 + extensions), INV-6 stat discipline (the companion-row probe — + metadata-only, never LO content), the `NoLivenessSources`/ + `ProtectFailed` abort shapes. **No kv-only rejection row** — no + kv-only pg-lo constructor exists (recorded; the postgres column + hosts that row for the SQL kv engines). +- **The durable-schedule sweep rows** — the SQL-backed columns' family + (README §The schedule-relative reading), run on **both factories**: + INV-2 dedup and INV-3 stage hygiene with counts read against the + staged schedule, the two-sweep exact-count row over the full + protection set (both tiers, the window observed — on the pg-lo host + for the mem-pglo factory, on the kv host for the postgres-pglo + factory), the sweep-vs-put race across the window delay + (execution-time re-arbitration), the holder-stages-only lease shape, + and INV-5 list correctness through durable-schedule outcomes. +- **The dual-tier dispatch rows** mirror the postgres column's family + (pure-function routing, threshold boundaries, fall-through per op) + with the placement oracle adapted to the LO medium: large entries + are companion rows + LOs (an independent pg connection counting + `lo_entries` and probing `pg_largeobject`), kv entries per the + factory's kv engine (the mem-pglo factory's kv placement is + in-process — the oracle's kv half follows the mem column's + introspection shape; recorded design point). +- **Virgin-store row**: reads are no-ops on a fresh schema (pool, + `lo_entries`, and the window state per the host rule all empty) — + the constructor always ensures the host's schema, this row being the + facade form. +- **Structural factory tests** (the postgres column's pattern): + factory isolation (fresh schema per open, nothing shared) and + teardown-against-a-panicking-row (schemas really dropped, LOs really + unlinked — zero residue probed). +- **README amendments** (the convention's normative test-side home): + the pg-lo rows table in §Columns beyond the reference set; the + column plan's SKIP-posture note naming the second landed column and + the per-factory feature-pair gating; current state (five columns). + +## Acceptance Criteria + +- [ ] The pg-lo column green **against the harness server** (`pglo-poc`, + :15432) under `--all-features`: the mirrored reference rows + the + durable-schedule sweep rows on both factories (the pg-lo-host + shape and the kv-host/dormancy shape) + the dispatch rows + the + virgin-store row, every row ADR-stamped and traceable to the + backlog +- [ ] The dormancy row asserts the consolidation shape's pg-lo schema + carries no `gc_window`; the mem-pglo rows assert the window on + the pg-lo host +- [ ] Server-less runs skip cleanly (the reachability probe; the + default-build gates stay green with the column skipped) +- [ ] Factory isolation + idempotent teardown hold (fresh schema per + open; schemas dropped, LOs unlinked; **zero `pg_largeobject` + residue** probed; teardown safe against a panicking row), + structurally exercised +- [ ] README amendments recorded (the pg-lo rows table; the + feature-pair gating note; current state) +- [ ] `cargo test` (default) green — the column compiles out; + `cargo test --features pg-lo` (and the mem/postgres pair + variants) green server-less (skipping) and green against the + server at acceptance; `cargo clippy --all-targets -- -D warnings` + and `cargo fmt --check` green (default + the pg-lo variants + + `--all-features`) + +## References + +- tests/conformance/README.md (the conventions this grows against; + §SKIP posture, §Columns beyond the reference set, §The + schedule-relative reading for SQL-backed columns) +- tests/conformance/postgres_column.rs (the SKIP-posture column + pattern — the factory shape, the medium oracle, the structural + tests this mirrors) +- tests/conformance/factory.rs (the factory contract this implements) +- tests/conformance/rows.rs (the reference row set being mirrored) +- docs/architecture/store-api.md (§Invariants — the test gate) +- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the + engine ADR the LO-medium rows stamp against) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md + §5 (the one-window-host rule — the dormancy row's authority), §6.7 + (the schedule the sweep rows assert against) +- tasks/engine-pg-lo.md (the LO-isolation design point this factory + inherits) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file