feat(pg-lo): the large-tier engine — LOs + companion table behind the Backend trait (wave 5)
Implement the pg-lo engine (ADR-009, POC #7 C1-C7) in src/pg_lo.rs behind feature pg-lo (default-off), plus the shared-helper home the task's design points resolve into src/pg_common.rs. - Engine: seven Backend methods + put_staged; the lo_entries companion table is the contract authority (list/size/has/CAS table-level; pg_largeobject read only by the orphan sweep's candidate enumeration); one-tx puts with 512 KiB lowrite statements (LOWRITE_CHUNK, the C2 knee), the descriptorless lo_get window cursor (READ_CHUNK; EOF clamping + the unexpected-EOF clean failure), the held-tx staging cursor aggregating 64 KiB windows, one-tx deletes, the complete single-snapshot list, the 42P01 read-only virgin-store split, and the crate-internal reap_orphans (scope-noted, planted-orphan contract test). Posture deltas in the module doc (catalog space never returned / VACUUM FULL runbook line / autovacuum inherited / the cached-get capacity facts). Nothing of EngineFleetState; the window family remains engine-pg-lo-window. - Shared helpers (pg_err, 42P01, quote_identifier, size clamp, i64-nanos encoding, build_pool, PgOpts, test janitors) move to src/pg_common.rs gated cfg(any(postgres, pg-lo)); PgOpts's public re-export widens (definition moved; crate::postgres::PgOpts path held by re-export); src/postgres.rs's changes are mechanical swaps. - LO isolation (C7 lesson): per-test dedicated databases + DatabaseJanitor (WITH (FORCE)); zero LO/schema/database residue on the shared harness db (pre-existing public.lo_entries exploration residue found and removed). - 18 in-module tests, env-driven (ALKBLOBS_PG_*), skipping cleanly server-less; all green against the pglo-poc harness server. Verified: cargo test (69/19/3), --features pg-lo live (87/19/3, 18/18 pg-lo), --features postgres live (103/36/3), --all-features (176/71/4); clippy -D warnings green for default, pg-lo, postgres, all-features, --no-default-features --features "pg-lo,mem" and "postgres,pg-lo"; cargo fmt --check; cargo doc --no-deps.
This commit is contained in:
1 parent
5a217f7bc7
commit
d69d9a345b
5 files changed
+2513
-225
No files matched your search
+109
-3
@@ -1,7 +1,7 @@
|
||||
---
|
||||
id: engine-pg-lo
|
||||
name: pg-lo large engine — Large Objects behind the Backend trait
|
||||
status: pending
|
||||
status: completed
|
||||
depends_on: []
|
||||
scope: moderate
|
||||
risk: medium
|
||||
@@ -230,8 +230,114 @@ engine and its in-module unit tests only.
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
> Implemented in `src/pg_lo.rs` (the engine, its cursors, the orphan
|
||||
> sweep, and 18 in-module tests), `src/pg_common.rs` (new — the
|
||||
> shared-helper home), and `src/{lib,postgres}.rs` (the shared-helper
|
||||
> move + gate widening; `src/postgres.rs`'s changes are mechanical
|
||||
> import swaps plus path-stable re-exports). Recorded design points,
|
||||
> as resolved:
|
||||
>
|
||||
> - **Shared-helper shape** — `src/pg_common.rs`, gated
|
||||
> `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per
|
||||
> file: `quote_identifier`/`qualified_table`, the 42P01 discriminant,
|
||||
> `err`/`pg_err` (the mapping keeps the `postgres:` prefix — both
|
||||
> engines attach to the same server family), `size_u64`,
|
||||
> `time_to_nanos` (the i64-nanos encoding), `build_pool`, [`PgOpts`],
|
||||
> and the `#[cfg(test)] test_support` janitors (`SchemaJanitor`
|
||||
> gated `#[cfg(feature = "postgres")]`, new `DatabaseJanitor` gated
|
||||
> `#[cfg(feature = "pg-lo")]`; `crate::postgres::test_support` remains
|
||||
> a path-stable re-export, so `src/store.rs`'s import needed no edit).
|
||||
> - **PgOpts** — one shared type; its definition moved to `pg_common`,
|
||||
> the public re-export widened to `any(postgres, pg-lo)` (the type is
|
||||
> unchanged; `alkblobs::postgres::PgOpts` still resolves through the
|
||||
> path-stable re-export). The `synchronous_commit` knob **carries**
|
||||
> (default `on`; the POC's server-side `off` was a measurement
|
||||
> posture, not the ship config — recorded in the module doc).
|
||||
> - **LO isolation (the C7 lesson resolved)** — Large Objects are
|
||||
> database-level; schema-per-test does not isolate them. The tests
|
||||
> run on **per-test dedicated databases** (the POC's proven
|
||||
> `open_fresh` shape): unique `abtlo_{tag}_{pid}_{seq}_{nanos}`
|
||||
> names, `CREATE DATABASE` through the `postgres` maintenance db, a
|
||||
> `DatabaseJanitor` drop-guard (`DROP DATABASE … WITH (FORCE)`) after
|
||||
> the backend's pool. Zero LO residue on the shared `blobs` db is
|
||||
> structural (no pg-lo test ever opens a pool against it) and gate-
|
||||
> tested (`shared_harness_db_holds_zero_lo_residue`).
|
||||
> - **`committed_at`** — the crate-wide i64-nanos encoding as `bigint`,
|
||||
> engine-stamped at insert (not a server-default column); the crate's
|
||||
> one timestamp encoding rather than the POC's `timestamptz default
|
||||
> now()`.
|
||||
> - **Staged reads** — verified live: `lo_get` inside the writing tx
|
||||
> sees its own uncommitted pages (read-your-own-writes), so the
|
||||
> verification pass reads the staged artifact through descriptorless
|
||||
> windows + the unflushed in-memory tail; no descriptor reads, no
|
||||
> `lo_lseek` anywhere (the C7 64-bit discipline: offsets ride bigint
|
||||
> only, `lowrite` is implicitly sequential).
|
||||
> - **Stage hygiene mechanics** — the staging cursor takes its
|
||||
> checkout in `commit` (`publish` over the taken `Object`), so every
|
||||
> > failure path converges on one ROLLBACK point; the dedup path
|
||||
> (CAS hit) ROLLBACKs and returns `Ok` (the staged LO is the residue
|
||||
> removed); `Drop` hands an abandoned cursor's open tx to a spawned
|
||||
> ROLLBACK (never returns a dirty checkout to the pool — deadpool's
|
||||
> `Fast` recycle pitfall), degrading to a one-shot thread when no
|
||||
> runtime is live. `#[cfg(test)] fail_commit` knob mirrors
|
||||
> mem/sqlite/postgres/local.
|
||||
> - **Harness-server residue found and cleaned** — the shared `blobs`
|
||||
> db carried pre-existing exploration residue (`public.lo_entries`
|
||||
> + 21 LOs, 9 472 catalog pages) from pre-task pg-lo probing;
|
||||
> unlinked + dropped (count now 0) so the
|
||||
> `pg_largeobject`-probe gate measures this suite's own runs. The
|
||||
> `public.blobs` kv table was left untouched (kv-engine content, out
|
||||
> of scope).
|
||||
> - **The literal `--no-default-features --features pg-lo` config
|
||||
> cannot compile the whole crate** — `store.rs`'s `KvEngine`/`KvCursor`
|
||||
> enums are empty there (E0004), identically on the pre-task tree
|
||||
> (verified by stash). Every constructor mode pairs a kv engine
|
||||
> (ADR-011 §4's table), so a kv-less store is not a shippable shape;
|
||||
> the clean configs are `--no-default-features --features "pg-lo,mem"`
|
||||
> and ADR-013 §4's pg-pure build
|
||||
> `--no-default-features --features "postgres,pg-lo"` — both clippy
|
||||
> clean. In the literal config the *shared pieces* compile (the 10
|
||||
> store.rs E0004s are the crate's only errors; nothing in
|
||||
> pg_common/pg_lo errors). Flagged for `store-pg-lo-construction` to
|
||||
> keep in view.
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
> Completed 2026-10-12 (wave 5). `src/pg_lo.rs` implements the
|
||||
> seven-method `Backend` + `put_staged` behind feature `pg-lo`
|
||||
> (default-off): LOs hold the bytes, the `lo_entries` companion table is
|
||||
> the contract authority (list/size/has/CAS all table-level; `pg_largeobject`
|
||||
> read by the engine only inside the orphan sweep's candidate
|
||||
> enumeration), one-tx puts (lo_create → 512 KiB chunked `lowrite`
|
||||
> (`LOWRITE_CHUNK`, the C2 knee) → row insert → COMMIT), the
|
||||
> descriptorless `lo_get(oid, off, len)` window cursor
|
||||
> (`READ_CHUNK`; `len()` metadata-only from the companion row; EOF
|
||||
> clamping per the cursor contract plus the unexpected-EOF clean
|
||||
> failure for a body short of its claimed length), the held-tx staging
|
||||
> cursor with aggregation of the store core's 64 KiB windows into
|
||||
> 512 KiB statements and both hygiene forms exact-count-tested, one-tx
|
||||
> deletes with the Missing discipline, the complete single-snapshot
|
||||
> `list`, idempotent schema+table bootstrap at the constructor with the
|
||||
> quoted-identifier boundary (hostile names tested), the 42P01
|
||||
> read-only virgin-store split (schema-less form tested), the
|
||||
> crate-internal `reap_orphans` sweep (scope-noted; planted-orphan
|
||||
> contract test with byte-exact committed-content survival), and the
|
||||
> posture deltas in the module doc (catalog space never returned +
|
||||
> `VACUUM FULL` runbook line + rel-size monitoring; autovacuum
|
||||
> inherited; the 20–50× cached-get capacity facts). Shared helpers
|
||||
> (`pg_err`, 42P01, quoting, timestamps, pool, `PgOpts`) moved to the
|
||||
> new `src/pg_common.rs` gated `any(postgres, pg-lo)`; `PgOpts`'s
|
||||
> public re-export widened (shared opts decision; the knob carries).
|
||||
> 18 in-module tests, per-test dedicated databases (`DatabaseJanitor`),
|
||||
> env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less. Verified:
|
||||
> `cargo test` (69/19/3 ok), `cargo test --features pg-lo` live
|
||||
> (87/19/3 ok, 18/18 pg-lo), `cargo test --features postgres` live
|
||||
> (103/36/3 ok), `cargo test --all-features` (176/71/4 ok), clippy
|
||||
> `-D warnings` green (default, `--features pg-lo`, `--features
|
||||
> postgres`, `--all-features`, `--no-default-features --features
|
||||
> "pg-lo,mem"`, `--no-default-features --features "postgres,pg-lo"`),
|
||||
> `cargo fmt --check` green, `cargo doc --no-deps` clean, zero LO +
|
||||
> schema + database residue on the harness server after full runs.
|
||||
> Engine-state window family remains `engine-pg-lo-window`
|
||||
> (`EngineWindowState` untouched; nothing of `EngineFleetState` exists
|
||||
> on this engine); constructor wiring remains `store-pg-lo-construction`.
|
||||
Reference in new issue
Block a user