feat(pg-lo): the large-tier engine — LOs + companion table behind the Backend trait (wave 5)

Implement the pg-lo engine (ADR-009, POC #7 C1-C7) in src/pg_lo.rs
behind feature pg-lo (default-off), plus the shared-helper home the
task's design points resolve into src/pg_common.rs.

- Engine: seven Backend methods + put_staged; the lo_entries companion
  table is the contract authority (list/size/has/CAS table-level;
  pg_largeobject read only by the orphan sweep's candidate enumeration);
  one-tx puts with 512 KiB lowrite statements (LOWRITE_CHUNK, the C2
  knee), the descriptorless lo_get window cursor (READ_CHUNK; EOF
  clamping + the unexpected-EOF clean failure), the held-tx staging
  cursor aggregating 64 KiB windows, one-tx deletes, the complete
  single-snapshot list, the 42P01 read-only virgin-store split, and the
  crate-internal reap_orphans (scope-noted, planted-orphan contract
  test). Posture deltas in the module doc (catalog space never
  returned / VACUUM FULL runbook line / autovacuum inherited / the
  cached-get capacity facts). Nothing of EngineFleetState; the window
  family remains engine-pg-lo-window.
- Shared helpers (pg_err, 42P01, quote_identifier, size clamp, i64-nanos
  encoding, build_pool, PgOpts, test janitors) move to src/pg_common.rs
  gated cfg(any(postgres, pg-lo)); PgOpts's public re-export widens
  (definition moved; crate::postgres::PgOpts path held by re-export);
  src/postgres.rs's changes are mechanical swaps.
- LO isolation (C7 lesson): per-test dedicated databases + DatabaseJanitor
  (WITH (FORCE)); zero LO/schema/database residue on the shared harness
  db (pre-existing public.lo_entries exploration residue found and
  removed).
- 18 in-module tests, env-driven (ALKBLOBS_PG_*), skipping cleanly
  server-less; all green against the pglo-poc harness server.

Verified: cargo test (69/19/3), --features pg-lo live (87/19/3, 18/18
pg-lo), --features postgres live (103/36/3), --all-features (176/71/4);
clippy -D warnings green for default, pg-lo, postgres, all-features,
--no-default-features --features "pg-lo,mem" and "postgres,pg-lo";
cargo fmt --check; cargo doc --no-deps.
This commit is contained in:
glm-5.3-flash committed 2026-10-12 07:31:46 +00:00
1 parent 5a217f7bc7
commit d69d9a345b
5 files changed
+2513 -225

No files matched your search

+109 -3
View File
@@ -1,7 +1,7 @@
---
id: engine-pg-lo
name: pg-lo large engine — Large Objects behind the Backend trait
status: pending
status: completed
depends_on: []
scope: moderate
risk: medium
@@ -230,8 +230,114 @@ engine and its in-module unit tests only.
## Notes
> To be filled by implementation agent
> Implemented in `src/pg_lo.rs` (the engine, its cursors, the orphan
> sweep, and 18 in-module tests), `src/pg_common.rs` (new — the
> shared-helper home), and `src/{lib,postgres}.rs` (the shared-helper
> move + gate widening; `src/postgres.rs`'s changes are mechanical
> import swaps plus path-stable re-exports). Recorded design points,
> as resolved:
>
> - **Shared-helper shape** — `src/pg_common.rs`, gated
> `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per
> file: `quote_identifier`/`qualified_table`, the 42P01 discriminant,
> `err`/`pg_err` (the mapping keeps the `postgres:` prefix — both
> engines attach to the same server family), `size_u64`,
> `time_to_nanos` (the i64-nanos encoding), `build_pool`, [`PgOpts`],
> and the `#[cfg(test)] test_support` janitors (`SchemaJanitor`
> gated `#[cfg(feature = "postgres")]`, new `DatabaseJanitor` gated
> `#[cfg(feature = "pg-lo")]`; `crate::postgres::test_support` remains
> a path-stable re-export, so `src/store.rs`'s import needed no edit).
> - **PgOpts** — one shared type; its definition moved to `pg_common`,
> the public re-export widened to `any(postgres, pg-lo)` (the type is
> unchanged; `alkblobs::postgres::PgOpts` still resolves through the
> path-stable re-export). The `synchronous_commit` knob **carries**
> (default `on`; the POC's server-side `off` was a measurement
> posture, not the ship config — recorded in the module doc).
> - **LO isolation (the C7 lesson resolved)** — Large Objects are
> database-level; schema-per-test does not isolate them. The tests
> run on **per-test dedicated databases** (the POC's proven
> `open_fresh` shape): unique `abtlo_{tag}_{pid}_{seq}_{nanos}`
> names, `CREATE DATABASE` through the `postgres` maintenance db, a
> `DatabaseJanitor` drop-guard (`DROP DATABASE … WITH (FORCE)`) after
> the backend's pool. Zero LO residue on the shared `blobs` db is
> structural (no pg-lo test ever opens a pool against it) and gate-
> tested (`shared_harness_db_holds_zero_lo_residue`).
> - **`committed_at`** — the crate-wide i64-nanos encoding as `bigint`,
> engine-stamped at insert (not a server-default column); the crate's
> one timestamp encoding rather than the POC's `timestamptz default
> now()`.
> - **Staged reads** — verified live: `lo_get` inside the writing tx
> sees its own uncommitted pages (read-your-own-writes), so the
> verification pass reads the staged artifact through descriptorless
> windows + the unflushed in-memory tail; no descriptor reads, no
> `lo_lseek` anywhere (the C7 64-bit discipline: offsets ride bigint
> only, `lowrite` is implicitly sequential).
> - **Stage hygiene mechanics** — the staging cursor takes its
> checkout in `commit` (`publish` over the taken `Object`), so every
> > failure path converges on one ROLLBACK point; the dedup path
> (CAS hit) ROLLBACKs and returns `Ok` (the staged LO is the residue
> removed); `Drop` hands an abandoned cursor's open tx to a spawned
> ROLLBACK (never returns a dirty checkout to the pool — deadpool's
> `Fast` recycle pitfall), degrading to a one-shot thread when no
> runtime is live. `#[cfg(test)] fail_commit` knob mirrors
> mem/sqlite/postgres/local.
> - **Harness-server residue found and cleaned** — the shared `blobs`
> db carried pre-existing exploration residue (`public.lo_entries`
> + 21 LOs, 9 472 catalog pages) from pre-task pg-lo probing;
> unlinked + dropped (count now 0) so the
> `pg_largeobject`-probe gate measures this suite's own runs. The
> `public.blobs` kv table was left untouched (kv-engine content, out
> of scope).
> - **The literal `--no-default-features --features pg-lo` config
> cannot compile the whole crate** — `store.rs`'s `KvEngine`/`KvCursor`
> enums are empty there (E0004), identically on the pre-task tree
> (verified by stash). Every constructor mode pairs a kv engine
> (ADR-011 §4's table), so a kv-less store is not a shippable shape;
> the clean configs are `--no-default-features --features "pg-lo,mem"`
> and ADR-013 §4's pg-pure build
> `--no-default-features --features "postgres,pg-lo"` — both clippy
> clean. In the literal config the *shared pieces* compile (the 10
> store.rs E0004s are the crate's only errors; nothing in
> pg_common/pg_lo errors). Flagged for `store-pg-lo-construction` to
> keep in view.
## Summary
> To be filled on completion
> Completed 2026-10-12 (wave 5). `src/pg_lo.rs` implements the
> seven-method `Backend` + `put_staged` behind feature `pg-lo`
> (default-off): LOs hold the bytes, the `lo_entries` companion table is
> the contract authority (list/size/has/CAS all table-level; `pg_largeobject`
> read by the engine only inside the orphan sweep's candidate
> enumeration), one-tx puts (lo_create → 512 KiB chunked `lowrite`
> (`LOWRITE_CHUNK`, the C2 knee) → row insert → COMMIT), the
> descriptorless `lo_get(oid, off, len)` window cursor
> (`READ_CHUNK`; `len()` metadata-only from the companion row; EOF
> clamping per the cursor contract plus the unexpected-EOF clean
> failure for a body short of its claimed length), the held-tx staging
> cursor with aggregation of the store core's 64 KiB windows into
> 512 KiB statements and both hygiene forms exact-count-tested, one-tx
> deletes with the Missing discipline, the complete single-snapshot
> `list`, idempotent schema+table bootstrap at the constructor with the
> quoted-identifier boundary (hostile names tested), the 42P01
> read-only virgin-store split (schema-less form tested), the
> crate-internal `reap_orphans` sweep (scope-noted; planted-orphan
> contract test with byte-exact committed-content survival), and the
> posture deltas in the module doc (catalog space never returned +
> `VACUUM FULL` runbook line + rel-size monitoring; autovacuum
> inherited; the 20–50× cached-get capacity facts). Shared helpers
> (`pg_err`, 42P01, quoting, timestamps, pool, `PgOpts`) moved to the
> new `src/pg_common.rs` gated `any(postgres, pg-lo)`; `PgOpts`'s
> public re-export widened (shared opts decision; the knob carries).
> 18 in-module tests, per-test dedicated databases (`DatabaseJanitor`),
> env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less. Verified:
> `cargo test` (69/19/3 ok), `cargo test --features pg-lo` live
> (87/19/3 ok, 18/18 pg-lo), `cargo test --features postgres` live
> (103/36/3 ok), `cargo test --all-features` (176/71/4 ok), clippy
> `-D warnings` green (default, `--features pg-lo`, `--features
> postgres`, `--all-features`, `--no-default-features --features
> "pg-lo,mem"`, `--no-default-features --features "postgres,pg-lo"`),
> `cargo fmt --check` green, `cargo doc --no-deps` clean, zero LO +
> schema + database residue on the harness server after full runs.
> Engine-state window family remains `engine-pg-lo-window`
> (`EngineWindowState` untouched; nothing of `EngineFleetState` exists
> on this engine); constructor wiring remains `store-pg-lo-construction`.