--- id: engine-pg-lo name: pg-lo large engine — Large Objects behind the Backend trait status: completed depends_on: [] scope: moderate risk: medium impact: component level: implementation tags: [wave-5, engines, pg-lo, postgres] --- ## Description Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off; ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base) against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the 2-line module-doc stub behind the pre-existing `#[cfg(feature = "pg-lo")]` gate). The postgres kv engine (`src/postgres.rs`, wave 4) is the structural twin for the driver posture (pool, prepared statements, schema bootstrap, error mapping); the local engine (`src/local.rs`) is the structural twin for the large-tier bodies (streaming read cursor, staged put). **`pg-lo` does not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml feature carries its own driver deps) — every shared piece must compile under the **pg-lo pure builds**: `--no-default-features --features "pg-lo,mem"` and `--no-default-features --features "postgres,pg-lo"`. **The literal kv-less config (`--no-default-features --features pg-lo`) is not a gate and cannot compile the whole crate** — with no kv feature enabled, `KvEngine`/`KvCursor` go empty and the facade's by-reference dispatch matches are non-exhaustive (E0004; references are always considered inhabited) — see `store-pg-lo-construction` for the full reading. - **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs hold the bytes; the companion table `lo_entries (key bytea PRIMARY KEY, loid oid, size bigint, committed_at)` is the **contract authority** — `list()`, `size()`, `has()`, CAS-existence are all table-level. **Never trust or read `pg_largeobject` for addressing** (the C6 sweep validates against the table only). One engine-owned PostgreSQL schema (the postgres engine's bootstrap pattern: idempotent DDL at the constructor, the quoted-identifier boundary, no migration machinery). The default schema name and the consolidation shape's schema sharing (both engines on one pg instance) are **recorded design points**. - **Driver posture (the postgres engine's, reused)** — deadpool-postgres over tokio-postgres, pool size configurable with a named shipped default, **per-connection `prepare_cached` discipline** (statements die with sessions), natively async — **no `spawn_blocking` seam**, no blocking I/O on the async path. **Shared-helper design point**: the postgres engine's crate-internal helpers (`quote_identifier`, the SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp helpers) live behind `#[cfg(feature = "postgres")]` today — decide and record the sharing shape (the natural one: a crate-internal pg-common module gated `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per file, so the pg-lo pure builds compile without duplication). The `PgOpts` question rides the same point: it is `pub` and re-exported behind `postgres` (wave 4's constructor signature put it on the public surface) — widening its gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a pg-lo-local opts type is a recorded design point; whether pg-lo carries the `synchronous_commit` knob at all (the POC ran the server with it off; the kv parity knob's posture may or may not carry) is recorded with it. - **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` → chunked `lowrite` → companion-row insert → COMMIT. **512 KiB statements** (POC #7 C2's measured knee; statement count, not LOBLKSIZE, is what costs) as a named shipped constant. CAS: companion-table existence check first — a re-put of an existing key is a no-op tx, never an overwrite (the C1 shape). The store core's put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates appends into 512 KiB `lowrite` statements (recorded design point). - **Staged put (the engine-level `put_staged` contract)** — the staging cursor body is a **held tx**: handout opens the checkout + tx + `lo_create`; `append` rides the chunked `lowrite` aggregation; `commit` publishes (companion-row insert + COMMIT — the row and the LO's pages land in one step, zero crash orphans structurally, C6); `discard` rolls back and returns the checkout. Every failure path converges on rollback — stage hygiene holds in both forms (discard → zero residue: no LO pages, no row; failed commit → self-discard), exact-count tested (the `#[cfg(test)]` fail-commit knob standing in for media failure, the mem/sqlite/postgres pattern). **Scope boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet* joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's ruling that pg-lo contributes LO writes + companion-row insert into the store-held tx); its API shape is a wave-6 design point. This task ships the engine-level staging cursor only. - **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3, the C4 posture)** — the read cursor closes over the companion row (oid + length; content immutable, so the row is authoritative), not over a tx-scoped descriptor. Range reads are single stateless statements, pool-friendly. `len()` reads the companion row — **metadata only, never LO content** (ADR-008's discipline). A range read beyond content length fails cleanly (`UnexpectedEof`-shaped → the engine's `Error::Io` mapping; the POC's `ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*` 64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any positioning is used; `lo_get`'s bigint offsets already are — the C7 int4 trap must not be inherited). The held-descriptor tx posture exists as the fallback (measured: no advantage even for sequential full reads) — named in the module doc as not-shipped, never coded. - **`has`** — companion-table EXISTS probe. **`size`** — the companion-row `size` probe (metadata-only; the ADR-009 companion-row probe the suite's INV-6 row names). **`delete`** — one tx (`lo_unlink` + row delete), rows-affected discipline (`Err(Error::Missing)` when absent); GC-participating (the delete window's arbitration is store-core and table-level — nothing fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`, complete (the POC's exact-count gate: list == companion table after mixed ops). - **Virgin-store posture** — the constructor creates the schema and table; the tolerance is the postgres engine's recorded read-only split (reads on a schema-less database map to absent/empty via the SQLSTATE 42P01 discriminant, writes surface the engine error). The schema-less test fixture mirrors the postgres engine's `#[cfg(test)] new_schemaless` pattern. - **Posture deltas in the module doc, stated not hidden (ADR-009 §5/§6)** — LO catalog space is reused but never returned (`pg_total_relation_size('pg_largeobject')` monitoring named; `VACUUM FULL`/`pg_repack` the only shrink path — an operator runbook line, never automatic); autovacuum inherited, keep it on; single-stream cached get 20–50× behind page-cache local fs at ≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture — named capacity facts, not solved problems. - **Orphan-recovery sweep (ADR-009 §5's requirement)** — a crate-internal `reap_orphans()` for the legacy/bypass class only (committed LOs without companion rows): enumerate `SELECT DISTINCT loid FROM pg_largeobject` as a candidate list **only**, check each against `lo_entries.loid` (the truth), `lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with oid count, not content size). Scope-noted `#[allow(dead_code)]` until a consumer exists (recorded: the ops/maintenance surface is wave 7+; no store-core verb is scoped this wave), contract-tested with a planted orphan and a byte-exact survival assertion for committed content. - **Unit tests** in-module: the method contract (virgin-store no-ops incl. the schema-less form, CAS re-put no-op, Missing discipline, size exactness + metadata-only, list completeness after mixed ops, ranged reads at window boundaries + beyond-length failure), the staged-put happy path and both hygiene forms, the orphan sweep. **Tests need a live server** — the alkstore harness convention: connection settings ride the environment (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly server-less, acceptance requires green against the server. The harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on :15432, db `blobs`, user/password `postgres`/`poc` (verified live 2026-10-12). - **LO isolation on the shared harness db (named design point — the C7 lesson, do not rediscover it)**: Large Objects are **database-level**, not schema-level — the schema-per-test convention does not isolate them, a `DROP SCHEMA` leaves the LO's pages in `pg_largeobject`, and shared-LO tests raced each other into spurious "large object NNNN does not exist" aborts in the POC. Decide and record the isolation shape: per-test `CREATE DATABASE` (the POC's proven `open_fresh` shape) or per-instance LOID tracking + teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor pattern extended). Either way acceptance includes **zero LO residue on the shared db** (`pg_largeobject` probed before/after full runs). - **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the schema-janitor discipline from the start; the wave-4 lesson (380 leaked schemas) is not to be re-learned. The `SchemaJanitor` lives in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the shared-helper design point. The window-family seam clauses (`EngineWindowState`) are `engine-pg-lo-window` (separate task); the constructor wiring is `store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep lock **no** — fleet pins live on the kv host). This task ships the engine and its in-module unit tests only. ## Acceptance Criteria - [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods + `put_staged`) behind feature `pg-lo` (default-off), with the descriptorless `lo_get` window cursor body and the held-tx staging body - [ ] The companion table is the contract authority (list/size/has/CAS all table-level); `pg_largeobject` is never read for addressing - [ ] One engine-owned schema, idempotent bootstrap at the constructor; the quoted-identifier boundary; the virgin-store read-only split (42P01 → absent/empty on reads, writes surface) unit-tested incl. the schema-less form - [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert → COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named constant with the C2 citation - [ ] Stage hygiene holds in both forms (discard → zero residue; failed commit → rollback self-discard), exact-count unit-tested - [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only from the companion row; beyond-length reads fail cleanly; the 64-bit-variants discipline recorded (the C7 trap not inherited) - [ ] `delete` = one tx (lo_unlink + row delete) with the Missing discipline; `list` complete after mixed ops - [ ] The orphan-recovery sweep exists (candidates from `pg_largeobject` only, validated against `lo_entries`), scope-noted, contract-tested with a planted orphan - [ ] Posture deltas recorded in the module doc (catalog space never returned + VACUUM FULL operator line + rel-size monitoring; autovacuum inherited; the cached-get capacity facts) - [ ] Shared-helper + PgOpts + LO-isolation design points decided and recorded; the pg-lo pure builds (`--no-default-features --features "pg-lo,mem"` and `--features "postgres,pg-lo"`) compile clean (the literal kv-less config is not a gate — see the Description) - [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less; green against the harness server at acceptance; zero schema + LO residue on the shared db after runs - [ ] `cargo test` (default — pg-lo compiles out) green; `cargo test --features pg-lo` green server-less (skipping); `cargo clippy --all-targets -- -D warnings` and `cargo fmt --check` green (default + `--features pg-lo` + the pg-lo pure builds) ## References - docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the engine ADR — §1–§4 the contract, §5/§6 the posture deltas) - docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan sweep, C7 the harness lessons) - docs/architecture/decisions/003-backend-contract-and-dispatch.md (the trait contract) - docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract; the large-tier cursor body families) - docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §4/§5 (the consolidation posture; the pg-lo applicability row — window family only, no fleet state) - docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build) - src/backend.rs (the trait + seam contracts this implements) - src/postgres.rs (the driver-posture twin: pool, prepare_cached, schema bootstrap, 42P01 split, test_support janitor) - src/local.rs (the large-tier body twin: streaming cursor, staged put hygiene shapes) - /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line PgLoBackend this engine productionizes) ## Notes > Implemented in `src/pg_lo.rs` (the engine, its cursors, the orphan > sweep, and 18 in-module tests), `src/pg_common.rs` (new — the > shared-helper home), and `src/{lib,postgres}.rs` (the shared-helper > move + gate widening; `src/postgres.rs`'s changes are mechanical > import swaps plus path-stable re-exports). Recorded design points, > as resolved: > > - **Shared-helper shape** — `src/pg_common.rs`, gated > `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per > file: `quote_identifier`/`qualified_table`, the 42P01 discriminant, > `err`/`pg_err` (the mapping keeps the `postgres:` prefix — both > engines attach to the same server family), `size_u64`, > `time_to_nanos` (the i64-nanos encoding), `build_pool`, [`PgOpts`], > and the `#[cfg(test)] test_support` janitors (`SchemaJanitor` > gated `#[cfg(feature = "postgres")]`, new `DatabaseJanitor` gated > `#[cfg(feature = "pg-lo")]`; `crate::postgres::test_support` remains > a path-stable re-export, so `src/store.rs`'s import needed no edit). > - **PgOpts** — one shared type; its definition moved to `pg_common`, > the public re-export widened to `any(postgres, pg-lo)` (the type is > unchanged; `alkblobs::postgres::PgOpts` still resolves through the > path-stable re-export). The `synchronous_commit` knob **carries** > (default `on`; the POC's server-side `off` was a measurement > posture, not the ship config — recorded in the module doc). > - **LO isolation (the C7 lesson resolved)** — Large Objects are > database-level; schema-per-test does not isolate them. The tests > run on **per-test dedicated databases** (the POC's proven > `open_fresh` shape): unique `abtlo_{tag}_{pid}_{seq}_{nanos}` > names, `CREATE DATABASE` through the `postgres` maintenance db, a > `DatabaseJanitor` drop-guard (`DROP DATABASE … WITH (FORCE)`) after > the backend's pool. Zero LO residue on the shared `blobs` db is > structural (no pg-lo test ever opens a pool against it) and gate- > tested (`shared_harness_db_holds_zero_lo_residue`). > - **`committed_at`** — the crate-wide i64-nanos encoding as `bigint`, > engine-stamped at insert (not a server-default column); the crate's > one timestamp encoding rather than the POC's `timestamptz default > now()`. > - **Staged reads** — verified live: `lo_get` inside the writing tx > sees its own uncommitted pages (read-your-own-writes), so the > verification pass reads the staged artifact through descriptorless > windows + the unflushed in-memory tail; no descriptor reads, no > `lo_lseek` anywhere (the C7 64-bit discipline: offsets ride bigint > only, `lowrite` is implicitly sequential). > - **Stage hygiene mechanics** — the staging cursor takes its > checkout in `commit` (`publish` over the taken `Object`), so every > > failure path converges on one ROLLBACK point; the dedup path > (CAS hit) ROLLBACKs and returns `Ok` (the staged LO is the residue > removed); `Drop` hands an abandoned cursor's open tx to a spawned > ROLLBACK (never returns a dirty checkout to the pool — deadpool's > `Fast` recycle pitfall), degrading to a one-shot thread when no > runtime is live. `#[cfg(test)] fail_commit` knob mirrors > mem/sqlite/postgres/local. > - **Harness-server residue found and cleaned** — the shared `blobs` > db carried pre-existing exploration residue (`public.lo_entries` > + 21 LOs, 9 472 catalog pages) from pre-task pg-lo probing; > unlinked + dropped (count now 0) so the > `pg_largeobject`-probe gate measures this suite's own runs. The > `public.blobs` kv table was left untouched (kv-engine content, out > of scope). > - **The literal `--no-default-features --features pg-lo` config > cannot compile the whole crate** — `store.rs`'s `KvEngine`/`KvCursor` > enums are empty there (E0004), identically on the pre-task tree > (verified by stash). Every constructor mode pairs a kv engine > (ADR-011 §4's table), so a kv-less store is not a shippable shape; > the clean configs are `--no-default-features --features "pg-lo,mem"` > and ADR-013 §4's pg-pure build > `--no-default-features --features "postgres,pg-lo"` — both clippy > clean. In the literal config the *shared pieces* compile (the 10 > store.rs E0004s are the crate's only errors; nothing in > pg_common/pg_lo errors). Flagged for `store-pg-lo-construction` to > keep in view. ## Summary > Completed 2026-10-12 (wave 5). `src/pg_lo.rs` implements the > seven-method `Backend` + `put_staged` behind feature `pg-lo` > (default-off): LOs hold the bytes, the `lo_entries` companion table is > the contract authority (list/size/has/CAS all table-level; `pg_largeobject` > read by the engine only inside the orphan sweep's candidate > enumeration), one-tx puts (lo_create → 512 KiB chunked `lowrite` > (`LOWRITE_CHUNK`, the C2 knee) → row insert → COMMIT), the > descriptorless `lo_get(oid, off, len)` window cursor > (`READ_CHUNK`; `len()` metadata-only from the companion row; EOF > clamping per the cursor contract plus the unexpected-EOF clean > failure for a body short of its claimed length), the held-tx staging > cursor with aggregation of the store core's 64 KiB windows into > 512 KiB statements and both hygiene forms exact-count-tested, one-tx > deletes with the Missing discipline, the complete single-snapshot > `list`, idempotent schema+table bootstrap at the constructor with the > quoted-identifier boundary (hostile names tested), the 42P01 > read-only virgin-store split (schema-less form tested), the > crate-internal `reap_orphans` sweep (scope-noted; planted-orphan > contract test with byte-exact committed-content survival), and the > posture deltas in the module doc (catalog space never returned + > `VACUUM FULL` runbook line + rel-size monitoring; autovacuum > inherited; the 20–50× cached-get capacity facts). Shared helpers > (`pg_err`, 42P01, quoting, timestamps, pool, `PgOpts`) moved to the > new `src/pg_common.rs` gated `any(postgres, pg-lo)`; `PgOpts`'s > public re-export widened (shared opts decision; the knob carries). > 18 in-module tests, per-test dedicated databases (`DatabaseJanitor`), > env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less. Verified: > `cargo test` (69/19/3 ok), `cargo test --features pg-lo` live > (87/19/3 ok, 18/18 pg-lo), `cargo test --features postgres` live > (103/36/3 ok), `cargo test --all-features` (176/71/4 ok), clippy > `-D warnings` green (default, `--features pg-lo`, `--features > postgres`, `--all-features`, `--no-default-features --features > "pg-lo,mem"`, `--no-default-features --features "postgres,pg-lo"`), > `cargo fmt --check` green, `cargo doc --no-deps` clean, zero LO + > schema + database residue on the harness server after full runs. > Engine-state window family remains `engine-pg-lo-window` > (`EngineWindowState` untouched; nothing of `EngineFleetState` exists > on this engine); constructor wiring remains `store-pg-lo-construction`.