alkblobs/tasks/engine-pg-lo.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
engine-pg-lo	pg-lo large engine — Large Objects behind the Backend trait	completed		moderate	medium	component	implementation	wave-5 engines pg-lo postgres

Description

Implement the pg-lo large-tier engine (feature pg-lo, default-off; ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base) against the wave-1 trait (src/backend.rs) in src/pg_lo.rs (the 2-line module-doc stub behind the pre-existing #[cfg(feature = "pg-lo")] gate). The postgres kv engine (src/postgres.rs, wave 4) is the structural twin for the driver posture (pool, prepared statements, schema bootstrap, error mapping); the local engine (src/local.rs) is the structural twin for the large-tier bodies (streaming read cursor, staged put). pg-lo does not imply postgres (ADR-013 §4's consequence; the Cargo.toml feature carries its own driver deps) — every shared piece must compile under the pg-lo pure builds: --no-default-features --features "pg-lo,mem" and --no-default-features --features "postgres,pg-lo". The literal kv-less config (--no-default-features --features pg-lo) is not a gate and cannot compile the whole crate — with no kv feature enabled, KvEngine/KvCursor go empty and the facade's by-reference dispatch matches are non-exhaustive (E0004; references are always considered inhabited) — see store-pg-lo-construction for the full reading.

• Engine shape (POC #7 C1 is the contract) — two artifacts: LOs hold the bytes; the companion table lo_entries (key bytea PRIMARY KEY, loid oid, size bigint, committed_at) is the contract authority — list(), size(), has(), CAS-existence are all table-level. Never trust or read pg_largeobject for addressing (the C6 sweep validates against the table only). One engine-owned PostgreSQL schema (the postgres engine's bootstrap pattern: idempotent DDL at the constructor, the quoted-identifier boundary, no migration machinery). The default schema name and the consolidation shape's schema sharing (both engines on one pg instance) are recorded design points.
• Driver posture (the postgres engine's, reused) — deadpool-postgres over tokio-postgres, pool size configurable with a named shipped default, per-connection prepare_cached discipline (statements die with sessions), natively async — no spawn_blocking seam, no blocking I/O on the async path. Shared-helper design point: the postgres engine's crate-internal helpers (quote_identifier, the SQLSTATE 42P01 discriminant, pg_err, the i64-nanos timestamp helpers) live behind #[cfg(feature = "postgres")] today — decide and record the sharing shape (the natural one: a crate-internal pg-common module gated cfg(any(feature = "postgres", feature = "pg-lo")), one module per file, so the pg-lo pure builds compile without duplication). The PgOpts question rides the same point: it is pub and re-exported behind postgres (wave 4's constructor signature put it on the public surface) — widening its gate to any(postgres, pg-lo) (re-export gate widened with it) vs a pg-lo-local opts type is a recorded design point; whether pg-lo carries the synchronous_commit knob at all (the POC ran the server with it off; the kv parity knob's posture may or may not carry) is recorded with it.
• Put = one transaction (ADR-009 §2) — BEGIN → lo_create → chunked lowrite → companion-row insert → COMMIT. 512 KiB statements (POC #7 C2's measured knee; statement count, not LOBLKSIZE, is what costs) as a named shipped constant. CAS: companion-table existence check first — a re-put of an existing key is a no-op tx, never an overwrite (the C1 shape). The store core's put loop hands 64 KiB windows (PUT_CHUNK); the engine aggregates appends into 512 KiB lowrite statements (recorded design point).
• Staged put (the engine-level put_staged contract) — the staging cursor body is a held tx: handout opens the checkout + tx + lo_create; append rides the chunked lowrite aggregation; commit publishes (companion-row insert + COMMIT — the row and the LO's pages land in one step, zero crash orphans structurally, C6); discard rolls back and returns the checkout. Every failure path converges on rollback — stage hygiene holds in both forms (discard → zero residue: no LO pages, no row; failed commit → self-discard), exact-count tested (the #[cfg(test)] fail-commit knob standing in for media failure, the mem/sqlite/postgres pattern). Scope boundary: ADR-010 §1's "a tx the store core holds" is the fleet joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's ruling that pg-lo contributes LO writes + companion-row insert into the store-held tx); its API shape is a wave-6 design point. This task ships the engine-level staging cursor only.
• Get = descriptorless lo_get(oid, off, len) windows (ADR-009 §3, the C4 posture) — the read cursor closes over the companion row (oid + length; content immutable, so the row is authoritative), not over a tx-scoped descriptor. Range reads are single stateless statements, pool-friendly. len() reads the companion row — metadata only, never LO content (ADR-008's discipline). A range read beyond content length fails cleanly (UnexpectedEof-shaped → the engine's Error::Io mapping; the POC's ranged_reads_windows_and_eof shape). Reads ride the lo_* 64-bit variants end-to-end (lo_lseek64/int8 offsets if any positioning is used; lo_get's bigint offsets already are — the C7 int4 trap must not be inherited). The held-descriptor tx posture exists as the fallback (measured: no advantage even for sequential full reads) — named in the module doc as not-shipped, never coded.
• has — companion-table EXISTS probe. size — the companion-row size probe (metadata-only; the ADR-009 companion-row probe the suite's INV-6 row names). delete — one tx (lo_unlink + row delete), rows-affected discipline (Err(Error::Missing) when absent); GC-participating (the delete window's arbitration is store-core and table-level — nothing fs-specific, ADR-009 §4). list — SELECT key FROM lo_entries, complete (the POC's exact-count gate: list == companion table after mixed ops).
• Virgin-store posture — the constructor creates the schema and table; the tolerance is the postgres engine's recorded read-only split (reads on a schema-less database map to absent/empty via the SQLSTATE 42P01 discriminant, writes surface the engine error). The schema-less test fixture mirrors the postgres engine's #[cfg(test)] new_schemaless pattern.
• Posture deltas in the module doc, stated not hidden (ADR-009 §5/§6) — LO catalog space is reused but never returned (pg_total_relation_size('pg_largeobject') monitoring named; VACUUM FULL/pg_repack the only shrink path — an operator runbook line, never automatic); autovacuum inherited, keep it on; single-stream cached get 20–50× behind page-cache local fs at ≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture — named capacity facts, not solved problems.
• Orphan-recovery sweep (ADR-009 §5's requirement) — a crate-internal reap_orphans() for the legacy/bypass class only (committed LOs without companion rows): enumerate SELECT DISTINCT loid FROM pg_largeobject as a candidate list only, check each against lo_entries.loid (the truth), lo_unlink the unclaimed (the C6/pgdiag10 shape — cost scales with oid count, not content size). Scope-noted #[allow(dead_code)] until a consumer exists (recorded: the ops/maintenance surface is wave 7+; no store-core verb is scoped this wave), contract-tested with a planted orphan and a byte-exact survival assertion for committed content.
• Unit tests in-module: the method contract (virgin-store no-ops incl. the schema-less form, CAS re-put no-op, Missing discipline, size exactness + metadata-only, list completeness after mixed ops, ranged reads at window boundaries + beyond-length failure), the staged-put happy path and both hygiene forms, the orphan sweep. Tests need a live server — the alkstore harness convention: connection settings ride the environment (ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB), tests skip cleanly server-less, acceptance requires green against the server. The harness server: dockerized pglo-poc (postgres:16-alpine) on :15432, db blobs, user/password postgres/poc (verified live 2026-10-12).
• LO isolation on the shared harness db (named design point — the C7 lesson, do not rediscover it): Large Objects are database-level, not schema-level — the schema-per-test convention does not isolate them, a DROP SCHEMA leaves the LO's pages in pg_largeobject, and shared-LO tests raced each other into spurious "large object NNNN does not exist" aborts in the POC. Decide and record the isolation shape: per-test CREATE DATABASE (the POC's proven open_fresh shape) or per-instance LOID tracking + teardown lo_unlink (a LoJanitor drop-guard, the SchemaJanitor pattern extended). Either way acceptance includes zero LO residue on the shared db (pg_largeobject probed before/after full runs).
• Review amendment A2 (wave-4 gate, owner: this task) — adopt the schema-janitor discipline from the start; the wave-4 lesson (380 leaked schemas) is not to be re-learned. The SchemaJanitor lives in src/postgres.rs's #[cfg(test)] test_support — its new shared home (gated cfg(test) + any(postgres, pg-lo)) rides the shared-helper design point.

The window-family seam clauses (EngineWindowState) are engine-pg-lo-window (separate task); the constructor wiring is store-pg-lo-construction. Nothing of EngineFleetState may exist on this engine (ADR-012 §5's pg-lo row: pin clauses no, sweep lock no — fleet pins live on the kv host). This task ships the engine and its in-module unit tests only.

Acceptance Criteria

• src/pg_lo.rs implements Backend completely (seven methods + put_staged) behind feature pg-lo (default-off), with the descriptorless lo_get window cursor body and the held-tx staging body
• The companion table is the contract authority (list/size/has/CAS all table-level); pg_largeobject is never read for addressing
• One engine-owned schema, idempotent bootstrap at the constructor; the quoted-identifier boundary; the virgin-store read-only split (42P01 → absent/empty on reads, writes surface) unit-tested incl. the schema-less form
• Put = one tx (lo_create → 512 KiB chunked lowrite → row insert → COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named constant with the C2 citation
• Stage hygiene holds in both forms (discard → zero residue; failed commit → rollback self-discard), exact-count unit-tested
• Reads ride lo_get(oid, off, len) windows; len() metadata-only from the companion row; beyond-length reads fail cleanly; the 64-bit-variants discipline recorded (the C7 trap not inherited)
• delete = one tx (lo_unlink + row delete) with the Missing discipline; list complete after mixed ops
• The orphan-recovery sweep exists (candidates from pg_largeobject only, validated against lo_entries), scope-noted, contract-tested with a planted orphan
• Posture deltas recorded in the module doc (catalog space never returned + VACUUM FULL operator line + rel-size monitoring; autovacuum inherited; the cached-get capacity facts)
• Shared-helper + PgOpts + LO-isolation design points decided and recorded; the pg-lo pure builds (--no-default-features --features "pg-lo,mem" and --features "postgres,pg-lo") compile clean (the literal kv-less config is not a gate — see the Description)
• Unit tests env-driven (ALKBLOBS_PG_*), skipping cleanly server-less; green against the harness server at acceptance; zero schema + LO residue on the shared db after runs
• cargo test (default — pg-lo compiles out) green; cargo test --features pg-lo green server-less (skipping); cargo clippy --all-targets -- -D warnings and cargo fmt --check green (default + --features pg-lo + the pg-lo pure builds)

References

• docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the engine ADR — §1–§4 the contract, §5/§6 the posture deltas)
• docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan sweep, C7 the harness lessons)
• docs/architecture/decisions/003-backend-contract-and-dispatch.md (the trait contract)
• docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract; the large-tier cursor body families)
• docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §4/§5 (the consolidation posture; the pg-lo applicability row — window family only, no fleet state)
• docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §4 (pg-lo does not imply postgres; the pg-lo-pure build)
• src/backend.rs (the trait + seam contracts this implements)
• src/postgres.rs (the driver-posture twin: pool, prepare_cached, schema bootstrap, 42P01 split, test_support janitor)
• src/local.rs (the large-tier body twin: streaming cursor, staged put hygiene shapes)
• /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line PgLoBackend this engine productionizes)

Notes

Implemented in src/pg_lo.rs (the engine, its cursors, the orphan sweep, and 18 in-module tests), src/pg_common.rs (new — the shared-helper home), and src/{lib,postgres}.rs (the shared-helper move + gate widening; src/postgres.rs's changes are mechanical import swaps plus path-stable re-exports). Recorded design points, as resolved:

• Shared-helper shape — src/pg_common.rs, gated cfg(any(feature = "postgres", feature = "pg-lo")), one module per file: quote_identifier/qualified_table, the 42P01 discriminant, err/pg_err (the mapping keeps the postgres: prefix — both engines attach to the same server family), size_u64, time_to_nanos (the i64-nanos encoding), build_pool, [PgOpts], and the #[cfg(test)] test_support janitors (SchemaJanitor gated #[cfg(feature = "postgres")], new DatabaseJanitor gated #[cfg(feature = "pg-lo")]; crate::postgres::test_support remains a path-stable re-export, so src/store.rs's import needed no edit).
• PgOpts — one shared type; its definition moved to pg_common, the public re-export widened to any(postgres, pg-lo) (the type is unchanged; alkblobs::postgres::PgOpts still resolves through the path-stable re-export). The synchronous_commit knob carries (default on; the POC's server-side off was a measurement posture, not the ship config — recorded in the module doc).
• LO isolation (the C7 lesson resolved) — Large Objects are database-level; schema-per-test does not isolate them. The tests run on per-test dedicated databases (the POC's proven open_fresh shape): unique abtlo_{tag}_{pid}_{seq}_{nanos} names, CREATE DATABASE through the postgres maintenance db, a DatabaseJanitor drop-guard (DROP DATABASE … WITH (FORCE)) after the backend's pool. Zero LO residue on the shared blobs db is structural (no pg-lo test ever opens a pool against it) and gate- tested (shared_harness_db_holds_zero_lo_residue).
• committed_at — the crate-wide i64-nanos encoding as bigint, engine-stamped at insert (not a server-default column); the crate's one timestamp encoding rather than the POC's timestamptz default now().
• Staged reads — verified live: lo_get inside the writing tx sees its own uncommitted pages (read-your-own-writes), so the verification pass reads the staged artifact through descriptorless windows + the unflushed in-memory tail; no descriptor reads, no lo_lseek anywhere (the C7 64-bit discipline: offsets ride bigint only, lowrite is implicitly sequential).
• Stage hygiene mechanics — the staging cursor takes its checkout in commit (publish over the taken Object), so every

  failure path converges on one ROLLBACK point; the dedup path (CAS hit) ROLLBACKs and returns Ok (the staged LO is the residue removed); Drop hands an abandoned cursor's open tx to a spawned ROLLBACK (never returns a dirty checkout to the pool — deadpool's Fast recycle pitfall), degrading to a one-shot thread when no runtime is live. #[cfg(test)] fail_commit knob mirrors mem/sqlite/postgres/local.

• Harness-server residue found and cleaned — the shared blobs db carried pre-existing exploration residue (public.lo_entries
  • 21 LOs, 9 472 catalog pages) from pre-task pg-lo probing; unlinked + dropped (count now 0) so the pg_largeobject-probe gate measures this suite's own runs. The public.blobs kv table was left untouched (kv-engine content, out of scope).
• The literal --no-default-features --features pg-lo config cannot compile the whole crate — store.rs's KvEngine/KvCursor enums are empty there (E0004), identically on the pre-task tree (verified by stash). Every constructor mode pairs a kv engine (ADR-011 §4's table), so a kv-less store is not a shippable shape; the clean configs are --no-default-features --features "pg-lo,mem" and ADR-013 §4's pg-pure build --no-default-features --features "postgres,pg-lo" — both clippy clean. In the literal config the shared pieces compile (the 10 store.rs E0004s are the crate's only errors; nothing in pg_common/pg_lo errors). Flagged for store-pg-lo-construction to keep in view.

Summary

Completed 2026-10-12 (wave 5). src/pg_lo.rs implements the seven-method Backend + put_staged behind feature pg-lo (default-off): LOs hold the bytes, the lo_entries companion table is the contract authority (list/size/has/CAS all table-level; pg_largeobject read by the engine only inside the orphan sweep's candidate enumeration), one-tx puts (lo_create → 512 KiB chunked lowrite (LOWRITE_CHUNK, the C2 knee) → row insert → COMMIT), the descriptorless lo_get(oid, off, len) window cursor (READ_CHUNK; len() metadata-only from the companion row; EOF clamping per the cursor contract plus the unexpected-EOF clean failure for a body short of its claimed length), the held-tx staging cursor with aggregation of the store core's 64 KiB windows into 512 KiB statements and both hygiene forms exact-count-tested, one-tx deletes with the Missing discipline, the complete single-snapshot list, idempotent schema+table bootstrap at the constructor with the quoted-identifier boundary (hostile names tested), the 42P01 read-only virgin-store split (schema-less form tested), the crate-internal reap_orphans sweep (scope-noted; planted-orphan contract test with byte-exact committed-content survival), and the posture deltas in the module doc (catalog space never returned + VACUUM FULL runbook line + rel-size monitoring; autovacuum inherited; the 20–50× cached-get capacity facts). Shared helpers (pg_err, 42P01, quoting, timestamps, pool, PgOpts) moved to the new src/pg_common.rs gated any(postgres, pg-lo); PgOpts's public re-export widened (shared opts decision; the knob carries). 18 in-module tests, per-test dedicated databases (DatabaseJanitor), env-driven (ALKBLOBS_PG_*), skipping cleanly server-less. Verified: cargo test (69/19/3 ok), cargo test --features pg-lo live (87/19/3 ok, 18/18 pg-lo), cargo test --features postgres live (103/36/3 ok), cargo test --all-features (176/71/4 ok), clippy -D warnings green (default, --features pg-lo, --features postgres, --all-features, --no-default-features --features "pg-lo,mem", --no-default-features --features "postgres,pg-lo"), cargo fmt --check green, cargo doc --no-deps clean, zero LO + schema + database residue on the harness server after full runs. Engine-state window family remains engine-pg-lo-window (EngineWindowState untouched; nothing of EngineFleetState exists on this engine); constructor wiring remains store-pg-lo-construction.