alkblobs/tasks/engine-pg-lo.md

---
id: engine-pg-lo
name: pg-lo large engine — Large Objects behind the Backend trait
status: completed
depends_on: []
scope: moderate
risk: medium
impact: component
level: implementation
tags: [wave-5, engines, pg-lo, postgres]
---

## Description

Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off;
ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base)
against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the
2-line module-doc stub behind the pre-existing
`#[cfg(feature = "pg-lo")]` gate). The postgres kv engine
(`src/postgres.rs`, wave 4) is the structural twin for the driver
posture (pool, prepared statements, schema bootstrap, error mapping);
the local engine (`src/local.rs`) is the structural twin for the
large-tier bodies (streaming read cursor, staged put). **`pg-lo` does
not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml
feature carries its own driver deps) — every shared piece must compile
under the **pg-lo pure builds**: `--no-default-features --features
"pg-lo,mem"` and `--no-default-features --features "postgres,pg-lo"`.
**The literal kv-less config (`--no-default-features --features
pg-lo`) is not a gate and cannot compile the whole crate** — with no kv
feature enabled, `KvEngine`/`KvCursor` go empty and the facade's
by-reference dispatch matches are non-exhaustive (E0004; references
are always considered inhabited) — see `store-pg-lo-construction` for
the full reading.

- **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs
  hold the bytes; the companion table
  `lo_entries (key bytea PRIMARY KEY, loid oid, size bigint,
  committed_at)` is the **contract authority** — `list()`, `size()`,
  `has()`, CAS-existence are all table-level. **Never trust or read
  `pg_largeobject` for addressing** (the C6 sweep validates against
  the table only). One engine-owned PostgreSQL schema (the postgres
  engine's bootstrap pattern: idempotent DDL at the constructor, the
  quoted-identifier boundary, no migration machinery). The default
  schema name and the consolidation shape's schema sharing (both
  engines on one pg instance) are **recorded design points**.
- **Driver posture (the postgres engine's, reused)** — deadpool-postgres
  over tokio-postgres, pool size configurable with a named shipped
  default, **per-connection `prepare_cached` discipline** (statements
  die with sessions), natively async — **no `spawn_blocking` seam**, no
  blocking I/O on the async path. **Shared-helper design point**: the
  postgres engine's crate-internal helpers (`quote_identifier`, the
  SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp
  helpers) live behind `#[cfg(feature = "postgres")]` today — decide
  and record the sharing shape (the natural one: a crate-internal
  pg-common module gated `cfg(any(feature = "postgres",
   feature = "pg-lo"))`, one module per file, so the pg-lo pure builds
   compile without duplication). The `PgOpts` question rides the same
  point: it is `pub` and re-exported behind `postgres` (wave 4's
  constructor signature put it on the public surface) — widening its
  gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a
  pg-lo-local opts type is a recorded design point; whether pg-lo
  carries the `synchronous_commit` knob at all (the POC ran the server
  with it off; the kv parity knob's posture may or may not carry) is
  recorded with it.
- **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` →
  chunked `lowrite` → companion-row insert → COMMIT. **512 KiB
  statements** (POC #7 C2's measured knee; statement count, not
  LOBLKSIZE, is what costs) as a named shipped constant. CAS:
  companion-table existence check first — a re-put of an existing key
  is a no-op tx, never an overwrite (the C1 shape). The store core's
  put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates
  appends into 512 KiB `lowrite` statements (recorded design point).
- **Staged put (the engine-level `put_staged` contract)** — the
  staging cursor body is a **held tx**: handout opens the checkout +
  tx + `lo_create`; `append` rides the chunked `lowrite` aggregation;
  `commit` publishes (companion-row insert + COMMIT — the row and the
  LO's pages land in one step, zero crash orphans structurally, C6);
  `discard` rolls back and returns the checkout. Every failure path
  converges on rollback — stage hygiene holds in both forms (discard →
  zero residue: no LO pages, no row; failed commit → self-discard),
  exact-count tested (the `#[cfg(test)]` fail-commit knob standing in
  for media failure, the mem/sqlite/postgres pattern). **Scope
  boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet*
  joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's
  ruling that pg-lo contributes LO writes + companion-row insert into
  the store-held tx); its API shape is a wave-6 design point. This
  task ships the engine-level staging cursor only.
- **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3,
  the C4 posture)** — the read cursor closes over the companion row
  (oid + length; content immutable, so the row is authoritative), not
  over a tx-scoped descriptor. Range reads are single stateless
  statements, pool-friendly. `len()` reads the companion row —
  **metadata only, never LO content** (ADR-008's discipline). A range
  read beyond content length fails cleanly (`UnexpectedEof`-shaped →
  the engine's `Error::Io` mapping; the POC's
  `ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*`
  64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any
  positioning is used; `lo_get`'s bigint offsets already are — the C7
  int4 trap must not be inherited). The held-descriptor tx posture
  exists as the fallback (measured: no advantage even for sequential
  full reads) — named in the module doc as not-shipped, never coded.
- **`has`** — companion-table EXISTS probe. **`size`** — the
  companion-row `size` probe (metadata-only; the ADR-009 companion-row
  probe the suite's INV-6 row names). **`delete`** — one tx
  (`lo_unlink` + row delete), rows-affected discipline
  (`Err(Error::Missing)` when absent); GC-participating (the delete
  window's arbitration is store-core and table-level — nothing
  fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`,
  complete (the POC's exact-count gate: list == companion table after
  mixed ops).
- **Virgin-store posture** — the constructor creates the schema and
  table; the tolerance is the postgres engine's recorded read-only
  split (reads on a schema-less database map to absent/empty via the
  SQLSTATE 42P01 discriminant, writes surface the engine error). The
  schema-less test fixture mirrors the postgres engine's
  `#[cfg(test)] new_schemaless` pattern.
- **Posture deltas in the module doc, stated not hidden (ADR-009
  §5/§6)** — LO catalog space is reused but never returned
  (`pg_total_relation_size('pg_largeobject')` monitoring named;
  `VACUUM FULL`/`pg_repack` the only shrink path — an operator
  runbook line, never automatic); autovacuum inherited, keep it on;
  single-stream cached get 20–50× behind page-cache local fs at
  ≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture —
  named capacity facts, not solved problems.
- **Orphan-recovery sweep (ADR-009 §5's requirement)** — a
  crate-internal `reap_orphans()` for the legacy/bypass class only
  (committed LOs without companion rows): enumerate
  `SELECT DISTINCT loid FROM pg_largeobject` as a candidate list
  **only**, check each against `lo_entries.loid` (the truth),
  `lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with
  oid count, not content size). Scope-noted `#[allow(dead_code)]`
  until a consumer exists (recorded: the ops/maintenance surface is
  wave 7+; no store-core verb is scoped this wave), contract-tested
  with a planted orphan and a byte-exact survival assertion for
  committed content.
- **Unit tests** in-module: the method contract (virgin-store no-ops
  incl. the schema-less form, CAS re-put no-op, Missing discipline,
  size exactness + metadata-only, list completeness after mixed ops,
  ranged reads at window boundaries + beyond-length failure), the
  staged-put happy path and both hygiene forms, the orphan sweep.
  **Tests need a live server** — the alkstore harness convention:
  connection settings ride the environment
  (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly
  server-less, acceptance requires green against the server. The
  harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on
  :15432, db `blobs`, user/password `postgres`/`poc` (verified live
  2026-10-12).
- **LO isolation on the shared harness db (named design point — the
  C7 lesson, do not rediscover it)**: Large Objects are
  **database-level**, not schema-level — the schema-per-test
  convention does not isolate them, a `DROP SCHEMA` leaves the LO's
  pages in `pg_largeobject`, and shared-LO tests raced each other into
  spurious "large object NNNN does not exist" aborts in the POC. Decide
  and record the isolation shape: per-test `CREATE DATABASE` (the
  POC's proven `open_fresh` shape) or per-instance LOID tracking +
  teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor
  pattern extended). Either way acceptance includes **zero LO residue
  on the shared db** (`pg_largeobject` probed before/after full runs).
- **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the
  schema-janitor discipline from the start; the wave-4 lesson (380
  leaked schemas) is not to be re-learned. The `SchemaJanitor` lives
  in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared
  home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the
  shared-helper design point.

The window-family seam clauses (`EngineWindowState`) are
`engine-pg-lo-window` (separate task); the constructor wiring is
`store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist
on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep
lock **no** — fleet pins live on the kv host). This task ships the
engine and its in-module unit tests only.

## Acceptance Criteria

- [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods +
      `put_staged`) behind feature `pg-lo` (default-off), with the
      descriptorless `lo_get` window cursor body and the held-tx
      staging body
- [ ] The companion table is the contract authority (list/size/has/CAS
      all table-level); `pg_largeobject` is never read for addressing
- [ ] One engine-owned schema, idempotent bootstrap at the
      constructor; the quoted-identifier boundary; the virgin-store
      read-only split (42P01 → absent/empty on reads, writes surface)
      unit-tested incl. the schema-less form
- [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert →
      COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named
      constant with the C2 citation
- [ ] Stage hygiene holds in both forms (discard → zero residue; failed
      commit → rollback self-discard), exact-count unit-tested
- [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only
      from the companion row; beyond-length reads fail cleanly; the
      64-bit-variants discipline recorded (the C7 trap not inherited)
- [ ] `delete` = one tx (lo_unlink + row delete) with the Missing
      discipline; `list` complete after mixed ops
- [ ] The orphan-recovery sweep exists (candidates from
      `pg_largeobject` only, validated against `lo_entries`),
      scope-noted, contract-tested with a planted orphan
- [ ] Posture deltas recorded in the module doc (catalog space never
      returned + VACUUM FULL operator line + rel-size monitoring;
      autovacuum inherited; the cached-get capacity facts)
- [ ] Shared-helper + PgOpts + LO-isolation design points decided and
      recorded; the pg-lo pure builds (`--no-default-features
      --features "pg-lo,mem"` and `--features "postgres,pg-lo"`)
      compile clean (the literal kv-less config is not a gate — see
      the Description)
- [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly
      server-less; green against the harness server at acceptance;
      zero schema + LO residue on the shared db after runs
- [ ] `cargo test` (default — pg-lo compiles out) green;
      `cargo test --features pg-lo` green server-less (skipping);
      `cargo clippy --all-targets -- -D warnings` and `cargo fmt
      --check` green (default + `--features pg-lo` + the pg-lo pure
      builds)

## References

- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the
  engine ADR — §1–§4 the contract, §5/§6 the posture deltas)
- docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence
  base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan
  sweep, C7 the harness lessons)
- docs/architecture/decisions/003-backend-contract-and-dispatch.md
  (the trait contract)
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
  §1 (the staged-put contract; the large-tier cursor body families)
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
  §4/§5 (the consolidation posture; the pg-lo applicability row —
  window family only, no fleet state)
- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md
  §4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build)
- src/backend.rs (the trait + seam contracts this implements)
- src/postgres.rs (the driver-posture twin: pool, prepare_cached,
  schema bootstrap, 42P01 split, test_support janitor)
- src/local.rs (the large-tier body twin: streaming cursor, staged
  put hygiene shapes)
- /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line
  PgLoBackend this engine productionizes)

## Notes

> Implemented in `src/pg_lo.rs` (the engine, its cursors, the orphan
> sweep, and 18 in-module tests), `src/pg_common.rs` (new — the
> shared-helper home), and `src/{lib,postgres}.rs` (the shared-helper
> move + gate widening; `src/postgres.rs`'s changes are mechanical
> import swaps plus path-stable re-exports). Recorded design points,
> as resolved:
>
> - **Shared-helper shape** — `src/pg_common.rs`, gated
>   `cfg(any(feature = "postgres", feature = "pg-lo"))`, one module per
>   file: `quote_identifier`/`qualified_table`, the 42P01 discriminant,
>   `err`/`pg_err` (the mapping keeps the `postgres:` prefix — both
>   engines attach to the same server family), `size_u64`,
>   `time_to_nanos` (the i64-nanos encoding), `build_pool`, [`PgOpts`],
>   and the `#[cfg(test)] test_support` janitors (`SchemaJanitor`
>   gated `#[cfg(feature = "postgres")]`, new `DatabaseJanitor` gated
>   `#[cfg(feature = "pg-lo")]`; `crate::postgres::test_support` remains
>   a path-stable re-export, so `src/store.rs`'s import needed no edit).
> - **PgOpts** — one shared type; its definition moved to `pg_common`,
>   the public re-export widened to `any(postgres, pg-lo)` (the type is
>   unchanged; `alkblobs::postgres::PgOpts` still resolves through the
>   path-stable re-export). The `synchronous_commit` knob **carries**
>   (default `on`; the POC's server-side `off` was a measurement
>   posture, not the ship config — recorded in the module doc).
> - **LO isolation (the C7 lesson resolved)** — Large Objects are
>   database-level; schema-per-test does not isolate them. The tests
>   run on **per-test dedicated databases** (the POC's proven
>   `open_fresh` shape): unique `abtlo_{tag}_{pid}_{seq}_{nanos}`
>   names, `CREATE DATABASE` through the `postgres` maintenance db, a
>   `DatabaseJanitor` drop-guard (`DROP DATABASE … WITH (FORCE)`) after
>   the backend's pool. Zero LO residue on the shared `blobs` db is
>   structural (no pg-lo test ever opens a pool against it) and gate-
>   tested (`shared_harness_db_holds_zero_lo_residue`).
> - **`committed_at`** — the crate-wide i64-nanos encoding as `bigint`,
>   engine-stamped at insert (not a server-default column); the crate's
>   one timestamp encoding rather than the POC's `timestamptz default
>   now()`.
> - **Staged reads** — verified live: `lo_get` inside the writing tx
>   sees its own uncommitted pages (read-your-own-writes), so the
>   verification pass reads the staged artifact through descriptorless
>   windows + the unflushed in-memory tail; no descriptor reads, no
>   `lo_lseek` anywhere (the C7 64-bit discipline: offsets ride bigint
>   only, `lowrite` is implicitly sequential).
> - **Stage hygiene mechanics** — the staging cursor takes its
>   checkout in `commit` (`publish` over the taken `Object`), so every
>   >   failure path converges on one ROLLBACK point; the dedup path
>   (CAS hit) ROLLBACKs and returns `Ok` (the staged LO is the residue
>   removed); `Drop` hands an abandoned cursor's open tx to a spawned
>   ROLLBACK (never returns a dirty checkout to the pool — deadpool's
>   `Fast` recycle pitfall), degrading to a one-shot thread when no
>   runtime is live. `#[cfg(test)] fail_commit` knob mirrors
>   mem/sqlite/postgres/local.
> - **Harness-server residue found and cleaned** — the shared `blobs`
>   db carried pre-existing exploration residue (`public.lo_entries`
>   + 21 LOs, 9 472 catalog pages) from pre-task pg-lo probing;
>   unlinked + dropped (count now 0) so the
>   `pg_largeobject`-probe gate measures this suite's own runs. The
>   `public.blobs` kv table was left untouched (kv-engine content, out
>   of scope).
> - **The literal `--no-default-features --features pg-lo` config
>   cannot compile the whole crate** — `store.rs`'s `KvEngine`/`KvCursor`
>   enums are empty there (E0004), identically on the pre-task tree
>   (verified by stash). Every constructor mode pairs a kv engine
>   (ADR-011 §4's table), so a kv-less store is not a shippable shape;
>   the clean configs are `--no-default-features --features "pg-lo,mem"`
>   and ADR-013 §4's pg-pure build
>   `--no-default-features --features "postgres,pg-lo"` — both clippy
>   clean. In the literal config the *shared pieces* compile (the 10
>   store.rs E0004s are the crate's only errors; nothing in
>   pg_common/pg_lo errors). Flagged for `store-pg-lo-construction` to
>   keep in view.

## Summary

> Completed 2026-10-12 (wave 5). `src/pg_lo.rs` implements the
> seven-method `Backend` + `put_staged` behind feature `pg-lo`
> (default-off): LOs hold the bytes, the `lo_entries` companion table is
> the contract authority (list/size/has/CAS all table-level; `pg_largeobject`
> read by the engine only inside the orphan sweep's candidate
> enumeration), one-tx puts (lo_create → 512 KiB chunked `lowrite`
> (`LOWRITE_CHUNK`, the C2 knee) → row insert → COMMIT), the
> descriptorless `lo_get(oid, off, len)` window cursor
> (`READ_CHUNK`; `len()` metadata-only from the companion row; EOF
> clamping per the cursor contract plus the unexpected-EOF clean
> failure for a body short of its claimed length), the held-tx staging
> cursor with aggregation of the store core's 64 KiB windows into
> 512 KiB statements and both hygiene forms exact-count-tested, one-tx
> deletes with the Missing discipline, the complete single-snapshot
> `list`, idempotent schema+table bootstrap at the constructor with the
> quoted-identifier boundary (hostile names tested), the 42P01
> read-only virgin-store split (schema-less form tested), the
> crate-internal `reap_orphans` sweep (scope-noted; planted-orphan
> contract test with byte-exact committed-content survival), and the
> posture deltas in the module doc (catalog space never returned +
> `VACUUM FULL` runbook line + rel-size monitoring; autovacuum
> inherited; the 20–50× cached-get capacity facts). Shared helpers
> (`pg_err`, 42P01, quoting, timestamps, pool, `PgOpts`) moved to the
> new `src/pg_common.rs` gated `any(postgres, pg-lo)`; `PgOpts`'s
> public re-export widened (shared opts decision; the knob carries).
> 18 in-module tests, per-test dedicated databases (`DatabaseJanitor`),
> env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less. Verified:
> `cargo test` (69/19/3 ok), `cargo test --features pg-lo` live
> (87/19/3 ok, 18/18 pg-lo), `cargo test --features postgres` live
> (103/36/3 ok), `cargo test --all-features` (176/71/4 ok), clippy
> `-D warnings` green (default, `--features pg-lo`, `--features
> postgres`, `--all-features`, `--no-default-features --features
> "pg-lo,mem"`, `--no-default-features --features "postgres,pg-lo"`),
> `cargo fmt --check` green, `cargo doc --no-deps` clean, zero LO +
> schema + database residue on the harness server after full runs.
> Engine-state window family remains `engine-pg-lo-window`
> (`EngineWindowState` untouched; nothing of `EngineFleetState` exists
> on this engine); constructor wiring remains `store-pg-lo-construction`.