alkblobs/tasks/engine-pg-lo.md

---
id: engine-pg-lo
name: pg-lo large engine — Large Objects behind the Backend trait
status: pending
depends_on: []
scope: moderate
risk: medium
impact: component
level: implementation
tags: [wave-5, engines, pg-lo, postgres]
---

## Description

Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off;
ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base)
against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the
2-line module-doc stub behind the pre-existing
`#[cfg(feature = "pg-lo")]` gate). The postgres kv engine
(`src/postgres.rs`, wave 4) is the structural twin for the driver
posture (pool, prepared statements, schema bootstrap, error mapping);
the local engine (`src/local.rs`) is the structural twin for the
large-tier bodies (streaming read cursor, staged put). **`pg-lo` does
not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml
feature carries its own driver deps) — every shared piece must
compile under `--no-default-features --features pg-lo` (the pg-lo-pure
build).

- **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs
  hold the bytes; the companion table
  `lo_entries (key bytea PRIMARY KEY, loid oid, size bigint,
  committed_at)` is the **contract authority** — `list()`, `size()`,
  `has()`, CAS-existence are all table-level. **Never trust or read
  `pg_largeobject` for addressing** (the C6 sweep validates against
  the table only). One engine-owned PostgreSQL schema (the postgres
  engine's bootstrap pattern: idempotent DDL at the constructor, the
  quoted-identifier boundary, no migration machinery). The default
  schema name and the consolidation shape's schema sharing (both
  engines on one pg instance) are **recorded design points**.
- **Driver posture (the postgres engine's, reused)** — deadpool-postgres
  over tokio-postgres, pool size configurable with a named shipped
  default, **per-connection `prepare_cached` discipline** (statements
  die with sessions), natively async — **no `spawn_blocking` seam**, no
  blocking I/O on the async path. **Shared-helper design point**: the
  postgres engine's crate-internal helpers (`quote_identifier`, the
  SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp
  helpers) live behind `#[cfg(feature = "postgres")]` today — decide
  and record the sharing shape (the natural one: a crate-internal
  pg-common module gated `cfg(any(feature = "postgres",
  feature = "pg-lo"))`, one module per file, so the pg-lo-pure build
  compiles without duplication). The `PgOpts` question rides the same
  point: it is `pub` and re-exported behind `postgres` (wave 4's
  constructor signature put it on the public surface) — widening its
  gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a
  pg-lo-local opts type is a recorded design point; whether pg-lo
  carries the `synchronous_commit` knob at all (the POC ran the server
  with it off; the kv parity knob's posture may or may not carry) is
  recorded with it.
- **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` →
  chunked `lowrite` → companion-row insert → COMMIT. **512 KiB
  statements** (POC #7 C2's measured knee; statement count, not
  LOBLKSIZE, is what costs) as a named shipped constant. CAS:
  companion-table existence check first — a re-put of an existing key
  is a no-op tx, never an overwrite (the C1 shape). The store core's
  put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates
  appends into 512 KiB `lowrite` statements (recorded design point).
- **Staged put (the engine-level `put_staged` contract)** — the
  staging cursor body is a **held tx**: handout opens the checkout +
  tx + `lo_create`; `append` rides the chunked `lowrite` aggregation;
  `commit` publishes (companion-row insert + COMMIT — the row and the
  LO's pages land in one step, zero crash orphans structurally, C6);
  `discard` rolls back and returns the checkout. Every failure path
  converges on rollback — stage hygiene holds in both forms (discard →
  zero residue: no LO pages, no row; failed commit → self-discard),
  exact-count tested (the `#[cfg(test)]` fail-commit knob standing in
  for media failure, the mem/sqlite/postgres pattern). **Scope
  boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet*
  joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's
  ruling that pg-lo contributes LO writes + companion-row insert into
  the store-held tx); its API shape is a wave-6 design point. This
  task ships the engine-level staging cursor only.
- **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3,
  the C4 posture)** — the read cursor closes over the companion row
  (oid + length; content immutable, so the row is authoritative), not
  over a tx-scoped descriptor. Range reads are single stateless
  statements, pool-friendly. `len()` reads the companion row —
  **metadata only, never LO content** (ADR-008's discipline). A range
  read beyond content length fails cleanly (`UnexpectedEof`-shaped →
  the engine's `Error::Io` mapping; the POC's
  `ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*`
  64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any
  positioning is used; `lo_get`'s bigint offsets already are — the C7
  int4 trap must not be inherited). The held-descriptor tx posture
  exists as the fallback (measured: no advantage even for sequential
  full reads) — named in the module doc as not-shipped, never coded.
- **`has`** — companion-table EXISTS probe. **`size`** — the
  companion-row `size` probe (metadata-only; the ADR-009 companion-row
  probe the suite's INV-6 row names). **`delete`** — one tx
  (`lo_unlink` + row delete), rows-affected discipline
  (`Err(Error::Missing)` when absent); GC-participating (the delete
  window's arbitration is store-core and table-level — nothing
  fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`,
  complete (the POC's exact-count gate: list == companion table after
  mixed ops).
- **Virgin-store posture** — the constructor creates the schema and
  table; the tolerance is the postgres engine's recorded read-only
  split (reads on a schema-less database map to absent/empty via the
  SQLSTATE 42P01 discriminant, writes surface the engine error). The
  schema-less test fixture mirrors the postgres engine's
  `#[cfg(test)] new_schemaless` pattern.
- **Posture deltas in the module doc, stated not hidden (ADR-009
  §5/§6)** — LO catalog space is reused but never returned
  (`pg_total_relation_size('pg_largeobject')` monitoring named;
  `VACUUM FULL`/`pg_repack` the only shrink path — an operator
  runbook line, never automatic); autovacuum inherited, keep it on;
  single-stream cached get 20–50× behind page-cache local fs at
  ≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture —
  named capacity facts, not solved problems.
- **Orphan-recovery sweep (ADR-009 §5's requirement)** — a
  crate-internal `reap_orphans()` for the legacy/bypass class only
  (committed LOs without companion rows): enumerate
  `SELECT DISTINCT loid FROM pg_largeobject` as a candidate list
  **only**, check each against `lo_entries.loid` (the truth),
  `lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with
  oid count, not content size). Scope-noted `#[allow(dead_code)]`
  until a consumer exists (recorded: the ops/maintenance surface is
  wave 7+; no store-core verb is scoped this wave), contract-tested
  with a planted orphan and a byte-exact survival assertion for
  committed content.
- **Unit tests** in-module: the method contract (virgin-store no-ops
  incl. the schema-less form, CAS re-put no-op, Missing discipline,
  size exactness + metadata-only, list completeness after mixed ops,
  ranged reads at window boundaries + beyond-length failure), the
  staged-put happy path and both hygiene forms, the orphan sweep.
  **Tests need a live server** — the alkstore harness convention:
  connection settings ride the environment
  (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly
  server-less, acceptance requires green against the server. The
  harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on
  :15432, db `blobs`, user/password `postgres`/`poc` (verified live
  2026-10-12).
- **LO isolation on the shared harness db (named design point — the
  C7 lesson, do not rediscover it)**: Large Objects are
  **database-level**, not schema-level — the schema-per-test
  convention does not isolate them, a `DROP SCHEMA` leaves the LO's
  pages in `pg_largeobject`, and shared-LO tests raced each other into
  spurious "large object NNNN does not exist" aborts in the POC. Decide
  and record the isolation shape: per-test `CREATE DATABASE` (the
  POC's proven `open_fresh` shape) or per-instance LOID tracking +
  teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor
  pattern extended). Either way acceptance includes **zero LO residue
  on the shared db** (`pg_largeobject` probed before/after full runs).
- **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the
  schema-janitor discipline from the start; the wave-4 lesson (380
  leaked schemas) is not to be re-learned. The `SchemaJanitor` lives
  in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared
  home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the
  shared-helper design point.

The window-family seam clauses (`EngineWindowState`) are
`engine-pg-lo-window` (separate task); the constructor wiring is
`store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist
on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep
lock **no** — fleet pins live on the kv host). This task ships the
engine and its in-module unit tests only.

## Acceptance Criteria

- [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods +
      `put_staged`) behind feature `pg-lo` (default-off), with the
      descriptorless `lo_get` window cursor body and the held-tx
      staging body
- [ ] The companion table is the contract authority (list/size/has/CAS
      all table-level); `pg_largeobject` is never read for addressing
- [ ] One engine-owned schema, idempotent bootstrap at the
      constructor; the quoted-identifier boundary; the virgin-store
      read-only split (42P01 → absent/empty on reads, writes surface)
      unit-tested incl. the schema-less form
- [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert →
      COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named
      constant with the C2 citation
- [ ] Stage hygiene holds in both forms (discard → zero residue; failed
      commit → rollback self-discard), exact-count unit-tested
- [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only
      from the companion row; beyond-length reads fail cleanly; the
      64-bit-variants discipline recorded (the C7 trap not inherited)
- [ ] `delete` = one tx (lo_unlink + row delete) with the Missing
      discipline; `list` complete after mixed ops
- [ ] The orphan-recovery sweep exists (candidates from
      `pg_largeobject` only, validated against `lo_entries`),
      scope-noted, contract-tested with a planted orphan
- [ ] Posture deltas recorded in the module doc (catalog space never
      returned + VACUUM FULL operator line + rel-size monitoring;
      autovacuum inherited; the cached-get capacity facts)
- [ ] Shared-helper + PgOpts + LO-isolation design points decided and
      recorded; the pg-lo-pure build (`--no-default-features
      --features pg-lo`) compiles clean
- [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly
      server-less; green against the harness server at acceptance;
      zero schema + LO residue on the shared db after runs
- [ ] `cargo test` (default — pg-lo compiles out) green;
      `cargo test --features pg-lo` green server-less (skipping);
      `cargo clippy --all-targets -- -D warnings` and `cargo fmt
      --check` green (default + `--features pg-lo` + the pg-lo-pure
      check)

## References

- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the
  engine ADR — §1–§4 the contract, §5/§6 the posture deltas)
- docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence
  base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan
  sweep, C7 the harness lessons)
- docs/architecture/decisions/003-backend-contract-and-dispatch.md
  (the trait contract)
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
  §1 (the staged-put contract; the large-tier cursor body families)
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
  §4/§5 (the consolidation posture; the pg-lo applicability row —
  window family only, no fleet state)
- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md
  §4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build)
- src/backend.rs (the trait + seam contracts this implements)
- src/postgres.rs (the driver-posture twin: pool, prepare_cached,
  schema bootstrap, 42P01 split, test_support janitor)
- src/local.rs (the large-tier body twin: streaming cursor, staged
  put hygiene shapes)
- /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line
  PgLoBackend this engine productionizes)

## Notes

> To be filled by implementation agent

## Summary

> To be filled on completion