docs(tasks): decompose wave 5 — the pg-lo large engine, the window host's large-tier arm, the LO-medium SKIP-posture column

- engine-pg-lo: the Backend impl behind feature pg-lo (default-off) —
  companion-table authority, one-tx staged put at the 512 KiB knee,
  descriptorless lo_get windows, the orphan-recovery sweep, the
  posture deltas recorded; the shared-helper/PgOpts/LO-isolation
  design points named (LOs are database-level — the C7 lesson cited
  so it is not rediscovered)
- engine-pg-lo-window: the EngineWindowState impl per ADR-012 §5's
  pg-lo row — gc_window only, no fleet family anywhere
- store-pg-lo-construction: LargeEngine::PgLo slotting (the large
  sub-enum design point), the consolidation + mem-pglo constructors
  (the fleet-valid set completed, implicitly non-fleet), and the
  one-window-host rule's store-core completion (dormancy + the
  large-host sweep arm)
- suite-column-pg-lo: the second SKIP-posture column, two factories
  spanning feature pairs, the dormancy row, zero LO residue in
  teardown
- review-wave-5: the engine-wave gate; amendment A2 (janitor
  adoption) verified here

Harness server cited in every server-touching task: dockerized
pglo-poc (postgres:16-alpine) on :15432, db blobs, postgres/poc
(verified live 2026-10-12); env-driven ALKBLOBS_PG_* settings, clean
server-less skips, acceptance requires green against the server.

Verification: cargo test 69/19/3 ok, clippy --all-targets -D warnings
green, fmt --check green (docs-only change).
This commit is contained in:
glm-5.3-flash committed 2026-10-12 07:13:51 +00:00
1 parent f3e8a88e2e
commit 5a217f7bc7
6 files changed
+785 -2

No files matched your search

+237
View File
@@ -0,0 +1,237 @@
---
id: engine-pg-lo
name: pg-lo large engine — Large Objects behind the Backend trait
status: pending
depends_on: []
scope: moderate
risk: medium
impact: component
level: implementation
tags: [wave-5, engines, pg-lo, postgres]
---
## Description
Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off;
ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base)
against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the
2-line module-doc stub behind the pre-existing
`#[cfg(feature = "pg-lo")]` gate). The postgres kv engine
(`src/postgres.rs`, wave 4) is the structural twin for the driver
posture (pool, prepared statements, schema bootstrap, error mapping);
the local engine (`src/local.rs`) is the structural twin for the
large-tier bodies (streaming read cursor, staged put). **`pg-lo` does
not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml
feature carries its own driver deps) — every shared piece must
compile under `--no-default-features --features pg-lo` (the pg-lo-pure
build).
- **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs
hold the bytes; the companion table
`lo_entries (key bytea PRIMARY KEY, loid oid, size bigint,
committed_at)` is the **contract authority** — `list()`, `size()`,
`has()`, CAS-existence are all table-level. **Never trust or read
`pg_largeobject` for addressing** (the C6 sweep validates against
the table only). One engine-owned PostgreSQL schema (the postgres
engine's bootstrap pattern: idempotent DDL at the constructor, the
quoted-identifier boundary, no migration machinery). The default
schema name and the consolidation shape's schema sharing (both
engines on one pg instance) are **recorded design points**.
- **Driver posture (the postgres engine's, reused)** — deadpool-postgres
over tokio-postgres, pool size configurable with a named shipped
default, **per-connection `prepare_cached` discipline** (statements
die with sessions), natively async — **no `spawn_blocking` seam**, no
blocking I/O on the async path. **Shared-helper design point**: the
postgres engine's crate-internal helpers (`quote_identifier`, the
SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp
helpers) live behind `#[cfg(feature = "postgres")]` today — decide
and record the sharing shape (the natural one: a crate-internal
pg-common module gated `cfg(any(feature = "postgres",
feature = "pg-lo"))`, one module per file, so the pg-lo-pure build
compiles without duplication). The `PgOpts` question rides the same
point: it is `pub` and re-exported behind `postgres` (wave 4's
constructor signature put it on the public surface) — widening its
gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a
pg-lo-local opts type is a recorded design point; whether pg-lo
carries the `synchronous_commit` knob at all (the POC ran the server
with it off; the kv parity knob's posture may or may not carry) is
recorded with it.
- **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` →
chunked `lowrite` → companion-row insert → COMMIT. **512 KiB
statements** (POC #7 C2's measured knee; statement count, not
LOBLKSIZE, is what costs) as a named shipped constant. CAS:
companion-table existence check first — a re-put of an existing key
is a no-op tx, never an overwrite (the C1 shape). The store core's
put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates
appends into 512 KiB `lowrite` statements (recorded design point).
- **Staged put (the engine-level `put_staged` contract)** — the
staging cursor body is a **held tx**: handout opens the checkout +
tx + `lo_create`; `append` rides the chunked `lowrite` aggregation;
`commit` publishes (companion-row insert + COMMIT — the row and the
LO's pages land in one step, zero crash orphans structurally, C6);
`discard` rolls back and returns the checkout. Every failure path
converges on rollback — stage hygiene holds in both forms (discard →
zero residue: no LO pages, no row; failed commit → self-discard),
exact-count tested (the `#[cfg(test)]` fail-commit knob standing in
for media failure, the mem/sqlite/postgres pattern). **Scope
boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet*
joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's
ruling that pg-lo contributes LO writes + companion-row insert into
the store-held tx); its API shape is a wave-6 design point. This
task ships the engine-level staging cursor only.
- **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3,
the C4 posture)** — the read cursor closes over the companion row
(oid + length; content immutable, so the row is authoritative), not
over a tx-scoped descriptor. Range reads are single stateless
statements, pool-friendly. `len()` reads the companion row —
**metadata only, never LO content** (ADR-008's discipline). A range
read beyond content length fails cleanly (`UnexpectedEof`-shaped →
the engine's `Error::Io` mapping; the POC's
`ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*`
64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any
positioning is used; `lo_get`'s bigint offsets already are — the C7
int4 trap must not be inherited). The held-descriptor tx posture
exists as the fallback (measured: no advantage even for sequential
full reads) — named in the module doc as not-shipped, never coded.
- **`has`** — companion-table EXISTS probe. **`size`** — the
companion-row `size` probe (metadata-only; the ADR-009 companion-row
probe the suite's INV-6 row names). **`delete`** — one tx
(`lo_unlink` + row delete), rows-affected discipline
(`Err(Error::Missing)` when absent); GC-participating (the delete
window's arbitration is store-core and table-level — nothing
fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`,
complete (the POC's exact-count gate: list == companion table after
mixed ops).
- **Virgin-store posture** — the constructor creates the schema and
table; the tolerance is the postgres engine's recorded read-only
split (reads on a schema-less database map to absent/empty via the
SQLSTATE 42P01 discriminant, writes surface the engine error). The
schema-less test fixture mirrors the postgres engine's
`#[cfg(test)] new_schemaless` pattern.
- **Posture deltas in the module doc, stated not hidden (ADR-009
§5/§6)** — LO catalog space is reused but never returned
(`pg_total_relation_size('pg_largeobject')` monitoring named;
`VACUUM FULL`/`pg_repack` the only shrink path — an operator
runbook line, never automatic); autovacuum inherited, keep it on;
single-stream cached get 20–50× behind page-cache local fs at
≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture —
named capacity facts, not solved problems.
- **Orphan-recovery sweep (ADR-009 §5's requirement)** — a
crate-internal `reap_orphans()` for the legacy/bypass class only
(committed LOs without companion rows): enumerate
`SELECT DISTINCT loid FROM pg_largeobject` as a candidate list
**only**, check each against `lo_entries.loid` (the truth),
`lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with
oid count, not content size). Scope-noted `#[allow(dead_code)]`
until a consumer exists (recorded: the ops/maintenance surface is
wave 7+; no store-core verb is scoped this wave), contract-tested
with a planted orphan and a byte-exact survival assertion for
committed content.
- **Unit tests** in-module: the method contract (virgin-store no-ops
incl. the schema-less form, CAS re-put no-op, Missing discipline,
size exactness + metadata-only, list completeness after mixed ops,
ranged reads at window boundaries + beyond-length failure), the
staged-put happy path and both hygiene forms, the orphan sweep.
**Tests need a live server** — the alkstore harness convention:
connection settings ride the environment
(`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly
server-less, acceptance requires green against the server. The
harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on
:15432, db `blobs`, user/password `postgres`/`poc` (verified live
2026-10-12).
- **LO isolation on the shared harness db (named design point — the
C7 lesson, do not rediscover it)**: Large Objects are
**database-level**, not schema-level — the schema-per-test
convention does not isolate them, a `DROP SCHEMA` leaves the LO's
pages in `pg_largeobject`, and shared-LO tests raced each other into
spurious "large object NNNN does not exist" aborts in the POC. Decide
and record the isolation shape: per-test `CREATE DATABASE` (the
POC's proven `open_fresh` shape) or per-instance LOID tracking +
teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor
pattern extended). Either way acceptance includes **zero LO residue
on the shared db** (`pg_largeobject` probed before/after full runs).
- **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the
schema-janitor discipline from the start; the wave-4 lesson (380
leaked schemas) is not to be re-learned. The `SchemaJanitor` lives
in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared
home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the
shared-helper design point.
The window-family seam clauses (`EngineWindowState`) are
`engine-pg-lo-window` (separate task); the constructor wiring is
`store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist
on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep
lock **no** — fleet pins live on the kv host). This task ships the
engine and its in-module unit tests only.
## Acceptance Criteria
- [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods +
`put_staged`) behind feature `pg-lo` (default-off), with the
descriptorless `lo_get` window cursor body and the held-tx
staging body
- [ ] The companion table is the contract authority (list/size/has/CAS
all table-level); `pg_largeobject` is never read for addressing
- [ ] One engine-owned schema, idempotent bootstrap at the
constructor; the quoted-identifier boundary; the virgin-store
read-only split (42P01 → absent/empty on reads, writes surface)
unit-tested incl. the schema-less form
- [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert →
COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named
constant with the C2 citation
- [ ] Stage hygiene holds in both forms (discard → zero residue; failed
commit → rollback self-discard), exact-count unit-tested
- [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only
from the companion row; beyond-length reads fail cleanly; the
64-bit-variants discipline recorded (the C7 trap not inherited)
- [ ] `delete` = one tx (lo_unlink + row delete) with the Missing
discipline; `list` complete after mixed ops
- [ ] The orphan-recovery sweep exists (candidates from
`pg_largeobject` only, validated against `lo_entries`),
scope-noted, contract-tested with a planted orphan
- [ ] Posture deltas recorded in the module doc (catalog space never
returned + VACUUM FULL operator line + rel-size monitoring;
autovacuum inherited; the cached-get capacity facts)
- [ ] Shared-helper + PgOpts + LO-isolation design points decided and
recorded; the pg-lo-pure build (`--no-default-features
--features pg-lo`) compiles clean
- [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly
server-less; green against the harness server at acceptance;
zero schema + LO residue on the shared db after runs
- [ ] `cargo test` (default — pg-lo compiles out) green;
`cargo test --features pg-lo` green server-less (skipping);
`cargo clippy --all-targets -- -D warnings` and `cargo fmt
--check` green (default + `--features pg-lo` + the pg-lo-pure
check)
## References
- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the
engine ADR — §1–§4 the contract, §5/§6 the posture deltas)
- docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence
base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan
sweep, C7 the harness lessons)
- docs/architecture/decisions/003-backend-contract-and-dispatch.md
(the trait contract)
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
§1 (the staged-put contract; the large-tier cursor body families)
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
§4/§5 (the consolidation posture; the pg-lo applicability row —
window family only, no fleet state)
- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md
§4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build)
- src/backend.rs (the trait + seam contracts this implements)
- src/postgres.rs (the driver-posture twin: pool, prepare_cached,
schema bootstrap, 42P01 split, test_support janitor)
- src/local.rs (the large-tier body twin: streaming cursor, staged
put hygiene shapes)
- /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line
PgLoBackend this engine productionizes)
## Notes
> To be filled by implementation agent
## Summary
> To be filled on completion