docs(tasks): decompose wave 5 — the pg-lo large engine, the window host's large-tier arm, the LO-medium SKIP-posture column
- engine-pg-lo: the Backend impl behind feature pg-lo (default-off) — companion-table authority, one-tx staged put at the 512 KiB knee, descriptorless lo_get windows, the orphan-recovery sweep, the posture deltas recorded; the shared-helper/PgOpts/LO-isolation design points named (LOs are database-level — the C7 lesson cited so it is not rediscovered) - engine-pg-lo-window: the EngineWindowState impl per ADR-012 §5's pg-lo row — gc_window only, no fleet family anywhere - store-pg-lo-construction: LargeEngine::PgLo slotting (the large sub-enum design point), the consolidation + mem-pglo constructors (the fleet-valid set completed, implicitly non-fleet), and the one-window-host rule's store-core completion (dormancy + the large-host sweep arm) - suite-column-pg-lo: the second SKIP-posture column, two factories spanning feature pairs, the dormancy row, zero LO residue in teardown - review-wave-5: the engine-wave gate; amendment A2 (janitor adoption) verified here Harness server cited in every server-touching task: dockerized pglo-poc (postgres:16-alpine) on :15432, db blobs, postgres/poc (verified live 2026-10-12); env-driven ALKBLOBS_PG_* settings, clean server-less skips, acceptance requires green against the server. Verification: cargo test 69/19/3 ok, clippy --all-targets -D warnings green, fmt --check green (docs-only change).
This commit is contained in:
1 parent
f3e8a88e2e
commit
5a217f7bc7
6 files changed
+785
-2
No files matched your search
@@ -0,0 +1,237 @@
|
||||
---
|
||||
id: engine-pg-lo
|
||||
name: pg-lo large engine — Large Objects behind the Backend trait
|
||||
status: pending
|
||||
depends_on: []
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-5, engines, pg-lo, postgres]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the `pg-lo` large-tier engine (feature `pg-lo`, default-off;
|
||||
ADR-009, admitted on POC #7 — findings C1–C7 are the evidence base)
|
||||
against the wave-1 trait (`src/backend.rs`) in `src/pg_lo.rs` (the
|
||||
2-line module-doc stub behind the pre-existing
|
||||
`#[cfg(feature = "pg-lo")]` gate). The postgres kv engine
|
||||
(`src/postgres.rs`, wave 4) is the structural twin for the driver
|
||||
posture (pool, prepared statements, schema bootstrap, error mapping);
|
||||
the local engine (`src/local.rs`) is the structural twin for the
|
||||
large-tier bodies (streaming read cursor, staged put). **`pg-lo` does
|
||||
not imply `postgres`** (ADR-013 §4's consequence; the Cargo.toml
|
||||
feature carries its own driver deps) — every shared piece must
|
||||
compile under `--no-default-features --features pg-lo` (the pg-lo-pure
|
||||
build).
|
||||
|
||||
- **Engine shape (POC #7 C1 is the contract)** — two artifacts: LOs
|
||||
hold the bytes; the companion table
|
||||
`lo_entries (key bytea PRIMARY KEY, loid oid, size bigint,
|
||||
committed_at)` is the **contract authority** — `list()`, `size()`,
|
||||
`has()`, CAS-existence are all table-level. **Never trust or read
|
||||
`pg_largeobject` for addressing** (the C6 sweep validates against
|
||||
the table only). One engine-owned PostgreSQL schema (the postgres
|
||||
engine's bootstrap pattern: idempotent DDL at the constructor, the
|
||||
quoted-identifier boundary, no migration machinery). The default
|
||||
schema name and the consolidation shape's schema sharing (both
|
||||
engines on one pg instance) are **recorded design points**.
|
||||
- **Driver posture (the postgres engine's, reused)** — deadpool-postgres
|
||||
over tokio-postgres, pool size configurable with a named shipped
|
||||
default, **per-connection `prepare_cached` discipline** (statements
|
||||
die with sessions), natively async — **no `spawn_blocking` seam**, no
|
||||
blocking I/O on the async path. **Shared-helper design point**: the
|
||||
postgres engine's crate-internal helpers (`quote_identifier`, the
|
||||
SQLSTATE 42P01 discriminant, `pg_err`, the i64-nanos timestamp
|
||||
helpers) live behind `#[cfg(feature = "postgres")]` today — decide
|
||||
and record the sharing shape (the natural one: a crate-internal
|
||||
pg-common module gated `cfg(any(feature = "postgres",
|
||||
feature = "pg-lo"))`, one module per file, so the pg-lo-pure build
|
||||
compiles without duplication). The `PgOpts` question rides the same
|
||||
point: it is `pub` and re-exported behind `postgres` (wave 4's
|
||||
constructor signature put it on the public surface) — widening its
|
||||
gate to `any(postgres, pg-lo)` (re-export gate widened with it) vs a
|
||||
pg-lo-local opts type is a recorded design point; whether pg-lo
|
||||
carries the `synchronous_commit` knob at all (the POC ran the server
|
||||
with it off; the kv parity knob's posture may or may not carry) is
|
||||
recorded with it.
|
||||
- **Put = one transaction (ADR-009 §2)** — BEGIN → `lo_create` →
|
||||
chunked `lowrite` → companion-row insert → COMMIT. **512 KiB
|
||||
statements** (POC #7 C2's measured knee; statement count, not
|
||||
LOBLKSIZE, is what costs) as a named shipped constant. CAS:
|
||||
companion-table existence check first — a re-put of an existing key
|
||||
is a no-op tx, never an overwrite (the C1 shape). The store core's
|
||||
put loop hands 64 KiB windows (`PUT_CHUNK`); the engine aggregates
|
||||
appends into 512 KiB `lowrite` statements (recorded design point).
|
||||
- **Staged put (the engine-level `put_staged` contract)** — the
|
||||
staging cursor body is a **held tx**: handout opens the checkout +
|
||||
tx + `lo_create`; `append` rides the chunked `lowrite` aggregation;
|
||||
`commit` publishes (companion-row insert + COMMIT — the row and the
|
||||
LO's pages land in one step, zero crash orphans structurally, C6);
|
||||
`discard` rolls back and returns the checkout. Every failure path
|
||||
converges on rollback — stage hygiene holds in both forms (discard →
|
||||
zero residue: no LO pages, no row; failed commit → self-discard),
|
||||
exact-count tested (the `#[cfg(test)]` fail-commit knob standing in
|
||||
for media failure, the mem/sqlite/postgres pattern). **Scope
|
||||
boundary**: ADR-010 §1's "a tx the store core holds" is the *fleet*
|
||||
joint form (wave 6's consolidation mechanics — ADR-012 §4/§5's
|
||||
ruling that pg-lo contributes LO writes + companion-row insert into
|
||||
the store-held tx); its API shape is a wave-6 design point. This
|
||||
task ships the engine-level staging cursor only.
|
||||
- **Get = descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3,
|
||||
the C4 posture)** — the read cursor closes over the companion row
|
||||
(oid + length; content immutable, so the row is authoritative), not
|
||||
over a tx-scoped descriptor. Range reads are single stateless
|
||||
statements, pool-friendly. `len()` reads the companion row —
|
||||
**metadata only, never LO content** (ADR-008's discipline). A range
|
||||
read beyond content length fails cleanly (`UnexpectedEof`-shaped →
|
||||
the engine's `Error::Io` mapping; the POC's
|
||||
`ranged_reads_windows_and_eof` shape). **Reads ride the `lo_*`
|
||||
64-bit variants end-to-end** (`lo_lseek64`/int8 offsets if any
|
||||
positioning is used; `lo_get`'s bigint offsets already are — the C7
|
||||
int4 trap must not be inherited). The held-descriptor tx posture
|
||||
exists as the fallback (measured: no advantage even for sequential
|
||||
full reads) — named in the module doc as not-shipped, never coded.
|
||||
- **`has`** — companion-table EXISTS probe. **`size`** — the
|
||||
companion-row `size` probe (metadata-only; the ADR-009 companion-row
|
||||
probe the suite's INV-6 row names). **`delete`** — one tx
|
||||
(`lo_unlink` + row delete), rows-affected discipline
|
||||
(`Err(Error::Missing)` when absent); GC-participating (the delete
|
||||
window's arbitration is store-core and table-level — nothing
|
||||
fs-specific, ADR-009 §4). **`list`** — `SELECT key FROM lo_entries`,
|
||||
complete (the POC's exact-count gate: list == companion table after
|
||||
mixed ops).
|
||||
- **Virgin-store posture** — the constructor creates the schema and
|
||||
table; the tolerance is the postgres engine's recorded read-only
|
||||
split (reads on a schema-less database map to absent/empty via the
|
||||
SQLSTATE 42P01 discriminant, writes surface the engine error). The
|
||||
schema-less test fixture mirrors the postgres engine's
|
||||
`#[cfg(test)] new_schemaless` pattern.
|
||||
- **Posture deltas in the module doc, stated not hidden (ADR-009
|
||||
§5/§6)** — LO catalog space is reused but never returned
|
||||
(`pg_total_relation_size('pg_largeobject')` monitoring named;
|
||||
`VACUUM FULL`/`pg_repack` the only shrink path — an operator
|
||||
runbook line, never automatic); autovacuum inherited, keep it on;
|
||||
single-stream cached get 20–50× behind page-cache local fs at
|
||||
≥1 MiB with the aggregate ~700 MB/s @ 16 readers serving picture —
|
||||
named capacity facts, not solved problems.
|
||||
- **Orphan-recovery sweep (ADR-009 §5's requirement)** — a
|
||||
crate-internal `reap_orphans()` for the legacy/bypass class only
|
||||
(committed LOs without companion rows): enumerate
|
||||
`SELECT DISTINCT loid FROM pg_largeobject` as a candidate list
|
||||
**only**, check each against `lo_entries.loid` (the truth),
|
||||
`lo_unlink` the unclaimed (the C6/pgdiag10 shape — cost scales with
|
||||
oid count, not content size). Scope-noted `#[allow(dead_code)]`
|
||||
until a consumer exists (recorded: the ops/maintenance surface is
|
||||
wave 7+; no store-core verb is scoped this wave), contract-tested
|
||||
with a planted orphan and a byte-exact survival assertion for
|
||||
committed content.
|
||||
- **Unit tests** in-module: the method contract (virgin-store no-ops
|
||||
incl. the schema-less form, CAS re-put no-op, Missing discipline,
|
||||
size exactness + metadata-only, list completeness after mixed ops,
|
||||
ranged reads at window boundaries + beyond-length failure), the
|
||||
staged-put happy path and both hygiene forms, the orphan sweep.
|
||||
**Tests need a live server** — the alkstore harness convention:
|
||||
connection settings ride the environment
|
||||
(`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB`), tests skip cleanly
|
||||
server-less, acceptance requires green against the server. The
|
||||
harness server: dockerized `pglo-poc` (`postgres:16-alpine`) on
|
||||
:15432, db `blobs`, user/password `postgres`/`poc` (verified live
|
||||
2026-10-12).
|
||||
- **LO isolation on the shared harness db (named design point — the
|
||||
C7 lesson, do not rediscover it)**: Large Objects are
|
||||
**database-level**, not schema-level — the schema-per-test
|
||||
convention does not isolate them, a `DROP SCHEMA` leaves the LO's
|
||||
pages in `pg_largeobject`, and shared-LO tests raced each other into
|
||||
spurious "large object NNNN does not exist" aborts in the POC. Decide
|
||||
and record the isolation shape: per-test `CREATE DATABASE` (the
|
||||
POC's proven `open_fresh` shape) or per-instance LOID tracking +
|
||||
teardown `lo_unlink` (a `LoJanitor` drop-guard, the SchemaJanitor
|
||||
pattern extended). Either way acceptance includes **zero LO residue
|
||||
on the shared db** (`pg_largeobject` probed before/after full runs).
|
||||
- **Review amendment A2 (wave-4 gate, owner: this task)** — adopt the
|
||||
schema-janitor discipline from the start; the wave-4 lesson (380
|
||||
leaked schemas) is not to be re-learned. The `SchemaJanitor` lives
|
||||
in `src/postgres.rs`'s `#[cfg(test)] test_support` — its new shared
|
||||
home (gated `cfg(test)` + `any(postgres, pg-lo)`) rides the
|
||||
shared-helper design point.
|
||||
|
||||
The window-family seam clauses (`EngineWindowState`) are
|
||||
`engine-pg-lo-window` (separate task); the constructor wiring is
|
||||
`store-pg-lo-construction`. **Nothing of `EngineFleetState` may exist
|
||||
on this engine** (ADR-012 §5's pg-lo row: pin clauses **no**, sweep
|
||||
lock **no** — fleet pins live on the kv host). This task ships the
|
||||
engine and its in-module unit tests only.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `src/pg_lo.rs` implements `Backend` completely (seven methods +
|
||||
`put_staged`) behind feature `pg-lo` (default-off), with the
|
||||
descriptorless `lo_get` window cursor body and the held-tx
|
||||
staging body
|
||||
- [ ] The companion table is the contract authority (list/size/has/CAS
|
||||
all table-level); `pg_largeobject` is never read for addressing
|
||||
- [ ] One engine-owned schema, idempotent bootstrap at the
|
||||
constructor; the quoted-identifier boundary; the virgin-store
|
||||
read-only split (42P01 → absent/empty on reads, writes surface)
|
||||
unit-tested incl. the schema-less form
|
||||
- [ ] Put = one tx (lo_create → 512 KiB chunked lowrite → row insert →
|
||||
COMMIT); CAS re-put is a no-op; the 512 KiB knee is a named
|
||||
constant with the C2 citation
|
||||
- [ ] Stage hygiene holds in both forms (discard → zero residue; failed
|
||||
commit → rollback self-discard), exact-count unit-tested
|
||||
- [ ] Reads ride `lo_get(oid, off, len)` windows; `len()` metadata-only
|
||||
from the companion row; beyond-length reads fail cleanly; the
|
||||
64-bit-variants discipline recorded (the C7 trap not inherited)
|
||||
- [ ] `delete` = one tx (lo_unlink + row delete) with the Missing
|
||||
discipline; `list` complete after mixed ops
|
||||
- [ ] The orphan-recovery sweep exists (candidates from
|
||||
`pg_largeobject` only, validated against `lo_entries`),
|
||||
scope-noted, contract-tested with a planted orphan
|
||||
- [ ] Posture deltas recorded in the module doc (catalog space never
|
||||
returned + VACUUM FULL operator line + rel-size monitoring;
|
||||
autovacuum inherited; the cached-get capacity facts)
|
||||
- [ ] Shared-helper + PgOpts + LO-isolation design points decided and
|
||||
recorded; the pg-lo-pure build (`--no-default-features
|
||||
--features pg-lo`) compiles clean
|
||||
- [ ] Unit tests env-driven (`ALKBLOBS_PG_*`), skipping cleanly
|
||||
server-less; green against the harness server at acceptance;
|
||||
zero schema + LO residue on the shared db after runs
|
||||
- [ ] `cargo test` (default — pg-lo compiles out) green;
|
||||
`cargo test --features pg-lo` green server-less (skipping);
|
||||
`cargo clippy --all-targets -- -D warnings` and `cargo fmt
|
||||
--check` green (default + `--features pg-lo` + the pg-lo-pure
|
||||
check)
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/decisions/009-pg-lo-large-tier-engine.md (the
|
||||
engine ADR — §1–§4 the contract, §5/§6 the posture deltas)
|
||||
- docs/research/poc-pglo-findings.md (findings C1–C7 — the evidence
|
||||
base; C2 the 512 KiB knee, C4 the window posture, C6 the orphan
|
||||
sweep, C7 the harness lessons)
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md
|
||||
(the trait contract)
|
||||
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md
|
||||
§1 (the staged-put contract; the large-tier cursor body families)
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
|
||||
§4/§5 (the consolidation posture; the pg-lo applicability row —
|
||||
window family only, no fleet state)
|
||||
- docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md
|
||||
§4 (`pg-lo` does not imply `postgres`; the pg-lo-pure build)
|
||||
- src/backend.rs (the trait + seam contracts this implements)
|
||||
- src/postgres.rs (the driver-posture twin: pool, prepare_cached,
|
||||
schema bootstrap, 42P01 split, test_support janitor)
|
||||
- src/local.rs (the large-tier body twin: streaming cursor, staged
|
||||
put hygiene shapes)
|
||||
- /workspace/alkblobs-pglo-poc (the POC crate — the ~400-line
|
||||
PgLoBackend this engine productionizes)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
Reference in new issue
Block a user