alkblobs/docs/research/poc-pglo-spec.md

status, title, last_updated

status	title	last_updated
passed	POC #7 — postgres Large Objects as the fs tier's pg-lo engine	2026-10-03

POC: postgres Large Objects as the fs tier's pg-lo engine — spec

POC register #7, added post-Phase-0 per the ADR-003/007 substitution door (the third use of it, after postgres-on-kv and redb-on-kv). Code: standalone crate /workspace/alkblobs-pglo-poc (clone the POC #5 crate's harness conventions; add a PgLoBackend implementing the ADR-008 trait contract including size). Findings land here regardless, per the established convention. Requested by REQ-2 (requirements.md — the fleet-with-large-blobs consolidation option; ADR-008 names pg-lo as the candidate fs-tier engine). Status: passed — run 2026-10-03; findings in poc-pglo-findings.md (curves C2, handle anatomy C3, window-vs-held C4, vacuum posture C5, crash-orphan C6). Verdict: the gate passes — companion-table engine shape, lo_get window gets, ~60–65 MB/s durable put ≈ fs's durable put, cached gets behind page-cache fs by 20–50× (bounded, monitorable), orphan-recovery sweep proven and crash-orphan behavior clean.

What this POC must decide

The ADR-008 question: can postgres Large Objects hold the fs tier's contract for the fleet topology — serving the ≥128 KiB regime (packfiles, large artifacts) from the same engine the kv tier already rides — at acceptable performance and with an honest ops posture? The fs tier's contract (ADR-003/008): get yields a pread-able handle, stage-then-commit semantics for unknown-length puts, complete list(), GC-participating delete, size without content fetch.

Large Objects' native shape differs from files in one structural way the ADR names: LO descriptors are transaction-scoped (lo_open → loread/lowrite must run inside a tx). The POC must measure what that does to the handle pattern under a connection pool, not assume it away.

Instruments

1. Inherited bench_small-shaped harness, LO arm — the POC #3/#5 methodology at fs-tier-relevant sizes: 128 KiB, 256 KiB, 1 MiB, 16 MiB, 128 MiB, sequential and random-replay, p50/p99. Compare against: local fs (the local engine baseline, same harness) and kv-tier pg (bytea) at the lower sizes, as cross-checks of methodology parity. (loread/lowrite via SQL functions through tokio-postgres — the postgres_large_object crate targets postgres 0.15 and is dead; do not use it.)
2. Handle-cost probe (pgdiag7): the tx-scoped descriptor under deadpool — cost of open-tx → lo_open → lseek/loread per range read, held vs. re-opened per read; pool pressure under N concurrent readers (the ops fetch handler's real shape).
3. Write path probe (pgdiag8): lo_create + chunked lowrite (the LOBLKSIZE ~2 KiB page layout matters here — measure write amplification as value size grows); unknown-length put shape (stage into one LO, companion row on commit); verify size-without-fetch (SELECT size from the companion table).
4. Churn/vacuum probe: repeated put/delete cycles at 1–16 MiB; measure pg_largeobject catalog growth, TOAST/vacuum behavior, autovacuum posture (the fs engine inherits ADR-008/007's maintenance-surface honesty requirement).
5. Crash/orphan probe: kill client mid-write-tx; verify no orphan LO and no half-commit; kill after lo_create before commit; write the orphan-recovery sweep story the engine ADR will need (compare lo_get-visible OIDs against the companion table — never trust the catalogs).

Decision gate (mirrors ADR-007's)

pg-lo is admitted as a shipped fs-tier engine iff:

• Performance: LO put/get at 128 KiB–16 MiB is within an order-of-magnitude of local fs on the same box (the tier is chosen for fleet correctness, not solo speed — but a 10× gap would put the fleet's packfile serving materially behind a single-node fs deployment, and the ADR must say so if measured);
• Contract: complete list() (companion table), GC-participating delete (lo_unlink under the delete window), virgin-store no-op reads, size probe — all hold, proven via the exact-count sweep-outcome test shape under --all-features;
• Ops posture: named deltas (tx-scoped handles, autovacuum/vacuum on pg_largeobject, orphan recovery sweep) with measured or documented-bounded costs.

Findings feed an engine ADR in the ADR-007 shape (measured curves, posture deltas as requirements). If the gate fails, ADR-008's recorded statement stands: shared-media or re-routing are REQ-2's fleet answers, and pg-lo's named cost was paid as evidence, not shipped blind.

Register note (phase-0.md OQ-BL-06 convention)

The canonical register is phase-0.md OQ-BL-06: #1 trait dispatch, #2 absorbed, #3 large-blob, #4 pooled-CAS (covered by #1), #5 postgres, #6 redb. The post-convergence POC files briefly carried inconsistent self-numberings (the postgres file titled itself "#4"); they are renumbered to the register. This POC is #7 — specified 2026-10-03, requested by REQ-2 (requirements.md; ADR-008 names pg-lo as the candidate fs-tier engine), run 2026-10-03 — see poc-pglo-findings.md.