alkblobs/docs/research/poc-pglo-findings.md

status, title, last_updated

status	title	last_updated
passed	POC #7 — postgres Large Objects as the fs tier's pg-lo engine: curves, handle anatomy, vacuum posture, crash-orphan behavior	2026-10-03

POC: postgres Large Objects as the fs-tier pg-lo engine — findings

POC register #7 (phase-0.md OQ-BL-06). Spec: poc-pglo-spec.md. Code: standalone crate /workspace/alkblobs-pglo-poc — PgLoBackend implementing the ADR-003/008 trait contract (has/get/put/delete/ list/name/size), instrumented per the spec. Date: 2026-10-03. Status: passed — 10/10 contract tests green, clippy -D warnings clean, fmt clean. Server: dockerized postgres:16-alpine on :15432 (--rm, the POC #5 convention), tuned fsync=off, synchronous_commit=off, shared_buffers=1GB (POC #5 B1 showed the local disk's fsync dominates single-op costs; the same trade was applied server-side so LO shape is measured, not disk fsync). Driver stack: tokio-postgres + deadpool-postgres, SQL lo_* functions — no new dependency (ADR-008 §Neutral holds).

What was decided

The ADR-008 question: can Large Objects hold the fs tier's contract (pread-able get, stage-then-commit, complete list(), GC-participating delete, size without content fetch) for the fleet topology — the ≥128 KiB packfile regime — at acceptable performance and honest ops? Answer: the contract holds, the performance gate holds, and the ops posture is real but bounded. The gate's three legs:

Gate leg	Verdict	Evidence
Performance (within an order of magnitude of local fs, 128 KiB–16 MiB)	passed	durable-put within ~1.5×→0.9× fs at ≥1 MiB; cached full-gets ~70–120 MB/s vs fs's page cache (2–5 GB/s) — a 20–50× cached-get gap at 16–128 MiB, honest in C2 below
Contract	passed	10/10 exact-count sweep-outcome tests (list() == companion table == LO catalog oids after mixed ops; virgin-store no-ops; CAS put; stage rollback; ranged reads at both handle postures)
Ops posture (named deltas, bounded costs)	passed	tx-scoped handles measured (C4); LO catalog vacuum story measured (C5); orphan recovery sweep proven (C6)

Result summary

Question	Verdict
LO read/write curves at the packfile regime	~60–65 MB/s durable put at ≥1 MiB; ~110–180 MB/s cache-warm single-stream get; chunk size matters less than statement count (finding C2)
tx-scoped descriptor under a connection pool	Real per-handle cost (~3.3 ms begin+open vs ~1 ms probe), but pool-friendly: the descriptorless lo_get(oid, off, len) window mode eliminates tx handling for range reads and keeps handle-acquire p99 at 1–6 ms under readers ≤ pool size (finding C4)
pg_largeobject vacuum/autovacuum posture	LO space is page-granular and churn-reused, not appended; autovacuum processes the table (it is a catalog heap); its rel file only shrinks by VACUUM FULL — bounded, monitorable, documented (finding C5)
Crash-orphan behavior	Clean: LO creation is transactional — kill/terminate mid-write-tx leaves zero orphan pages, zero half-commits; only a committed lo_create without a companion row can orphan, and the sweep reaps it (finding C6)
Whole-put CAS semantics	ON CONFLICT DO NOTHING-alike via companion-table check; same-content re-put is a no-op tx (finding C1)

Findings

Finding C1 (engine shape): the companion table is the engine; the LO is the content

PgLoBackend (~400 lines) is two artifacts: LOs hold the bytes; lo_entries (key bytea PK, loid oid, size bigint, committed_at) holds the index. Everything the fs tier's shared pool semantics need lives in the table — list(), size(), has(), CAS-existence — and everything the bytes need lives in the LO. The mapping is lo_entries.loid → (lo_open, descriptor ops), and the table is the contract-complete authority (never pg_largeobject — proven in C6 and the sweep-outcome tests). This is bytea-kv's row shape split in two, with the value moved out of TOAST's compression/size regime. The engine swap is smaller than the fs local engine's dedup machinery: put = BEGIN → lo_create → chunked lowrite → table insert → COMMIT; delete = BEGIN → lo_unlink → row delete → COMMIT (GC-participating: the delete window's executor calls this post-arbitration; ADR-008's fleet delete arbitration is all table-level, so nothing fs-specific is needed).

Finding C2 (performance curves): durable put lands at 60–65 MB/s ≥1 MiB and beats durable local fs below 1 MiB on this box; cached get lands ~70–180 MB/s

pgdiag8 / dbg14-shaped fresh-key whole-put runs (durable fs = write + sync_all per commit; pg-lo = the full put tx incl. catalog insert + commit):

size     pg-lo put MB/s   fs put MB/s (fsync)     pg-lo get MB/s (cached)   fs get MB/s (cached)
128 KiB      8.7–15           1.0                        21–27                    5 800–7 600
256 KiB     13.6–14           7.2                        34–39                    6 400–7 600
1 MiB       20.6–28.2        18.2                        56–72                    3 500–5 500
16 MiB      40.8–59.3        68.9                        73–110                   2 400–5 000
128 MiB     61.1–65          70.7                        73–122                   1 575–1 588

Readings:

• Durable puts: pg-lo ≈ fs at ≥1 MiB (within 1.5×, converging at 16–128 MiB to within 5–10%) and pg-lo wins below 1 MiB — because one server-side WAL flush per commit amortizes across the LO's pages while local fsync-per-file pays the disk's ~18 MB/s dsync per file. This disk's fsync is exceptional (POC #5 B2); on media with normal fsync the sub-1MiB inversion likely shrinks, but the shape (LO flattening per-commit costs) stands.
• Cached gets: the honest gap is 20–50× at ≥1 MiB. The fs arm's page-cache hit (3–7 GB/s) dwarfs the LO path (~70–180 MB/s). The LO ceiling is server-side per-statement cost: loread(512 KiB) p50 ~7.5–8.6 ms (pgdiag7's anatomy), i.e. ~65–70 MB/s per in-flight statement; larger read windows (8 MiB) don't beat it (~150–182 MB/s) — the ceiling is the server's LO page-walk + copy, not statement framing. Concurrency scales it: 2 readers → 200 MB/s aggregate, 8 → ~670, 16 → ~700 (server-side parallelism; an 8-core box). For the fleet's use case (packfile serving to many users) the aggregate is what matters and it multiplies to ~0.7 GB/s on this box; a single 128 MiB clone stream gets ~120 MB/s.
• Put-side statement framing: lowrite chunk sweeps (64 MiB tx) show 6 MB/s @8 KiB chunks → 39–43 MB/s @512 KiB → flattened above (39 @4 MiB, 38 @8 MiB). The 512 KiB default WRITE_CHUNK is at the knee; per-statement p50 goes 1.4 ms → 9.5 ms → 83 ms as the window grows (same ~65 MB/s per-statement ceiling). 512 KiB statements, not LOBLKSIZE, is the relevant granularity — catalog rows are ceil(len/2048) irrespective of transport chunking (pgdiag8: 3 158 073 B → 1543 rows = ceil(len/2048) exactly).
• The bench_small-shaped harness at fs-tier sizes (bench_lo, 128 KiB–128 MiB, seq+rand, p50/p99) with the dedup-CAS put path (the shipped put semantics): pg-lo CAS-put is ~350–380 ops/s flat across 128 KiB–128 MiB — the per-op floor is the existence probe + BEGIN/COMMIT round-trips (~2.7–2.8 ms p50), size-independent. fs's CAS-put is an existence-check stat (~4 µs, page-cached). The harness numbers are the fleet's re-put path cost (dedup hit = ~2.8 ms); fresh-content writes are the durable-put curves above. Cross-check: the bytea kv arm (pg) at 128–256 KiB p50 ~1.7–2.3 ms vs pg-lo ~2.7–2.8 ms — LO adds one round-trip class (begin+open), ~1 ms.

Finding C3 (methodology parity): POC #5's numbers reproduce; the bytea arm agrees with the LO arm's floor

bench_lo's bytea arm reproduced POC #5's shapes (kv-band puts at 1.7–5.2 ms depending on size — same ~1 ms round-trip floor plus value transfer; parity holds). The pg-lo arm's ~2.8 ms floor decomposes (pgdiag7 anatomy, on the tuned server):

probe(select loid,size)   p50   0.99 ms   (every get pays this)
BEGIN + lo_open           p50   3.3  ms   (held-descriptor surcharge)
loread(512 KiB)           p50   7.5  ms   (~70 MB/s per statement)
lo_get(oid,0,512K)        p50   3.7  ms   (descriptorless, halves the loread cost)

begin+open's p50 (3.3 ms) is measured per full open-close cycle (incl. its COMMIT); the steady-state held handle pays it once per handle, not per read.

Finding C4 (the spec's structural question): tx-scoped handles are real but two pool-compatible postures exist, and the cheap one (lo_get windows) is the shipped answer

The ADR-008 named delta was the transaction-scoped descriptor — lo_open's fd valid only inside the opening tx, which under a pool means a held descriptor pins a pool connection for the handle's lifetime. Measured (pgdiag7):

• Held mode (BEGIN + lo_open; descriptor held; all reads over one tx): handle-acquire p50 6.2 ms under 8 readers/pool 8 (contention tail p99 ~24 ms), whole-value throughput identical to window mode at every reader count (44 gets/s at 16 readers/pool 8 — pool-wait bound). Cost per open handle: one pinned connection for the handle's life. Under N > pool readers, handle-acquire p99 climbs to ~0.2–0.26 s (queue wait) — the pool wait, not the tx.
• Window mode (lo_get(loid, off, len) per range, no descriptor, no explicit tx): handle-acquire p50 0.7–1.6 ms, p99 1.5–5.6 ms at ≤8 readers; identical aggregate throughput. The descriptor's only loss is per-pread positioning (each window restates the offset); pgdiag8's verification (size probe + byte-exact stage round-trip via Window) shows nothing else rides descriptor state.
• The ops fetch handler's real shape (ranged reads, pool-shared) is therefore not exposed to the tx-scoped cost at all: ranged range reads are single lo_get statements, pool-friendly and stateless. The held descriptor earns its keep only for the streaming-get (sequential full read) path, where it amortizes to one open per object — and even there window mode matched it (read_to_end over windows ≈ held in every measurement; 227 vs 222 ms for 16 MiB).

Conclusion: lo_get(oid, off, len) windows become the shipped get shape; the held-descriptor tx exists as the fallback posture. The fs-tier contract's "get yields a pread-able handle" holds — the handle just closes over the oid + length (a companion-table row) rather than over a descriptor. The one caveat: a get handle's length comes from the companion row (content is immutable so the row is authoritative); a range read beyond content length fails cleanly (UnexpectedEof; test ranged_reads_windows_and_eof).

Finding C5 (churn/vacuum): LO catalog churn is page-reused and autovacuum-visible; the honest delta is "space never returned, monitor rel size"

pgdiag9 (put/delete cycles, 20 live slots, the rest deleted, at 1 and 16 MiB per cycle):

• Catalog growth is bounded by live content, not by churn: after 40–60 cycles at 16 MiB (640 MB written over ~20 live), pg_largeobject held 163 840 rows ≈ 46 MB rel — deleted LOs' pages are freed at delete-commit and reused by later puts. The dead-page count after churn was 0 (delete-then-reuse cycles reuse the freed pages).
• A vacuum did not shrink pg_largeobject (nothing to reclaim during churn — the freed pages were already reused). After a full delete of everything, rows = 0 but the rel file stayed at its peak (22–46 MB on a fresh store) — space is reused, never returned; VACUUM FULL/pg_repack is the only shrink path. This is exactly the fs tier's "deleted file space returns to the OS immediately" inversion: the fs engine's GC frees media unconditionally; pg-lo's does inside-postgres.
• Autovacuum posture is maintained, not broken: pg_largeobject is a relkind-'r' heap tracked by pg_stat_sys_tables; autovacuum ran on it in-session (last_autovacuum set, autovacuum_count > 0, dead tuples back to 0). It inherits the database's autovacuum tuning — no special engine work — but the engine ADR must name (a) rel-size monitoring (pg_total_relation_size('pg_largeobject')), and (b) an operator-visible periodic VACUUM note for deployments that disable autovacuum.
• Effective churn throughput (put+delete round trips): ~36–48 MB/s at 1–16 MiB blobs — the delete side is cheap (one tx: unlink + row delete).

Finding C6 (crash/orphan): LO creation is transactional — there is no orphan window unless someone bypasses the companion table

pgdiag10:

1. Kill client mid-write-tx (connection dropped with lowrite pages in flight): pg_largeobject delta 0, companion row absent. The server rolls back the LO's pages with the tx. No orphan sweep is needed for the crash case.
2. Kill after lo_create, before commit: delta 0 — even the allocated oid leaves no pages.
3. pg_terminate_backend mid-tx: delta 0; in-progress pages vanish.
4. The residual orphan class: a tx that commits an LO outside the companion-table discipline (planted: lo_create + lowrite + COMMIT, no row — the shape a legacy/mixed deployment could hold). pgdiag10 plants one, then runs the orphan recovery sweep: enumerate SELECT DISTINCT loid FROM pg_largeobject (candidate list only), check each against lo_entries.loid (the truth), lo_unlink the unclaimed. Sweep cost: 2 oids in 14 ms cold (~7 ms/oid; scales with oid count, not content size — the check is an indexed table probe). Committed content survives the sweep byte-exact (asserted).
5. Half-commit state does not exist structurally: the companion row and the LO's pages commit in the one tx (LO lifecycle is transactional in modern postgres) — a committed row always has its pages, a present LO without a row is by construction the sweep-reapable class.

This is the fs local engine inverted: stage files on fs do leave crash-orphans (crashed mid-rename → .stage-* residue; the fs engine needs its recovery sweep of stage files), while pg-lo's equivalent crash state cleans itself server-side. pg-lo's orphan risk only exists where the engine invariant ("every LO is owned by a companion row published in the same tx") is violated deliberately.

Finding C7 (harness lessons — recorded for the engine ADR's test shape)

• Shared-table tests unsound under parallel cargo: tests sharing one lo_entries raced each other's purge() (an entry's LO unlinked while its owner-test held it) — visible as spurious "large object NNNN does not exist" tx aborts. Per-test CREATE DATABASE isolation (open_fresh) is the sound shape. Engine ADR consequence: fleet GC's one-sweeper advisory lock (ADR-008) is not a nicety — without any coordination, independent actors unlinking LOs surface exactly these errors.
• The dead postgres_large_object crate is correctly avoided: raw SQL lo_* functions over tokio-postgres (this POC's whole engine) are ~15 SQL statement shapes; no io-trait glue needed.
• lo_lseek is int4-positioned (2 GiB cap); lo_lseek64 is the int8 variant — the fs tier serves >2 GiB blobs, so the shipped engine must standardize on the 64 variants end-to-end (lo_get's bigint offset already is).

Decision-gate verdict

pg-lo passes the gate — recommend it as a Phase 1 engine candidate behind its ADR, with the posture deltas named above as requirements:

• fs-tier ADR engine shape: companion table is the contract authority; gets ride lo_get windows by default (held descriptor as fallback); puts ride one-tx lo_create+lowrite+row-insert (stage writer for unknown length; rollback discards); delete is unlink+row-delete in one tx under the delete window's arbitration.
• Named deltas for ops: (1) catalog rel-size monitoring — space is reusable but never returned; (2) autovacuum inherited, keep it on; (3) orphan-recovery sweep required for pre-migration/legacy stores, unnecessary for crash recovery (cleaner than fs).
• Named performance deltas: cached-get 20–50× behind page-cache fs (aggregate ~0.7 GB/s over 16 readers on this box — a fleet's serving picture, not a single clone's), 60–65 MB/s durable put ≈ fs's durable put, CAS-put floor ~2.8 ms (fleet re-put path).

REQ-2's deployment choice remains what ADR-008 says (shared media or re-routing are the immediate answers); this POC's evidence says the consolidation option is real and carries quantified, monitorable costs.

Reproduce

docker run --rm --name pglo-poc -d -p 15432:5432 \
  -e POSTGRES_PASSWORD=poc -e POSTGRES_USER=postgres -e POSTGRES_DB=blobs \
  postgres:16-alpine -c fsync=off -c synchronous_commit=off -c shared_buffers=1GB
cd /workspace/alkblobs-pglo-poc
cargo test --release          # 10 contract tests
cargo run --release --bin bench_lo -- 3
cargo run --release --example pgdiag7 -- 16777216 8 16 4
cargo run --release --example pgdiag8
cargo run --release --example pgdiag9 -- 40 16777216
cargo run --release --example pgdiag10

Register note (phase-0.md OQ-BL-06)

The canonical register is phase-0.md OQ-BL-06; this is #7 (spec: poc-pglo-spec.md, specified 2026-10-03 per REQ-2/ADR-008). Crate: /workspace/alkblobs-pglo-poc (clone of the POC #5 harness conventions; PgLoBackend over the ADR-008 trait contract including size).