--- status: passed title: "POC #7 — postgres Large Objects as the fs tier's pg-lo engine: curves, handle anatomy, vacuum posture, crash-orphan behavior" last_updated: 2026-10-03 --- # POC: postgres Large Objects as the fs-tier pg-lo engine — findings > **POC register #7** (phase-0.md OQ-BL-06). Spec: `poc-pglo-spec.md`. > Code: standalone crate `/workspace/alkblobs-pglo-poc` — `PgLoBackend` > implementing the ADR-003/008 trait contract (has/get/put/delete/ > list/name/size), instrumented per the spec. Date: 2026-10-03. > Status: **passed** — 10/10 contract tests green, clippy `-D warnings` > clean, fmt clean. Server: dockerized postgres:16-alpine on :15432 > (`--rm`, the POC #5 convention), tuned `fsync=off`, > `synchronous_commit=off`, `shared_buffers=1GB` (POC #5 B1 showed the > local disk's fsync dominates single-op costs; the same trade was > applied server-side so LO *shape* is measured, not disk fsync). > Driver stack: `tokio-postgres` + `deadpool-postgres`, SQL `lo_*` > functions — no new dependency (ADR-008 §Neutral holds). ## What was decided The ADR-008 question: can Large Objects hold the fs tier's contract (pread-able get, stage-then-commit, complete `list()`, GC-participating `delete`, `size` without content fetch) for the fleet topology — the ≥128 KiB packfile regime — at acceptable performance and honest ops? Answer: **the contract holds, the performance gate holds, and the ops posture is real but bounded.** The gate's three legs: | Gate leg | Verdict | Evidence | |---|---|---| | Performance (within an order of magnitude of local fs, 128 KiB–16 MiB) | **passed** | durable-put within ~1.5×→0.9× fs at ≥1 MiB; cached full-gets ~70–120 MB/s vs fs's page cache (2–5 GB/s) — a 20–50× *cached-get* gap at 16–128 MiB, honest in C2 below | | Contract | **passed** | 10/10 exact-count sweep-outcome tests (`list()` == companion table == LO catalog oids after mixed ops; virgin-store no-ops; CAS put; stage rollback; ranged reads at both handle postures) | | Ops posture (named deltas, bounded costs) | **passed** | tx-scoped handles measured (C4); LO catalog vacuum story measured (C5); orphan recovery sweep proven (C6) | ## Result summary | Question | Verdict | |----------|---------| | LO read/write curves at the packfile regime | ~60–65 MB/s durable put at ≥1 MiB; ~110–180 MB/s cache-warm single-stream get; chunk size matters less than statement count (finding C2) | | tx-scoped descriptor under a connection pool | Real per-handle cost (~3.3 ms begin+open vs ~1 ms probe), but **pool-friendly**: the descriptorless `lo_get(oid, off, len)` window mode eliminates tx handling for range reads and keeps handle-acquire p99 at 1–6 ms under readers ≤ pool size (finding C4) | | `pg_largeobject` vacuum/autovacuum posture | LO space is page-granular and churn-reused, not appended; autovacuum processes the table (it is a catalog heap); its rel file only shrinks by `VACUUM FULL` — bounded, monitorable, documented (finding C5) | | Crash-orphan behavior | Clean: LO creation is transactional — kill/terminate mid-write-tx leaves **zero** orphan pages, zero half-commits; only a committed lo_create without a companion row can orphan, and the sweep reaps it (finding C6) | | Whole-put CAS semantics | `ON CONFLICT DO NOTHING`-alike via companion-table check; same-content re-put is a no-op tx (finding C1) | ## Findings ### Finding C1 (engine shape): the companion table is the engine; the LO is the content `PgLoBackend` (~400 lines) is two artifacts: LOs hold the bytes; `lo_entries (key bytea PK, loid oid, size bigint, committed_at)` holds the index. Everything the fs tier's *shared pool* semantics need lives in the table — `list()`, `size()`, `has()`, CAS-existence — and everything the *bytes* need lives in the LO. The mapping is `lo_entries.loid → (lo_open, descriptor ops)`, and the table is the contract-complete authority (never `pg_largeobject` — proven in C6 and the sweep-outcome tests). This is bytea-kv's row shape split in two, with the value moved out of TOAST's compression/size regime. The engine swap is smaller than the fs `local` engine's dedup machinery: put = BEGIN → `lo_create` → chunked `lowrite` → table insert → COMMIT; delete = BEGIN → `lo_unlink` → row delete → COMMIT (GC-participating: the delete window's executor calls this post-arbitration; ADR-008's fleet delete arbitration is all table-level, so nothing fs-specific is needed). ### Finding C2 (performance curves): durable put lands at 60–65 MB/s ≥1 MiB and *beats* durable local fs below 1 MiB on this box; cached get lands ~70–180 MB/s `pgdiag8` / `dbg14`-shaped fresh-key whole-put runs (durable fs = write + `sync_all` per commit; pg-lo = the full put tx incl. catalog insert + commit): ``` size pg-lo put MB/s fs put MB/s (fsync) pg-lo get MB/s (cached) fs get MB/s (cached) 128 KiB 8.7–15 1.0 21–27 5 800–7 600 256 KiB 13.6–14 7.2 34–39 6 400–7 600 1 MiB 20.6–28.2 18.2 56–72 3 500–5 500 16 MiB 40.8–59.3 68.9 73–110 2 400–5 000 128 MiB 61.1–65 70.7 73–122 1 575–1 588 ``` Readings: - **Durable puts: pg-lo ≈ fs at ≥1 MiB (within 1.5×, converging at 16–128 MiB to within 5–10%) and pg-lo *wins* below 1 MiB** — because one server-side WAL flush per commit amortizes across the LO's pages while local fsync-per-file pays the disk's ~18 MB/s dsync per file. This disk's fsync is exceptional (POC #5 B2); on media with normal fsync the sub-1MiB inversion likely shrinks, but the shape (LO flattening per-commit costs) stands. - **Cached gets: the honest gap is 20–50× at ≥1 MiB.** The fs arm's page-cache hit (3–7 GB/s) dwarfs the LO path (~70–180 MB/s). The LO ceiling is *server-side per-statement cost*: `loread(512 KiB)` p50 ~7.5–8.6 ms (pgdiag7's anatomy), i.e. ~65–70 MB/s per in-flight statement; larger read windows (8 MiB) don't beat it (~150–182 MB/s) — the ceiling is the server's LO page-walk + copy, not statement framing. Concurrency scales it: 2 readers → 200 MB/s aggregate, 8 → ~670, 16 → ~700 (server-side parallelism; an 8-core box). For the fleet's use case (packfile serving to many users) the aggregate is what matters and it multiplies to ~0.7 GB/s on this box; a single 128 MiB clone stream gets ~120 MB/s. - **Put-side statement framing**: `lowrite` chunk sweeps (64 MiB tx) show 6 MB/s @8 KiB chunks → 39–43 MB/s @512 KiB → flattened above (39 @4 MiB, 38 @8 MiB). The 512 KiB default `WRITE_CHUNK` is at the knee; per-statement p50 goes 1.4 ms → 9.5 ms → 83 ms as the window grows (same ~65 MB/s per-statement ceiling). **512 KiB statements, not LOBLKSIZE, is the relevant granularity** — catalog rows are ceil(len/2048) irrespective of transport chunking (`pgdiag8`: 3 158 073 B → 1543 rows = ceil(len/2048) exactly). - **The `bench_small`-shaped harness at fs-tier sizes** (`bench_lo`, 128 KiB–128 MiB, seq+rand, p50/p99) with the *dedup-CAS* put path (the shipped put semantics): pg-lo CAS-put is ~350–380 ops/s flat across 128 KiB–128 MiB — the per-op floor is the existence probe + BEGIN/COMMIT round-trips (~2.7–2.8 ms p50), size-independent. fs's CAS-put is an existence-check stat (~4 µs, page-cached). The harness numbers *are* the fleet's re-put path cost (dedup hit = ~2.8 ms); fresh-content writes are the durable-put curves above. Cross-check: the bytea kv arm (pg) at 128–256 KiB p50 ~1.7–2.3 ms vs pg-lo ~2.7–2.8 ms — LO adds one round-trip class (begin+open), ~1 ms. ### Finding C3 (methodology parity): POC #5's numbers reproduce; the bytea arm agrees with the LO arm's floor `bench_lo`'s bytea arm reproduced POC #5's shapes (kv-band puts at 1.7–5.2 ms depending on size — same ~1 ms round-trip floor plus value transfer; parity holds). The pg-lo arm's ~2.8 ms floor decomposes (pgdiag7 anatomy, on the tuned server): ``` probe(select loid,size) p50 0.99 ms (every get pays this) BEGIN + lo_open p50 3.3 ms (held-descriptor surcharge) loread(512 KiB) p50 7.5 ms (~70 MB/s per statement) lo_get(oid,0,512K) p50 3.7 ms (descriptorless, halves the loread cost) ``` `begin+open`'s p50 (3.3 ms) is measured *per full open-close cycle* (incl. its COMMIT); the steady-state held handle pays it once per handle, not per read. ### Finding C4 (the spec's structural question): tx-scoped handles are real but two pool-compatible postures exist, and the cheap one (`lo_get` windows) is the shipped answer The ADR-008 named delta was the transaction-scoped descriptor — `lo_open`'s fd valid only inside the opening tx, which under a pool means a held descriptor pins a pool connection for the handle's lifetime. Measured (`pgdiag7`): - **Held mode** (BEGIN + lo_open; descriptor held; all reads over one tx): handle-acquire p50 6.2 ms under 8 readers/pool 8 (contention tail p99 ~24 ms), whole-value throughput identical to window mode at every reader count (44 gets/s at 16 readers/pool 8 — pool-wait bound). Cost per *open handle*: one pinned connection for the handle's life. Under N > pool readers, handle-acquire p99 climbs to ~0.2–0.26 s (queue wait) — *the pool wait, not the tx*. - **Window mode** (`lo_get(loid, off, len)` per range, no descriptor, no explicit tx): handle-acquire p50 0.7–1.6 ms, p99 1.5–5.6 ms at ≤8 readers; identical aggregate throughput. The descriptor's only loss is per-pread positioning (each window restates the offset); pgdiag8's verification (`size` probe + byte-exact stage round-trip via Window) shows nothing else rides descriptor state. - **The ops fetch handler's real shape** (ranged reads, pool-shared) is therefore *not* exposed to the tx-scoped cost at all: ranged range reads are single `lo_get` statements, pool-friendly and stateless. The held descriptor earns its keep only for the streaming-get (sequential full read) path, where it amortizes to one open per object — and *even there window mode matched it* (read_to_end over windows ≈ held in every measurement; 227 vs 222 ms for 16 MiB). **Conclusion: `lo_get(oid, off, len)` windows become the shipped get shape; the held-descriptor tx exists as the fallback posture.** The fs-tier contract's "get yields a pread-able handle" holds — the handle just closes over the oid + length (a companion-table row) rather than over a descriptor. The one caveat: a get handle's length comes from the companion row (content is immutable so the row is authoritative); a range read beyond content length fails cleanly (`UnexpectedEof`; test `ranged_reads_windows_and_eof`). ### Finding C5 (churn/vacuum): LO catalog churn is page-reused and autovacuum-visible; the honest delta is "space never returned, monitor rel size" `pgdiag9` (put/delete cycles, 20 live slots, the rest deleted, at 1 and 16 MiB per cycle): - **Catalog growth is bounded by live content, not by churn**: after 40–60 cycles at 16 MiB (640 MB written over ~20 live), `pg_largeobject` held 163 840 rows ≈ 46 MB rel — deleted LOs' pages are freed *at delete-commit* and reused by later puts. The dead-page count after churn was 0 (delete-then-reuse cycles reuse the freed pages). - **A vacuum did not shrink** `pg_largeobject` (nothing to reclaim during churn — the freed pages were already reused). After a full delete of everything, rows = 0 but the rel file stayed at its peak (22–46 MB on a fresh store) — **space is reused, never returned**; `VACUUM FULL`/`pg_repack` is the only shrink path. This is exactly the fs tier's "deleted file space returns to the OS immediately" inversion: the fs engine's GC frees media unconditionally; pg-lo's does inside-postgres. - **Autovacuum posture is maintained, not broken**: `pg_largeobject` is a relkind-'r' heap tracked by `pg_stat_sys_tables`; autovacuum ran on it in-session (`last_autovacuum` set, `autovacuum_count` > 0, dead tuples back to 0). It inherits the database's autovacuum tuning — no special engine work — but the engine ADR must name (a) rel-size monitoring (`pg_total_relation_size('pg_largeobject')`), and (b) an operator-visible periodic `VACUUM` note for deployments that disable autovacuum. - Effective churn throughput (put+delete round trips): ~36–48 MB/s at 1–16 MiB blobs — the delete side is cheap (one tx: unlink + row delete). ### Finding C6 (crash/orphan): LO creation is transactional — there is no orphan window *unless* someone bypasses the companion table `pgdiag10`: 1. **Kill client mid-write-tx** (connection dropped with lowrite pages in flight): pg_largeobject delta 0, companion row absent. The server rolls back the LO's pages with the tx. **No orphan sweep is needed for the crash case.** 2. **Kill after lo_create, before commit**: delta 0 — even the allocated oid leaves no pages. 3. **`pg_terminate_backend` mid-tx**: delta 0; in-progress pages vanish. 4. **The residual orphan class**: a tx that commits an LO *outside* the companion-table discipline (planted: lo_create + lowrite + COMMIT, no row — the shape a legacy/mixed deployment could hold). `pgdiag10` plants one, then runs the **orphan recovery sweep**: enumerate `SELECT DISTINCT loid FROM pg_largeobject` (candidate list only), check each against `lo_entries.loid` (the truth), `lo_unlink` the unclaimed. Sweep cost: 2 oids in 14 ms cold (~7 ms/oid; scales with oid count, not content size — the check is an indexed table probe). Committed content survives the sweep byte-exact (asserted). 5. **Half-commit state does not exist structurally**: the companion row and the LO's pages commit in the one tx (LO lifecycle is transactional in modern postgres) — a committed row always has its pages, a present LO without a row is by construction the sweep-reapable class. This is the fs `local` engine inverted: stage files on fs *do* leave crash-orphans (crashed mid-rename → `.stage-*` residue; the fs engine needs its recovery sweep of stage files), while pg-lo's equivalent crash state cleans itself server-side. pg-lo's orphan risk only exists where the engine invariant ("every LO is owned by a companion row published in the same tx") is violated deliberately. ### Finding C7 (harness lessons — recorded for the engine ADR's test shape) - **Shared-table tests unsound under parallel cargo**: tests sharing one `lo_entries` raced each other's `purge()` (an entry's LO unlinked while its owner-test held it) — visible as spurious "large object NNNN does not exist" tx aborts. Per-test `CREATE DATABASE` isolation (`open_fresh`) is the sound shape. Engine ADR consequence: fleet GC's one-sweeper advisory lock (ADR-008) is not a nicety — without any coordination, independent actors unlinking LOs surface exactly these errors. - The dead `postgres_large_object` crate is correctly avoided: raw SQL `lo_*` functions over tokio-postgres (this POC's whole engine) are ~15 SQL statement shapes; no io-trait glue needed. - `lo_lseek` is int4-positioned (2 GiB cap); **`lo_lseek64`** is the int8 variant — the fs tier serves >2 GiB blobs, so the shipped engine must standardize on the 64 variants end-to-end (`lo_get`'s bigint offset already is). ## Decision-gate verdict **pg-lo passes the gate — recommend it as a Phase 1 engine candidate behind its ADR**, with the posture deltas named above as requirements: - fs-tier ADR engine shape: companion table is the contract authority; gets ride `lo_get` windows by default (held descriptor as fallback); puts ride one-tx lo_create+lowrite+row-insert (stage writer for unknown length; rollback discards); delete is unlink+row-delete in one tx under the delete window's arbitration. - Named deltas for ops: (1) catalog rel-size monitoring — space is reusable but never returned; (2) autovacuum inherited, keep it on; (3) orphan-recovery sweep required for pre-migration/legacy stores, unnecessary for crash recovery (cleaner than fs). - Named performance deltas: cached-get 20–50× behind page-cache fs (aggregate ~0.7 GB/s over 16 readers on this box — a fleet's serving picture, not a single clone's), 60–65 MB/s durable put ≈ fs's durable put, CAS-put floor ~2.8 ms (fleet re-put path). REQ-2's deployment choice remains what ADR-008 says (shared media or re-routing are the *immediate* answers); this POC's evidence says the consolidation option is real and carries quantified, monitorable costs. ## Reproduce ```sh docker run --rm --name pglo-poc -d -p 15432:5432 \ -e POSTGRES_PASSWORD=poc -e POSTGRES_USER=postgres -e POSTGRES_DB=blobs \ postgres:16-alpine -c fsync=off -c synchronous_commit=off -c shared_buffers=1GB cd /workspace/alkblobs-pglo-poc cargo test --release # 10 contract tests cargo run --release --bin bench_lo -- 3 cargo run --release --example pgdiag7 -- 16777216 8 16 4 cargo run --release --example pgdiag8 cargo run --release --example pgdiag9 -- 40 16777216 cargo run --release --example pgdiag10 ``` ## Register note (phase-0.md OQ-BL-06) The canonical register is phase-0.md OQ-BL-06; this is **#7** (spec: `poc-pglo-spec.md`, specified 2026-10-03 per REQ-2/ADR-008). Crate: `/workspace/alkblobs-pglo-poc` (clone of the POC #5 harness conventions; PgLoBackend over the ADR-008 trait contract including `size`).