glm-5.3-flash
cee97defba
The ADR-008 pg-lo admission POC ran in a standalone crate (/workspace/alkblobs-pglo-poc): PgLoBackend over the ADR-003/008 trait contract (including size), 10/10 exact-count sweep-outcome contract tests, clippy/fmt clean; dockerized postgres:16-alpine on :15432, POC #5 driver stack (tokio-postgres + deadpool) via SQL lo_* functions, no new dependency. Gate verdict: passed, with named deltas. - Performance: durable put 60-65 MB/s at >=1 MiB, within 1.5x of — and below 1 MiB beating — durable local fs on this fsync-slow disk; cached gets 70-180 MB/s single-stream, ~0.7 GB/s aggregate over 16 readers (20-50x behind page-cache fs — the honest named delta) - Contract: companion table is the list()/size()/CAS authority (never the catalogs); stage-then-commit; GC-participating lo_unlink delete - Handles: the tx-scoped descriptor is real but pool-compatible via descriptorless lo_get(oid, off, len) windows — window gets keep handle-acquire p99 at 1-6 ms under readers <= pool; held descriptor is the fallback posture - Vacuum: pg_largeobject pages churn-reused, never returned; tracked by autovacuum; rel-size monitoring named as an ops requirement - Crash/orphan: LO creation is transactional — kill/terminate mid-write-tx leaves zero orphan pages; the only orphan class is a committed LO bypassing the companion table (planted, reaped by the ~7 ms/oid sweep; committed content survives byte-exact) - Harness lessons: lo_lseek is int4 — the 64 variants are the >2 GiB discipline; shared-table parallel tests are unsound (per-test CREATE DATABASE isolation) Docs: new poc-pglo-findings.md; poc-pglo-spec.md status passed; register OQ-BL-06 #7 marked passed; ADR-008 pg-lo bullet updated (duplicate bullet removed) + backends-and-dispatch/open-questions cross-references. Verification: cargo test --release (10 passed), clippy -D warnings, fmt --check in /workspace/alkblobs-pglo-poc.
101 lines
5.1 KiB
Markdown
101 lines
5.1 KiB
Markdown
---
|
||
status: passed
|
||
title: "POC #7 — postgres Large Objects as the fs tier's pg-lo engine"
|
||
last_updated: 2026-10-03
|
||
---
|
||
|
||
# POC: postgres Large Objects as the fs tier's pg-lo engine — spec
|
||
|
||
> **POC register #7**, added post-Phase-0 per the ADR-003/007
|
||
> substitution door (the third use of it, after postgres-on-kv and
|
||
> redb-on-kv). Code: standalone crate
|
||
> `/workspace/alkblobs-pglo-poc` (clone the POC #5 crate's harness
|
||
> conventions; add a `PgLoBackend` implementing the ADR-008 trait
|
||
> contract including `size`). Findings land here regardless, per the
|
||
> established convention. Requested by REQ-2 (requirements.md — the
|
||
> fleet-with-large-blobs consolidation option; ADR-008 names pg-lo as
|
||
> the candidate fs-tier engine). Status: **passed** — run 2026-10-03;
|
||
> findings in `poc-pglo-findings.md` (curves C2, handle anatomy C3,
|
||
> window-vs-held C4, vacuum posture C5, crash-orphan C6). Verdict: the
|
||
> gate passes — companion-table engine shape, `lo_get` window gets,
|
||
> ~60–65 MB/s durable put ≈ fs's durable put, cached gets behind
|
||
> page-cache fs by 20–50× (bounded, monitorable), orphan-recovery
|
||
> sweep proven and crash-orphan behavior clean.
|
||
|
||
## What this POC must decide
|
||
|
||
The ADR-008 question: can postgres Large Objects hold the fs tier's
|
||
contract for the fleet topology — serving the ≥128 KiB regime
|
||
(packfiles, large artifacts) from the same engine the kv tier already
|
||
rides — at acceptable performance and with an honest ops posture? The
|
||
fs tier's contract (ADR-003/008): `get` yields a pread-able handle,
|
||
stage-then-commit semantics for unknown-length puts, complete
|
||
`list()`, GC-participating `delete`, `size` without content fetch.
|
||
|
||
Large Objects' native shape differs from files in one structural way
|
||
the ADR names: **LO descriptors are transaction-scoped** (lo_open →
|
||
loread/lowrite must run inside a tx). The POC must measure what that
|
||
does to the handle pattern under a connection pool, not assume it away.
|
||
|
||
## Instruments
|
||
|
||
1. **Inherited `bench_small`-shaped harness, LO arm** — the POC #3/#5
|
||
methodology at fs-tier-relevant sizes: **128 KiB, 256 KiB, 1 MiB,
|
||
16 MiB, 128 MiB**, sequential and random-replay, p50/p99. Compare
|
||
against: local fs (the `local` engine baseline, same harness) and
|
||
kv-tier pg (bytea) at the lower sizes, as cross-checks of
|
||
methodology parity. (`loread`/`lowrite` via SQL functions through
|
||
tokio-postgres — the `postgres_large_object` crate targets
|
||
postgres 0.15 and is dead; do not use it.)
|
||
2. **Handle-cost probe** (`pgdiag7`): the tx-scoped descriptor under
|
||
deadpool — cost of open-tx → lo_open → lseek/loread per range read,
|
||
held vs. re-opened per read; pool pressure under N concurrent
|
||
readers (the ops fetch handler's real shape).
|
||
3. **Write path probe** (`pgdiag8`): lo_create + chunked lowrite
|
||
(the LOBLKSIZE ~2 KiB page layout matters here — measure write
|
||
amplification as value size grows); unknown-length put shape
|
||
(stage into one LO, companion row on commit); verify
|
||
`size`-without-fetch (`SELECT size` from the companion table).
|
||
4. **Churn/vacuum probe**: repeated put/delete cycles at 1–16 MiB;
|
||
measure `pg_largeobject` catalog growth, TOAST/vacuum behavior,
|
||
autovacuum posture (the fs engine inherits ADR-008/007's
|
||
maintenance-surface honesty requirement).
|
||
5. **Crash/orphan probe**: kill client mid-write-tx; verify no orphan
|
||
LO and no half-commit; kill after lo_create before commit; write
|
||
the orphan-recovery sweep story the engine ADR will need (compare
|
||
`lo_get`-visible OIDs against the companion table — never trust
|
||
the catalogs).
|
||
|
||
## Decision gate (mirrors ADR-007's)
|
||
|
||
pg-lo is admitted as a shipped fs-tier engine iff:
|
||
|
||
- **Performance:** LO put/get at 128 KiB–16 MiB is within an
|
||
order-of-magnitude of local fs on the same box (the tier is chosen
|
||
for fleet correctness, not solo speed — but a 10× gap would put the
|
||
fleet's packfile serving materially behind a single-node fs
|
||
deployment, and the ADR must say so if measured);
|
||
- **Contract:** complete `list()` (companion table), GC-participating
|
||
delete (`lo_unlink` under the delete window), virgin-store no-op
|
||
reads, `size` probe — all hold, proven via the exact-count
|
||
sweep-outcome test shape under `--all-features`;
|
||
- **Ops posture:** named deltas (tx-scoped handles, autovacuum/vacuum
|
||
on `pg_largeobject`, orphan recovery sweep) with measured or
|
||
documented-bounded costs.
|
||
|
||
Findings feed an engine ADR in the ADR-007 shape (measured curves,
|
||
posture deltas as requirements). If the gate fails, ADR-008's
|
||
recorded statement stands: shared-media or re-routing are REQ-2's
|
||
fleet answers, and pg-lo's named cost was paid as evidence, not
|
||
shipped blind.
|
||
|
||
## Register note (phase-0.md OQ-BL-06 convention)
|
||
|
||
The canonical register is phase-0.md OQ-BL-06: #1 trait dispatch,
|
||
#2 absorbed, #3 large-blob, #4 pooled-CAS (covered by #1), #5
|
||
postgres, #6 redb. The post-convergence POC files briefly carried
|
||
inconsistent self-numberings (the postgres file titled itself "#4");
|
||
they are renumbered to the register. This POC is **#7** — specified
|
||
2026-10-03, requested by REQ-2 (requirements.md; ADR-008 names pg-lo
|
||
as the candidate fs-tier engine), run 2026-10-03 — see
|
||
`poc-pglo-findings.md`. |