alkblobs/docs/research/poc-postgres-kv-findings.md

---
status: passed
title: "POC #5 — postgres as the kv engine: single-conn penalty measured, concurrency scale-out measured"
last_updated: 2026-10-03
---

# POC: postgres as the kv engine — findings

> **POC register #5**, added after Phase 0 convergence (the register was
> complete at #3; this is the ADR-003 "substitution seam" input arriving
> early). *Renumbered 2026-10-03 from a self-assigned "#4" to the
> canonical register (phase-0.md OQ-BL-06), where #4 is the pooled-CAS
> POC covered by #1; the redb file's cross-references were fixed in the
> same pass.)* Code: standalone crate `/workspace/alkblobs-postgres-poc`
> (a copy of the POC #3 crate with a `PgKv` backend added; findings land
> here regardless, per the established convention). Date: 2026-10-02.
> Status: **passed** — the inherited POC #3 test suite (14 tests, byte-exact
> vs `git hash-object` CLI) passes unchanged, clippy `-D warnings` clean,
> fmt clean. Server: dockerized postgres:16, `--rm` container. The pg arm
> rides `tokio-postgres` + `deadpool-postgres` (client-side pool).

## What this POC set out to decide

The motivating asymmetry: the crate ships two backends (kv/sqlite, fs).
A downstream whose deployment already runs postgres (the distributed-git
replicator case: many users against one node) must either accept sqlite
*because we pinned it* or implement a third Backend with its own
sweep-safety story. The question was purely mechanical: **does postgres
hold the "kv beats fs for small blobs" property that the whole two-tier
design rests on (POC #3 A4 / finding A2 of the original appfileformat
citations)?** If yes, postgres is a viable kv engine under the same
trait contract; the engine pin (ADR-003) is a choice, not a structural
necessity.

Two benchmark instruments:

1. **The inherited `bench_small` harness** (sqlite/fs/pg arms at
   1-256 KiB, 2k distinct 32-byte git-sha keys, random replay) —
   byte-identical methodology to POC #3 A4, so the numbers are directly
   comparable.
2. **New concurrency probes** (`pgdiag4/5/6`, `sqlite_conc`) — N
   workers, one connection each (own-connection discipline, no
   cross-worker contention in the client), head-to-head write and
   read scaling. This is the axis POC #3 never measured: its benchmark
   was single-threaded single-connection, which structurally favors the
   sqlite arm (embedded, zero-framing) and structurally understates
   postgres's whole reason to exist (connection scale-out).

## Result summary

| Question | Verdict |
|----------|---------|
| Does pg hold "kv beats fs" for small blobs? | **Not below ~64 KiB** — fs wins at 1-16 KiB on this disk (finding B1); pg wins at 64-128 KiB |
| Single-conn pg put latency | **~1 ms floor** — ~10-50× sqlite per-op, dominated by round-trips + commit-path fsync (finding B2) |
| Does pg scale out under concurrency? | **Yes, decisively** — puts scale near-linearly 1→4 conns (~2.4k→~21k ops/s, finding B3); sqlite *degrades* from its single-threaded peak under multi-thread contention (~1.2k ops/s ceiling, ~0.5-1× of its 42k solo number) |
| Get-side scale-out | Same shape: 3.3k (1 conn) → ~30-44k ops/s (8-24 conns, finding B4) |
| Storage efficiency | ~40× sqlite's file size for the same 2k×1 KiB working set (finding B6: fixed page granularity + block overhead) |

**The headline conclusion (B1+B3): the comparison is two-dimensional,
and the original question was one-dimensional.** POC #3 asked "what's
the fastest backend one worker can use?" and answered sqlite. The
distributed-git question is "what's the fastest backend N concurrent
workers can share?" — and on that axis sqlite's WAL-write-lock
serialization caps it ~1.2k ops/s while postgres multiplies. Single-user
/private deployments: sqlite's numbers stand. Many-replicator-client
nodes: the crossover flips at least an order of magnitude below the
1-16 KiB zone.

## Findings

### Finding B1 (benchmark, single-worker): postgres holds kv-beats-fs only in the 64-128 KiB zone on this disk; below it fs wins

Inherited harness, 1-2 s/arm, same 2k-key working set as POC #3 A4.
Representative run (this box's SSD; disk measured ~19 MB/s `dd oflag=dsync`
— see B2 for why pg cares):

```
backend  size       puts/s      gets/s   put p50  put p99   get p50   get p99
pg       1024          ~1000        ~900    ~1000    ~1550     ~1050     ~1600
sqlite   1024         ~44000      ~47000      ~21       50       ~20       ~27
fs       1024          ~5400       ~3500     ~200      ~280     ~280      ~390
pg       16384          ~900        ~900    ~1050    ~1600     ~1200     ~1650
sqlite   16384        ~13500      ~14500      ~70      160       ~66       ~87
fs       16384         ~4200       ~2300     ~240      ~370     ~440      ~560
pg       65536          ~800        ~840    ~1350    ~1850     ~1150     ~1600
sqlite   65536         ~4100       ~4200     ~250      390      ~240      ~370
fs       65536         ~2900       ~2300     ~370      ~530     ~390      ~740
```

(The 256 KiB arm kept timing out inside the pg arm's runtime windows and
was dropped from final runs — see the POC-quality-notes caveats.)

Readings:

- **At 1-16 KiB (the git small-blob regime), postgres does *not* hold
  the kv-beats-fs property in this deployment**: the ~1 ms pg round-trip
  floor is 2-5× fs's ~200-450 µs syscall chain (and 10-50× sqlite).
  POC #3's fs numbers reproduce within noise, confirming methodology
  parity.
- **At 64-128 KiB postgres sits between fs and sqlite** (~800-1000 vs
  fs ~2300-2900 on gets — *behind* fs on some arms; sqlite still leads
  ~4.1k at 64 KiB). The honest reading: pg holds no decisive win in
  the 1-128 KiB single-worker band on this disk; **the pg-vs-fs
  crossover, where it exists at all, sits higher than sqlite-vs-fs's**.
  Note: pg's decisive advantage was never the solo 1-D curve — it is
  the concurrency axis (B3/B4), where it multiplies and sqlite
  serializes.
- The sqlite numbers reproduce A4 within noise (44k vs 48k at 1 KiB),
  so the methodology transfer is sound.

### Finding B2: the ~1 ms single-op pg floor is protocol + fsync, and on this disk fsync dominates

Isolated probes (raw `tokio-postgres`, single conn, prepared stmts,
`synchronous_commit=on` default ship config):

- **Prepared round-trips on an idle keepalive connection are ~150-500 µs**
  (p50 ~140-220 µs for a miss-probe, ~160-330 µs for inserts over
  repeated runs) — that is the wire-protocol + query-execution floor.
- **A cold `pgdiag`-style loop (fresh process, anon statements, no
  prepared cache) lands at ~1 ms p50 with ~1.5-1.6 ms p99s** —
  matching the bench arm's ~1000 µs floor. Decomposition:
  ~0.85 ms is the **commit-round-trip** (synchronous fsync of WAL,
  fdatasync on this disk measured 18.7 MB/s at `dd oflag=dsync` —
  plausibly ~5-8 ms worst-case per fsync, and the 8.3-15 ms p99 INSERTs
  in isolated runs smell like fsync batching edges), and the rest is
  parse/plan (anon statements re-parse every op; prepared statements
  drop ~0.6-0.8 ms off, see pgdiag2's 150-250 µs).
- `fsync=off` + `synchronous_commit=off` (restart) moved the
  fresh-conn INSERT p50 from ~24k µs-equivalent to ~450 µs —
  **fsync is the dominant single-op cost on this disk**, exactly the
  axis sqlite's `synchronous=NORMAL` trades down (and the sqlite arm
  in A4 also doesn't full-fsync per op — the comparison at NORMAL-WAL
  durability was fair as-measured and remains the shipped tier).
- **Fresh-connection cost is enormous**: ~19-25 ms p50 per new session
  (pgdiag3), i.e. connection churn must be pooled — deadpool-style
  keepalive pools are the correct shape (this validates ADR-003's
  prepared-statements + bounded-reads discipline carrying over).

### Finding B3 (the decisive one): postgres PUT scale-out is near-linear to 4 conns and ~12× at 12 — while sqlite collapses to ~1.2k ops/s multi-threaded

`pgdiag6` vs `sqlite_conc` — same op (INSERT-or-ignore, 32-byte key,
1 KiB value), N workers each with one private connection/thread,
distinct keys, wall-clock window:

```
workers    pg ops/s (3 separate runs)         sqlite ops/s (3 separate runs)
1          2.5k, 3.0k, 2.5k  (~2.7-3.2k peak)   0.67k, 0.72k, 0.78k
2          4.6-5.5k                            0.96k, 1.04k, 1.13k
4          20.1k, 20.8k, 21.3k                 0.63k, 0.89k, 0.95k
8          27.6k, 27.8k, 27.8k                 0.75k, 1.19k, 1.25k
12         30.2k, 32.2k, 32.2k                 (not run: sqlite plateaued)
24         36.9k, 37.6k                        —
48         37.4k                               —
```

Readings:

- **pg scales ~5.7× by 8 conns and ~12× by 12 conns** (vs its own
  single-conn number), flattening at 24-48 (the 8-core box; postgres
  CPU itself is the limit, not the fs). The 4-vs-1 discontinuity
  (~2.7k → ~20k) is WAL-group-commit payback: the first extra
  connection amortizes the fsync across concurrent commits.
- **sqlite degrades under the same discipline**: its own solo number in
  this concurrent harness is ~0.7-0.8k (vs ~42k in the single-threaded
  bench_small loop — the difference is prepared-stmt cache + the
  POC #3 harness not crossing threads through a mutex), and N threads
  contending on the one write lock plateau at **~1.2k ops/s — the
  known sqlite WAL writer-serialization ceiling** — with occasional
  *inversions* (4 threads slower than 1: lock thrash). Multi-threaded
  sqlite writes are where its embedded advantage ends.
- Per-op pg latency inside the scaled loop is still ~200-500 µs
  (measured in B2's probes) — so ~21-37k ops/s is not batching
  artifacts, it genuinely reflects the WAL-group-commit amortization.

### Finding B4 (get-side): pg read scale-out is the same shape — 3.3k solo → ~30k at 8 conns, ~44k at 24

`pgdiag5` (SELECT-probe over a 2k×1 KiB set, spread across conns):

```
conns    1        2        4        8        12       24
ops/s    3.3k     8.6k    21.9k    29.9k    34.3k    44.0k
```

Reads scale cleanly (no writer serialization to fight) and at 8+
conns land in sqlite-bench_small territory (~30k vs ~47k) — sqlite
still leads solo reads on a private connection, but pg *serves many
clients concurrently* which a single-file database physically cannot.

### Finding B5: the engine swap is small, but the durability-tuning posture differs by engine

The swap itself (`kv_postgres.rs`, ~180 lines): same table shape
(`key bytea PK / value bytea / size bigint`, fillfactor 90), same
INSERT-on-conflict-DO-NOTHING CAS semantics (dead-tuple count returned;
`ON CONFLICT` matches `INSERT OR IGNORE`), same prepared-statement +
bounded-read discipline. Differences that matter:

- **No `synchronous=NORMAL` equivalent at table level** — postgres's
  matching knob is `synchronous_commit` (session/system), default
  `on`. The durability parity with ADR-003's shipped tier is
  *configurable but not free*: a deployment wanting sqlite-NORMAL-alike
  latency sets it at the pool/session level, and our benchmark showed
  that moves the single-op floor by ~2-3× on this disk.
- **Connection pooling is load-bearing**, not an optimization: fresh
  sessions cost ~20 ms (B2) and prepared statements must be
  per-connection (cached statements die with sessions; the probe hit
  the "prepared statement s1 does not exist" error exactly when a
  pool-recycled conn lost its prepared statement — deadpool's
  statement cache handled it, but the pattern must be encoded in the
  backend impl, not hoped for).
- **Vacuum/autovacuum is a real maintenance surface** for a
  continuously-churning CAS pool (dead tuples from every re-put) —
  the sqlite arm has nothing analogous. A backend ADR for pg would need
  an explicit autovacuum tuning statement, not silence.
- Table metadata: the row shape needs `fillfactor` tuning for a
  mostly-append immutable workload; default (100) already produced a
  bloated-looking heap in early runs that needed explicit VACUUM before
  honest reads.

### Finding B6: pg's storage overhead is material at the small tier — ~40× sqlite's footprint for the same working set

Same 2k×1 KiB working set: sqlite file ~10.5 MB (db+wal); pg table
after VACUUM ~92 MB heap + 6.7 MB idx at *82829 rows* (~1.1 KB/row —
the 8.7× bigger-than-value row floor comes from page-fill and block
overhead), i.e. ~40 MB for 2k×1 KiB of real content vs sqlite's
~10.5 MB. At 8 KiB+ this gap narrows relatively (the per-row overhead
amortizes), but at 1-4 KiB it's real. For the small-blob tier this
matters mostly for disk-space-vs-benefit calculus at small-node scale —
a deployment running the *fs tier anyway* has none of either cost.

## Consequences for the phase-0/1 register

- **A4's crossover was confirmed but shown to be one axis of a
  2D question**: the sqlite-vs-fs and pg-vs-fs crossovers are ~
  128-256 KiB and ~16-64 KiB respectively on this box, and the
  single-connection framing (A4's whole shape) is the *only* framing in
  which sqlite's 42k solo number is the relevant one.
- **The "engine substitution" door in ADR-003** ("a downstream wanting
  a different engine there would be new ADRs") is now informed, not
  hypothetical: postgres can hold the kv tier's contract (the CAS
  semantics, complete-list, GC-participating delete all map), but it is
  a *durability-and-ops posture change*, not a free drop-in: B5's
  differences (synchronous_commit posture, pooling discipline,
  autovacuum surface, ~40× storage floor) are the ADR content any
  postgres Backend impl would carry.
- **Decision status (as originally recorded): no ADR action from a
  single POC at the time of writing** — sqlite stayed the pinned
  engine, postgres the named candidate. *Amended 2026-10-02:* the
  decision has since been made on the accumulated evidence of this
  POC + POC #6 (with POC #3's A4 sqlite baseline) — ADR-007 ships
  postgres as the kv tier's second engine (feature-gated,
  constructor-selected); this section's measured content above stands
  as the ADR's evidence base.
- **OQ-BL-02 residuals**: unchanged in substance. The one new residual:
  if a postgres Backend were adopted, its GC-sweep implementation is
  `TRUNCATE`-friendly (whole-table dead-set rebuilds) but the
  list()-complete-by-contract test would need the vacuum-aware
  equivalent of POC #1's exact-count sweep assertions.

## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #5/#6)

The three alkblobs-relevant git/vfs use cases, mapped onto the measured
curves; this was the conclusion OQ-10 carried before its resolution
in ADR-007):

| Engine | Covers | Evidence |
|---|---|---|
| **sqlite** | self-hosted git serving (gitea-replacement scale), agent-workspace pools, small/personal replicators | 44k solo puts/s at 1-16 KiB (POC #3 A4, reproduced here); WAL single-writer ceiling ~1.2k/s under contention is comfortably above single-node push rates; zero-ops, no daemon; honker rides the same file (same WAL-NORMAL posture, its defaults are literally the shipped tier) |
| **postgres** | public multi-tenant replicator nodes (many users pushing, gossip catch-up, bulk ingest), consolidated-ops deployments (already-running pg) | near-linear put scale-out to ~21k at 4 conns, ~28k at 8, ~32k at 12, ~37k at 24-48 (B3); reads scale to ~44k at 24 conns (B4); the only engine whose throughput *increases* under concurrency |
| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #5/#6 reproductions |

The decision rule the two POCs establish: **the trigger is topology,
not throughput** — cross-MACHINE write access to one pool makes postgres
mandatory; sustained-rate pressure (sqlite's flatline failure mode:
latency climbs without bound under overload) is the early-warning that
precedes it; single-node/private/small-team cases stay sqlite by
measurement, with honker-parity features (notify/queue/stream) available
free on the sqlite file and via `LISTEN`/`NOTIFY` + pgboss-rs on a
future pg deployment (a consumer-layer concern either way — the store
crate ships no queue).

**Per-node choice is the form the fork takes** (constructor parameter,
ADR-003): a heterogeneous replicator fleet — personal nodes on sqlite,
community nodes on pg — syncs over the same wire above store-core seams
that are engine-blind. Work-above-the-trait (store core, GC mechanism,
ops, consumers) is written once; engine work parallelizes against the
fixed contract (two engines, one trait, same sweep-safety proof shape).

The deployment economics paragraph this table encodes, now stated
outright (recorded in ADR-007): the classical downstream stack — a git
server, a vfs node — needs three storage systems (kv + fs + relational);
this design collapses the kv into the relational engine, so a node
provisions **one relational engine + fs**, and which relational engine
is the operator's per-node choice among the shipped engines.

Sequencing (amended 2026-10-02 by ADR-007): postgres **ships** as the
kv tier's second engine (feature `postgres`, default-off),
constructor-selected per node — sqlite stays the default and Phase 1's
mainline; the pg impl is normal Phase 1 implementation work,
parallelizable against the fixed contract (two engines, one trait,
same sweep-safety proof shape, CI-gated via `--all-features`). The
earlier framing here and in OQ-10 — a "standing offer" whose ADR
opens on a first multi-tenant replicator deployment — was caught at
review as circular hedging: such a deployment runs alkblobs, so it
cannot cross the trigger until the pg engine ships, which the trigger
then forbids shipping. Superseded by
[ADR-007](../architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md),
which records the decision this POC's evidence supports; the
deployment-trigger rule survives in corrected form as deployment
guidance (which engine a *node* chooses — ADR-007's decision rule),
not as a gate on the crate's own work.

## POC quality notes

- The inherited 14-test suite (12 integration + 2 git-CLI cross-checks)
  passes unchanged against the swapped crate — the pg backend is
  additive, so the POC #3 contract tests all still hold for the
  sqlite/fs arms; the pg arm has its own smoke (CAS semantics,
  size/get/has/delete round-trip, 256 KiB blob round-trip).
- clippy `-D warnings` clean, `cargo fmt` clean (diagnostics: the
  2s-per-arm pg arm of the full bench run *does* exceed the bench
  harness's 120 s default shell timeout — the run above is 1s/arm).
- **Honest caveats (all repeat POC #3's)**: single dev box, one SSD
  (measured 18.7 MB/s at dsync, which drives B2's conclusions, and
  ~zero-cache-warm on some arms), dockerized postgres shares the same
  disk, 8 cores total. The 256 KiB pg arm timed out across several runs
  (the harness's 300 s shell budget with 2s/arm × 8 arm-runs; the
  1 KiB-128 KiB pg numbers are reproducible within noise across three
  runs but 256 KiB's pg-vs-* conclusion is *dropped* rather than
  guessed), and pg's `fsync`-off-vs-on decomposition (B2) is a
  *deliberate instrumentation stop* — it explains the shape, it does
  not ship a config. The concurrent sqlite number is one process
  sharing one `parking_lot` mutex — a *multi-process* deployment would
  not have that mutex but would still have WAL's single-writer
  serialization, so ~1.2k ops/s stands as the real ceiling shape.
- Instruments: `examples/pgdiag.rs` (anon-stmt latency decomposition),
  `pgdiag2.rs` (prepared stmts + keepalive), `pgdiag3.rs` (fresh-conn
  cost), `pgdiag4.rs` (write scale-out, pooled pool with
  per-worker-connection discipline), `pgdiag5.rs` (read scale-out),
  `pgdiag6.rs` (write scale-out wall-clock head-to-head),
  `sqlite_conc.rs` (the same discipline against the POC #3 sqlite
  shape). Docker: `docker run --rm -e POSTGRES_PASSWORD=poc
  POSTGRES_DB=blobs -p 127.0.0.1:15432:5432 postgres:16`.