alkblobs/docs/research/poc-redb-kv-findings.md

---
status: passed
title: "POC #6 — redb as the kv engine: the 2-7×-over-sqlite claim does not survive crash-consistent measurement"
last_updated: 2026-10-02
---

# POC: redb as the kv engine — findings

> **POC register #6**, added post-convergence (same standing as #5: an
> ADR-003 substitution-seam input, not a Phase 0 gate). Code: standalone
> crate `/workspace/alkblobs-redb-poc` — a copy of the POC crate with the
> pg arm replaced by a `RedbKv` backend (`redb` 4.3, the engine iroh-blobs'
> fs store uses for its metadata tables). The inherited sqlite/fs arms and
> the 14-test suite are byte-identical to POC #3/#5's methodology.
> Date: 2026-10-02. Status: **passed** (14 tests, clippy `-D warnings`
> clean, fmt clean).

## What this POC set out to decide

The "opposite route" from POC #5: postgres is complicated, we only need a
kv — what about a high-performance *embedded* kv like redb, which the
DESIGN.md lineage (iroh-blobs' "faster than the filesystem for small
items") and downstream folklore put anywhere from **2-7× over sqlite**,
which was itself 7-9× over fs (POC #3 A4)? If redb beats sqlite by that
margin at equal semantics, the kv tier's engine pin (ADR-003) is again
informed rather than hypothetical — and so is the anti-recommendation if
it doesn't.

Same 2D instrument as POC #5 (the shape of the question, learned there):

1.Solo sweep (inherited `bench_small` harness, redb/sqlite/fs arms at
   1-256 KiB, identical working set) — where the 2-7× claim should show up.
2. **Worker scale-out probe** (`redb_conc`, plus the POC #5 `sqlite_conc`
   head-to-head) — redb is a single-file single-writer engine; does it
   degrade like sqlite's WAL or survive like postgres?

## Result summary

| Question | Verdict |
|----------|---------|
| redb 2-7× over sqlite for small blobs? | **No, inverted: sqlite ~430× over redb on crash-consistent puts** (finding C1) |
| redb single-put floor | **~8-20 ms** at `Durability::Immediate` — one fdatasync per commit whose latency *is* the commit (finding C2, strace-verified) |
| redb at `Durability::None` (crash-inconsistent) | ~25-28k puts/s solo — but **degrades under threads** (24.7k → 12k at 8, finding C4) and can leave the file unreopenable after a crash |
| redb reads | **Catastrophically fast and ~free**: ~530-575k gets/s (p50 1-4 µs) — mmap/page-cache reads, 1000× sqlite (finding C3) |
| redb write scale-out | **Flat ~40-44 ops/s at 1, 2, 4, 8 threads** — pure serialization, worse than sqlite's WAL (which at least reached ~1.1k), nowhere near pg's ~28k (finding C5) |
| Is the swap small like pg's? | **Yes** — ~230 lines, same table shape, but the durability API is coarser (finding C6) |

**The headline conclusion (C1+C2): the 2-7× claim is a *durability-tier
confusion*, not a measured property.** It implicitly compares workloads
where fsync costs are hidden (in-memory, or crash-inconsistent commits).
Under the shipped tier this crate would actually tolerate — redb
`Immediate` vs sqlite `synchronous=NORMAL` WAL, both crash-consistent —
sqlite's WAL commits are fsync-free (checkpoint-time only) while redb's
Immediate commits are one-fdatasync-each: the 430× inversion. redb's
genuine weapon is reads (C3), and reads are the thing the fs backend and
page cache were never losing at.

## Findings

### Finding C1 (benchmark, solo): sqlite ~430× faster than redb on crash-consistent puts at 1 KiB; the 2-7× claim inverted

Inherited harness, 1 s/arms, representative run (see the harness for a
re-run; all sizes measured, table shows key ones):

```
backend  size      puts/s    gets/s    put p50µs  put p99µs  get p50µs  get p99µs
redb     1024         ~100   ~540000         8334      ~45000         1.0          5
sqlite   1024       ~44000    ~51000           21          64        19.0         24
fs       1024        ~5000     ~3500          200         285       282.0        365
redb     16384        ~105   ~530000         8333      ~42000         1.0          5
sqlite   16384      ~13000    ~14800           73         205        66.0         79
fs       16384       ~4700     ~2870          210         350       410.0        580
redb     65536        ~100   ~530000         8334      ~70000         1.0          5
sqlite   65536       ~3700     ~4080          270         495       243.0        310
fs       65536       ~2550     ~1840          410         550       543.0        665
redb     131072        ~103   ~527000         8330      ~73000         1.0          5
sqlite   131072       ~1900     ~1930          530         795       507.0        768
fs       131072       ~1930     ~1540          527         795       629.0        840
redb     262144         ~86   ~516000         8333      ~70000         1.0          5
sqlite   262144        ~900     ~990         1080        2825      1008.0       1194
fs       262144       ~1530     ~2470          588        1375       381.0        580
```

Readings:

- **redb puts are flat ~100/s regardless of size** (1 KiB through
  256 KiB — the put cost is the commit, not the value). 8.3 ms p50,
  40-85 ms p99. sqlite remains 440×/1300×/4×/2×/0.6× ahead at
  1/16/64/128/256 KiB respectively.
- **The fs crossover survives** (fs wins 256 KiB puts, and puts get
  close at 128 KiB) — A4's threshold story is unchanged by a fourth arm.
- sqlite's numbers reproduce A4 within noise again (44k at 1 KiB):
  methodology parity across all four POCs holds.

### Finding C2: redb's Immediate commit is *exactly one fdatasync* — and on a loaded mdraid that sync costs 8-30 ms; the p50 tracks it 1:1

Decomposition probes (`redbdiag*`):

- **Warm Immediate puts: p50 8.3-19.7 ms depending on device pressure,
  p99 40-135 ms.** `strace -c` on a full bench run: 5784 fdatasync,
  ~300 µs mean each; `strace -tt -T` on a warm solo run: 363 fdatasync
  accounting for **95% of wall time** (10.2 s of 10.7 s), per-fdatasync
  19-30 ms p50-p90 on the loaded disk. The commit latency *is* the
  fdatasync latency.
- The fdatasync count per put is ~1 (~300 Immediate commits + ~13
  startup/savepoint commits vs 313 calls in the isolated probe) — no
  2-phase double-sync; it's a single sync, just an expensive one on a
  device with dirty-page pressure (the pg container's WAL churn was
  co-resident through part of this POC — noted in the caveats; the
  *relative* redb-vs-sqlite conclusion is unaffected, and redbdiag was
  re-run clean at the end: p50 18.5 ms).
- **Durability::None puts: 27-33 µs p50** — the fsync removal alone
  accounts for the 300× spread. Confirms the mechanism precisely.

### Finding C3: redb reads are ~530-575k gets/s (p50 1-4 µs) — the page cache through an mmap-like path

The has/get probes return in 1-4 µs p50 (5 µs p99) across every size —
that's not a B-tree probe, that's the OS page cache served from
process-mapped memory. sqlite's 19-23 µs p50 is a syscall-per-probe; fs's
282 µs is open+read syscalls. **This is redb's real 10-1000× win — on the
operation that matters least**, because (a) the fs tier already serves
large blobs from the page cache the same way, and (b) small-tier gets
were never the bottleneck axis (sqlite was already the fastest get at
19 µs, amortized by the store's digest checks anyway).

### Finding C4: redb's Durability::None concurrency *degrades* — 24.7k solo → 20k at 4 threads → 12k at 8

`redb_conc` in none-mode: the None arm has no fdatasync to amortize, so
adding threads only adds internal transaction-lock contention plus the
None-durability freelist growth penalty (pages freed between durable
commits accumulate until an Immediate commit collects them — redb's
documented "rapid growth" behavior). Contrast POC #5: sqlite's
contentioned WAL plateaued ~1.2k, pg *scaled* to 28k+. redb's None mode
is the only arm in three POCs where throughput goes **down** as workers
go up.

### Finding C5 (the decisive one): redb write scale-out is perfectly flat — ~40-44 ops/s at 1, 2, 4, and 8 threads

`redb_conc` Immediate mode (same fresh-key discipline as POC #5's
`pgdiag6`/`sqlite_conc`):

| workers | redb Immediate | sqlite (same probe) | pg (from POC #5) |
|---------|---------------|---------------------|------------------|
| 1 | ~42-44 ops/s | ~700-800/s | ~2.5-3.2k/s |
| 2 | ~40 ops/s | ~1.0-1.1k/s | ~4.6-5.5k/s |
| 4 | ~40-43 ops/s | ~0.6-0.9k/s | ~20-21k/s |
| 8 | ~43 ops/s | ~1.1-1.2k/s | ~28k/s |

- redb's single-writer file lock + internal transaction serialization
  admits **zero concurrency**: flat at the solo commit cost (~24 ms/put
  fresh-key; ~8-19 ms re-put). sqlite's WAL at least *queues* writers
  behind the one flush lock and gets modest throughput from contention
  efficiency; redb just serializes full fdatasync latencies.
- Against pg's curve this is the whole argument in one row set: the
  embedded engines' curves are flat/degrading; the client-server curve
  is the only increasing one.

### Finding C6: the swap is small but the durability API is coarser than sqlite's

`kv_redb.rs` (~230 lines): same row shape (`TableDefinition<&[u8],
(&[u8], i64)>` — the value/size tuple matches sqlite's row exactly;
`WITHOUT ROWID`-equivalent PK ordering is native B-tree). Differences:

- **redb 4.x offers only `Durability::None` and `Durability::Immediate`**
  — no middle tier. sqlite's `synchronous=NORMAL` (consistent-on-crash,
  not-fsynced-per-commit — the shipped tier this crate benchmarks and
  documents) has **no redb equivalent**: `None` is not
  crash-consistent (unpersisted commits can leave the file failing to
  open until repaired — the freelist bookkeeping itself is lost on
  crash), `Immediate` pays a full fdatasync per commit. The
  durability-posture comparison that POC #5 could tune continuously
  (`synchronous_commit=off` on pg) doesn't exist here.
- redb's `Database` is process-lifetime; concurrency modes
  (`ExclusiveWriter` default, `SingleWriter`, `MultiWriter`) only matter
  multi-process — in-process it's one writer transaction at a time,
  structurally.
- `create`-then-repair behavior on a `None`-durability crash and the
  `has+insert` CAS dance inside one write tx are correct but slower
  (the bench's re-put path costs a table probe per commit).

## Consequences for the phase-0/1 register

- **redb is ruled out as the shipped kv engine at a measured tier
  mismatch, not a benchmark quibble**: the only durability postures it
  offers are "lose acknowledged data on crash" or "pay one fdatasync
  per commit" — the crate's shipped durability tier
  (`synchronous=NORMAL` WAL, crash-consistent, no per-commit fsync)
  is not expressible. sqlite keeps ADR-003's pin on both economics
  (44k vs 100 puts/s at 1 KiB) and posture.
- **The iroh-blobs DESIGN.md lineage ("faster than the filesystem for
  small items") is now fully triangulated**: its kv-tier claim is
  true for its *inline small items with batched/durable-None commits*
  (iroh batches writes through a serialized actor with batch windows —
  `BatchOptions.max_write_batch`, the same amortization pg gets from
  WAL group-commit *for free across connections*). Our harnesses
  measure per-commit put discipline; iroh's actor batching is the
  reconciliation, and it is also why our POC #3 sqlite numbers (44k)
  are the honest ceiling for a per-op CAS store.
- **Batched commits are a real Phase 1 lever, proven by three
  independent engines**: pg's group-commit (B3), iroh's write actor
  windows, redb's None+flush emulation (arm C4's structure). The
  store's put seam could take a `WriteBatch`-shaped commit later — an
  API question for Phase 1 (pinned batch puts exist in ADR-005's
  surface already; the *durability-window* question is adjacent).
- **Reads: no case for redb on merit** — page-cache reads are what any
  of these give the large tier for free; sqlite's 19 µs probe was
  already fine.
- **Anti-recommendation adopted (ADR-007, 2026-10-02): redb stays
  ruled out** — recorded in the decision so the question does not
  reopen implicitly. SQLite's engine pin now has four-way triangulated
  evidence (POC #3 fs, POC #5 pg, POC #6 redb, plus the lineage
  citation), and ADR-007 ships the kv tier as sqlite + postgres. If a
  future consumer wanted redb-shaped durability trade-offs (ephemeral
  cache tier), that's a *different* tier than the kv engine slot.

## POC quality notes

- The inherited 14-test suite passes unchanged (the redb backend is
  additive; sqlite/fs arms untouched). The CAS semantics of
  put-if-absent are exercised through the smoke path; redb's
  insert-returns-prior-guard shape documented in `kv_redb.rs` docs.
- clippy `-D warnings` clean, fmt clean.
- **Honest caveats**: mdraid single dev box whose fdatasync latency
  ranged 0.24-30 ms during the session under co-resident load (pg
  container churn from POC #5 ran through the first bench; stopped
  before the final redbdiag re-run — relative conclusions unaffected,
  absolute redb-Immediate numbers are *device-pressure-dependent* and
  the honest statement is "1 fdatasync per commit, cost = device
  fdatasync latency, observed 8-30 ms here"); the 2s full bench was
  dropped from the record (its 128/256 KiB redb arms hit shell
  timeouts; 1s arms with explicit size lists used instead — the harness
  gained a size-list argument for this); the 1g redb working-set files
  (~270 MB at 128 KiB arm) cleaned from /tmp after diagnosis.
- Instruments: `src/kv_redb.rs` (the backend: Immediate + None + flush
  emulation methods), `src/bin/bench_small.rs` (inherited + redb arm +
  size-list arg), `examples/redbdiag.rs` (durability decomposition),
  `redbdiag2.rs` (None + periodic-flush group-commit emulation),
  `redbdiag3.rs` (commit-vs-insert split, per-put file growth),
  `redbdiag4.rs` (warm distribution), `redb_conc.rs` (scale-out, both
  durability modes), `sqlite_conc.rs` (copied head-to-head from #5).