alkblobs/docs/research/poc-redb-kv-findings.md

status, title, last_updated

status	title	last_updated
passed	POC #6 — redb as the kv engine: the 2-7×-over-sqlite claim does not survive crash-consistent measurement	2026-10-02

POC: redb as the kv engine — findings

POC register #6, added post-convergence (same standing as #5: an ADR-003 substitution-seam input, not a Phase 0 gate). Code: standalone crate /workspace/alkblobs-redb-poc — a copy of the POC crate with the pg arm replaced by a RedbKv backend (redb 4.3, the engine iroh-blobs' fs store uses for its metadata tables). The inherited sqlite/fs arms and the 14-test suite are byte-identical to POC #3/#5's methodology. Date: 2026-10-02. Status: passed (14 tests, clippy -D warnings clean, fmt clean).

What this POC set out to decide

The "opposite route" from POC #5: postgres is complicated, we only need a kv — what about a high-performance embedded kv like redb, which the DESIGN.md lineage (iroh-blobs' "faster than the filesystem for small items") and downstream folklore put anywhere from 2-7× over sqlite, which was itself 7-9× over fs (POC #3 A4)? If redb beats sqlite by that margin at equal semantics, the kv tier's engine pin (ADR-003) is again informed rather than hypothetical — and so is the anti-recommendation if it doesn't.

Same 2D instrument as POC #5 (the shape of the question, learned there):

1.Solo sweep (inherited bench_small harness, redb/sqlite/fs arms at 1-256 KiB, identical working set) — where the 2-7× claim should show up. 2. Worker scale-out probe (redb_conc, plus the POC #5 sqlite_conc head-to-head) — redb is a single-file single-writer engine; does it degrade like sqlite's WAL or survive like postgres?

Result summary

Question	Verdict
redb 2-7× over sqlite for small blobs?	No, inverted: sqlite ~430× over redb on crash-consistent puts (finding C1)
redb single-put floor	~8-20 ms at Durability::Immediate — one fdatasync per commit whose latency is the commit (finding C2, strace-verified)
redb at Durability::None (crash-inconsistent)	~25-28k puts/s solo — but degrades under threads (24.7k → 12k at 8, finding C4) and can leave the file unreopenable after a crash
redb reads	Catastrophically fast and ~free: ~530-575k gets/s (p50 1-4 µs) — mmap/page-cache reads, 1000× sqlite (finding C3)
redb write scale-out	Flat ~40-44 ops/s at 1, 2, 4, 8 threads — pure serialization, worse than sqlite's WAL (which at least reached ~1.1k), nowhere near pg's ~28k (finding C5)
Is the swap small like pg's?	Yes — ~230 lines, same table shape, but the durability API is coarser (finding C6)

The headline conclusion (C1+C2): the 2-7× claim is a durability-tier confusion, not a measured property. It implicitly compares workloads where fsync costs are hidden (in-memory, or crash-inconsistent commits). Under the shipped tier this crate would actually tolerate — redb Immediate vs sqlite synchronous=NORMAL WAL, both crash-consistent — sqlite's WAL commits are fsync-free (checkpoint-time only) while redb's Immediate commits are one-fdatasync-each: the 430× inversion. redb's genuine weapon is reads (C3), and reads are the thing the fs backend and page cache were never losing at.

Findings

Finding C1 (benchmark, solo): sqlite ~430× faster than redb on crash-consistent puts at 1 KiB; the 2-7× claim inverted

Inherited harness, 1 s/arms, representative run (see the harness for a re-run; all sizes measured, table shows key ones):

backend  size      puts/s    gets/s    put p50µs  put p99µs  get p50µs  get p99µs
redb     1024         ~100   ~540000         8334      ~45000         1.0          5
sqlite   1024       ~44000    ~51000           21          64        19.0         24
fs       1024        ~5000     ~3500          200         285       282.0        365
redb     16384        ~105   ~530000         8333      ~42000         1.0          5
sqlite   16384      ~13000    ~14800           73         205        66.0         79
fs       16384       ~4700     ~2870          210         350       410.0        580
redb     65536        ~100   ~530000         8334      ~70000         1.0          5
sqlite   65536       ~3700     ~4080          270         495       243.0        310
fs       65536       ~2550     ~1840          410         550       543.0        665
redb     131072        ~103   ~527000         8330      ~73000         1.0          5
sqlite   131072       ~1900     ~1930          530         795       507.0        768
fs       131072       ~1930     ~1540          527         795       629.0        840
redb     262144         ~86   ~516000         8333      ~70000         1.0          5
sqlite   262144        ~900     ~990         1080        2825      1008.0       1194
fs       262144       ~1530     ~2470          588        1375       381.0        580

Readings:

• redb puts are flat ~100/s regardless of size (1 KiB through 256 KiB — the put cost is the commit, not the value). 8.3 ms p50, 40-85 ms p99. sqlite remains 440×/1300×/4×/2×/0.6× ahead at 1/16/64/128/256 KiB respectively.
• The fs crossover survives (fs wins 256 KiB puts, and puts get close at 128 KiB) — A4's threshold story is unchanged by a fourth arm.
• sqlite's numbers reproduce A4 within noise again (44k at 1 KiB): methodology parity across all four POCs holds.

Finding C2: redb's Immediate commit is exactly one fdatasync — and on a loaded mdraid that sync costs 8-30 ms; the p50 tracks it 1:1

Decomposition probes (redbdiag*):

• Warm Immediate puts: p50 8.3-19.7 ms depending on device pressure, p99 40-135 ms. strace -c on a full bench run: 5784 fdatasync, ~300 µs mean each; strace -tt -T on a warm solo run: 363 fdatasync accounting for 95% of wall time (10.2 s of 10.7 s), per-fdatasync 19-30 ms p50-p90 on the loaded disk. The commit latency is the fdatasync latency.
• The fdatasync count per put is ~1 (~300 Immediate commits + ~13 startup/savepoint commits vs 313 calls in the isolated probe) — no 2-phase double-sync; it's a single sync, just an expensive one on a device with dirty-page pressure (the pg container's WAL churn was co-resident through part of this POC — noted in the caveats; the relative redb-vs-sqlite conclusion is unaffected, and redbdiag was re-run clean at the end: p50 18.5 ms).
• Durability::None puts: 27-33 µs p50 — the fsync removal alone accounts for the 300× spread. Confirms the mechanism precisely.

Finding C3: redb reads are ~530-575k gets/s (p50 1-4 µs) — the page cache through an mmap-like path

The has/get probes return in 1-4 µs p50 (5 µs p99) across every size — that's not a B-tree probe, that's the OS page cache served from process-mapped memory. sqlite's 19-23 µs p50 is a syscall-per-probe; fs's 282 µs is open+read syscalls. This is redb's real 10-1000× win — on the operation that matters least, because (a) the fs tier already serves large blobs from the page cache the same way, and (b) small-tier gets were never the bottleneck axis (sqlite was already the fastest get at 19 µs, amortized by the store's digest checks anyway).

Finding C4: redb's Durability::None concurrency degrades — 24.7k solo → 20k at 4 threads → 12k at 8

redb_conc in none-mode: the None arm has no fdatasync to amortize, so adding threads only adds internal transaction-lock contention plus the None-durability freelist growth penalty (pages freed between durable commits accumulate until an Immediate commit collects them — redb's documented "rapid growth" behavior). Contrast POC #5: sqlite's contentioned WAL plateaued ~1.2k, pg scaled to 28k+. redb's None mode is the only arm in three POCs where throughput goes down as workers go up.

Finding C5 (the decisive one): redb write scale-out is perfectly flat — ~40-44 ops/s at 1, 2, 4, and 8 threads

redb_conc Immediate mode (same fresh-key discipline as POC #5's pgdiag6/sqlite_conc):

workers	redb Immediate	sqlite (same probe)	pg (from POC #5)
1	~42-44 ops/s	~700-800/s	~2.5-3.2k/s
2	~40 ops/s	~1.0-1.1k/s	~4.6-5.5k/s
4	~40-43 ops/s	~0.6-0.9k/s	~20-21k/s
8	~43 ops/s	~1.1-1.2k/s	~28k/s

• redb's single-writer file lock + internal transaction serialization admits zero concurrency: flat at the solo commit cost (~24 ms/put fresh-key; ~8-19 ms re-put). sqlite's WAL at least queues writers behind the one flush lock and gets modest throughput from contention efficiency; redb just serializes full fdatasync latencies.
• Against pg's curve this is the whole argument in one row set: the embedded engines' curves are flat/degrading; the client-server curve is the only increasing one.

Finding C6: the swap is small but the durability API is coarser than sqlite's

kv_redb.rs (~230 lines): same row shape (TableDefinition<&[u8], (&[u8], i64)> — the value/size tuple matches sqlite's row exactly; WITHOUT ROWID-equivalent PK ordering is native B-tree). Differences:

• redb 4.x offers only Durability::None and Durability::Immediate — no middle tier. sqlite's synchronous=NORMAL (consistent-on-crash, not-fsynced-per-commit — the shipped tier this crate benchmarks and documents) has no redb equivalent: None is not crash-consistent (unpersisted commits can leave the file failing to open until repaired — the freelist bookkeeping itself is lost on crash), Immediate pays a full fdatasync per commit. The durability-posture comparison that POC #5 could tune continuously (synchronous_commit=off on pg) doesn't exist here.
• redb's Database is process-lifetime; concurrency modes (ExclusiveWriter default, SingleWriter, MultiWriter) only matter multi-process — in-process it's one writer transaction at a time, structurally.
• create-then-repair behavior on a None-durability crash and the has+insert CAS dance inside one write tx are correct but slower (the bench's re-put path costs a table probe per commit).

Consequences for the phase-0/1 register

• redb is ruled out as the shipped kv engine at a measured tier mismatch, not a benchmark quibble: the only durability postures it offers are "lose acknowledged data on crash" or "pay one fdatasync per commit" — the crate's shipped durability tier (synchronous=NORMAL WAL, crash-consistent, no per-commit fsync) is not expressible. sqlite keeps ADR-003's pin on both economics (44k vs 100 puts/s at 1 KiB) and posture.
• The iroh-blobs DESIGN.md lineage ("faster than the filesystem for small items") is now fully triangulated: its kv-tier claim is true for its inline small items with batched/durable-None commits (iroh batches writes through a serialized actor with batch windows — BatchOptions.max_write_batch, the same amortization pg gets from WAL group-commit for free across connections). Our harnesses measure per-commit put discipline; iroh's actor batching is the reconciliation, and it is also why our POC #3 sqlite numbers (44k) are the honest ceiling for a per-op CAS store.
• Batched commits are a real Phase 1 lever, proven by three independent engines: pg's group-commit (B3), iroh's write actor windows, redb's None+flush emulation (arm C4's structure). The store's put seam could take a WriteBatch-shaped commit later — an API question for Phase 1 (pinned batch puts exist in ADR-005's surface already; the durability-window question is adjacent).
• Reads: no case for redb on merit — page-cache reads are what any of these give the large tier for free; sqlite's 19 µs probe was already fine.
• Anti-recommendation adopted (ADR-007, 2026-10-02): redb stays ruled out — recorded in the decision so the question does not reopen implicitly. SQLite's engine pin now has four-way triangulated evidence (POC #3 fs, POC #5 pg, POC #6 redb, plus the lineage citation), and ADR-007 ships the kv tier as sqlite + postgres. If a future consumer wanted redb-shaped durability trade-offs (ephemeral cache tier), that's a different tier than the kv engine slot.

POC quality notes

• The inherited 14-test suite passes unchanged (the redb backend is additive; sqlite/fs arms untouched). The CAS semantics of put-if-absent are exercised through the smoke path; redb's insert-returns-prior-guard shape documented in kv_redb.rs docs.
• clippy -D warnings clean, fmt clean.
• Honest caveats: mdraid single dev box whose fdatasync latency ranged 0.24-30 ms during the session under co-resident load (pg container churn from POC #5 ran through the first bench; stopped before the final redbdiag re-run — relative conclusions unaffected, absolute redb-Immediate numbers are device-pressure-dependent and the honest statement is "1 fdatasync per commit, cost = device fdatasync latency, observed 8-30 ms here"); the 2s full bench was dropped from the record (its 128/256 KiB redb arms hit shell timeouts; 1s arms with explicit size lists used instead — the harness gained a size-list argument for this); the 1g redb working-set files (~270 MB at 128 KiB arm) cleaned from /tmp after diagnosis.
• Instruments: src/kv_redb.rs (the backend: Immediate + None + flush emulation methods), src/bin/bench_small.rs (inherited + redb arm + size-list arg), examples/redbdiag.rs (durability decomposition), redbdiag2.rs (None + periodic-flush group-commit emulation), redbdiag3.rs (commit-vs-insert split, per-put file growth), redbdiag4.rs (warm distribution), redb_conc.rs (scale-out, both durability modes), sqlite_conc.rs (copied head-to-head from #5).