alkblobs/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md

# ADR-007: Two kv engines — sqlite and postgres — one trait, constructor-selected

## Status

Accepted

## Context

ADR-003/004 settled the crate's physical shape: two backends (kv for
small blobs, fs for large), size-threshold dispatch between them, and
the "two backends" count as a *tier* count. Within the kv tier, sqlite
was pinned as the shipped engine, on single-connection economics (POC
#3 A4). That pin left a real downstream shape unserved: the classical
stack for the consumers this crate exists for — git servers, vfs/
appfile nodes, replicators — is *three* storage systems (a kv for small
content, fs for large content, a relational database for refs,
manifests, ACLs, queues). Forcing such a node to run sqlite *beside*
its relational engine multiplies engines, WAL files, backup stories,
and ops postures for no benefit.

POC #5 (`poc-postgres-kv-findings.md`, findings B1-B6) measured
postgres holding the kv tier's full contract and POC #6
(`poc-redb-kv-findings.md`, findings C1-C6) ruled out redb: its
durability API cannot express the shipped tier (crash-consistent
without per-commit fsync). Together with POC #3 A4 this is four-way
triangulated engine evidence, complete.

The deciding facts are all in hand and none is a fact only a future
deployment of this crate could create (the deferral rule — see the
Schrödinger's-code corollary in README.md, and the OQ-09 precedent):

- **Topology, not throughput, is the trigger** — but topology is a
  property of the node the operator builds *today*, not a future
  event: a private single-machine node has one writer and sqlite wins
  by measurement (44k solo puts/s at 1-16 KiB, POC #3 A4, reproduced
  in POC #5); a multi-machine node writing one pool needs postgres
  structurally, not merely faster (sqlite's WAL single-writer ceiling
  ~1.2k objects/s degrades under contention and flatlines — latency
  climbs without bound — while postgres scales near-linearly to ~37k
  puts/s across workers, POC #5 B3/B4).
- **The engine cost delta is measured and bounded**: ~180-line impl on
  the same table shape (B5), with named posture deltas (see Decision).
- **The sweep-safety gate is testable in CI**, not in production: list
  correctness is observable only through GC (POC #1 finding 2), and
  the exact-count sweep-outcome test shape runs against any engine the
  contract covers — including dockerized postgres under `--all-features`.

Recording this as an open question gated on "a replicator-shaped
deployment materializes" would be circular hedging: such a deployment
runs alkblobs, so it cannot cross the trigger until the pg engine
ships, and the engine would not ship until the deployment exists.
That deferral shape was already rejected once, on the same reasoning,
in OQ-09.

## Decision

**The kv tier ships two engines — sqlite (default) and postgres —
behind one `Backend` trait; engine selection is a constructor
parameter on the backend's configuration.** A node provisions one
relational engine plus the fs tier; the kv tier rides whichever
engine the operator chose. This collapses the classical three-engine
downstream stack (kv + fs + relational) to two, and makes which
relational engine a per-node choice rather than a crate fork.

1. **sqlite stays the default engine**: zero-ops, no daemon, solo
   economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy
   rides the same file (honker — the alk* family's SQLite
   notify/queue extension, `/workspace/honker` — ships WAL + the
   NORMAL-synchronous posture, so its defaults are literally the
   shipped tier). Single-machine nodes are sqlite by measurement, and
   remain so even at sustained rates well above observed push traffic
   (~1.2k/s WAL ceiling vs. single-node push rates an order below it).
2. **postgres ships as an engine, feature-gated (`postgres`,
   default-off)**: `tokio-postgres` + `deadpool-postgres`, the POC #5
   arm's stack. It exists for the topology cases measured in POC #5
   B3/B4: cross-machine write access to one pool, many-client
   replicator/hub nodes, and consolidated-ops deployments already
   running postgres. A postgres node's consumer tables (refs,
   manifests, queues) co-tenant the same instance by choice — the
   deployment's own consolidation, beside the pool per ADR-004.
3. **The `Backend` trait does not change.** Both engines implement the
   identical contract: opaque byte keys/values, CAS semantics
   (`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
   `INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
   virgin-store no-op reads, namespace-blindness. The dispatch layer
   (size threshold, kv↔fs routing, fall-through) is engine-blind and
   unchanged.
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
   as requirements** (they are the difference between a port and an
   engine posture):
   - `synchronous_commit` posture per pool/session, configurable —
     the parity knob with sqlite's `synchronous=NORMAL`-WAL shipped
     tier (latency moves ~2-3× across settings on the POC disk);
   - pooled connections with per-connection prepared-statement
     discipline (fresh sessions cost ~20 ms, B2; prepared statements
     die with sessions — deadpool's statement cache encodes the
     pattern);
   - autovacuum tuning stated for the churning CAS table, plus
     `fillfactor` (90 measured) — a real maintenance surface sqlite
     does not have;
   - ~40× storage overhead at the 1-4 KiB tier accepted as the
     small-tier cost of a page-based server engine (B6) — documented,
     not hidden.
5. **Sweep-safety proof per ADR-003's admission gate**: for pg, `list()`
   is a stable `SELECT key` cursor over a vacuuming table (MVCC-ghost
   stability across a long sweep), batch-delete via `key = ANY($1)`;
   the exact-count sweep-outcome tests (store-api.md invariants 4/5)
   run in CI against dockerized postgres under `--all-features`, same
   gate shape as every backend.
6. **Non-production posture defaults are named, deferred costs** (the
   ADR-006 pattern, not hedges): the B5 tuning numbers are POC-derived
   from one dev-box disk; the first real pg deployment verifies them
   against its media and records deltas. No decision here waits on
   that verification.
7. **redb stays ruled out** (POC #6) — recorded here so the question
   does not reopen implicitly: its durability API cannot express the
   shipped crash-consistent-without-per-commit-fsync tier, and its
   ~40/s write scale-out is below even sqlite's WAL ceiling.

The ADR-003 substitution door ("a downstream wanting a different
engine there would be new ADRs") is now exercised in-repo, ahead of
any downstream needing it: the downstream's third-engine question
collapses to its own deployment choice among shipped engines.

## Consequences

**Positive**

- The classical downstream stack lands at two engines per node (one
  relational + fs), eliminating the "sqlite *beside* the real
  database" duplication this crate's consumers would otherwise carry
  forever.
- Per-node engine choice (constructor parameter, per ADR-003/004)
  makes a heterogeneous fleet — personal nodes on sqlite, multi-tenant
  hubs on postgres — a deployment topology, not a fork; everything
  above the trait is written once and is engine-blind.
- The evidence for both engines is first-party measured (POC #3 A4,
  POC #5 B1-B6, POC #6 C1-C6) and complete; this ADR stands on
  evidence in hand, no deferred decision remains in the engine story.
- CI covers the pg backend's contract via the same sweep-outcome test
  gate as sqlite, behind `--all-features`.

**Negative**

- The pg engine adds a dependency cluster (tokio-postgres,
  deadpool-postgres) behind a default-off feature — the base crate
  stays lean per convention 8, but the crate now carries a second
  engine impl and its posture surface to maintain.
- Postgres storage overhead at the small tier (~40×) is real (B6) and
  accepted; deployments dominated by tiny blobs on constrained disks
  stay better served by sqlite.
- Two engines means two durability/ops postures to keep honest across
  upstream changes (sqlite version/file-compat upstream guarantees;
  pg autovacuum behavior is config-dependent).

**Neutral**

- Implementation sequencing: sqlite is Phase 1's mainline; the pg impl
  is parallelizable against the fixed contract (POC #5 scaffold
  exists). No consumer wait gates it; it lands as normal Phase 1
  implementation work.
- A third engine (S3-like, etc.) remains the ADR-003 path: a new ADR
  with its own sweep-safety proof. Nothing in this decision closes or
  opens that door further.

## References

- `docs/research/poc-postgres-kv-findings.md` (POC #5, findings B1-B6)
- `docs/research/poc-redb-kv-findings.md` (POC #6, findings C1-C6)
- `docs/research/poc-largeblob-findings.md` (POC #3, finding A4)
- `docs/research/poc-trait-dispatch-findings.md` (POC #1, finding 2)
- ADR-003 (the trait contract and dispatch this decision rides),
  ADR-004 (tier count ≠ engine count; the beside-the-pool boundary),
  ADR-006 (the deferred-cost pattern this decision's posture defaults
  follow)
- [open-questions.md](../open-questions.md) — OQ-10, resolved by this
  ADR; OQ-09 (the anti-circularity precedent)
- [backends-and-dispatch.md](../backends-and-dispatch.md) — the spec
  surface this ADR amends