alkblobs/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md

ADR-007: Two kv engines — sqlite and postgres — one trait, constructor-selected

Status

Accepted (the engine story's I/O seams and GC-state home are ruled by ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as its third engine — is pinned by ADR-011)

Context

ADR-003/004 settled the crate's physical shape: two backends (kv for small blobs, fs for large), size-threshold dispatch between them, and the "two backends" count as a tier count. Within the kv tier, sqlite was pinned as the shipped engine, on single-connection economics (POC #3 A4). That pin left a real downstream shape unserved: the classical stack for the consumers this crate exists for — git servers, vfs/ appfile nodes, replicators — is three storage systems (a kv for small content, fs for large content, a relational database for refs, manifests, ACLs, queues). Forcing such a node to run sqlite beside its relational engine multiplies engines, WAL files, backup stories, and ops postures for no benefit.

POC #5 (poc-postgres-kv-findings.md, findings B1-B6) measured postgres holding the kv tier's full contract and POC #6 (poc-redb-kv-findings.md, findings C1-C6) ruled out redb: its durability API cannot express the shipped tier (crash-consistent without per-commit fsync). Together with POC #3 A4 this is four-way triangulated engine evidence, complete.

The deciding facts are all in hand and none is a fact only a future deployment of this crate could create (the deferral rule — see the Schrödinger's-code corollary in README.md, and the OQ-09 precedent):

• Topology, not throughput, is the trigger — but topology is a property of the node the operator builds today, not a future event: a private single-machine node has one writer and sqlite wins by measurement (44k solo puts/s at 1-16 KiB, POC #3 A4, reproduced in POC #5); a multi-machine node writing one pool needs postgres structurally, not merely faster (sqlite's WAL single-writer ceiling ~1.2k objects/s degrades under contention and flatlines — latency climbs without bound — while postgres scales near-linearly to ~37k puts/s across workers, POC #5 B3/B4).
• The engine cost delta is measured and bounded: ~180-line impl on the same table shape (B5), with named posture deltas (see Decision).
• The sweep-safety gate is testable in CI, not in production: list correctness is observable only through GC (POC #1 finding 2), and the exact-count sweep-outcome test shape runs against any engine the contract covers — including dockerized postgres under --all-features.

Recording this as an open question gated on "a replicator-shaped deployment materializes" would be circular hedging: such a deployment runs alkblobs, so it cannot cross the trigger until the pg engine ships, and the engine would not ship until the deployment exists. That deferral shape was already rejected once, on the same reasoning, in OQ-09.

Decision

The kv tier ships two engines — sqlite (default) and postgres — behind one Backend trait; engine selection is a constructor parameter on the backend's configuration. A node provisions one relational engine plus the large tier; the kv tier rides whichever engine the operator chose. This collapses the classical three-engine downstream stack (kv + large content + relational) to two, and makes which relational engine a per-node choice rather than a crate fork.

1. sqlite stays the default engine: zero-ops, no daemon, solo economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy rides the same file (honker — the alk* family's SQLite notify/queue extension, /workspace/honker — ships WAL + the NORMAL-synchronous posture, so its defaults are literally the shipped tier). Single-machine nodes are sqlite by measurement, and remain so even at sustained rates well above observed push traffic (~1.2k/s WAL ceiling vs. single-node push rates an order below it).
2. postgres ships as an engine, feature-gated (postgres, default-off): tokio-postgres + deadpool-postgres, the POC #5 arm's stack. It exists for the topology cases measured in POC #5 B3/B4: cross-machine write access to one pool, many-client replicator/hub nodes, and consolidated-ops deployments already running postgres. A postgres node's consumer tables (refs, manifests, queues) co-tenant the same instance by choice — the deployment's own consolidation, beside the pool per ADR-004.
3. The Backend trait does not change. Both engines implement the identical contract: opaque byte keys/values, CAS semantics (INSERT ... ON CONFLICT DO NOTHING maps sqlite's INSERT OR IGNORE), complete list(), GC-participating delete, virgin-store no-op reads, namespace-blindness. The dispatch layer (size threshold, kv↔large routing, fall-through) is engine-blind and unchanged.
4. The pg engine's impl contract carries POC #5 finding B5's deltas as requirements (they are the difference between a port and an engine posture):
   • synchronous_commit posture per pool/session, configurable — the parity knob with sqlite's synchronous=NORMAL-WAL shipped tier (latency moves ~2-3× across settings on the POC disk);
   • pooled connections with per-connection prepared-statement discipline (fresh sessions cost ~20 ms, B2; prepared statements die with sessions — deadpool's statement cache encodes the pattern);
   • autovacuum tuning stated for the churning CAS table, plus fillfactor (90 measured) — a real maintenance surface sqlite does not have;
   • ~40× storage overhead at the 1-4 KiB tier accepted as the small-tier cost of a page-based server engine (B6) — documented, not hidden.
5. Sweep-safety proof per ADR-003's admission gate: for pg, list() is a stable SELECT key cursor over a vacuuming table (MVCC-ghost stability across a long sweep), batch-delete via key = ANY($1); the exact-count sweep-outcome tests (store-api.md invariants 4/5) run in CI against dockerized postgres under --all-features, same gate shape as every backend.
6. Non-production posture defaults are named, deferred costs (the ADR-006 pattern, not hedges): the B5 tuning numbers are POC-derived from one dev-box disk; the first real pg deployment verifies them against its media and records deltas. No decision here waits on that verification.
7. redb stays ruled out (POC #6) — recorded here so the question does not reopen implicitly: its durability API cannot express the shipped crash-consistent-without-per-commit-fsync tier, and its ~40/s write scale-out is below even sqlite's WAL ceiling.

The ADR-003 substitution door ("a downstream wanting a different engine there would be new ADRs") is now exercised in-repo, ahead of any downstream needing it: the downstream's third-engine question collapses to its own deployment choice among shipped engines.

Consequences

Positive

• The classical downstream stack lands at two engines per node (one relational engine + large tier), eliminating the "sqlite beside the real database" duplication this crate's consumers would otherwise carry forever.
• Per-node engine choice (constructor parameter, per ADR-003/004) makes a heterogeneous fleet — personal nodes on sqlite, multi-tenant hubs on postgres — a deployment topology, not a fork; everything above the trait is written once and is engine-blind.
• The evidence for both engines is first-party measured (POC #3 A4, POC #5 B1-B6, POC #6 C1-C6) and complete; this ADR stands on evidence in hand, no deferred decision remains in the engine story.
• CI covers the pg backend's contract via the same sweep-outcome test gate as sqlite, behind --all-features.

Negative

• The pg engine adds a dependency cluster (tokio-postgres, deadpool-postgres) behind a default-off feature — the base crate stays lean per convention 8, but the crate now carries a second engine impl and its posture surface to maintain.
• Postgres storage overhead at the small tier (~40×) is real (B6) and accepted; deployments dominated by tiny blobs on constrained disks stay better served by sqlite.
• Two engines means two durability/ops postures to keep honest across upstream changes (sqlite version/file-compat upstream guarantees; pg autovacuum behavior is config-dependent).

Neutral

• Implementation sequencing: sqlite is Phase 1's mainline; the pg impl is parallelizable against the fixed contract (POC #5 scaffold exists). No consumer wait gates it; it lands as normal Phase 1 implementation work.
• A third engine (S3-like, etc.) remains the ADR-003 path: a new ADR with its own sweep-safety proof. Nothing in this decision closes or opens that door further.

References

• docs/research/poc-postgres-kv-findings.md (POC #5, findings B1-B6)
• docs/research/poc-redb-kv-findings.md (POC #6, findings C1-C6)
• docs/research/poc-largeblob-findings.md (POC #3, finding A4)
• docs/research/poc-trait-dispatch-findings.md (POC #1, finding 2)
• ADR-003 (the trait contract and dispatch this decision rides), ADR-004 (tier count ≠ engine count; the beside-the-pool boundary), ADR-006 (the deferred-cost pattern this decision's posture defaults follow)
• open-questions.md — OQ-10, resolved by this ADR; OQ-09 (the anti-circularity precedent)
• backends-and-dispatch.md — the spec surface this ADR amends