docs(architecture): ADR-007 — two kv engines (sqlite + postgres) behind one trait

Dissolve OQ-10's circular "standing offer" framing and record the
engine decision the POC evidence already supported:

- ADR-007 (new): kv tier ships sqlite (default) + postgres (feature
  `postgres`, default-off), constructor-selected per node; records the
  deployment economics (classical kv+fs+relational downstream stack
  collapses to one relational engine + fs), the pg impl contract
  (POC #5 B5 posture deltas), the CI sweep-safety gate under
  --all-features, redb ruled out (POC #6), and posture defaults as
  deferred cost in the ADR-006 pattern
- ADR-003: amended status (engine pin widened by ADR-007); kv entry
  notes the widened engine set
- ADR-004: reconciliation note — the "two backends" count is tiers,
  not engines
- backends-and-dispatch.md: tier≠engine distinction, pg engine
  section, co-tenancy note (same-file consumer writes share sqlite's
  single-writer ceiling), substitution-door precedent
- open-questions.md: OQ-10 resolved-by-ADR-007, with the
  why-this-is-not-Schrödinger's-code record (a deployment running
  alkblobs cannot pre-exist the pg engine — OQ-09 precedent); parked
  index narrowed to OQ-07/OQ-08
- overview.md: layer map + dependency posture reflect the `postgres`
  feature; README.md: ADR-007 in tables + corollary applications
- phase-0.md / pg+redb findings: promotion + supersession notes

Trigger framing survives only as deployment guidance (which engine a
node chooses — topology, not throughput), not as a gate on the crate's
own work.

Verified: cargo test, clippy --all-targets -D warnings, fmt --check.
Docs-only change (Phase 1 architecture); no implementation yet.
This commit is contained in:
glm-5.3-flash committed 2026-10-02 19:29:42 +00:00
1 parent 1af9399900
commit 395d58cf3f
10 files changed
+369 -100

No files matched your search

@@ -2,7 +2,9 @@
## Status
Accepted
Accepted (engine pin amended by ADR-007 — the kv tier ships two
engines, sqlite and postgres, behind this contract; the dispatch layer
and trait are unchanged)
## Context
@@ -53,7 +55,8 @@ Evidence in hand (no further POC or consumer-waiting needed):
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
the regime holding most git objects, workspace files, and manifests.
WAL + `synchronous=NORMAL` as shipped durability; prepared
statements; no-tables-yet tolerance.
statements; no-tables-yet tolerance. (The tier's engine set was
widened to include postgres by ADR-007.)
- **`fs` (default-on)** for large blobs — flat
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
conclusion); stage-then-commit-rename; pread range reads (sound
@@ -36,6 +36,15 @@ and by contract.** (A non-production ephemeral `mem` tier exists for
tests — ADR-003; it is not a storage story and does not extend this
boundary.)
**Scope note (post-ADR-007 reconciliation): the "two" here counts
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
engines behind it to a set (sqlite default + postgres, feature-gated)
so a node's kv rides its relational engine of choice. The count that
matters to this ADR — trait impls *this crate ships* as CAS tiers —
remains two (kv + fs); a second kv engine does not make a third
backend, and nothing in the engine set touches the manifest-layer
boundary below.
1. **Two production backends is the complete set.** kv/sqlite (small)
+ fs (large) cover the scale economics with measured evidence
(ADR-003); no other physical tier is needed by any current or
@@ -0,0 +1,177 @@
# ADR-007: Two kv engines — sqlite and postgres — one trait, constructor-selected
## Status
Accepted
## Context
ADR-003/004 settled the crate's physical shape: two backends (kv for
small blobs, fs for large), size-threshold dispatch between them, and
the "two backends" count as a *tier* count. Within the kv tier, sqlite
was pinned as the shipped engine, on single-connection economics (POC
#3 A4). That pin left a real downstream shape unserved: the classical
stack for the consumers this crate exists for — git servers, vfs/
appfile nodes, replicators — is *three* storage systems (a kv for small
content, fs for large content, a relational database for refs,
manifests, ACLs, queues). Forcing such a node to run sqlite *beside*
its relational engine multiplies engines, WAL files, backup stories,
and ops postures for no benefit.
POC #5 (`poc-postgres-kv-findings.md`, findings B1-B6) measured
postgres holding the kv tier's full contract and POC #6
(`poc-redb-kv-findings.md`, findings C1-C6) ruled out redb: its
durability API cannot express the shipped tier (crash-consistent
without per-commit fsync). Together with POC #3 A4 this is four-way
triangulated engine evidence, complete.
The deciding facts are all in hand and none is a fact only a future
deployment of this crate could create (the deferral rule — see the
Schrödinger's-code corollary in README.md, and the OQ-09 precedent):
- **Topology, not throughput, is the trigger** — but topology is a
property of the node the operator builds *today*, not a future
event: a private single-machine node has one writer and sqlite wins
by measurement (44k solo puts/s at 1-16 KiB, POC #3 A4, reproduced
in POC #5); a multi-machine node writing one pool needs postgres
structurally, not merely faster (sqlite's WAL single-writer ceiling
~1.2k objects/s degrades under contention and flatlines — latency
climbs without bound — while postgres scales near-linearly to ~37k
puts/s across workers, POC #5 B3/B4).
- **The engine cost delta is measured and bounded**: ~180-line impl on
the same table shape (B5), with named posture deltas (see Decision).
- **The sweep-safety gate is testable in CI**, not in production: list
correctness is observable only through GC (POC #1 finding 2), and
the exact-count sweep-outcome test shape runs against any engine the
contract covers — including dockerized postgres under `--all-features`.
Recording this as an open question gated on "a replicator-shaped
deployment materializes" would be circular hedging: such a deployment
runs alkblobs, so it cannot cross the trigger until the pg engine
ships, and the engine would not ship until the deployment exists.
That deferral shape was already rejected once, on the same reasoning,
in OQ-09.
## Decision
**The kv tier ships two engines — sqlite (default) and postgres —
behind one `Backend` trait; engine selection is a constructor
parameter on the backend's configuration.** A node provisions one
relational engine plus the fs tier; the kv tier rides whichever
engine the operator chose. This collapses the classical three-engine
downstream stack (kv + fs + relational) to two, and makes which
relational engine a per-node choice rather than a crate fork.
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy
rides the same file (honker's WAL-NORMAL defaults are literally the
shipped tier). Single-machine nodes are sqlite by measurement, and
remain so even at sustained rates well above observed push traffic
(~1.2k/s WAL ceiling vs. single-node push rates an order below it).
2. **postgres ships as an engine, feature-gated (`postgres`,
default-off)**: `tokio-postgres` + `deadpool-postgres`, the POC #5
arm's stack. It exists for the topology cases measured in POC #5
B3/B4: cross-machine write access to one pool, many-client
replicator/hub nodes, and consolidated-ops deployments already
running postgres. A postgres node's consumer tables (refs,
manifests, queues) co-tenant the same instance by choice — the
deployment's own consolidation, beside the pool per ADR-004.
3. **The `Backend` trait does not change.** Both engines implement the
identical contract: opaque byte keys/values, CAS semantics
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
virgin-store no-op reads, namespace-blindness. The dispatch layer
(size threshold, kv↔fs routing, fall-through) is engine-blind and
unchanged.
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
as requirements** (they are the difference between a port and an
engine posture):
- `synchronous_commit` posture per pool/session, configurable —
the parity knob with sqlite's `synchronous=NORMAL`-WAL shipped
tier (latency moves ~2-3× across settings on the POC disk);
- pooled connections with per-connection prepared-statement
discipline (fresh sessions cost ~20 ms, B2; prepared statements
die with sessions — deadpool's statement cache encodes the
pattern);
- autovacuum tuning stated for the churning CAS table, plus
`fillfactor` (90 measured) — a real maintenance surface sqlite
does not have;
- ~40× storage overhead at the 1-4 KiB tier accepted as the
small-tier cost of a page-based server engine (B6) — documented,
not hidden.
5. **Sweep-safety proof per ADR-003's admission gate**: for pg, `list()`
is a stable `SELECT key` cursor over a vacuuming table (MVCC-ghost
stability across a long sweep), batch-delete via `key = ANY($1)`;
the exact-count sweep-outcome tests (store-api.md invariants 4/5)
run in CI against dockerized postgres under `--all-features`, same
gate shape as every backend.
6. **Non-production posture defaults are named, deferred costs** (the
ADR-006 pattern, not hedges): the B5 tuning numbers are POC-derived
from one dev-box disk; the first real pg deployment verifies them
against its media and records deltas. No decision here waits on
that verification.
7. **redb stays ruled out** (POC #6) — recorded here so the question
does not reopen implicitly: its durability API cannot express the
shipped crash-consistent-without-per-commit-fsync tier, and its
~40/s write scale-out is below even sqlite's WAL ceiling.
The ADR-003 substitution door ("a downstream wanting a different
engine there would be new ADRs") is now exercised in-repo, ahead of
any downstream needing it: the downstream's third-engine question
collapses to its own deployment choice among shipped engines.
## Consequences
**Positive**
- The classical downstream stack lands at two engines per node (one
relational + fs), eliminating the "sqlite *beside* the real
database" duplication this crate's consumers would otherwise carry
forever.
- Per-node engine choice (constructor parameter, per ADR-003/004)
makes a heterogeneous fleet — personal nodes on sqlite, multi-tenant
hubs on postgres — a deployment topology, not a fork; everything
above the trait is written once and is engine-blind.
- The evidence for both engines is first-party measured (POC #3 A4,
POC #5 B1-B6, POC #6 C1-C6) and complete; this ADR stands on
evidence in hand, no deferred decision remains in the engine story.
- CI covers the pg backend's contract via the same sweep-outcome test
gate as sqlite, behind `--all-features`.
**Negative**
- The pg engine adds a dependency cluster (tokio-postgres,
deadpool-postgres) behind a default-off feature — the base crate
stays lean per convention 8, but the crate now carries a second
engine impl and its posture surface to maintain.
- Postgres storage overhead at the small tier (~40×) is real (B6) and
accepted; deployments dominated by tiny blobs on constrained disks
stay better served by sqlite.
- Two engines means two durability/ops postures to keep honest across
upstream changes (sqlite version/file-compat upstream guarantees;
pg autovacuum behavior is config-dependent).
**Neutral**
- Implementation sequencing: sqlite is Phase 1's mainline; the pg impl
is parallelizable against the fixed contract (POC #5 scaffold
exists). No consumer wait gates it; it lands as normal Phase 1
implementation work.
- A third engine (S3-like, etc.) remains the ADR-003 path: a new ADR
with its own sweep-safety proof. Nothing in this decision closes or
opens that door further.
## References
- `docs/research/poc-postgres-kv-findings.md` (POC #5, findings B1-B6)
- `docs/research/poc-redb-kv-findings.md` (POC #6, findings C1-C6)
- `docs/research/poc-largeblob-findings.md` (POC #3, finding A4)
- `docs/research/poc-trait-dispatch-findings.md` (POC #1, finding 2)
- ADR-003 (the trait contract and dispatch this decision rides),
ADR-004 (tier count ≠ engine count; the beside-the-pool boundary),
ADR-006 (the deferred-cost pattern this decision's posture defaults
follow)
- [open-questions.md](../open-questions.md) — OQ-10, resolved by this
ADR; OQ-09 (the anti-circularity precedent)
- [backends-and-dispatch.md](../backends-and-dispatch.md) — the spec
surface this ADR amends