docs(architecture): ADR-007 — two kv engines (sqlite + postgres) behind one trait
Dissolve OQ-10's circular "standing offer" framing and record the engine decision the POC evidence already supported: - ADR-007 (new): kv tier ships sqlite (default) + postgres (feature `postgres`, default-off), constructor-selected per node; records the deployment economics (classical kv+fs+relational downstream stack collapses to one relational engine + fs), the pg impl contract (POC #5 B5 posture deltas), the CI sweep-safety gate under --all-features, redb ruled out (POC #6), and posture defaults as deferred cost in the ADR-006 pattern - ADR-003: amended status (engine pin widened by ADR-007); kv entry notes the widened engine set - ADR-004: reconciliation note — the "two backends" count is tiers, not engines - backends-and-dispatch.md: tier≠engine distinction, pg engine section, co-tenancy note (same-file consumer writes share sqlite's single-writer ceiling), substitution-door precedent - open-questions.md: OQ-10 resolved-by-ADR-007, with the why-this-is-not-Schrödinger's-code record (a deployment running alkblobs cannot pre-exist the pg engine — OQ-09 precedent); parked index narrowed to OQ-07/OQ-08 - overview.md: layer map + dependency posture reflect the `postgres` feature; README.md: ADR-007 in tables + corollary applications - phase-0.md / pg+redb findings: promotion + supersession notes Trigger framing survives only as deployment guidance (which engine a node chooses — topology, not throughput), not as a gate on the crate's own work. Verified: cargo test, clippy --all-targets -D warnings, fmt --check. Docs-only change (Phase 1 architecture); no implementation yet.
This commit is contained in:
1 parent
1af9399900
commit
395d58cf3f
10 files changed
+369
-100
No files matched your search
@@ -2,7 +2,9 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
||||
engines, sqlite and postgres, behind this contract; the dispatch layer
|
||||
and trait are unchanged)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -53,7 +55,8 @@ Evidence in hand (no further POC or consumer-waiting needed):
|
||||
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
|
||||
the regime holding most git objects, workspace files, and manifests.
|
||||
WAL + `synchronous=NORMAL` as shipped durability; prepared
|
||||
statements; no-tables-yet tolerance.
|
||||
statements; no-tables-yet tolerance. (The tier's engine set was
|
||||
widened to include postgres by ADR-007.)
|
||||
- **`fs` (default-on)** for large blobs — flat
|
||||
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
|
||||
conclusion); stage-then-commit-rename; pread range reads (sound
|
||||
|
||||
@@ -36,6 +36,15 @@ and by contract.** (A non-production ephemeral `mem` tier exists for
|
||||
tests — ADR-003; it is not a storage story and does not extend this
|
||||
boundary.)
|
||||
|
||||
**Scope note (post-ADR-007 reconciliation): the "two" here counts
|
||||
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
|
||||
engines behind it to a set (sqlite default + postgres, feature-gated)
|
||||
so a node's kv rides its relational engine of choice. The count that
|
||||
matters to this ADR — trait impls *this crate ships* as CAS tiers —
|
||||
remains two (kv + fs); a second kv engine does not make a third
|
||||
backend, and nothing in the engine set touches the manifest-layer
|
||||
boundary below.
|
||||
|
||||
1. **Two production backends is the complete set.** kv/sqlite (small)
|
||||
+ fs (large) cover the scale economics with measured evidence
|
||||
(ADR-003); no other physical tier is needed by any current or
|
||||
|
||||
@@ -0,0 +1,177 @@
|
||||
# ADR-007: Two kv engines — sqlite and postgres — one trait, constructor-selected
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
ADR-003/004 settled the crate's physical shape: two backends (kv for
|
||||
small blobs, fs for large), size-threshold dispatch between them, and
|
||||
the "two backends" count as a *tier* count. Within the kv tier, sqlite
|
||||
was pinned as the shipped engine, on single-connection economics (POC
|
||||
#3 A4). That pin left a real downstream shape unserved: the classical
|
||||
stack for the consumers this crate exists for — git servers, vfs/
|
||||
appfile nodes, replicators — is *three* storage systems (a kv for small
|
||||
content, fs for large content, a relational database for refs,
|
||||
manifests, ACLs, queues). Forcing such a node to run sqlite *beside*
|
||||
its relational engine multiplies engines, WAL files, backup stories,
|
||||
and ops postures for no benefit.
|
||||
|
||||
POC #5 (`poc-postgres-kv-findings.md`, findings B1-B6) measured
|
||||
postgres holding the kv tier's full contract and POC #6
|
||||
(`poc-redb-kv-findings.md`, findings C1-C6) ruled out redb: its
|
||||
durability API cannot express the shipped tier (crash-consistent
|
||||
without per-commit fsync). Together with POC #3 A4 this is four-way
|
||||
triangulated engine evidence, complete.
|
||||
|
||||
The deciding facts are all in hand and none is a fact only a future
|
||||
deployment of this crate could create (the deferral rule — see the
|
||||
Schrödinger's-code corollary in README.md, and the OQ-09 precedent):
|
||||
|
||||
- **Topology, not throughput, is the trigger** — but topology is a
|
||||
property of the node the operator builds *today*, not a future
|
||||
event: a private single-machine node has one writer and sqlite wins
|
||||
by measurement (44k solo puts/s at 1-16 KiB, POC #3 A4, reproduced
|
||||
in POC #5); a multi-machine node writing one pool needs postgres
|
||||
structurally, not merely faster (sqlite's WAL single-writer ceiling
|
||||
~1.2k objects/s degrades under contention and flatlines — latency
|
||||
climbs without bound — while postgres scales near-linearly to ~37k
|
||||
puts/s across workers, POC #5 B3/B4).
|
||||
- **The engine cost delta is measured and bounded**: ~180-line impl on
|
||||
the same table shape (B5), with named posture deltas (see Decision).
|
||||
- **The sweep-safety gate is testable in CI**, not in production: list
|
||||
correctness is observable only through GC (POC #1 finding 2), and
|
||||
the exact-count sweep-outcome test shape runs against any engine the
|
||||
contract covers — including dockerized postgres under `--all-features`.
|
||||
|
||||
Recording this as an open question gated on "a replicator-shaped
|
||||
deployment materializes" would be circular hedging: such a deployment
|
||||
runs alkblobs, so it cannot cross the trigger until the pg engine
|
||||
ships, and the engine would not ship until the deployment exists.
|
||||
That deferral shape was already rejected once, on the same reasoning,
|
||||
in OQ-09.
|
||||
|
||||
## Decision
|
||||
|
||||
**The kv tier ships two engines — sqlite (default) and postgres —
|
||||
behind one `Backend` trait; engine selection is a constructor
|
||||
parameter on the backend's configuration.** A node provisions one
|
||||
relational engine plus the fs tier; the kv tier rides whichever
|
||||
engine the operator chose. This collapses the classical three-engine
|
||||
downstream stack (kv + fs + relational) to two, and makes which
|
||||
relational engine a per-node choice rather than a crate fork.
|
||||
|
||||
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
||||
economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy
|
||||
rides the same file (honker's WAL-NORMAL defaults are literally the
|
||||
shipped tier). Single-machine nodes are sqlite by measurement, and
|
||||
remain so even at sustained rates well above observed push traffic
|
||||
(~1.2k/s WAL ceiling vs. single-node push rates an order below it).
|
||||
2. **postgres ships as an engine, feature-gated (`postgres`,
|
||||
default-off)**: `tokio-postgres` + `deadpool-postgres`, the POC #5
|
||||
arm's stack. It exists for the topology cases measured in POC #5
|
||||
B3/B4: cross-machine write access to one pool, many-client
|
||||
replicator/hub nodes, and consolidated-ops deployments already
|
||||
running postgres. A postgres node's consumer tables (refs,
|
||||
manifests, queues) co-tenant the same instance by choice — the
|
||||
deployment's own consolidation, beside the pool per ADR-004.
|
||||
3. **The `Backend` trait does not change.** Both engines implement the
|
||||
identical contract: opaque byte keys/values, CAS semantics
|
||||
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
|
||||
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
|
||||
virgin-store no-op reads, namespace-blindness. The dispatch layer
|
||||
(size threshold, kv↔fs routing, fall-through) is engine-blind and
|
||||
unchanged.
|
||||
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
|
||||
as requirements** (they are the difference between a port and an
|
||||
engine posture):
|
||||
- `synchronous_commit` posture per pool/session, configurable —
|
||||
the parity knob with sqlite's `synchronous=NORMAL`-WAL shipped
|
||||
tier (latency moves ~2-3× across settings on the POC disk);
|
||||
- pooled connections with per-connection prepared-statement
|
||||
discipline (fresh sessions cost ~20 ms, B2; prepared statements
|
||||
die with sessions — deadpool's statement cache encodes the
|
||||
pattern);
|
||||
- autovacuum tuning stated for the churning CAS table, plus
|
||||
`fillfactor` (90 measured) — a real maintenance surface sqlite
|
||||
does not have;
|
||||
- ~40× storage overhead at the 1-4 KiB tier accepted as the
|
||||
small-tier cost of a page-based server engine (B6) — documented,
|
||||
not hidden.
|
||||
5. **Sweep-safety proof per ADR-003's admission gate**: for pg, `list()`
|
||||
is a stable `SELECT key` cursor over a vacuuming table (MVCC-ghost
|
||||
stability across a long sweep), batch-delete via `key = ANY($1)`;
|
||||
the exact-count sweep-outcome tests (store-api.md invariants 4/5)
|
||||
run in CI against dockerized postgres under `--all-features`, same
|
||||
gate shape as every backend.
|
||||
6. **Non-production posture defaults are named, deferred costs** (the
|
||||
ADR-006 pattern, not hedges): the B5 tuning numbers are POC-derived
|
||||
from one dev-box disk; the first real pg deployment verifies them
|
||||
against its media and records deltas. No decision here waits on
|
||||
that verification.
|
||||
7. **redb stays ruled out** (POC #6) — recorded here so the question
|
||||
does not reopen implicitly: its durability API cannot express the
|
||||
shipped crash-consistent-without-per-commit-fsync tier, and its
|
||||
~40/s write scale-out is below even sqlite's WAL ceiling.
|
||||
|
||||
The ADR-003 substitution door ("a downstream wanting a different
|
||||
engine there would be new ADRs") is now exercised in-repo, ahead of
|
||||
any downstream needing it: the downstream's third-engine question
|
||||
collapses to its own deployment choice among shipped engines.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The classical downstream stack lands at two engines per node (one
|
||||
relational + fs), eliminating the "sqlite *beside* the real
|
||||
database" duplication this crate's consumers would otherwise carry
|
||||
forever.
|
||||
- Per-node engine choice (constructor parameter, per ADR-003/004)
|
||||
makes a heterogeneous fleet — personal nodes on sqlite, multi-tenant
|
||||
hubs on postgres — a deployment topology, not a fork; everything
|
||||
above the trait is written once and is engine-blind.
|
||||
- The evidence for both engines is first-party measured (POC #3 A4,
|
||||
POC #5 B1-B6, POC #6 C1-C6) and complete; this ADR stands on
|
||||
evidence in hand, no deferred decision remains in the engine story.
|
||||
- CI covers the pg backend's contract via the same sweep-outcome test
|
||||
gate as sqlite, behind `--all-features`.
|
||||
|
||||
**Negative**
|
||||
|
||||
- The pg engine adds a dependency cluster (tokio-postgres,
|
||||
deadpool-postgres) behind a default-off feature — the base crate
|
||||
stays lean per convention 8, but the crate now carries a second
|
||||
engine impl and its posture surface to maintain.
|
||||
- Postgres storage overhead at the small tier (~40×) is real (B6) and
|
||||
accepted; deployments dominated by tiny blobs on constrained disks
|
||||
stay better served by sqlite.
|
||||
- Two engines means two durability/ops postures to keep honest across
|
||||
upstream changes (sqlite version/file-compat upstream guarantees;
|
||||
pg autovacuum behavior is config-dependent).
|
||||
|
||||
**Neutral**
|
||||
|
||||
- Implementation sequencing: sqlite is Phase 1's mainline; the pg impl
|
||||
is parallelizable against the fixed contract (POC #5 scaffold
|
||||
exists). No consumer wait gates it; it lands as normal Phase 1
|
||||
implementation work.
|
||||
- A third engine (S3-like, etc.) remains the ADR-003 path: a new ADR
|
||||
with its own sweep-safety proof. Nothing in this decision closes or
|
||||
opens that door further.
|
||||
|
||||
## References
|
||||
|
||||
- `docs/research/poc-postgres-kv-findings.md` (POC #5, findings B1-B6)
|
||||
- `docs/research/poc-redb-kv-findings.md` (POC #6, findings C1-C6)
|
||||
- `docs/research/poc-largeblob-findings.md` (POC #3, finding A4)
|
||||
- `docs/research/poc-trait-dispatch-findings.md` (POC #1, finding 2)
|
||||
- ADR-003 (the trait contract and dispatch this decision rides),
|
||||
ADR-004 (tier count ≠ engine count; the beside-the-pool boundary),
|
||||
ADR-006 (the deferred-cost pattern this decision's posture defaults
|
||||
follow)
|
||||
- [open-questions.md](../open-questions.md) — OQ-10, resolved by this
|
||||
ADR; OQ-09 (the anti-circularity precedent)
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) — the spec
|
||||
surface this ADR amends
|
||||
Reference in new issue
Block a user