docs(architecture): OQ-10 — postgres as the second shipped kv engine, a standing offer on a deployment trigger
Records the two-engine story the POC #5/#6 round established, in the terms this crate's boundary uses (engine, not backend; trait contract as the admission gate): - OQ-10 (externally-owned standing offer): open the postgres-kv ADR when a multi-tenant replicator deployment materializes. Trigger is topology, not throughput — cross-machine write access to one pool (throughput pressure / sqlite's flatline is the early warning). Names the trigger-time work: PgKv impl + engine-posture deltas (synchronous_commit, pooling/prepared-statement discipline, autovacuum, ~40x storage overhead) + the sweep-safety proof (vacuum-stable list() cursor, exact-count sweep tests). Sequenced: sqlite stays Phase 1 mainline; evidence base is done and waiting. - poc-postgres-kv-findings.md: engine deployment posture section — the three use cases (self-hosted git serving, p2p replicator nodes, agent-workspace pools) mapped onto the measured curves; per-node choice as the form of the fork; honker parity stack named for the pg side (LISTEN/NOTIFY + pgboss-rs, deployment layer). No code changes; engine work parallelizes against the fixed contract.
This commit is contained in:
1 parent
5eea656295
commit
1af9399900
2 files changed
+87
-2
No files matched your search
@@ -31,8 +31,8 @@ make before shipping. Phase 0's residual list was swept under this
|
||||
rule: every residual either resolved into an ADR (the evidence was
|
||||
already in hand) or re-owned as below.
|
||||
|
||||
**Index of active OQs:** OQ-07, OQ-08 (externally-owned, carried for
|
||||
visibility). OQ-09 is resolved (recorded below). Promoted Phase 0
|
||||
**Index of active OQs:** OQ-07, OQ-08, OQ-10 (externally-owned, carried
|
||||
for visibility). OQ-09 is resolved (recorded below). Promoted Phase 0
|
||||
questions OQ-BL-01..06 are recorded here with their resolutions for
|
||||
traceability.
|
||||
|
||||
@@ -77,6 +77,56 @@ traceability.
|
||||
liveness seams (ADR-003/004/005) without baking an alkfs shape in.
|
||||
- **Cross-references**: OQ-07, ADR-004
|
||||
|
||||
### OQ-10: Postgres as the second shipped kv engine
|
||||
|
||||
- **Origin**: `docs/research/poc-postgres-kv-findings.md`,
|
||||
`poc-redb-kv-findings.md`, [backends-and-dispatch.md](backends-and-dispatch.md)
|
||||
- **Status**: externally-owned (the deployment trigger) — carried here
|
||||
as a standing offer, not a blocker
|
||||
- **Priority**: medium-high (the evidence base is done; only the
|
||||
deployment trigger is outstanding)
|
||||
- **Owner**: whichever consumer deployment crosses the trigger
|
||||
(per OQ-07: alkgit's replicator is the named candidate)
|
||||
- **Question**: when a replicator-shaped deployment materializes
|
||||
(multi-tenant public node: many users pushing, gossip catch-up, bulk
|
||||
ingest), open the postgres-kv ADR. The measured case for it:
|
||||
sqlite's WAL single-writer ceiling is ~1.2k objects/s under
|
||||
contention and *flatlines* (latency climbs without bound under
|
||||
overload) while postgres scales out to ~28-37k puts/s across
|
||||
workers with group-commit amortization (POC #5 findings B3/B4);
|
||||
single-user/private/small-replicator nodes stay sqlite by
|
||||
measurement (POC #5 A4/B1). Also settled by the same POCs: redb is
|
||||
ruled out (POC #6 — its durability API cannot express the shipped
|
||||
WAL-NORMAL-tier: crash-consistent without per-commit fsync), and
|
||||
the trigger is **topology, not throughput** — cross-MACHINE write
|
||||
access to one pool is the condition that makes postgres mandatory
|
||||
rather than merely faster.
|
||||
- **Shape of the work when triggered** (per-node choice is a
|
||||
constructor parameter, per ADR-003/004; the impl is parallelizable
|
||||
against the fixed contract — two engines, one trait):
|
||||
1. `PgKv` backend impl (~180 lines, POC #5 scaffold exists)
|
||||
with the engine-posture deltas POC #5 finding B5 named:
|
||||
per-pool `synchronous_commit` posture, pooled-connection +
|
||||
prepared-statement discipline (prepared statements die with
|
||||
sessions), autovacuum tuning for a churning CAS table,
|
||||
fillfactor, ~40× storage overhead accepted as the small-tier
|
||||
cost;
|
||||
2. the sweep-safety proof ADR-003's admission gate requires:
|
||||
`list()` as a stable `SELECT key` cursor over a vacuuming table
|
||||
(MVCC-ghost stability across a long sweep), batch-delete via
|
||||
`key = ANY($1)`, exact-count sweep-outcome tests (the POC #1
|
||||
finding-2 gate applied to pg);
|
||||
3. consumer-layer note: honker's notify/queue/stream features ride
|
||||
sqlite's file; the pg deployment's parity stack is
|
||||
`LISTEN`/`NOTIFY` + pgboss-rs (honker's own README points
|
||||
postgres users that direction) — a deployment concern, not a
|
||||
store-crate one.
|
||||
- **Resolution**: none — waiting on the deployment trigger. The ADR
|
||||
opens the day a real replicator consumer exists to keep its
|
||||
sweep-safety proof honest; building it earlier would leave an impl
|
||||
with no first consumer.
|
||||
- **Cross-references**: OQ-07, ADR-003, ADR-004, POC #5, POC #6
|
||||
|
||||
---
|
||||
|
||||
## Theme: ops surface
|
||||
@@ -186,6 +236,7 @@ traceability.
|
||||
|---|---|---|
|
||||
| OQ-07 | externally-owned | alkgit's architecture process (answerable on paper anytime; gates nothing here) |
|
||||
| OQ-08 | externally-owned | alkfs Phase 0 intake |
|
||||
| OQ-10 | externally-owned (standing offer) | the first multi-tenant replicator deployment (POC #5 is the ready evidence base; the ADR opens on the trigger) |
|
||||
|
||||
OQ-09 was `deferred(scope)` on the first embedded ops deployment — a
|
||||
deciding fact that could only exist once the ops module ships, i.e. a
|
||||
|
||||
@@ -250,6 +250,40 @@ a deployment running the *fs tier anyway* has none of either cost.
|
||||
list()-complete-by-contract test would need the vacuum-aware
|
||||
equivalent of POC #1's exact-count sweep assertions.
|
||||
|
||||
## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #4/#6)
|
||||
|
||||
The three alkblobs-relevant git/vfs use cases, mapped onto the measured
|
||||
curves; this is the conclusion OQ-10 carries:
|
||||
|
||||
| Engine | Covers | Evidence |
|
||||
|---|---|---|
|
||||
| **sqlite** | self-hosted git serving (gitea-replacement scale), agent-workspace pools, small/personal replicators | 44k solo puts/s at 1-16 KiB (POC #3 A4, reproduced here); WAL single-writer ceiling ~1.2k/s under contention is comfortably above single-node push rates; zero-ops, no daemon; honker rides the same file (same WAL-NORMAL posture, its defaults are literally the shipped tier) |
|
||||
| **postgres** | public multi-tenant replicator nodes (many users pushing, gossip catch-up, bulk ingest), consolidated-ops deployments (already-running pg) | near-linear put scale-out to ~21k at 4 conns, ~28k at 8, ~32k at 12, ~37k at 24-48 (B3); reads scale to ~44k at 24 conns (B4); the only engine whose throughput *increases* under concurrency |
|
||||
| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #4/#6 reproductions |
|
||||
|
||||
The decision rule the two POCs establish: **the trigger is topology,
|
||||
not throughput** — cross-MACHINE write access to one pool makes postgres
|
||||
mandatory; sustained-rate pressure (sqlite's flatline failure mode:
|
||||
latency climbs without bound under overload) is the early-warning that
|
||||
precedes it; single-node/private/small-team cases stay sqlite by
|
||||
measurement, with honker-parity features (notify/queue/stream) available
|
||||
free on the sqlite file and via `LISTEN`/`NOTIFY` + pgboss-rs on a
|
||||
future pg deployment (a consumer-layer concern either way — the store
|
||||
crate ships no queue).
|
||||
|
||||
**Per-node choice is the form the fork takes** (constructor parameter,
|
||||
ADR-003): a heterogeneous replicator fleet — personal nodes on sqlite,
|
||||
community nodes on pg — syncs over the same wire above store-core seams
|
||||
that are engine-blind. Work-above-the-trait (store core, GC mechanism,
|
||||
ops, consumers) is written once; engine work parallelizes against the
|
||||
fixed contract (two engines, one trait, same sweep-safety proof shape).
|
||||
|
||||
Sequencing (recorded in OQ-10): sqlite stays Phase 1's mainline; the pg
|
||||
ADR opens on the deployment trigger, not before — building it earlier
|
||||
would leave an impl with no first consumer to keep the sweep-safety
|
||||
proof honest. The evidence base (this doc + POC #6's redb guardrail) is
|
||||
complete and waiting.
|
||||
|
||||
## POC quality notes
|
||||
|
||||
- The inherited 14-test suite (12 integration + 2 git-CLI cross-checks)
|
||||
|
||||
Reference in new issue
Block a user