docs(architecture): ADR-008 — trait size probe, fleet GC, fs-tier engines; requirements anchor
- requirements.md (new): the three use cases pinned as REQ-1..4 with the node/pool/fleet/engine vocabulary defined once — ends per-session re-derivation of consumer facts. REQ-2 (replicator fleet over one shared pg pool, incl. large-blob serving) is recorded as a planning fact predating all POCs, on the operator's authority. - ADR-008: Backend trait gains size(key) length probe (before any backend ships data — ADR-002 one-way-door discipline); fleet GC mechanism (DB-backed pin rows committed atomically with entries, TTL+renewal semantics, liveness = embedder-owned table, protect callback is single-node-only, advisory-locked single sweeper, staged re-arbitrated delete window on both engines); fs tier becomes engine-selectable (local default; pg-lo named candidate) with the fleet locality contract (shared media or re-routing; mixed tiers are a documented deployment invariant, not a constructor-provable one). - poc-pglo-spec.md (new): POC #7 spec — pg Large Objects as the fs tier's pg-lo engine; instruments, decision gate, registered in phase-0.md OQ-BL-06. - ADR-003/005 status amendments point to ADR-008's extensions; specs ripple (backends/store-api/gc/ops/overview/README). - open-questions.md: deferral-policy header gains the decisions-vs- sequenced-work distinction; pg-lo's why-not-parked audit trail recorded. - research fixes: postgres POC renumbered #4->#5 to the canonical register (phase-0 OQ-BL-06), redb cross-refs fixed, thinking- artifact sentence in B1 replaced with the honest reading. Verification: docs-only change; reference-integrity sweep across the tree (ADR/REQ/POC refs resolve); architecture-reviewer pass on the delta — original 3 criticals addressed, its follow-up (fleet liveness form, pin TTL, staged-delete semantics, enforceability, shared-media caveats) fixed in this commit.
This commit is contained in:
glm-5.3-flash committed
1 parent
395d58cf3f
commit
281c37876e
16 files changed
+689
-88
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# alkblobs — Architecture
|
||||
@@ -36,12 +36,24 @@ there — not inherited as hedges:
|
||||
(POCs #3/#5/#6); the would-be "deployment trigger" framing was caught
|
||||
at review as circular hedging (a deployment running alkblobs cannot
|
||||
pre-exist the engine) and closed on the OQ-09 precedent.
|
||||
- **ADR-008** (2026-10-03) repairs the seams ADR-007 stressed, against
|
||||
the pinned consumer requirement REQ-2 (requirements.md — a replicator
|
||||
node may run a **fleet** of store instances sharing one pg pool, a
|
||||
planning fact predating all POCs): the `Backend` trait gains a
|
||||
`size` probe (before any backend ships data); fleet GC gets
|
||||
DB-backed pins, an advisory-locked single sweeper, and SQL-level
|
||||
delete arbitration (single-node postures unchanged); and the fs tier
|
||||
becomes engine-selectable (`local` default; `pg-lo` named candidate,
|
||||
admission-gated on POC #7). It also adds
|
||||
[requirements.md](requirements.md) — the REQ anchor that ends
|
||||
per-session re-derivation of consumer facts.
|
||||
|
||||
## Architecture Documents
|
||||
|
||||
| Document | Scope | Status |
|
||||
|---|---|---|
|
||||
| [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft |
|
||||
| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, node/pool/fleet vocabulary | draft |
|
||||
| [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft |
|
||||
| [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft |
|
||||
| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, shipped backends, size-threshold dispatch | draft |
|
||||
@@ -60,6 +72,7 @@ there — not inherited as hedges:
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted |
|
||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted |
|
||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted |
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -68,7 +81,11 @@ Tracked in [open-questions.md](open-questions.md). The Phase 0 register
|
||||
remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping,
|
||||
alkfs intake; carried for visibility, gating nothing here). OQ-09
|
||||
(namespace-visibility default) resolved closed-by-default; OQ-10
|
||||
(second kv engine) resolved by ADR-007. **Deferral
|
||||
(second kv engine) resolved by ADR-007. The pg-lo fs-engine question
|
||||
resolved to *sequenced work*, not a parked question: REQ-2/ADR-008
|
||||
named it the candidate fs-tier engine and POC #7
|
||||
(`docs/research/poc-pglo-spec.md`) is its admission evidence — a task,
|
||||
not a wait. **Deferral
|
||||
policy (the "Schrödinger's code" rule):** a *decision this crate needs
|
||||
before shipping* may not be deferred on a dependency that is itself
|
||||
waiting for this crate to exist. Two parking kinds remain legitimate:
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Backends and dispatch
|
||||
@@ -39,10 +39,16 @@ is ever reachable through that door.
|
||||
is also the persistence boundary — a kv row or an fs filename has no
|
||||
schema-migration story, so the byte layout must be self-describing
|
||||
(algorithm-tagged keys).
|
||||
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name`.
|
||||
Streaming put/get shapes ride the store layer's seams (store-api.md);
|
||||
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` /
|
||||
`size` (the length probe added by ADR-008). Streaming put/get shapes
|
||||
ride the store layer's seams (store-api.md);
|
||||
backends provide byte- or handle-level primitives underneath
|
||||
(`get` on the fs tier yields a file handle, not a loaded buffer).
|
||||
`size(key)` returns the entry's length or `None` if absent —
|
||||
metadata only, never a content fetch: kv engines probe the indexed
|
||||
`size` column both row shapes carry (POC #5 B5); the `local` fs
|
||||
engine stats the file. Store-level `stat` and pool accounting ride
|
||||
this method.
|
||||
- **`list()` complete by contract.** A *malformed* list is as deadly as
|
||||
an incomplete one: list correctness is only observable through GC
|
||||
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
|
||||
@@ -122,18 +128,43 @@ sweep-outcome CI gate, run against dockerized postgres under
|
||||
near-linearly to ~37k puts/s at 24 conns — the only engine whose
|
||||
throughput *increases* under concurrency.
|
||||
|
||||
### fs backend (feature `fs`, default-on)
|
||||
### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (candidate)
|
||||
|
||||
Large blobs. Flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}`
|
||||
sharding survives from iroh's conclusion (limits directory size on
|
||||
huge pools); stage-then-commit-rename for the two-pass unknown-length
|
||||
path; pread-based range reads (POC #3 finding A2 — local range serving
|
||||
is sound, e.g. for packfiles).
|
||||
Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract,
|
||||
multiple engines** — the medium is a per-node constructor choice
|
||||
(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing
|
||||
for fleets):
|
||||
|
||||
- **`local` engine** (the default; the ADR-003 backend as originally
|
||||
specced): flat sharded layout:
|
||||
`{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's
|
||||
conclusion (limits directory size on huge pools); stage-then-commit-
|
||||
rename for the two-pass unknown-length path; pread-based range
|
||||
reads (POC #3 finding A2 — local range serving is sound, e.g. for
|
||||
packfiles).
|
||||
- **`pg-lo` engine** (candidate, *not shipped*; ADR-008): postgres
|
||||
Large Objects as the fs tier's storage — the same engine-behind-
|
||||
one-trait move as ADR-007, one tier over. Admission is gated on
|
||||
POC #7's measured evidence and its own engine ADR (see
|
||||
`docs/research/poc-pglo-spec.md`).
|
||||
- **Fleet locality contract (ADR-008):** over one shared pool, the fs
|
||||
tier is either shared media (every node's `local` root on the same
|
||||
fleet-shared media — soundness caveats and the required media
|
||||
guarantees in ADR-008) or re-routed (one storage node serves pool
|
||||
large-blob content via the ops surface; client nodes are
|
||||
kv-tier-only over pool content). Mixed per-node-local fs tiers over
|
||||
one pool is the partitioning failure this contract exists to
|
||||
prevent — a documented deployment invariant the constructor cannot
|
||||
fully prove but must be declared against (ADR-008 names the
|
||||
enforceable seam and the detection symptom).
|
||||
|
||||
### mem backend (feature `mem`, default-off)
|
||||
|
||||
`BTreeMap`-shaped ephemeral backend for tests and in-process
|
||||
ephemerality. A testing/utility tier, never a production story.
|
||||
Implements the full trait contract including `size` (trivially) — it
|
||||
exists partly as the reference impl every other backend's conformance
|
||||
tests can mirror.
|
||||
|
||||
## Size-threshold dispatch (ADR-003)
|
||||
|
||||
@@ -156,12 +187,13 @@ ephemerality. A testing/utility tier, never a production story.
|
||||
The trait is open to future implementations (network stores, S3-like
|
||||
tiers), but nothing in the current consumer set requires one, and the
|
||||
contract is deliberately hostile to half-implementations (complete
|
||||
`list()`, GC-participating `delete`). Any future backend — or third kv
|
||||
engine — is a new ADR carrying its own sweep-safety story (ADR-007 is
|
||||
the precedent for an engine addition: measured evidence first, impl
|
||||
contract naming the durability/ops posture deltas). This crate's
|
||||
roadmap is not blocked on one (see open-questions.md — alkfs intake
|
||||
may name needs externally; OQ-08).
|
||||
`list()`, GC-participating `delete`). Any future backend — or a third
|
||||
kv engine or second fs engine — is a new ADR carrying its own
|
||||
sweep-safety story (ADR-007 is the precedent for a kv engine
|
||||
addition; ADR-008 named pg-lo as the candidate second fs engine with
|
||||
POC #7 as its admission evidence — the door is exercised twice). This
|
||||
crate's roadmap is not blocked on one (see open-questions.md — alkfs
|
||||
intake may name needs externally; OQ-08).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
@@ -171,6 +203,7 @@ may name needs externally; OQ-08).
|
||||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
|
||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -4,7 +4,11 @@
|
||||
|
||||
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
||||
engines, sqlite and postgres, behind this contract; the dispatch layer
|
||||
and trait are unchanged)
|
||||
and trait are unchanged. The trait was extended by ADR-008 with a
|
||||
`size` length probe — before any backend ships data — and the fs
|
||||
tier became engine-selectable (`local` default; pg-lo candidate), so
|
||||
this ADR's "engines"/"backends" counts are read through ADR-004's
|
||||
tier≠engine scope note)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -38,9 +42,10 @@ Evidence in hand (no further POC or consumer-waiting needed):
|
||||
generic-over-`K` alternative was considered and rejected in POC #1
|
||||
finding 1: with one key type, generics buy nothing and break the
|
||||
persistence boundary.)
|
||||
- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`. Streaming and
|
||||
range shapes ride the store layer above (fs `get` yields a file
|
||||
handle, not a loaded buffer).
|
||||
- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`/`size` (the
|
||||
length probe added by ADR-008 before any backend shipped data).
|
||||
Streaming and range shapes ride the store layer above (fs `get`
|
||||
yields a file handle, not a loaded buffer).
|
||||
- **`list()` complete by contract** — and *well-formed*: a malformed
|
||||
list (wrong destructure, values-as-keys) is as deadly as an
|
||||
incomplete one, and silent until a sweep deletes the wrong things
|
||||
|
||||
@@ -2,7 +2,13 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (the sweep-safety protocol here is the single-node posture;
|
||||
ADR-008 extends the mechanism to fleet topologies — DB-backed pins,
|
||||
advisory-locked single sweeper, SQL-level delete arbitration under
|
||||
shared-pool engines (REQ-2) — while keeping this ADR's invariants
|
||||
and the in-process protocol unchanged everywhere else. The "one pool
|
||||
per node" phrasing below reads as "per deployment" for fleets; the
|
||||
vocabulary is pinned in requirements.md)
|
||||
|
||||
## Context
|
||||
|
||||
|
||||
@@ -64,7 +64,9 @@ relational engine a per-node choice rather than a crate fork.
|
||||
|
||||
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
||||
economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy
|
||||
rides the same file (honker's WAL-NORMAL defaults are literally the
|
||||
rides the same file (honker — the alk* family's SQLite
|
||||
notify/queue extension, `/workspace/honker` — ships WAL + the
|
||||
NORMAL-synchronous posture, so its defaults are literally the
|
||||
shipped tier). Single-machine nodes are sqlite by measurement, and
|
||||
remain so even at sustained rates well above observed push traffic
|
||||
(~1.2k/s WAL ceiling vs. single-node push rates an order below it).
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Three defects surfaced in the post-ADR-007 architecture review, all
|
||||
concentrated at the seams ADR-007 touched and all load-bearing for
|
||||
REQ-2 (a replicator node may run multiple store instances — a fleet —
|
||||
sharing one pool via postgres; requirements.md):
|
||||
|
||||
1. **The `Backend` trait has no length probe.** The store surface
|
||||
promises `stat(key) → Option<EntryMeta>` ("cheap length/type
|
||||
probe"; store-api.md) and pool accounting rides `list` + `stat`
|
||||
(ADR-005), but the trait's methods are `has/get/put/delete/list/
|
||||
name` with no value-metadata access. The kv engines' row shapes
|
||||
both carry a `size` column (POC #5 B5), and the fs tier can stat a
|
||||
file — but with no trait method, kv `stat` degenerates to a full
|
||||
value load. The trait is the crate's second stable seam (ADR-003);
|
||||
discovering this during implementation would be a breaking trait
|
||||
change after backends exist.
|
||||
2. **ADR-005's sweep-safety protocol is per-node by construction.**
|
||||
Pins, the pin/liveness arbitration lock, and delete-window
|
||||
arbitration all live inside one process (ADR-005 §Decision). That
|
||||
is sound for sqlite and single-instance postgres (REQ-1, REQ-3
|
||||
topologies) — but under a fleet (REQ-2), node A's sweep enumerates
|
||||
the *shared* pool and arbitrates against A's in-process pin map;
|
||||
node B's in-flight put is invisible. Data loss, structurally: the
|
||||
topology REQ-2 requires does not satisfy the protocol ADR-005
|
||||
specifies.
|
||||
3. **The fs tier is node-local, silently partitioning a fleet.** The
|
||||
fs backend stores to a sharded layout on the node's own filesystem
|
||||
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
|
||||
put via node A exists only on A's disk; node B's get fall-through
|
||||
queries its own fs and misses, and `list()` unions disagree per
|
||||
node. REQ-2 makes this topology a shipping requirement, not a
|
||||
hypothetical.
|
||||
|
||||
These three travel together: the fleet topology forces #2 and #3, and
|
||||
#3's resolution constrains what the fs tier must support, while #1 is
|
||||
a trait-level gap any topology hits.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. `size(key) → Option<u64>` joins the `Backend` trait
|
||||
|
||||
The trait gains a length probe: `size(key)` returns the entry's byte
|
||||
length or `None` if absent. Required of all engines; semantics per
|
||||
tier:
|
||||
|
||||
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
|
||||
never a value load. The `size` column both POC row shapes carry
|
||||
becomes contract-relevant.
|
||||
- **fs engine (`local`)**: stat/metadata read, no content fetch.
|
||||
- The store facade's `stat` (store-api.md) and the accounting path
|
||||
(ADR-005) ride this method; the `has` method remains a pure
|
||||
existence probe.
|
||||
|
||||
`size` is a trait addition *before* any backend ships data — the
|
||||
one-way-door discipline of ADR-002/003 applied to the trait itself.
|
||||
|
||||
### 2. Fleet GC: the delete window is durable and re-arbitrated in the database; pins are DB-backed under fleet engines
|
||||
|
||||
The store facade's GC invariant is unchanged — **a visible pool entry
|
||||
is never deleted while liveness protects it, and an in-flight put is
|
||||
never deleted by a sweep started before it committed** (ADR-005).
|
||||
Under a fleet, the mechanism moves into the shared engine:
|
||||
|
||||
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
|
||||
inserts in one transaction on the fleet engine: the entry row and
|
||||
its pin row commit together (or neither does). The pin row carries
|
||||
an owner (node id) and expiry: the pin guard renews its row every
|
||||
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
||||
parameters, not API). **Protection = an un-expired pin row**: at
|
||||
delete-time arbitration, only `expiry > now()` pins count; an
|
||||
expired pin is abandoned (its holder failed to renew — the same
|
||||
liveness-failure signal as an in-process guard's drop) and is reaped
|
||||
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
||||
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
||||
remains the correctness backstop — content is re-put byte-identically
|
||||
under the same key.
|
||||
- **Registered liveness sources have one fleet form: a liveness
|
||||
table.** A fleet embedder registers its liveness as a named table of
|
||||
`(key, …)` rows in the shared engine (schema pinned by the embedder,
|
||||
shape contractually `(key bytea, …)` — any columns it needs beyond
|
||||
the key). In-process callback *views* are not a fleet liveness form:
|
||||
delete-time arbitration queries/finger-locks tables per key, so a
|
||||
source must be a table for the SQL protocol to see it. Single-node
|
||||
embedders keep callback sources unchanged (ADR-005).
|
||||
- **The protect callback is single-node-only.** A fleet sweeper cannot
|
||||
consult another node's in-process callback, so under fleet engines
|
||||
the callback's fleet expression is: the embedder materializes its
|
||||
protect set into its liveness table (or pin rows) *before* invoking
|
||||
sweep — the callback's abort semantics remain for
|
||||
single-node/topology-local sweeps that cannot express their
|
||||
protection as rows.
|
||||
- **One sweeper per pool.** A fleet node acquires a pg advisory lock
|
||||
before sweeping and holds it for the whole sweep. Two nodes
|
||||
sweeping concurrently is not a state this crate tolerates — the
|
||||
lock is the admission gate, and a second-node sweep returns the
|
||||
typed `GcAborted` (no deletion) rather than racing. `GcAborted`
|
||||
thus carries two cause classes (protection-source failure —
|
||||
ADR-005; sweeper lock contention — this ADR); the typed error's
|
||||
fleet variant identifies which.
|
||||
- **Delete-time arbitration is SQL-level.** The delete window's
|
||||
re-check (ADR-005: "arbitrate under the pin/liveness lock at delete
|
||||
time") executes as `SELECT … FOR UPDATE` against pin rows and
|
||||
liveness tables at delete time, then batch-deletes. The **staged
|
||||
re-arbitrated delete**: during mark, candidate-dead keys are staged
|
||||
as rows carrying `visible_after = mark_end + window` (the delete
|
||||
window's delay — default 5 min, constructor-tunable per
|
||||
deployment); the executor (the sweeper, or the next sweep's owner)
|
||||
re-arbitrates each staged candidate *at execution time* against
|
||||
pins and liveness tables before deleting. This is the durable
|
||||
delete window on every engine — the window table is engine rows on
|
||||
sqlite as well; what differs by topology is only where the liveness
|
||||
state lives (in-process maps and callbacks for single nodes;
|
||||
tables and SQL arbitration for fleets), not whether staging exists.
|
||||
No external queue dependency is required (the pattern is
|
||||
engine-native transactions; consumer-side queues like honker or
|
||||
pg-boss-lineage tools remain consumer-layer options, per ADR-004's
|
||||
boundary).
|
||||
- **Single-node postures are unchanged.** sqlite and single-instance
|
||||
postgres keep ADR-005's in-process protocol verbatim: the pin map,
|
||||
RAII guards, and arbitration lock stay in-process; only the
|
||||
*persistence* of the delete window (the staged-candidate table)
|
||||
is shared machinery. The pin-row expiry/ttl machinery exists only
|
||||
where fleet engines are active.
|
||||
|
||||
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
|
||||
|
||||
The fs tier is a tier, not a medium. It ships engines the way the kv
|
||||
tier does (ADR-007), and a deployment's topology determines which are
|
||||
valid:
|
||||
|
||||
- **`local` (default):** the sharded-layout filesystem backend,
|
||||
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
|
||||
REQ-4) and for fleet nodes using **shared media** (below).
|
||||
- **Fleet + shared media (deployment requirement):** multi-node pools
|
||||
may keep the `local` engine by pointing every node's fs tier at
|
||||
shared/replicated media. The soundness claim is *qualified*, not
|
||||
assumed: immutable-once-committed files plus stage-then-commit-
|
||||
rename are sound on media that provides (a) atomic rename within a
|
||||
directory, (b) close-to-open consistency (a written+closed file is
|
||||
complete to any later reader — NFSv3+ with the close-to-open
|
||||
discipline; NFSv4 and EFS-class stores qualify), and (c) no
|
||||
cross-node POSIX locking reliance (this store takes none — its
|
||||
cross-node coordination is the fleet engine's locks, never file
|
||||
locks; honker's NFS warning is about sqlite's lock protocol, not
|
||||
this shape). Verifying (a)-(c) on the deployment's actual media is
|
||||
the deployment's responsibility; a verification pass is recorded
|
||||
ops work, not crate code.
|
||||
- **Fleet re-routing (topology alternative):** a fleet whose nodes do
|
||||
not share media designates **one storage node**; large-blob writes
|
||||
and reads over the pool route through it via the ops surface
|
||||
(verified fetch/put; ADR-001). Client nodes' constructors are
|
||||
configured with a **kv-tier-only dispatch** (or an fs tier reserved
|
||||
for node-private, consumer-owned content — which is outside the
|
||||
pool and outside this crate's GC; it is the consumer's own storage
|
||||
beside the pool per ADR-004, not pool content). The facade shape on
|
||||
a client node is its own store for sub-threshold content plus its
|
||||
registered ops-put path for large content — a consumer-side
|
||||
composition, not a store-level mode.
|
||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
||||
as the fs tier's storage — the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
||||
with its admission gated on measured evidence (POC #7: LO write/read
|
||||
curves at the packfile regime, tx-scoped handle cost under
|
||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||
pg-lo is the consolidation option if those are unacceptable.
|
||||
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
|
||||
the enforceable seam.** Cross-node configuration cannot be validated
|
||||
by any one constructor (it sees only its own node). What the
|
||||
constructor *can* enforce: a fleet-mode constructor (fleet engines
|
||||
active) requires an explicit fs-tier declaration — `local` (with the
|
||||
operator's assertion that its root is fleet-shared media), routed
|
||||
(no local fs over pool content), or — when shipped — `pg-lo`. The
|
||||
cross-node truth ("is node B's `local` actually the same media as
|
||||
node A's?") is a deployment invariant this crate documents and ops
|
||||
verifies, not something a constructor can prove; a partitioned
|
||||
tier's observable signature (get fall-through misses for entries
|
||||
another node wrote) is the documented detection symptom.
|
||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
||||
as the fs tier's storage — the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
||||
with its admission gated on measured evidence (POC #7: LO write/read
|
||||
curves at the packfile regime, tx-scoped handle cost under
|
||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||
pg-lo is the consolidation option if those are unacceptable.
|
||||
|
||||
In sum: a fleet's fs tier over pool content is either shared media (all
|
||||
nodes' `local` roots on the same fleet-shared media) or routed (one
|
||||
storage node); mixing per-node-local media over one pool is the
|
||||
partitioning failure this decision exists to prevent.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The trait now expressibly supports `stat` and pool accounting on
|
||||
every engine; the "cheap probe" claim of store-api.md is true rather
|
||||
than aspirational.
|
||||
- REQ-2 (fleet over one pool) has a complete, specced GC story: pins
|
||||
move to the shared engine atomically with put-commits, one sweeper
|
||||
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
|
||||
invariant extends to fleets as a *stated* mechanism, not an
|
||||
accident of single-process luck.
|
||||
- The fs tier's fleet problem has named, shipped-code-free answers
|
||||
(shared media, re-routing) and a named consolidation candidate
|
||||
(pg-lo) whose admission path is the same POC-first door as ADR-007.
|
||||
|
||||
**Negative**
|
||||
|
||||
- The `Backend` trait gains a method before any backend ships — the
|
||||
correct moment, but every engine's task list grows by one method.
|
||||
- Fleet GC is real state machinery: pin rows with expiry (GC-of-GC
|
||||
surface for orphaned pins at node crash — the expiry is the cleanup
|
||||
handle), advisory-lock coordination, and SQL-level arbitration
|
||||
added on top of the already-intricate delete-window state machine.
|
||||
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
|
||||
re-route pays for its ADR + POC before shipping large blobs — the
|
||||
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
|
||||
|
||||
**Neutral**
|
||||
|
||||
- The sqlite and single-node-postgres postures are unchanged; this ADR
|
||||
costs existing paths nothing at runtime.
|
||||
- `pg-lo`, if admitted, rides the same driver stack ADR-007 already
|
||||
ships (tokio-postgres + deadpool) via SQL `lo_*` functions — no
|
||||
new driver dependency (the `postgres_large_object` crate is a dead
|
||||
0.15-era io-trait glue; unnecessary).
|
||||
|
||||
## References
|
||||
|
||||
- [requirements.md](../requirements.md) — REQ-2 (the fleet fact this
|
||||
ADR binds), REQ-1/3/4 (the topologies left unchanged)
|
||||
- POC #5 (`poc-postgres-kv-findings.md` B5 — row shapes with `size`
|
||||
columns; the driver stack pg-lo would ride)
|
||||
- ADR-002 (the one-way-door discipline this trait change applies),
|
||||
ADR-003 (the trait contract and admission gate `size` and pg-lo
|
||||
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
|
||||
ADR-005 (the GC invariant and single-node protocol extended here,
|
||||
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
|
||||
follows), ADR-007 (the engine-behind-one-trait precedent)
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
|
||||
and tier surface this amends);
|
||||
[store-api.md](../store-api.md) (`stat`'s contract);
|
||||
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
||||
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
||||
topology's transport)
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Pooling, namespaces, and GC
|
||||
@@ -16,11 +16,17 @@ price, and this document specifies the mechanism the POCs validated
|
||||
|
||||
## The pooled CAS
|
||||
|
||||
- **One pool per node.** Repos, workspaces, and appfile stores are all
|
||||
- **One pool per deployment; a node is one store instance** (the
|
||||
engine/vocabulary definitions live in requirements.md). Repos,
|
||||
workspaces, and appfile stores are all
|
||||
*sets of hash references* over the same CAS — never walled-off
|
||||
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
|
||||
object pools) is the same idea done awkwardly; the pool generalizes it
|
||||
to unrelated repos and to p2p replication.
|
||||
to unrelated repos and to p2p replication. **A fleet (REQ-2) — two or
|
||||
more nodes sharing one pool — is the one topology where the GC
|
||||
mechanism below extends beyond a node: ADR-008 pins, sweeper
|
||||
locking, and delete arbitration move into the shared engine;
|
||||
the invariants are unchanged.**
|
||||
- **One address space.** Under the canonical hash (ADR-002), the same
|
||||
file content has the same address whether it entered via a git oid or
|
||||
a workspace manifest — cross-consumer dedup is free by construction,
|
||||
@@ -116,16 +122,19 @@ existence flips atomically per key).
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
|
||||
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
|
||||
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere |
|
||||
|
||||
## Open Questions
|
||||
|
||||
None owned by this document. The p2p replicator's cross-node GC
|
||||
coordination is a property of the replicator policy layer (above the
|
||||
crate), governed by the same seam — it raises a question here only
|
||||
when a consumer *names a requirement* (OQ-07/OQ-08 intake), which is
|
||||
re-entry via a new decision at a new ADR, not a parked question: no
|
||||
decision here waits on a consumer, and a future requirement does not
|
||||
create one retroactively.
|
||||
None owned by this document. Shared-pool GC coordination within a
|
||||
deployment is now in-crate (fleet mechanism: ADR-008). What remains a
|
||||
replicator-policy-layer property (above the crate) is *cross-pool*
|
||||
coordination — gossip sync between separately-pooled replicator nodes
|
||||
deciding what each keeps — governed by the same seam; it raises a
|
||||
question here only when a consumer *names a requirement*
|
||||
(OQ-07/OQ-08 intake), which is re-entry via a new decision at a new
|
||||
ADR, not a parked question: no decision here waits on a consumer, and
|
||||
a future requirement does not create one retroactively.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Open Questions
|
||||
@@ -31,6 +31,19 @@ make before shipping. Phase 0's residual list was swept under this
|
||||
rule: every residual either resolved into an ADR (the evidence was
|
||||
already in hand) or re-owned as below.
|
||||
|
||||
**The line between the two kinds** (added 2026-10-03, after the
|
||||
pg-lo round): deferring on a consumer's *design decision* is valid —
|
||||
the deciding fact is obtainable independently of this crate (OQ-07,
|
||||
OQ-08). Deferring on a consumer's *deployment of this crate* is
|
||||
invalid — that fact is created by this crate (OQ-10's original
|
||||
framing, OQ-09's original framing). And a third thing is neither:
|
||||
**decided-but-sequenced work**. Once a question resolves to a decision
|
||||
(the pg-lo engine is the named fs-tier candidate under REQ-2/ADR-008),
|
||||
what remains is the evidence pipeline the decision's own admission
|
||||
gate requires (POC #7) — a task with an owner, not a parked question,
|
||||
and not a hedge. A decision is not "unmade" because its verification
|
||||
is pending; deferral policy applies to decisions, not to work.
|
||||
|
||||
**Index of active OQs:** OQ-07, OQ-08 (externally-owned, carried
|
||||
for visibility). OQ-09 and OQ-10 are resolved (recorded below).
|
||||
Promoted Phase 0 questions OQ-BL-01..06 are recorded here with their
|
||||
@@ -205,9 +218,12 @@ resolutions for traceability.
|
||||
- **Origin**: `docs/research/phase-0.md`
|
||||
- **Status**: resolved (complete: #1, #3 passed; #2 absorbed; #4
|
||||
covered in miniature, concurrency half specified as architecture in
|
||||
ADR-005)
|
||||
- **Resolution**: register complete; Phase 0 ended; evidence trail in
|
||||
`docs/research/`.
|
||||
ADR-005; post-convergence additions #5 postgres and #6 redb passed
|
||||
and fed ADR-007; #7 pg-lo specified — `docs/research/poc-pglo-spec.md`,
|
||||
requested by REQ-2/ADR-008)
|
||||
- **Resolution**: register complete and extended post-convergence;
|
||||
Phase 0 ended; evidence trail in `docs/research/`. The register's
|
||||
canonical numbering lives in phase-0.md OQ-BL-06.
|
||||
- **Cross-references**: all ADRs above
|
||||
|
||||
---
|
||||
@@ -229,6 +245,18 @@ a deployment trigger — the same invalid shape, caught at review: the
|
||||
trigger was a fact only this crate could create. It resolved to
|
||||
ADR-007 (two kv engines; see its entry above).
|
||||
|
||||
The pg-lo fs-engine question (raised in the ADR-008 review round,
|
||||
2026-10-03) never became a parked question: REQ-2 made the
|
||||
multi-instance-with-large-blobs topology a standing requirement
|
||||
(requirements.md — a planning fact predating all POCs), so there was
|
||||
nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the
|
||||
candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
|
||||
as its admission evidence — sequenced work, per the deferral-policy
|
||||
header's third category. An earlier draft of that OQ parked on
|
||||
"alkgit's architecture process names the topology" — rendered moot the
|
||||
same day by the REQ-2 fact; recorded here because the audit trail of
|
||||
*why not parked* is part of this document's job.
|
||||
|
||||
OQ-07/OQ-08 are owned by other repos' processes and are not alkblobs
|
||||
tracker tasks (their outcome arrives *through* their owners, not
|
||||
through any artifact this repo creates).
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Ops surface (alkcall-backed network operations)
|
||||
@@ -142,7 +142,11 @@ side channel: fetch handlers call `get`/`read_range`/`stat`; put
|
||||
handlers call `put`; both honor pins and delete windows. There is no
|
||||
ops-internal path around the store facade (ADR-001 §Consequences —
|
||||
this is what makes "re-homing the ops surface later loses nothing"
|
||||
true).
|
||||
true). One fleet topology additionally rides this module: **re-routing
|
||||
(ADR-008)** — a fleet whose nodes do not share fs media routes
|
||||
large-blob access through a single storage node via these ops, with
|
||||
client nodes' fs tiers serving only node-private content. The module
|
||||
supports this shape without change; no fleet-specific op exists.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
@@ -150,6 +154,7 @@ true).
|
||||
|---|---|---|
|
||||
| [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale |
|
||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-02
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Overview
|
||||
@@ -10,9 +10,15 @@ last_updated: 2026-10-02
|
||||
`alkblobs` is content-addressed blob storage in the alk* family: a
|
||||
put/get/verify store keyed under one canonical content hash (the
|
||||
git-family derivation; legacy SHA-1-tolerant), with pluggable backends
|
||||
(small blobs in a kv store, large blobs on a filesystem fallback), split
|
||||
from any wire/ops-protocol layer. Network ops ride the alkcall substrate
|
||||
behind a feature-gated `ops` module (ADR-001).
|
||||
(small blobs in a kv store, large blobs on a filesystem fallback),
|
||||
split from any wire/ops-protocol layer. Network ops ride the alkcall
|
||||
substrate behind a feature-gated `ops` module (ADR-001).
|
||||
|
||||
Deployment-relevant consumer facts — what each use case forces on this
|
||||
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
|
||||
with the **node/pool/fleet/engine** vocabulary defined there). This
|
||||
tree references REQs by ID; per-session re-derivation from vision prose
|
||||
is exactly what that document exists to prevent.
|
||||
|
||||
## Why it exists — the extracted shared core
|
||||
|
||||
@@ -60,10 +66,11 @@ or three times.
|
||||
│ size-threshold dispatch (ADR-003) │
|
||||
└───────┬───────────────────┬───────────────────────────────┘
|
||||
▼ ▼
|
||||
kv backend fs backend (ADR-003: exactly two tiers;
|
||||
│ ADR-007: kv = 2 engines)
|
||||
▼
|
||||
engine per node: sqlite (default) | postgres (feature "postgres")
|
||||
│ kv backend fs backend (ADR-003: exactly two tiers;
|
||||
│ │ ADR-007: kv = 2 engines;
|
||||
│ ▼ ADR-008: fs engines too)
|
||||
│ engine per node: sqlite (default) | postgres (feature "postgres")
|
||||
│ fs engine per node: local (default) | pg-lo (candidate, POC #7)
|
||||
|
||||
transport: none, anywhere. alkgit/alkfs or the embedder dials;
|
||||
an alkcall Connection is handed in at the ops boundary.
|
||||
@@ -74,15 +81,22 @@ or three times.
|
||||
| Layer | Dependencies | Notes |
|
||||
|---|---|---|
|
||||
| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
|
||||
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||||
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||||
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
|
||||
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
|
||||
|
||||
Backends are feature-gated; both `kv` and `fs` ship default-on
|
||||
(batteries-included, the alkgit feature-model pattern); the `kv` tier's
|
||||
postgres engine (`postgres`, ADR-007) and the `ops` feature are
|
||||
default-off. Wasm: the backends are not a wasm story; a wasm client is a
|
||||
consumer of the `ops` surface (which rides alkcall, itself
|
||||
(batteries-included, the alkgit feature-model pattern — default-on
|
||||
features, opt-out rather than opt-in); the `kv` tier's postgres engine
|
||||
(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only
|
||||
kv node (postgres engine without rusqlite) is **not** a supported
|
||||
configuration this crate optimizes for — feature-graph work for it is
|
||||
deferred until a named requirement exists (the base `kv` feature
|
||||
pulling rusqlite is the accepted v1 shape; convention 8's
|
||||
lean-base principle applies to *default* builds, which stay lean since
|
||||
`kv` is what the feature model controls, not what default-on means
|
||||
downstream). Wasm: the backends are not a wasm story; a wasm client is
|
||||
a consumer of the `ops` surface (which rides alkcall, itself
|
||||
wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
||||
|
||||
## Non-goals (each enforced by an ADR)
|
||||
@@ -111,6 +125,7 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
|
||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
|
||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -125,6 +140,10 @@ crate level:
|
||||
open-read requirement)
|
||||
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
|
||||
shipped behind one trait; per-node constructor choice)
|
||||
- **POC #7** (`docs/research/poc-pglo-spec.md`): pg Large Objects as
|
||||
the fs tier's `pg-lo` engine — spec'd **sequenced work** (not a
|
||||
parked question; open-questions.md's policy header distinguishes the
|
||||
two): admission evidence for ADR-008's named candidate (REQ-2)
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Requirements
|
||||
|
||||
The deployment-relevant facts each use case forces on this crate.
|
||||
Specs and ADRs reference these by ID (REQ-n); a future session must be
|
||||
able to read this document and never re-derive a requirement from
|
||||
conversation or vision prose. **This document is the anchor: when a
|
||||
consumer fact changes, it changes here first, with a dated note, and
|
||||
the ADRs re-derive from it.**
|
||||
|
||||
## Vocabulary (defined once; used everywhere)
|
||||
|
||||
- **Engine** — a concrete storage implementation of one tier's
|
||||
contract: the kv tier's engines are sqlite and postgres (ADR-007);
|
||||
the fs tier's engines are `local` (default) and `pg-lo` (candidate —
|
||||
see REQ-2, ADR-008).
|
||||
- **Pool** — the flat content-addressed namespace of one deployment
|
||||
(`hash → bytes` everywhere; ADR-005).
|
||||
- **Node** — one running store instance (one process embedder; its fs
|
||||
tier is its own media). Node is the GC arbitration domain for
|
||||
single-node topologies (ADR-005's pins, liveness locks, and delete
|
||||
windows live inside a node); under a fleet, arbitration moves into
|
||||
the shared engine (ADR-008).
|
||||
- **Fleet** — two or more nodes over **one shared pool** (e.g. via one
|
||||
postgres instance). A fleet is the only topology in which
|
||||
cross-node GC and fs-locality questions exist (ADR-008).
|
||||
- **Deployment** — one operator's whole system: typically one fleet
|
||||
(multi-node) or one node; consumer tables, op registrations, and
|
||||
sweep cadence are deployment choices.
|
||||
|
||||
## The use cases and their requirements
|
||||
|
||||
### UC-1: Self-hosted git serving (gitea-replacement scale)
|
||||
|
||||
Git repos for a user/team on one machine. Small-object regime dominated
|
||||
by loose objects in the kv tier; packs and large artifacts (git-lfs
|
||||
shape) on the fs tier.
|
||||
|
||||
- **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics:
|
||||
sqlite is the default kv engine; single-writer WAL ceiling (~1.2k
|
||||
objects/s under contention) is comfortably above single-node push
|
||||
rates. **Zero-ops is a requirement**: no daemon beyond the node
|
||||
process itself.
|
||||
|
||||
### UC-2: Distributed git with replicator nodes hosting repos for many users
|
||||
|
||||
The p2p/OSS sketch (phase-0 §p2p shape): replicators are push/clone
|
||||
endpoints serving many concurrent users; gossip-style sync between
|
||||
replicators; ACL/ownership above (alkcall layer).
|
||||
|
||||
- **REQ-2 (traces: ADR-007, ADR-008)** — **a replicator node may run
|
||||
multiple store instances (a fleet) sharing one pool** via a shared
|
||||
postgres kv engine, including serving large (>threshold) pool
|
||||
content from that fleet. This is stated as a requirement, dated
|
||||
2026-10-03 (recording a planning fact from the use case's origin
|
||||
discussion; it predates all POCs and architecture docs, recorded
|
||||
here on the operator's authority as the use case's stated shape).
|
||||
Consequences are derived in ADR-008: fleet GC (DB-backed pins,
|
||||
advisory-locked sweeper, SQL delete arbitration), the fs-tier
|
||||
engine concept with its locality contract, and the pg-lo candidate
|
||||
engine (POC #7).
|
||||
- **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out:
|
||||
the fleet's hub-class nodes need put throughput that *increases*
|
||||
under concurrency (pg: near-linear to ~37k puts/s at 24 conns);
|
||||
sqlite's WAL ceiling (~1.2k/s, latency-climbs-unbounded) bounds
|
||||
single-node sustained write pressure regardless of client count.
|
||||
This is the measured basis for the per-node engine split.
|
||||
|
||||
### UC-3: Agent workspaces (many agents, near-identical content)
|
||||
|
||||
Agents working simultaneously in mostly-identical workspaces, each
|
||||
making small edits; a shared pool dedups by construction; per-workspace
|
||||
path→hash manifests above.
|
||||
|
||||
- **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space
|
||||
across workspaces; dedup must be structural (not configured). Small
|
||||
edits mean the small-blob regime dominates: sqlite-class kv
|
||||
performance is what matters; the fs tier is hit rarely (large
|
||||
workspace artifacts).
|
||||
|
||||
## What this document is not
|
||||
|
||||
- **Not consumer architecture**: how alkgit maps refs/manifests onto
|
||||
the store remains OQ-07 (externally-owned); how alkfs shapes
|
||||
workspaces remains OQ-08. This document records what each use case
|
||||
*forces on this crate*, not what each consumer *is*.
|
||||
- **Not a requirements-change process substitute**: a new or changed
|
||||
REQ arrives here with a dated note and owner; ADRs that depended on
|
||||
the old fact state their reopen conditions and reopen when this
|
||||
file changes.
|
||||
|
||||
## Decision trace
|
||||
|
||||
| REQ | Introduced | Source | ADRs |
|
||||
|---|---|---|---|
|
||||
| REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 |
|
||||
| REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 |
|
||||
| REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 |
|
||||
| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-005 |
|
||||
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-01
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# Store API
|
||||
@@ -11,7 +11,9 @@ The store core's public surface: the typed facade above the backends
|
||||
that consumers (alkgit, alkfs, the ops module) program against. It owns
|
||||
typed keys and hashing (ADR-002), size-threshold dispatch (ADR-003),
|
||||
the put-pinning seam, and the liveness/sweep seam (ADR-005) — and
|
||||
nothing else: no paths, no manifests, no wire (ADR-004, ADR-001).
|
||||
nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). The
|
||||
facade is **engine-blind** (ADR-007/008): which kv/fs engine a node
|
||||
runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
||||
|
||||
## Public surface
|
||||
|
||||
@@ -57,7 +59,9 @@ nothing else: no paths, no manifests, no wire (ADR-004, ADR-001).
|
||||
primitive and the packfile-serving surface rides this). `EntryMeta`
|
||||
carries `len: u64` and the key's algorithm (the "type" from the
|
||||
store's perspective is the hash algorithm — there is no other type at
|
||||
this layer).
|
||||
this layer). `stat` rides the `Backend` trait's `size` method
|
||||
(ADR-008): an indexed-column probe on kv engines, a file stat on the
|
||||
fs `local` engine — never a value/content load.
|
||||
- **`read_range(key, range) → (slice, slice_digest)`** — slice plus an
|
||||
out-of-band slice digest: SHA-256 over the slice bytes, fixed by
|
||||
convention so callers and ops handlers agree without negotiation
|
||||
@@ -85,7 +89,12 @@ nothing else: no paths, no manifests, no wire (ADR-004, ADR-001).
|
||||
(finding 7) is why the verb/name is pinned here).
|
||||
- **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own
|
||||
the cadence (ADR-005, no ambient timers). A sweep with no registered
|
||||
liveness sources aborts without deleting — the safe default.
|
||||
liveness sources aborts without deleting — the safe default. Under
|
||||
a fleet (multiple nodes over one pool — requirements.md REQ-2),
|
||||
sweep/put/delete semantics are mechanism-extended by ADR-008: pins
|
||||
are DB-backed rows committed atomically with entries, one sweeper
|
||||
holds the advisory lock, and delete arbitration is SQL-level — the
|
||||
facade's invariant text is unchanged.
|
||||
|
||||
## Error model
|
||||
|
||||
@@ -96,7 +105,9 @@ tests (AGENTS convention 2). Distinguished failure families:
|
||||
- `Verification` — put/get hash-check failure (content ≠ key)
|
||||
- `Io(String)` — backend media failure (stringly because
|
||||
`std::io::Error` is not stable across versions)
|
||||
- `GcAborted` — a protection source failed; nothing deleted (ADR-005)
|
||||
- `GcAborted` — nothing deleted; sweep not run. Two cause classes
|
||||
(ADR-005: a protection source failed; ADR-008: another node holds
|
||||
the fleet sweeper lock) — the typed error distinguishes them.
|
||||
- `KeyInvalid` — malformed key bytes at the boundary (hashing doc)
|
||||
|
||||
Virgin-store semantics: read paths on a fresh store see absent/empty,
|
||||
@@ -117,18 +128,27 @@ lesson generalizes to any kv engine).
|
||||
## Invariants (the test gate)
|
||||
|
||||
1. Whole-blob put/get round-trips byte-identical under the canonical
|
||||
derivation, both tiers, both put paths (interop with real git
|
||||
remains the source of truth, per POC #1's lesson about hardcoded
|
||||
vectors).
|
||||
derivation, both tiers, both put paths, **and every shipped engine
|
||||
of both tiers** (kv: sqlite + postgres; fs: `local`; ADR-007/008)
|
||||
(interop with real git remains the source of truth, per POC #1's
|
||||
lesson about hardcoded vectors).
|
||||
2. Dedup: putting identical content twice (same or different path) is
|
||||
one pool entry.
|
||||
3. Stage hygiene: any failure mid-put leaves zero stage files; any
|
||||
success leaves exactly one committed entry.
|
||||
4. Sweep safety: with correct liveness registered, sweep counts are
|
||||
exact; with aborting sources, sweep deletes nothing.
|
||||
exact; with aborting sources, sweep deletes nothing. **Engine-gated
|
||||
form (ADR-008):** single-node postures assert this under the
|
||||
in-process protocol; fleet postures (shared-pool engines) assert it
|
||||
under DB-backed pins + the advisory-locked sweep, with the
|
||||
exact-count sweep-outcome shape run against dockerized postgres
|
||||
under `--all-features`.
|
||||
5. `list()` correctness is observable only through GC — list-related
|
||||
tests assert through sweep outcomes (POC #1 finding 2's lesson
|
||||
codified).
|
||||
6. `size`/`stat` on any engine returns the entry length without
|
||||
fetching content (asserted per-backend: kv probe, fs stat —
|
||||
ADR-008).
|
||||
|
||||
## Design Decisions
|
||||
|
||||
@@ -138,6 +158,7 @@ lesson generalizes to any kv engine).
|
||||
| [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering |
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics |
|
||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed |
|
||||
|
||||
## Open Questions
|
||||
|
||||
|
||||
@@ -1,8 +1,6 @@
|
||||
---
|
||||
status: converged
|
||||
last_updated: 2026-10-02 (Phase 0 convergence — all POCs passed, OQs
|
||||
settled into decisions + residuals; phase-0 plan complete;
|
||||
OQ-BL-01 ops-surface substrate settled: alkcall)
|
||||
last_updated: 2026-10-03 (POC #7 spec added to the register OQ-BL-06; Phase 0 remains closed)
|
||||
---
|
||||
|
||||
# alkblobs — Phase 0 (Exploration)
|
||||
@@ -595,6 +593,7 @@ POC #3 pack analysis):**
|
||||
| 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Covered in miniature by #1** — namespace tables + sweep + recover validated single-threaded; the concurrency half is Phase 1 implementation work, not a POC gate | `poc-trait-dispatch-findings.md` findings 3/7/8 |
|
||||
| 5 | Postgres as the kv engine (added post-convergence): inherited sqlite/fs/pg benchmark arms + write/read concurrency scale-out probes | **Passed 2026-10-02** (6 findings B1-B6: single-conn pg floor ~1 ms fsync-dominated, ~40× storage overhead; pg PUT scale-out ~12× at 12 conns / ~37k ops/s vs sqlite's ~1.2k WAL-serialized ceiling) | `poc-postgres-kv-findings.md`; code: `/workspace/alkblobs-postgres-poc` |
|
||||
| 6 | redb as the kv engine (added post-convergence, same standing as #5): inherited sqlite/fs arms + redb durability decomposition + scale-out probe | **Passed 2026-10-02** (6 findings C1-C6: the "2-7× over sqlite" claim inverted — sqlite ~430× over redb at crash-consistent puts; redb Immediate = 1 fdatasync/commit, 8-30 ms on this disk; reads ~530k/s but irrelevant; write scale-out flat ~42/s; ruled out at a durability-tier mismatch, not a benchmark quibble) | `poc-redb-kv-findings.md`; code: `/workspace/alkblobs-redb-poc` |
|
||||
| 7 | Postgres Large Objects as the fs tier's `pg-lo` engine (added 2026-10-03, per ADR-008 naming it the candidate fs-tier engine for REQ-2 fleets): LO write/read curves at the packfile regime, tx-scoped handle cost under pooling, pg_largeobject/vacuum posture under churn, crash-orphan behavior | **Specified, not run** — `poc-pglo-spec.md`; admission evidence for the pg-lo engine ADR | `poc-pglo-spec.md` |
|
||||
|
||||
Sequencing outcome: #1 and #3 passed; #2 absorbed/validated under #1;
|
||||
#4's single-threaded core covered by #1 (the mechanism choice it
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
status: specified
|
||||
title: "POC #7 — postgres Large Objects as the fs tier's pg-lo engine"
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# POC: postgres Large Objects as the fs tier's pg-lo engine — spec
|
||||
|
||||
> **POC register #7**, added post-Phase-0 per the ADR-003/007
|
||||
> substitution door (the third use of it, after postgres-on-kv and
|
||||
> redb-on-kv). Code: standalone crate
|
||||
> `/workspace/alkblobs-pglo-poc` (clone the POC #5 crate's harness
|
||||
> conventions; add a `PgLoBackend` implementing the ADR-008 trait
|
||||
> contract including `size`). Findings land here regardless, per the
|
||||
> established convention. Requested by REQ-2 (requirements.md — the
|
||||
> fleet-with-large-blobs consolidation option; ADR-008 names pg-lo as
|
||||
> the candidate fs-tier engine). Status: **specified, not run**.
|
||||
|
||||
## What this POC must decide
|
||||
|
||||
The ADR-008 question: can postgres Large Objects hold the fs tier's
|
||||
contract for the fleet topology — serving the ≥128 KiB regime
|
||||
(packfiles, large artifacts) from the same engine the kv tier already
|
||||
rides — at acceptable performance and with an honest ops posture? The
|
||||
fs tier's contract (ADR-003/008): `get` yields a pread-able handle,
|
||||
stage-then-commit semantics for unknown-length puts, complete
|
||||
`list()`, GC-participating `delete`, `size` without content fetch.
|
||||
|
||||
Large Objects' native shape differs from files in one structural way
|
||||
the ADR names: **LO descriptors are transaction-scoped** (lo_open →
|
||||
loread/lowrite must run inside a tx). The POC must measure what that
|
||||
does to the handle pattern under a connection pool, not assume it away.
|
||||
|
||||
## Instruments
|
||||
|
||||
1. **Inherited `bench_small`-shaped harness, LO arm** — the POC #3/#5
|
||||
methodology at fs-tier-relevant sizes: **128 KiB, 256 KiB, 1 MiB,
|
||||
16 MiB, 128 MiB**, sequential and random-replay, p50/p99. Compare
|
||||
against: local fs (the `local` engine baseline, same harness) and
|
||||
kv-tier pg (bytea) at the lower sizes, as cross-checks of
|
||||
methodology parity. (`loread`/`lowrite` via SQL functions through
|
||||
tokio-postgres — the `postgres_large_object` crate targets
|
||||
postgres 0.15 and is dead; do not use it.)
|
||||
2. **Handle-cost probe** (`pgdiag7`): the tx-scoped descriptor under
|
||||
deadpool — cost of open-tx → lo_open → lseek/loread per range read,
|
||||
held vs. re-opened per read; pool pressure under N concurrent
|
||||
readers (the ops fetch handler's real shape).
|
||||
3. **Write path probe** (`pgdiag8`): lo_create + chunked lowrite
|
||||
(the LOBLKSIZE ~2 KiB page layout matters here — measure write
|
||||
amplification as value size grows); unknown-length put shape
|
||||
(stage into one LO, companion row on commit); verify
|
||||
`size`-without-fetch (`SELECT size` from the companion table).
|
||||
4. **Churn/vacuum probe**: repeated put/delete cycles at 1–16 MiB;
|
||||
measure `pg_largeobject` catalog growth, TOAST/vacuum behavior,
|
||||
autovacuum posture (the fs engine inherits ADR-008/007's
|
||||
maintenance-surface honesty requirement).
|
||||
5. **Crash/orphan probe**: kill client mid-write-tx; verify no orphan
|
||||
LO and no half-commit; kill after lo_create before commit; write
|
||||
the orphan-recovery sweep story the engine ADR will need (compare
|
||||
`lo_get`-visible OIDs against the companion table — never trust
|
||||
the catalogs).
|
||||
|
||||
## Decision gate (mirrors ADR-007's)
|
||||
|
||||
pg-lo is admitted as a shipped fs-tier engine iff:
|
||||
|
||||
- **Performance:** LO put/get at 128 KiB–16 MiB is within an
|
||||
order-of-magnitude of local fs on the same box (the tier is chosen
|
||||
for fleet correctness, not solo speed — but a 10× gap would put the
|
||||
fleet's packfile serving materially behind a single-node fs
|
||||
deployment, and the ADR must say so if measured);
|
||||
- **Contract:** complete `list()` (companion table), GC-participating
|
||||
delete (`lo_unlink` under the delete window), virgin-store no-op
|
||||
reads, `size` probe — all hold, proven via the exact-count
|
||||
sweep-outcome test shape under `--all-features`;
|
||||
- **Ops posture:** named deltas (tx-scoped handles, autovacuum/vacuum
|
||||
on `pg_largeobject`, orphan recovery sweep) with measured or
|
||||
documented-bounded costs.
|
||||
|
||||
Findings feed an engine ADR in the ADR-007 shape (measured curves,
|
||||
posture deltas as requirements). If the gate fails, ADR-008's
|
||||
recorded statement stands: shared-media or re-routing are REQ-2's
|
||||
fleet answers, and pg-lo's named cost was paid as evidence, not
|
||||
shipped blind.
|
||||
|
||||
## Register note (phase-0.md OQ-BL-06 convention)
|
||||
|
||||
The canonical register is phase-0.md OQ-BL-06: #1 trait dispatch,
|
||||
#2 absorbed, #3 large-blob, #4 pooled-CAS (covered by #1), #5
|
||||
postgres, #6 redb. The post-convergence POC files briefly carried
|
||||
inconsistent self-numberings (the postgres file titled itself "#4");
|
||||
they are renumbered to the register. This POC is **#7** — specified
|
||||
2026-10-03, requested by REQ-2 (requirements.md; ADR-008 names pg-lo
|
||||
as the candidate fs-tier engine).
|
||||
@@ -1,14 +1,17 @@
|
||||
---
|
||||
status: passed
|
||||
title: "POC #4 — postgres as the kv engine: single-conn penalty measured, concurrency scale-out measured"
|
||||
last_updated: 2026-10-02
|
||||
title: "POC #5 — postgres as the kv engine: single-conn penalty measured, concurrency scale-out measured"
|
||||
last_updated: 2026-10-03
|
||||
---
|
||||
|
||||
# POC: postgres as the kv engine — findings
|
||||
|
||||
> **POC register #4**, added after Phase 0 convergence (the register was
|
||||
> **POC register #5**, added after Phase 0 convergence (the register was
|
||||
> complete at #3; this is the ADR-003 "substitution seam" input arriving
|
||||
> early). Code: standalone crate `/workspace/alkblobs-postgres-poc`
|
||||
> early). *Renumbered 2026-10-03 from a self-assigned "#4" to the
|
||||
> canonical register (phase-0.md OQ-BL-06), where #4 is the pooled-CAS
|
||||
> POC covered by #1; the redb file's cross-references were fixed in the
|
||||
> same pass.)* Code: standalone crate `/workspace/alkblobs-postgres-poc`
|
||||
> (a copy of the POC #3 crate with a `PgKv` backend added; findings land
|
||||
> here regardless, per the established convention). Date: 2026-10-02.
|
||||
> Status: **passed** — the inherited POC #3 test suite (14 tests, byte-exact
|
||||
@@ -94,13 +97,14 @@ Readings:
|
||||
floor is 2-5× fs's ~200-450 µs syscall chain (and 10-50× sqlite).
|
||||
POC #3's fs numbers reproduce within noise, confirming methodology
|
||||
parity.
|
||||
- **At 64-128 KiB postgres pulls ahead of sqlite** (~800-1000 vs
|
||||
~2050-4100 — actually *behind* sqlite, but ahead of fs on gets and
|
||||
comparable on puts) — wait, no: at 64 KiB sqlite still wins
|
||||
(~4.1k). The honest reading: pg crosses fs somewhere in the 16-64 KiB
|
||||
band for puts and is ~at-parity through 64-128 KiB. **The pg-vs-fs
|
||||
crossover is real but sits an order of magnitude higher than
|
||||
sqlite-vs-fs.**
|
||||
- **At 64-128 KiB postgres sits between fs and sqlite** (~800-1000 vs
|
||||
fs ~2300-2900 on gets — *behind* fs on some arms; sqlite still leads
|
||||
~4.1k at 64 KiB). The honest reading: pg holds no decisive win in
|
||||
the 1-128 KiB single-worker band on this disk; **the pg-vs-fs
|
||||
crossover, where it exists at all, sits higher than sqlite-vs-fs's**.
|
||||
Note: pg's decisive advantage was never the solo 1-D curve — it is
|
||||
the concurrency axis (B3/B4), where it multiplies and sqlite
|
||||
serializes.
|
||||
- The sqlite numbers reproduce A4 within noise (44k vs 48k at 1 KiB),
|
||||
so the methodology transfer is sound.
|
||||
|
||||
@@ -252,7 +256,7 @@ a deployment running the *fs tier anyway* has none of either cost.
|
||||
list()-complete-by-contract test would need the vacuum-aware
|
||||
equivalent of POC #1's exact-count sweep assertions.
|
||||
|
||||
## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #4/#6)
|
||||
## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #5/#6)
|
||||
|
||||
The three alkblobs-relevant git/vfs use cases, mapped onto the measured
|
||||
curves; this was the conclusion OQ-10 carried before its resolution
|
||||
@@ -262,7 +266,7 @@ in ADR-007):
|
||||
|---|---|---|
|
||||
| **sqlite** | self-hosted git serving (gitea-replacement scale), agent-workspace pools, small/personal replicators | 44k solo puts/s at 1-16 KiB (POC #3 A4, reproduced here); WAL single-writer ceiling ~1.2k/s under contention is comfortably above single-node push rates; zero-ops, no daemon; honker rides the same file (same WAL-NORMAL posture, its defaults are literally the shipped tier) |
|
||||
| **postgres** | public multi-tenant replicator nodes (many users pushing, gossip catch-up, bulk ingest), consolidated-ops deployments (already-running pg) | near-linear put scale-out to ~21k at 4 conns, ~28k at 8, ~32k at 12, ~37k at 24-48 (B3); reads scale to ~44k at 24 conns (B4); the only engine whose throughput *increases* under concurrency |
|
||||
| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #4/#6 reproductions |
|
||||
| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #5/#6 reproductions |
|
||||
|
||||
The decision rule the two POCs establish: **the trigger is topology,
|
||||
not throughput** — cross-MACHINE write access to one pool makes postgres
|
||||
|
||||
@@ -11,13 +11,13 @@ last_updated: 2026-10-02
|
||||
> crate `/workspace/alkblobs-redb-poc` — a copy of the POC crate with the
|
||||
> pg arm replaced by a `RedbKv` backend (`redb` 4.3, the engine iroh-blobs'
|
||||
> fs store uses for its metadata tables). The inherited sqlite/fs arms and
|
||||
> the 14-test suite are byte-identical to POC #3/#4's methodology.
|
||||
> the 14-test suite are byte-identical to POC #3/#5's methodology.
|
||||
> Date: 2026-10-02. Status: **passed** (14 tests, clippy `-D warnings`
|
||||
> clean, fmt clean).
|
||||
|
||||
## What this POC set out to decide
|
||||
|
||||
The "opposite route" from POC #4: postgres is complicated, we only need a
|
||||
The "opposite route" from POC #5: postgres is complicated, we only need a
|
||||
kv — what about a high-performance *embedded* kv like redb, which the
|
||||
DESIGN.md lineage (iroh-blobs' "faster than the filesystem for small
|
||||
items") and downstream folklore put anywhere from **2-7× over sqlite**,
|
||||
@@ -26,11 +26,11 @@ margin at equal semantics, the kv tier's engine pin (ADR-003) is again
|
||||
informed rather than hypothetical — and so is the anti-recommendation if
|
||||
it doesn't.
|
||||
|
||||
Same 2D instrument as POC #4 (the shape of the question, learned there):
|
||||
Same 2D instrument as POC #5 (the shape of the question, learned there):
|
||||
|
||||
1.Solo sweep (inherited `bench_small` harness, redb/sqlite/fs arms at
|
||||
1-256 KiB, identical working set) — where the 2-7× claim should show up.
|
||||
2. **Worker scale-out probe** (`redb_conc`, plus the POC #4 `sqlite_conc`
|
||||
2. **Worker scale-out probe** (`redb_conc`, plus the POC #5 `sqlite_conc`
|
||||
head-to-head) — redb is a single-file single-writer engine; does it
|
||||
degrade like sqlite's WAL or survive like postgres?
|
||||
|
||||
@@ -129,17 +129,17 @@ were never the bottleneck axis (sqlite was already the fastest get at
|
||||
adding threads only adds internal transaction-lock contention plus the
|
||||
None-durability freelist growth penalty (pages freed between durable
|
||||
commits accumulate until an Immediate commit collects them — redb's
|
||||
documented "rapid growth" behavior). Contrast POC #4: sqlite's
|
||||
documented "rapid growth" behavior). Contrast POC #5: sqlite's
|
||||
contentioned WAL plateaued ~1.2k, pg *scaled* to 28k+. redb's None mode
|
||||
is the only arm in three POCs where throughput goes **down** as workers
|
||||
go up.
|
||||
|
||||
### Finding C5 (the decisive one): redb write scale-out is perfectly flat — ~40-44 ops/s at 1, 2, 4, and 8 threads
|
||||
|
||||
`redb_conc` Immediate mode (same fresh-key discipline as POC #4's
|
||||
`redb_conc` Immediate mode (same fresh-key discipline as POC #5's
|
||||
`pgdiag6`/`sqlite_conc`):
|
||||
|
||||
| workers | redb Immediate | sqlite (same probe) | pg (from POC #4) |
|
||||
| workers | redb Immediate | sqlite (same probe) | pg (from POC #5) |
|
||||
|---------|---------------|---------------------|------------------|
|
||||
| 1 | ~42-44 ops/s | ~700-800/s | ~2.5-3.2k/s |
|
||||
| 2 | ~40 ops/s | ~1.0-1.1k/s | ~4.6-5.5k/s |
|
||||
@@ -168,7 +168,7 @@ go up.
|
||||
crash-consistent (unpersisted commits can leave the file failing to
|
||||
open until repaired — the freelist bookkeeping itself is lost on
|
||||
crash), `Immediate` pays a full fdatasync per commit. The
|
||||
durability-posture comparison that POC #4 could tune continuously
|
||||
durability-posture comparison that POC #5 could tune continuously
|
||||
(`synchronous_commit=off` on pg) doesn't exist here.
|
||||
- redb's `Database` is process-lifetime; concurrency modes
|
||||
(`ExclusiveWriter` default, `SingleWriter`, `MultiWriter`) only matter
|
||||
@@ -208,7 +208,7 @@ go up.
|
||||
- **Anti-recommendation adopted (ADR-007, 2026-10-02): redb stays
|
||||
ruled out** — recorded in the decision so the question does not
|
||||
reopen implicitly. SQLite's engine pin now has four-way triangulated
|
||||
evidence (POC #3 fs, POC #4/#5 pg, POC #6 redb, plus the lineage
|
||||
evidence (POC #3 fs, POC #5 pg, POC #6 redb, plus the lineage
|
||||
citation), and ADR-007 ships the kv tier as sqlite + postgres. If a
|
||||
future consumer wanted redb-shaped durability trade-offs (ephemeral
|
||||
cache tier), that's a *different* tier than the kv engine slot.
|
||||
@@ -222,7 +222,7 @@ go up.
|
||||
- clippy `-D warnings` clean, fmt clean.
|
||||
- **Honest caveats**: mdraid single dev box whose fdatasync latency
|
||||
ranged 0.24-30 ms during the session under co-resident load (pg
|
||||
container churn from POC #4 ran through the first bench; stopped
|
||||
container churn from POC #5 ran through the first bench; stopped
|
||||
before the final redbdiag re-run — relative conclusions unaffected,
|
||||
absolute redb-Immediate numbers are *device-pressure-dependent* and
|
||||
the honest statement is "1 fdatasync per commit, cost = device
|
||||
@@ -237,4 +237,4 @@ go up.
|
||||
`redbdiag2.rs` (None + periodic-flush group-commit emulation),
|
||||
`redbdiag3.rs` (commit-vs-insert split, per-put file growth),
|
||||
`redbdiag4.rs` (warm distribution), `redb_conc.rs` (scale-out, both
|
||||
durability modes), `sqlite_conc.rs` (copied head-to-head from #4).
|
||||
durability modes), `sqlite_conc.rs` (copied head-to-head from #5).
|
||||
Reference in new issue
Block a user