From 281c37876ed7738aeebd6317de50b6b1197fd185 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Sat, 3 Oct 2026 03:41:49 +0000 Subject: [PATCH] =?UTF-8?q?docs(architecture):=20ADR-008=20=E2=80=94=20tra?= =?UTF-8?q?it=20size=20probe,=20fleet=20GC,=20fs-tier=20engines;=20require?= =?UTF-8?q?ments=20anchor?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - requirements.md (new): the three use cases pinned as REQ-1..4 with the node/pool/fleet/engine vocabulary defined once — ends per-session re-derivation of consumer facts. REQ-2 (replicator fleet over one shared pg pool, incl. large-blob serving) is recorded as a planning fact predating all POCs, on the operator's authority. - ADR-008: Backend trait gains size(key) length probe (before any backend ships data — ADR-002 one-way-door discipline); fleet GC mechanism (DB-backed pin rows committed atomically with entries, TTL+renewal semantics, liveness = embedder-owned table, protect callback is single-node-only, advisory-locked single sweeper, staged re-arbitrated delete window on both engines); fs tier becomes engine-selectable (local default; pg-lo named candidate) with the fleet locality contract (shared media or re-routing; mixed tiers are a documented deployment invariant, not a constructor-provable one). - poc-pglo-spec.md (new): POC #7 spec — pg Large Objects as the fs tier's pg-lo engine; instruments, decision gate, registered in phase-0.md OQ-BL-06. - ADR-003/005 status amendments point to ADR-008's extensions; specs ripple (backends/store-api/gc/ops/overview/README). - open-questions.md: deferral-policy header gains the decisions-vs- sequenced-work distinction; pg-lo's why-not-parked audit trail recorded. - research fixes: postgres POC renumbered #4->#5 to the canonical register (phase-0 OQ-BL-06), redb cross-refs fixed, thinking- artifact sentence in B1 replaced with the honest reading. Verification: docs-only change; reference-integrity sweep across the tree (ADR/REQ/POC refs resolve); architecture-reviewer pass on the delta — original 3 criticals addressed, its follow-up (fleet liveness form, pin TTL, staged-delete semantics, enforceability, shared-media caveats) fixed in this commit. --- docs/architecture/README.md | 21 +- docs/architecture/backends-and-dispatch.md | 63 ++++- .../003-backend-contract-and-dispatch.md | 13 +- .../005-pooled-cas-and-gc-mechanism.md | 8 +- .../007-two-kv-engines-sqlite-and-postgres.md | 4 +- ...rait-size-probe-fleet-gc-and-fs-engines.md | 256 ++++++++++++++++++ docs/architecture/gc-and-namespaces.md | 29 +- docs/architecture/open-questions.md | 36 ++- docs/architecture/ops-surface.md | 9 +- docs/architecture/overview.md | 45 ++- docs/architecture/requirements.md | 103 +++++++ docs/architecture/store-api.md | 39 ++- docs/research/phase-0.md | 5 +- docs/research/poc-pglo-spec.md | 94 +++++++ docs/research/poc-postgres-kv-findings.md | 30 +- docs/research/poc-redb-kv-findings.md | 22 +- 16 files changed, 689 insertions(+), 88 deletions(-) create mode 100644 docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md create mode 100644 docs/architecture/requirements.md create mode 100644 docs/research/poc-pglo-spec.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 5b293ce..2eee107 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # alkblobs — Architecture @@ -36,12 +36,24 @@ there — not inherited as hedges: (POCs #3/#5/#6); the would-be "deployment trigger" framing was caught at review as circular hedging (a deployment running alkblobs cannot pre-exist the engine) and closed on the OQ-09 precedent. +- **ADR-008** (2026-10-03) repairs the seams ADR-007 stressed, against + the pinned consumer requirement REQ-2 (requirements.md — a replicator + node may run a **fleet** of store instances sharing one pg pool, a + planning fact predating all POCs): the `Backend` trait gains a + `size` probe (before any backend ships data); fleet GC gets + DB-backed pins, an advisory-locked single sweeper, and SQL-level + delete arbitration (single-node postures unchanged); and the fs tier + becomes engine-selectable (`local` default; `pg-lo` named candidate, + admission-gated on POC #7). It also adds + [requirements.md](requirements.md) — the REQ anchor that ends + per-session re-derivation of consumer facts. ## Architecture Documents | Document | Scope | Status | |---|---|---| | [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft | +| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, node/pool/fleet vocabulary | draft | | [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft | | [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft | | [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, shipped backends, size-threshold dispatch | draft | @@ -60,6 +72,7 @@ there — not inherited as hedges: | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted | | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted | ## Open Questions @@ -68,7 +81,11 @@ Tracked in [open-questions.md](open-questions.md). The Phase 0 register remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping, alkfs intake; carried for visibility, gating nothing here). OQ-09 (namespace-visibility default) resolved closed-by-default; OQ-10 -(second kv engine) resolved by ADR-007. **Deferral +(second kv engine) resolved by ADR-007. The pg-lo fs-engine question +resolved to *sequenced work*, not a parked question: REQ-2/ADR-008 +named it the candidate fs-tier engine and POC #7 +(`docs/research/poc-pglo-spec.md`) is its admission evidence — a task, +not a wait. **Deferral policy (the "Schrödinger's code" rule):** a *decision this crate needs before shipping* may not be deferred on a dependency that is itself waiting for this crate to exist. Two parking kinds remain legitimate: diff --git a/docs/architecture/backends-and-dispatch.md b/docs/architecture/backends-and-dispatch.md index b74dd63..7e689fc 100644 --- a/docs/architecture/backends-and-dispatch.md +++ b/docs/architecture/backends-and-dispatch.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # Backends and dispatch @@ -39,10 +39,16 @@ is ever reachable through that door. is also the persistence boundary — a kv row or an fs filename has no schema-migration story, so the byte layout must be self-describing (algorithm-tagged keys). -- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name`. - Streaming put/get shapes ride the store layer's seams (store-api.md); +- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` / + `size` (the length probe added by ADR-008). Streaming put/get shapes + ride the store layer's seams (store-api.md); backends provide byte- or handle-level primitives underneath (`get` on the fs tier yields a file handle, not a loaded buffer). + `size(key)` returns the entry's length or `None` if absent — + metadata only, never a content fetch: kv engines probe the indexed + `size` column both row shapes carry (POC #5 B5); the `local` fs + engine stats the file. Store-level `stat` and pool accounting ride + this method. - **`list()` complete by contract.** A *malformed* list is as deadly as an incomplete one: list correctness is only observable through GC (POC #1 finding 2 — the redb key-vs-value trap produced a sweep that @@ -122,18 +128,43 @@ sweep-outcome CI gate, run against dockerized postgres under near-linearly to ~37k puts/s at 24 conns — the only engine whose throughput *increases* under concurrency. -### fs backend (feature `fs`, default-on) +### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (candidate) -Large blobs. Flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}` -sharding survives from iroh's conclusion (limits directory size on -huge pools); stage-then-commit-rename for the two-pass unknown-length -path; pread-based range reads (POC #3 finding A2 — local range serving -is sound, e.g. for packfiles). +Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract, +multiple engines** — the medium is a per-node constructor choice +(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing +for fleets): + +- **`local` engine** (the default; the ADR-003 backend as originally + specced): flat sharded layout: + `{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's + conclusion (limits directory size on huge pools); stage-then-commit- + rename for the two-pass unknown-length path; pread-based range + reads (POC #3 finding A2 — local range serving is sound, e.g. for + packfiles). +- **`pg-lo` engine** (candidate, *not shipped*; ADR-008): postgres + Large Objects as the fs tier's storage — the same engine-behind- + one-trait move as ADR-007, one tier over. Admission is gated on + POC #7's measured evidence and its own engine ADR (see + `docs/research/poc-pglo-spec.md`). +- **Fleet locality contract (ADR-008):** over one shared pool, the fs + tier is either shared media (every node's `local` root on the same + fleet-shared media — soundness caveats and the required media + guarantees in ADR-008) or re-routed (one storage node serves pool + large-blob content via the ops surface; client nodes are + kv-tier-only over pool content). Mixed per-node-local fs tiers over + one pool is the partitioning failure this contract exists to + prevent — a documented deployment invariant the constructor cannot + fully prove but must be declared against (ADR-008 names the + enforceable seam and the detection symptom). ### mem backend (feature `mem`, default-off) `BTreeMap`-shaped ephemeral backend for tests and in-process ephemerality. A testing/utility tier, never a production story. +Implements the full trait contract including `size` (trivially) — it +exists partly as the reference impl every other backend's conformance +tests can mirror. ## Size-threshold dispatch (ADR-003) @@ -156,12 +187,13 @@ ephemerality. A testing/utility tier, never a production story. The trait is open to future implementations (network stores, S3-like tiers), but nothing in the current consumer set requires one, and the contract is deliberately hostile to half-implementations (complete -`list()`, GC-participating `delete`). Any future backend — or third kv -engine — is a new ADR carrying its own sweep-safety story (ADR-007 is -the precedent for an engine addition: measured evidence first, impl -contract naming the durability/ops posture deltas). This crate's -roadmap is not blocked on one (see open-questions.md — alkfs intake -may name needs externally; OQ-08). +`list()`, GC-participating `delete`). Any future backend — or a third +kv engine or second fs engine — is a new ADR carrying its own +sweep-safety story (ADR-007 is the precedent for a kv engine +addition; ADR-008 named pg-lo as the candidate second fs engine with +POC #7 as its admission evidence — the door is exercised twice). This +crate's roadmap is not blocked on one (see open-questions.md — alkfs +intake may name needs externally; OQ-08). ## Design Decisions @@ -171,6 +203,7 @@ may name needs externally; OQ-08). | [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption | | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only | | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) | ## Open Questions diff --git a/docs/architecture/decisions/003-backend-contract-and-dispatch.md b/docs/architecture/decisions/003-backend-contract-and-dispatch.md index ac0733c..195b10e 100644 --- a/docs/architecture/decisions/003-backend-contract-and-dispatch.md +++ b/docs/architecture/decisions/003-backend-contract-and-dispatch.md @@ -4,7 +4,11 @@ Accepted (engine pin amended by ADR-007 — the kv tier ships two engines, sqlite and postgres, behind this contract; the dispatch layer -and trait are unchanged) +and trait are unchanged. The trait was extended by ADR-008 with a +`size` length probe — before any backend ships data — and the fs +tier became engine-selectable (`local` default; pg-lo candidate), so +this ADR's "engines"/"backends" counts are read through ADR-004's +tier≠engine scope note) ## Context @@ -38,9 +42,10 @@ Evidence in hand (no further POC or consumer-waiting needed): generic-over-`K` alternative was considered and rejected in POC #1 finding 1: with one key type, generics buy nothing and break the persistence boundary.) -- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`. Streaming and - range shapes ride the store layer above (fs `get` yields a file - handle, not a loaded buffer). +- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`/`size` (the + length probe added by ADR-008 before any backend shipped data). + Streaming and range shapes ride the store layer above (fs `get` + yields a file handle, not a loaded buffer). - **`list()` complete by contract** — and *well-formed*: a malformed list (wrong destructure, values-as-keys) is as deadly as an incomplete one, and silent until a sweep deletes the wrong things diff --git a/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md b/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md index 94add42..348391b 100644 --- a/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md +++ b/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md @@ -2,7 +2,13 @@ ## Status -Accepted +Accepted (the sweep-safety protocol here is the single-node posture; +ADR-008 extends the mechanism to fleet topologies — DB-backed pins, +advisory-locked single sweeper, SQL-level delete arbitration under +shared-pool engines (REQ-2) — while keeping this ADR's invariants +and the in-process protocol unchanged everywhere else. The "one pool +per node" phrasing below reads as "per deployment" for fleets; the +vocabulary is pinned in requirements.md) ## Context diff --git a/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md b/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md index 85374ba..d149857 100644 --- a/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md +++ b/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md @@ -64,7 +64,9 @@ relational engine a per-node choice rather than a crate fork. 1. **sqlite stays the default engine**: zero-ops, no daemon, solo economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy - rides the same file (honker's WAL-NORMAL defaults are literally the + rides the same file (honker — the alk* family's SQLite + notify/queue extension, `/workspace/honker` — ships WAL + the + NORMAL-synchronous posture, so its defaults are literally the shipped tier). Single-machine nodes are sqlite by measurement, and remain so even at sustained rates well above observed push traffic (~1.2k/s WAL ceiling vs. single-node push rates an order below it). diff --git a/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md b/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md new file mode 100644 index 0000000..653f172 --- /dev/null +++ b/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md @@ -0,0 +1,256 @@ +# ADR-008: Trait length probe, fleet GC, and fs-tier engines + +## Status + +Accepted + +## Context + +Three defects surfaced in the post-ADR-007 architecture review, all +concentrated at the seams ADR-007 touched and all load-bearing for +REQ-2 (a replicator node may run multiple store instances — a fleet — +sharing one pool via postgres; requirements.md): + +1. **The `Backend` trait has no length probe.** The store surface + promises `stat(key) → Option` ("cheap length/type + probe"; store-api.md) and pool accounting rides `list` + `stat` + (ADR-005), but the trait's methods are `has/get/put/delete/list/ + name` with no value-metadata access. The kv engines' row shapes + both carry a `size` column (POC #5 B5), and the fs tier can stat a + file — but with no trait method, kv `stat` degenerates to a full + value load. The trait is the crate's second stable seam (ADR-003); + discovering this during implementation would be a breaking trait + change after backends exist. +2. **ADR-005's sweep-safety protocol is per-node by construction.** + Pins, the pin/liveness arbitration lock, and delete-window + arbitration all live inside one process (ADR-005 §Decision). That + is sound for sqlite and single-instance postgres (REQ-1, REQ-3 + topologies) — but under a fleet (REQ-2), node A's sweep enumerates + the *shared* pool and arbitrates against A's in-process pin map; + node B's in-flight put is invisible. Data loss, structurally: the + topology REQ-2 requires does not satisfy the protocol ADR-005 + specifies. +3. **The fs tier is node-local, silently partitioning a fleet.** The + fs backend stores to a sharded layout on the node's own filesystem + (ADR-003). Under a fleet sharing one kv pool, a >threshold blob + put via node A exists only on A's disk; node B's get fall-through + queries its own fs and misses, and `list()` unions disagree per + node. REQ-2 makes this topology a shipping requirement, not a + hypothetical. + +These three travel together: the fleet topology forces #2 and #3, and +#3's resolution constrains what the fs tier must support, while #1 is +a trait-level gap any topology hits. + +## Decision + +### 1. `size(key) → Option` joins the `Backend` trait + +The trait gains a length probe: `size(key)` returns the entry's byte +length or `None` if absent. Required of all engines; semantics per +tier: + +- **kv engines**: an indexed column probe (`SELECT size …WHERE key`), + never a value load. The `size` column both POC row shapes carry + becomes contract-relevant. +- **fs engine (`local`)**: stat/metadata read, no content fetch. +- The store facade's `stat` (store-api.md) and the accounting path + (ADR-005) ride this method; the `has` method remains a pure + existence probe. + +`size` is a trait addition *before* any backend ships data — the +one-way-door discipline of ADR-002/003 applied to the trait itself. + +### 2. Fleet GC: the delete window is durable and re-arbitrated in the database; pins are DB-backed under fleet engines + +The store facade's GC invariant is unchanged — **a visible pool entry +is never deleted while liveness protects it, and an in-flight put is +never deleted by a sweep started before it committed** (ADR-005). +Under a fleet, the mechanism moves into the shared engine: + +- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two + inserts in one transaction on the fleet engine: the entry row and + its pin row commit together (or neither does). The pin row carries + an owner (node id) and expiry: the pin guard renews its row every + TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor + parameters, not API). **Protection = an un-expired pin row**: at + delete-time arbitration, only `expiry > now()` pins count; an + expired pin is abandoned (its holder failed to renew — the same + liveness-failure signal as an in-process guard's drop) and is reaped + by the sweeper's maintenance step (the GC-of-GC cleanup). A node + that pauses past TTL loses its pins; delete-then-recover (ADR-005) + remains the correctness backstop — content is re-put byte-identically + under the same key. +- **Registered liveness sources have one fleet form: a liveness + table.** A fleet embedder registers its liveness as a named table of + `(key, …)` rows in the shared engine (schema pinned by the embedder, + shape contractually `(key bytea, …)` — any columns it needs beyond + the key). In-process callback *views* are not a fleet liveness form: + delete-time arbitration queries/finger-locks tables per key, so a + source must be a table for the SQL protocol to see it. Single-node + embedders keep callback sources unchanged (ADR-005). +- **The protect callback is single-node-only.** A fleet sweeper cannot + consult another node's in-process callback, so under fleet engines + the callback's fleet expression is: the embedder materializes its + protect set into its liveness table (or pin rows) *before* invoking + sweep — the callback's abort semantics remain for + single-node/topology-local sweeps that cannot express their + protection as rows. +- **One sweeper per pool.** A fleet node acquires a pg advisory lock + before sweeping and holds it for the whole sweep. Two nodes + sweeping concurrently is not a state this crate tolerates — the + lock is the admission gate, and a second-node sweep returns the + typed `GcAborted` (no deletion) rather than racing. `GcAborted` + thus carries two cause classes (protection-source failure — + ADR-005; sweeper lock contention — this ADR); the typed error's + fleet variant identifies which. +- **Delete-time arbitration is SQL-level.** The delete window's + re-check (ADR-005: "arbitrate under the pin/liveness lock at delete + time") executes as `SELECT … FOR UPDATE` against pin rows and + liveness tables at delete time, then batch-deletes. The **staged + re-arbitrated delete**: during mark, candidate-dead keys are staged + as rows carrying `visible_after = mark_end + window` (the delete + window's delay — default 5 min, constructor-tunable per + deployment); the executor (the sweeper, or the next sweep's owner) + re-arbitrates each staged candidate *at execution time* against + pins and liveness tables before deleting. This is the durable + delete window on every engine — the window table is engine rows on + sqlite as well; what differs by topology is only where the liveness + state lives (in-process maps and callbacks for single nodes; + tables and SQL arbitration for fleets), not whether staging exists. + No external queue dependency is required (the pattern is + engine-native transactions; consumer-side queues like honker or + pg-boss-lineage tools remain consumer-layer options, per ADR-004's + boundary). +- **Single-node postures are unchanged.** sqlite and single-instance + postgres keep ADR-005's in-process protocol verbatim: the pin map, + RAII guards, and arbitration lock stay in-process; only the + *persistence* of the delete window (the staged-candidate table) + is shared machinery. The pin-row expiry/ttl machinery exists only + where fleet engines are active. + +### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible + +The fs tier is a tier, not a medium. It ships engines the way the kv +tier does (ADR-007), and a deployment's topology determines which are +valid: + +- **`local` (default):** the sharded-layout filesystem backend, + unchanged (ADR-003). Valid for any single-node topology (REQ-1, + REQ-4) and for fleet nodes using **shared media** (below). +- **Fleet + shared media (deployment requirement):** multi-node pools + may keep the `local` engine by pointing every node's fs tier at + shared/replicated media. The soundness claim is *qualified*, not + assumed: immutable-once-committed files plus stage-then-commit- + rename are sound on media that provides (a) atomic rename within a + directory, (b) close-to-open consistency (a written+closed file is + complete to any later reader — NFSv3+ with the close-to-open + discipline; NFSv4 and EFS-class stores qualify), and (c) no + cross-node POSIX locking reliance (this store takes none — its + cross-node coordination is the fleet engine's locks, never file + locks; honker's NFS warning is about sqlite's lock protocol, not + this shape). Verifying (a)-(c) on the deployment's actual media is + the deployment's responsibility; a verification pass is recorded + ops work, not crate code. +- **Fleet re-routing (topology alternative):** a fleet whose nodes do + not share media designates **one storage node**; large-blob writes + and reads over the pool route through it via the ops surface + (verified fetch/put; ADR-001). Client nodes' constructors are + configured with a **kv-tier-only dispatch** (or an fs tier reserved + for node-private, consumer-owned content — which is outside the + pool and outside this crate's GC; it is the consumer's own storage + beside the pool per ADR-004, not pool content). The facade shape on + a client node is its own store for sub-threshold content plus its + registered ops-put path for large content — a consumer-side + composition, not a store-level mode. +- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects + as the fs tier's storage — the same "engine behind one contract" + move as ADR-007, one tier over. Named here so the door is explicit, + with its admission gated on measured evidence (POC #7: LO write/read + curves at the packfile regime, tx-scoped handle cost under + pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan + behavior) and its own engine ADR. **Not shipped speculatively;** + REQ-2's immediate fleet answers are shared media or re-routing; + pg-lo is the consolidation option if those are unacceptable. +- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at + the enforceable seam.** Cross-node configuration cannot be validated + by any one constructor (it sees only its own node). What the + constructor *can* enforce: a fleet-mode constructor (fleet engines + active) requires an explicit fs-tier declaration — `local` (with the + operator's assertion that its root is fleet-shared media), routed + (no local fs over pool content), or — when shipped — `pg-lo`. The + cross-node truth ("is node B's `local` actually the same media as + node A's?") is a deployment invariant this crate documents and ops + verifies, not something a constructor can prove; a partitioned + tier's observable signature (get fall-through misses for entries + another node wrote) is the documented detection symptom. +- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects + as the fs tier's storage — the same "engine behind one contract" + move as ADR-007, one tier over. Named here so the door is explicit, + with its admission gated on measured evidence (POC #7: LO write/read + curves at the packfile regime, tx-scoped handle cost under + pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan + behavior) and its own engine ADR. **Not shipped speculatively;** + REQ-2's immediate fleet answers are shared media or re-routing; + pg-lo is the consolidation option if those are unacceptable. + +In sum: a fleet's fs tier over pool content is either shared media (all +nodes' `local` roots on the same fleet-shared media) or routed (one +storage node); mixing per-node-local media over one pool is the +partitioning failure this decision exists to prevent. + +## Consequences + +**Positive** + +- The trait now expressibly supports `stat` and pool accounting on + every engine; the "cheap probe" claim of store-api.md is true rather + than aspirational. +- REQ-2 (fleet over one pool) has a complete, specced GC story: pins + move to the shared engine atomically with put-commits, one sweeper + holds the lock, deletes re-arbitrate in-transaction. The ADR-005 + invariant extends to fleets as a *stated* mechanism, not an + accident of single-process luck. +- The fs tier's fleet problem has named, shipped-code-free answers + (shared media, re-routing) and a named consolidation candidate + (pg-lo) whose admission path is the same POC-first door as ADR-007. + +**Negative** + +- The `Backend` trait gains a method before any backend ships — the + correct moment, but every engine's task list grows by one method. +- Fleet GC is real state machinery: pin rows with expiry (GC-of-GC + surface for orphaned pins at node crash — the expiry is the cleanup + handle), advisory-lock coordination, and SQL-level arbitration + added on top of the already-intricate delete-window state machine. +- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot + re-route pays for its ADR + POC before shipping large blobs — the + measured, deliberate deferral of the ADR-006 pattern, not a hedge. + +**Neutral** + +- The sqlite and single-node-postgres postures are unchanged; this ADR + costs existing paths nothing at runtime. +- `pg-lo`, if admitted, rides the same driver stack ADR-007 already + ships (tokio-postgres + deadpool) via SQL `lo_*` functions — no + new driver dependency (the `postgres_large_object` crate is a dead + 0.15-era io-trait glue; unnecessary). + +## References + +- [requirements.md](../requirements.md) — REQ-2 (the fleet fact this + ADR binds), REQ-1/3/4 (the topologies left unchanged) +- POC #5 (`poc-postgres-kv-findings.md` B5 — row shapes with `size` + columns; the driver stack pg-lo would ride) +- ADR-002 (the one-way-door discipline this trait change applies), + ADR-003 (the trait contract and admission gate `size` and pg-lo + amend/enter), ADR-004 (tier≠engine, extended to the fs tier), + ADR-005 (the GC invariant and single-node protocol extended here, + not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission + follows), ADR-007 (the engine-behind-one-trait precedent) +- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait + and tier surface this amends); + [store-api.md](../store-api.md) (`stat`'s contract); + [gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC + mechanism); [ops-surface.md](../ops-surface.md) (the re-routing + topology's transport) \ No newline at end of file diff --git a/docs/architecture/gc-and-namespaces.md b/docs/architecture/gc-and-namespaces.md index 5a589bb..10d78ac 100644 --- a/docs/architecture/gc-and-namespaces.md +++ b/docs/architecture/gc-and-namespaces.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # Pooling, namespaces, and GC @@ -16,11 +16,17 @@ price, and this document specifies the mechanism the POCs validated ## The pooled CAS -- **One pool per node.** Repos, workspaces, and appfile stores are all +- **One pool per deployment; a node is one store instance** (the + engine/vocabulary definitions live in requirements.md). Repos, + workspaces, and appfile stores are all *sets of hash references* over the same CAS — never walled-off per-repo stores. Git's `alternates`/object-pool mechanism (GitLab object pools) is the same idea done awkwardly; the pool generalizes it - to unrelated repos and to p2p replication. + to unrelated repos and to p2p replication. **A fleet (REQ-2) — two or + more nodes sharing one pool — is the one topology where the GC + mechanism below extends beyond a node: ADR-008 pins, sweeper + locking, and delete arbitration move into the shared engine; + the invariants are unchanged.** - **One address space.** Under the canonical hash (ADR-002), the same file content has the same address whether it entered via a git oid or a workspace manifest — cross-consumer dedup is free by construction, @@ -116,16 +122,19 @@ existence flips atomically per key). | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling | | [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate | | [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere | ## Open Questions -None owned by this document. The p2p replicator's cross-node GC -coordination is a property of the replicator policy layer (above the -crate), governed by the same seam — it raises a question here only -when a consumer *names a requirement* (OQ-07/OQ-08 intake), which is -re-entry via a new decision at a new ADR, not a parked question: no -decision here waits on a consumer, and a future requirement does not -create one retroactively. +None owned by this document. Shared-pool GC coordination within a +deployment is now in-crate (fleet mechanism: ADR-008). What remains a +replicator-policy-layer property (above the crate) is *cross-pool* +coordination — gossip sync between separately-pooled replicator nodes +deciding what each keeps — governed by the same seam; it raises a +question here only when a consumer *names a requirement* +(OQ-07/OQ-08 intake), which is re-entry via a new decision at a new +ADR, not a parked question: no decision here waits on a consumer, and +a future requirement does not create one retroactively. ## References diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index ee39081..5e31313 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # Open Questions @@ -31,6 +31,19 @@ make before shipping. Phase 0's residual list was swept under this rule: every residual either resolved into an ADR (the evidence was already in hand) or re-owned as below. +**The line between the two kinds** (added 2026-10-03, after the +pg-lo round): deferring on a consumer's *design decision* is valid — +the deciding fact is obtainable independently of this crate (OQ-07, +OQ-08). Deferring on a consumer's *deployment of this crate* is +invalid — that fact is created by this crate (OQ-10's original +framing, OQ-09's original framing). And a third thing is neither: +**decided-but-sequenced work**. Once a question resolves to a decision +(the pg-lo engine is the named fs-tier candidate under REQ-2/ADR-008), +what remains is the evidence pipeline the decision's own admission +gate requires (POC #7) — a task with an owner, not a parked question, +and not a hedge. A decision is not "unmade" because its verification +is pending; deferral policy applies to decisions, not to work. + **Index of active OQs:** OQ-07, OQ-08 (externally-owned, carried for visibility). OQ-09 and OQ-10 are resolved (recorded below). Promoted Phase 0 questions OQ-BL-01..06 are recorded here with their @@ -205,9 +218,12 @@ resolutions for traceability. - **Origin**: `docs/research/phase-0.md` - **Status**: resolved (complete: #1, #3 passed; #2 absorbed; #4 covered in miniature, concurrency half specified as architecture in - ADR-005) -- **Resolution**: register complete; Phase 0 ended; evidence trail in - `docs/research/`. + ADR-005; post-convergence additions #5 postgres and #6 redb passed + and fed ADR-007; #7 pg-lo specified — `docs/research/poc-pglo-spec.md`, + requested by REQ-2/ADR-008) +- **Resolution**: register complete and extended post-convergence; + Phase 0 ended; evidence trail in `docs/research/`. The register's + canonical numbering lives in phase-0.md OQ-BL-06. - **Cross-references**: all ADRs above --- @@ -229,6 +245,18 @@ a deployment trigger — the same invalid shape, caught at review: the trigger was a fact only this crate could create. It resolved to ADR-007 (two kv engines; see its entry above). +The pg-lo fs-engine question (raised in the ADR-008 review round, +2026-10-03) never became a parked question: REQ-2 made the +multi-instance-with-large-blobs topology a standing requirement +(requirements.md — a planning fact predating all POCs), so there was +nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the +candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`) +as its admission evidence — sequenced work, per the deferral-policy +header's third category. An earlier draft of that OQ parked on +"alkgit's architecture process names the topology" — rendered moot the +same day by the REQ-2 fact; recorded here because the audit trail of +*why not parked* is part of this document's job. + OQ-07/OQ-08 are owned by other repos' processes and are not alkblobs tracker tasks (their outcome arrives *through* their owners, not through any artifact this repo creates). \ No newline at end of file diff --git a/docs/architecture/ops-surface.md b/docs/architecture/ops-surface.md index 2641829..245ae58 100644 --- a/docs/architecture/ops-surface.md +++ b/docs/architecture/ops-surface.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # Ops surface (alkcall-backed network operations) @@ -142,7 +142,11 @@ side channel: fetch handlers call `get`/`read_range`/`stat`; put handlers call `put`; both honor pins and delete windows. There is no ops-internal path around the store facade (ADR-001 §Consequences — this is what makes "re-homing the ops surface later loses nothing" -true). +true). One fleet topology additionally rides this module: **re-routing +(ADR-008)** — a fleet whose nodes do not share fs media routes +large-blob access through a single storage node via these ops, with +client nodes' fs tiers serving only node-private content. The module +supports this shape without change; no fleet-specific op exists. ## Design Decisions @@ -150,6 +154,7 @@ true). |---|---|---| | [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops | ## Open Questions diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 5ad67dd..40a06b1 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-02 +last_updated: 2026-10-03 --- # Overview @@ -10,9 +10,15 @@ last_updated: 2026-10-02 `alkblobs` is content-addressed blob storage in the alk* family: a put/get/verify store keyed under one canonical content hash (the git-family derivation; legacy SHA-1-tolerant), with pluggable backends -(small blobs in a kv store, large blobs on a filesystem fallback), split -from any wire/ops-protocol layer. Network ops ride the alkcall substrate -behind a feature-gated `ops` module (ADR-001). +(small blobs in a kv store, large blobs on a filesystem fallback), +split from any wire/ops-protocol layer. Network ops ride the alkcall +substrate behind a feature-gated `ops` module (ADR-001). + +Deployment-relevant consumer facts — what each use case forces on this +crate — are pinned in [requirements.md](requirements.md) (REQ-1..4, +with the **node/pool/fleet/engine** vocabulary defined there). This +tree references REQs by ID; per-session re-derivation from vision prose +is exactly what that document exists to prevent. ## Why it exists — the extracted shared core @@ -60,10 +66,11 @@ or three times. │ size-threshold dispatch (ADR-003) │ └───────┬───────────────────┬───────────────────────────────┘ ▼ ▼ - kv backend fs backend (ADR-003: exactly two tiers; - │ ADR-007: kv = 2 engines) - ▼ - engine per node: sqlite (default) | postgres (feature "postgres") +│ kv backend fs backend (ADR-003: exactly two tiers; +│ │ ADR-007: kv = 2 engines; +│ ▼ ADR-008: fs engines too) +│ engine per node: sqlite (default) | postgres (feature "postgres") +│ fs engine per node: local (default) | pg-lo (candidate, POC #7) transport: none, anywhere. alkgit/alkfs or the embedder dials; an alkcall Connection is handed in at the ops boundary. @@ -74,15 +81,22 @@ or three times. | Layer | Dependencies | Notes | |---|---|---| | base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks | -| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | same `Backend` contract, own sweep-safety CI gate under `--all-features` | +| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` | | `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 | | consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs | Backends are feature-gated; both `kv` and `fs` ship default-on -(batteries-included, the alkgit feature-model pattern); the `kv` tier's -postgres engine (`postgres`, ADR-007) and the `ops` feature are -default-off. Wasm: the backends are not a wasm story; a wasm client is a -consumer of the `ops` surface (which rides alkcall, itself +(batteries-included, the alkgit feature-model pattern — default-on +features, opt-out rather than opt-in); the `kv` tier's postgres engine +(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only +kv node (postgres engine without rusqlite) is **not** a supported +configuration this crate optimizes for — feature-graph work for it is +deferred until a named requirement exists (the base `kv` feature +pulling rusqlite is the accepted v1 shape; convention 8's +lean-base principle applies to *default* builds, which stay lean since +`kv` is what the feature model controls, not what default-on means +downstream). Wasm: the backends are not a wasm story; a wasm client is +a consumer of the `ops` surface (which rides alkcall, itself wasm-clean), never an in-process embedder (ADR-001 §Consequences). ## Non-goals (each enforced by an ADR) @@ -111,6 +125,7 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences). | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping | | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable | ## Open Questions @@ -125,6 +140,10 @@ crate level: open-read requirement) - **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres shipped behind one trait; per-node constructor choice) +- **POC #7** (`docs/research/poc-pglo-spec.md`): pg Large Objects as + the fs tier's `pg-lo` engine — spec'd **sequenced work** (not a + parked question; open-questions.md's policy header distinguishes the + two): admission evidence for ADR-008's named candidate (REQ-2) ## References diff --git a/docs/architecture/requirements.md b/docs/architecture/requirements.md new file mode 100644 index 0000000..db49cfc --- /dev/null +++ b/docs/architecture/requirements.md @@ -0,0 +1,103 @@ +--- +status: draft +last_updated: 2026-10-03 +--- + +# Requirements + +The deployment-relevant facts each use case forces on this crate. +Specs and ADRs reference these by ID (REQ-n); a future session must be +able to read this document and never re-derive a requirement from +conversation or vision prose. **This document is the anchor: when a +consumer fact changes, it changes here first, with a dated note, and +the ADRs re-derive from it.** + +## Vocabulary (defined once; used everywhere) + +- **Engine** — a concrete storage implementation of one tier's + contract: the kv tier's engines are sqlite and postgres (ADR-007); + the fs tier's engines are `local` (default) and `pg-lo` (candidate — + see REQ-2, ADR-008). +- **Pool** — the flat content-addressed namespace of one deployment + (`hash → bytes` everywhere; ADR-005). +- **Node** — one running store instance (one process embedder; its fs + tier is its own media). Node is the GC arbitration domain for + single-node topologies (ADR-005's pins, liveness locks, and delete + windows live inside a node); under a fleet, arbitration moves into + the shared engine (ADR-008). +- **Fleet** — two or more nodes over **one shared pool** (e.g. via one + postgres instance). A fleet is the only topology in which + cross-node GC and fs-locality questions exist (ADR-008). +- **Deployment** — one operator's whole system: typically one fleet + (multi-node) or one node; consumer tables, op registrations, and + sweep cadence are deployment choices. + +## The use cases and their requirements + +### UC-1: Self-hosted git serving (gitea-replacement scale) + +Git repos for a user/team on one machine. Small-object regime dominated +by loose objects in the kv tier; packs and large artifacts (git-lfs +shape) on the fs tier. + +- **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics: + sqlite is the default kv engine; single-writer WAL ceiling (~1.2k + objects/s under contention) is comfortably above single-node push + rates. **Zero-ops is a requirement**: no daemon beyond the node + process itself. + +### UC-2: Distributed git with replicator nodes hosting repos for many users + +The p2p/OSS sketch (phase-0 §p2p shape): replicators are push/clone +endpoints serving many concurrent users; gossip-style sync between +replicators; ACL/ownership above (alkcall layer). + +- **REQ-2 (traces: ADR-007, ADR-008)** — **a replicator node may run + multiple store instances (a fleet) sharing one pool** via a shared + postgres kv engine, including serving large (>threshold) pool + content from that fleet. This is stated as a requirement, dated + 2026-10-03 (recording a planning fact from the use case's origin + discussion; it predates all POCs and architecture docs, recorded + here on the operator's authority as the use case's stated shape). + Consequences are derived in ADR-008: fleet GC (DB-backed pins, + advisory-locked sweeper, SQL delete arbitration), the fs-tier + engine concept with its locality contract, and the pg-lo candidate + engine (POC #7). +- **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out: + the fleet's hub-class nodes need put throughput that *increases* + under concurrency (pg: near-linear to ~37k puts/s at 24 conns); + sqlite's WAL ceiling (~1.2k/s, latency-climbs-unbounded) bounds + single-node sustained write pressure regardless of client count. + This is the measured basis for the per-node engine split. + +### UC-3: Agent workspaces (many agents, near-identical content) + +Agents working simultaneously in mostly-identical workspaces, each +making small edits; a shared pool dedups by construction; per-workspace +path→hash manifests above. + +- **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space + across workspaces; dedup must be structural (not configured). Small + edits mean the small-blob regime dominates: sqlite-class kv + performance is what matters; the fs tier is hit rarely (large + workspace artifacts). + +## What this document is not + +- **Not consumer architecture**: how alkgit maps refs/manifests onto + the store remains OQ-07 (externally-owned); how alkfs shapes + workspaces remains OQ-08. This document records what each use case + *forces on this crate*, not what each consumer *is*. +- **Not a requirements-change process substitute**: a new or changed + REQ arrives here with a dated note and owner; ADRs that depended on + the old fact state their reopen conditions and reopen when this + file changes. + +## Decision trace + +| REQ | Introduced | Source | ADRs | +|---|---|---|---| +| REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 | +| REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 | +| REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 | +| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-005 | \ No newline at end of file diff --git a/docs/architecture/store-api.md b/docs/architecture/store-api.md index ea9820c..83ce281 100644 --- a/docs/architecture/store-api.md +++ b/docs/architecture/store-api.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-01 +last_updated: 2026-10-03 --- # Store API @@ -11,7 +11,9 @@ The store core's public surface: the typed facade above the backends that consumers (alkgit, alkfs, the ops module) program against. It owns typed keys and hashing (ADR-002), size-threshold dispatch (ADR-003), the put-pinning seam, and the liveness/sweep seam (ADR-005) — and -nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). +nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). The +facade is **engine-blind** (ADR-007/008): which kv/fs engine a node +runs changes nothing here except fleet-GC mechanism wiring (ADR-008). ## Public surface @@ -57,7 +59,9 @@ nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). primitive and the packfile-serving surface rides this). `EntryMeta` carries `len: u64` and the key's algorithm (the "type" from the store's perspective is the hash algorithm — there is no other type at - this layer). + this layer). `stat` rides the `Backend` trait's `size` method + (ADR-008): an indexed-column probe on kv engines, a file stat on the + fs `local` engine — never a value/content load. - **`read_range(key, range) → (slice, slice_digest)`** — slice plus an out-of-band slice digest: SHA-256 over the slice bytes, fixed by convention so callers and ops handlers agree without negotiation @@ -85,7 +89,12 @@ nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). (finding 7) is why the verb/name is pinned here). - **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own the cadence (ADR-005, no ambient timers). A sweep with no registered - liveness sources aborts without deleting — the safe default. + liveness sources aborts without deleting — the safe default. Under + a fleet (multiple nodes over one pool — requirements.md REQ-2), + sweep/put/delete semantics are mechanism-extended by ADR-008: pins + are DB-backed rows committed atomically with entries, one sweeper + holds the advisory lock, and delete arbitration is SQL-level — the + facade's invariant text is unchanged. ## Error model @@ -96,7 +105,9 @@ tests (AGENTS convention 2). Distinguished failure families: - `Verification` — put/get hash-check failure (content ≠ key) - `Io(String)` — backend media failure (stringly because `std::io::Error` is not stable across versions) -- `GcAborted` — a protection source failed; nothing deleted (ADR-005) +- `GcAborted` — nothing deleted; sweep not run. Two cause classes + (ADR-005: a protection source failed; ADR-008: another node holds + the fleet sweeper lock) — the typed error distinguishes them. - `KeyInvalid` — malformed key bytes at the boundary (hashing doc) Virgin-store semantics: read paths on a fresh store see absent/empty, @@ -117,18 +128,27 @@ lesson generalizes to any kv engine). ## Invariants (the test gate) 1. Whole-blob put/get round-trips byte-identical under the canonical - derivation, both tiers, both put paths (interop with real git - remains the source of truth, per POC #1's lesson about hardcoded - vectors). + derivation, both tiers, both put paths, **and every shipped engine + of both tiers** (kv: sqlite + postgres; fs: `local`; ADR-007/008) + (interop with real git remains the source of truth, per POC #1's + lesson about hardcoded vectors). 2. Dedup: putting identical content twice (same or different path) is one pool entry. 3. Stage hygiene: any failure mid-put leaves zero stage files; any success leaves exactly one committed entry. 4. Sweep safety: with correct liveness registered, sweep counts are - exact; with aborting sources, sweep deletes nothing. + exact; with aborting sources, sweep deletes nothing. **Engine-gated + form (ADR-008):** single-node postures assert this under the + in-process protocol; fleet postures (shared-pool engines) assert it + under DB-backed pins + the advisory-locked sweep, with the + exact-count sweep-outcome shape run against dockerized postgres + under `--all-features`. 5. `list()` correctness is observable only through GC — list-related tests assert through sweep outcomes (POC #1 finding 2's lesson codified). +6. `size`/`stat` on any engine returns the entry length without + fetching content (asserted per-backend: kv probe, fs stat — + ADR-008). ## Design Decisions @@ -138,6 +158,7 @@ lesson generalizes to any kv engine). | [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering | | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band | +| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed | ## Open Questions diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md index 9be0a11..6a92b3e 100644 --- a/docs/research/phase-0.md +++ b/docs/research/phase-0.md @@ -1,8 +1,6 @@ --- status: converged -last_updated: 2026-10-02 (Phase 0 convergence — all POCs passed, OQs -settled into decisions + residuals; phase-0 plan complete; -OQ-BL-01 ops-surface substrate settled: alkcall) +last_updated: 2026-10-03 (POC #7 spec added to the register OQ-BL-06; Phase 0 remains closed) --- # alkblobs — Phase 0 (Exploration) @@ -595,6 +593,7 @@ POC #3 pack analysis):** | 4 | Pooled CAS + GC: namespace reference tables over the flat pool; mark-and-sweep with protect-callback + TempTag pinning; delete-then-recover semantics | **Covered in miniature by #1** — namespace tables + sweep + recover validated single-threaded; the concurrency half is Phase 1 implementation work, not a POC gate | `poc-trait-dispatch-findings.md` findings 3/7/8 | | 5 | Postgres as the kv engine (added post-convergence): inherited sqlite/fs/pg benchmark arms + write/read concurrency scale-out probes | **Passed 2026-10-02** (6 findings B1-B6: single-conn pg floor ~1 ms fsync-dominated, ~40× storage overhead; pg PUT scale-out ~12× at 12 conns / ~37k ops/s vs sqlite's ~1.2k WAL-serialized ceiling) | `poc-postgres-kv-findings.md`; code: `/workspace/alkblobs-postgres-poc` | | 6 | redb as the kv engine (added post-convergence, same standing as #5): inherited sqlite/fs arms + redb durability decomposition + scale-out probe | **Passed 2026-10-02** (6 findings C1-C6: the "2-7× over sqlite" claim inverted — sqlite ~430× over redb at crash-consistent puts; redb Immediate = 1 fdatasync/commit, 8-30 ms on this disk; reads ~530k/s but irrelevant; write scale-out flat ~42/s; ruled out at a durability-tier mismatch, not a benchmark quibble) | `poc-redb-kv-findings.md`; code: `/workspace/alkblobs-redb-poc` | +| 7 | Postgres Large Objects as the fs tier's `pg-lo` engine (added 2026-10-03, per ADR-008 naming it the candidate fs-tier engine for REQ-2 fleets): LO write/read curves at the packfile regime, tx-scoped handle cost under pooling, pg_largeobject/vacuum posture under churn, crash-orphan behavior | **Specified, not run** — `poc-pglo-spec.md`; admission evidence for the pg-lo engine ADR | `poc-pglo-spec.md` | Sequencing outcome: #1 and #3 passed; #2 absorbed/validated under #1; #4's single-threaded core covered by #1 (the mechanism choice it diff --git a/docs/research/poc-pglo-spec.md b/docs/research/poc-pglo-spec.md new file mode 100644 index 0000000..79c6da4 --- /dev/null +++ b/docs/research/poc-pglo-spec.md @@ -0,0 +1,94 @@ +--- +status: specified +title: "POC #7 — postgres Large Objects as the fs tier's pg-lo engine" +last_updated: 2026-10-03 +--- + +# POC: postgres Large Objects as the fs tier's pg-lo engine — spec + +> **POC register #7**, added post-Phase-0 per the ADR-003/007 +> substitution door (the third use of it, after postgres-on-kv and +> redb-on-kv). Code: standalone crate +> `/workspace/alkblobs-pglo-poc` (clone the POC #5 crate's harness +> conventions; add a `PgLoBackend` implementing the ADR-008 trait +> contract including `size`). Findings land here regardless, per the +> established convention. Requested by REQ-2 (requirements.md — the +> fleet-with-large-blobs consolidation option; ADR-008 names pg-lo as +> the candidate fs-tier engine). Status: **specified, not run**. + +## What this POC must decide + +The ADR-008 question: can postgres Large Objects hold the fs tier's +contract for the fleet topology — serving the ≥128 KiB regime +(packfiles, large artifacts) from the same engine the kv tier already +rides — at acceptable performance and with an honest ops posture? The +fs tier's contract (ADR-003/008): `get` yields a pread-able handle, +stage-then-commit semantics for unknown-length puts, complete +`list()`, GC-participating `delete`, `size` without content fetch. + +Large Objects' native shape differs from files in one structural way +the ADR names: **LO descriptors are transaction-scoped** (lo_open → +loread/lowrite must run inside a tx). The POC must measure what that +does to the handle pattern under a connection pool, not assume it away. + +## Instruments + +1. **Inherited `bench_small`-shaped harness, LO arm** — the POC #3/#5 + methodology at fs-tier-relevant sizes: **128 KiB, 256 KiB, 1 MiB, + 16 MiB, 128 MiB**, sequential and random-replay, p50/p99. Compare + against: local fs (the `local` engine baseline, same harness) and + kv-tier pg (bytea) at the lower sizes, as cross-checks of + methodology parity. (`loread`/`lowrite` via SQL functions through + tokio-postgres — the `postgres_large_object` crate targets + postgres 0.15 and is dead; do not use it.) +2. **Handle-cost probe** (`pgdiag7`): the tx-scoped descriptor under + deadpool — cost of open-tx → lo_open → lseek/loread per range read, + held vs. re-opened per read; pool pressure under N concurrent + readers (the ops fetch handler's real shape). +3. **Write path probe** (`pgdiag8`): lo_create + chunked lowrite + (the LOBLKSIZE ~2 KiB page layout matters here — measure write + amplification as value size grows); unknown-length put shape + (stage into one LO, companion row on commit); verify + `size`-without-fetch (`SELECT size` from the companion table). +4. **Churn/vacuum probe**: repeated put/delete cycles at 1–16 MiB; + measure `pg_largeobject` catalog growth, TOAST/vacuum behavior, + autovacuum posture (the fs engine inherits ADR-008/007's + maintenance-surface honesty requirement). +5. **Crash/orphan probe**: kill client mid-write-tx; verify no orphan + LO and no half-commit; kill after lo_create before commit; write + the orphan-recovery sweep story the engine ADR will need (compare + `lo_get`-visible OIDs against the companion table — never trust + the catalogs). + +## Decision gate (mirrors ADR-007's) + +pg-lo is admitted as a shipped fs-tier engine iff: + +- **Performance:** LO put/get at 128 KiB–16 MiB is within an + order-of-magnitude of local fs on the same box (the tier is chosen + for fleet correctness, not solo speed — but a 10× gap would put the + fleet's packfile serving materially behind a single-node fs + deployment, and the ADR must say so if measured); +- **Contract:** complete `list()` (companion table), GC-participating + delete (`lo_unlink` under the delete window), virgin-store no-op + reads, `size` probe — all hold, proven via the exact-count + sweep-outcome test shape under `--all-features`; +- **Ops posture:** named deltas (tx-scoped handles, autovacuum/vacuum + on `pg_largeobject`, orphan recovery sweep) with measured or + documented-bounded costs. + +Findings feed an engine ADR in the ADR-007 shape (measured curves, +posture deltas as requirements). If the gate fails, ADR-008's +recorded statement stands: shared-media or re-routing are REQ-2's +fleet answers, and pg-lo's named cost was paid as evidence, not +shipped blind. + +## Register note (phase-0.md OQ-BL-06 convention) + +The canonical register is phase-0.md OQ-BL-06: #1 trait dispatch, +#2 absorbed, #3 large-blob, #4 pooled-CAS (covered by #1), #5 +postgres, #6 redb. The post-convergence POC files briefly carried +inconsistent self-numberings (the postgres file titled itself "#4"); +they are renumbered to the register. This POC is **#7** — specified +2026-10-03, requested by REQ-2 (requirements.md; ADR-008 names pg-lo +as the candidate fs-tier engine). \ No newline at end of file diff --git a/docs/research/poc-postgres-kv-findings.md b/docs/research/poc-postgres-kv-findings.md index 5d7fe18..7ba4dcf 100644 --- a/docs/research/poc-postgres-kv-findings.md +++ b/docs/research/poc-postgres-kv-findings.md @@ -1,14 +1,17 @@ --- status: passed -title: "POC #4 — postgres as the kv engine: single-conn penalty measured, concurrency scale-out measured" -last_updated: 2026-10-02 +title: "POC #5 — postgres as the kv engine: single-conn penalty measured, concurrency scale-out measured" +last_updated: 2026-10-03 --- # POC: postgres as the kv engine — findings -> **POC register #4**, added after Phase 0 convergence (the register was +> **POC register #5**, added after Phase 0 convergence (the register was > complete at #3; this is the ADR-003 "substitution seam" input arriving -> early). Code: standalone crate `/workspace/alkblobs-postgres-poc` +> early). *Renumbered 2026-10-03 from a self-assigned "#4" to the +> canonical register (phase-0.md OQ-BL-06), where #4 is the pooled-CAS +> POC covered by #1; the redb file's cross-references were fixed in the +> same pass.)* Code: standalone crate `/workspace/alkblobs-postgres-poc` > (a copy of the POC #3 crate with a `PgKv` backend added; findings land > here regardless, per the established convention). Date: 2026-10-02. > Status: **passed** — the inherited POC #3 test suite (14 tests, byte-exact @@ -94,13 +97,14 @@ Readings: floor is 2-5× fs's ~200-450 µs syscall chain (and 10-50× sqlite). POC #3's fs numbers reproduce within noise, confirming methodology parity. -- **At 64-128 KiB postgres pulls ahead of sqlite** (~800-1000 vs - ~2050-4100 — actually *behind* sqlite, but ahead of fs on gets and - comparable on puts) — wait, no: at 64 KiB sqlite still wins - (~4.1k). The honest reading: pg crosses fs somewhere in the 16-64 KiB - band for puts and is ~at-parity through 64-128 KiB. **The pg-vs-fs - crossover is real but sits an order of magnitude higher than - sqlite-vs-fs.** +- **At 64-128 KiB postgres sits between fs and sqlite** (~800-1000 vs + fs ~2300-2900 on gets — *behind* fs on some arms; sqlite still leads + ~4.1k at 64 KiB). The honest reading: pg holds no decisive win in + the 1-128 KiB single-worker band on this disk; **the pg-vs-fs + crossover, where it exists at all, sits higher than sqlite-vs-fs's**. + Note: pg's decisive advantage was never the solo 1-D curve — it is + the concurrency axis (B3/B4), where it multiplies and sqlite + serializes. - The sqlite numbers reproduce A4 within noise (44k vs 48k at 1 KiB), so the methodology transfer is sound. @@ -252,7 +256,7 @@ a deployment running the *fs tier anyway* has none of either cost. list()-complete-by-contract test would need the vacuum-aware equivalent of POC #1's exact-count sweep assertions. -## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #4/#6) +## Engine deployment posture (recorded 2026-10-02 — the two-engine story, post-POC #5/#6) The three alkblobs-relevant git/vfs use cases, mapped onto the measured curves; this was the conclusion OQ-10 carried before its resolution @@ -262,7 +266,7 @@ in ADR-007): |---|---|---| | **sqlite** | self-hosted git serving (gitea-replacement scale), agent-workspace pools, small/personal replicators | 44k solo puts/s at 1-16 KiB (POC #3 A4, reproduced here); WAL single-writer ceiling ~1.2k/s under contention is comfortably above single-node push rates; zero-ops, no daemon; honker rides the same file (same WAL-NORMAL posture, its defaults are literally the shipped tier) | | **postgres** | public multi-tenant replicator nodes (many users pushing, gossip catch-up, bulk ingest), consolidated-ops deployments (already-running pg) | near-linear put scale-out to ~21k at 4 conns, ~28k at 8, ~32k at 12, ~37k at 24-48 (B3); reads scale to ~44k at 24 conns (B4); the only engine whose throughput *increases* under concurrency | -| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #4/#6 reproductions | +| fs (tier, not engine) | ≥128-256 KiB content, packfiles | crossover measured (POC #3 A4), stable across #5/#6 reproductions | The decision rule the two POCs establish: **the trigger is topology, not throughput** — cross-MACHINE write access to one pool makes postgres diff --git a/docs/research/poc-redb-kv-findings.md b/docs/research/poc-redb-kv-findings.md index e77ee4e..332428e 100644 --- a/docs/research/poc-redb-kv-findings.md +++ b/docs/research/poc-redb-kv-findings.md @@ -11,13 +11,13 @@ last_updated: 2026-10-02 > crate `/workspace/alkblobs-redb-poc` — a copy of the POC crate with the > pg arm replaced by a `RedbKv` backend (`redb` 4.3, the engine iroh-blobs' > fs store uses for its metadata tables). The inherited sqlite/fs arms and -> the 14-test suite are byte-identical to POC #3/#4's methodology. +> the 14-test suite are byte-identical to POC #3/#5's methodology. > Date: 2026-10-02. Status: **passed** (14 tests, clippy `-D warnings` > clean, fmt clean). ## What this POC set out to decide -The "opposite route" from POC #4: postgres is complicated, we only need a +The "opposite route" from POC #5: postgres is complicated, we only need a kv — what about a high-performance *embedded* kv like redb, which the DESIGN.md lineage (iroh-blobs' "faster than the filesystem for small items") and downstream folklore put anywhere from **2-7× over sqlite**, @@ -26,11 +26,11 @@ margin at equal semantics, the kv tier's engine pin (ADR-003) is again informed rather than hypothetical — and so is the anti-recommendation if it doesn't. -Same 2D instrument as POC #4 (the shape of the question, learned there): +Same 2D instrument as POC #5 (the shape of the question, learned there): 1.Solo sweep (inherited `bench_small` harness, redb/sqlite/fs arms at 1-256 KiB, identical working set) — where the 2-7× claim should show up. -2. **Worker scale-out probe** (`redb_conc`, plus the POC #4 `sqlite_conc` +2. **Worker scale-out probe** (`redb_conc`, plus the POC #5 `sqlite_conc` head-to-head) — redb is a single-file single-writer engine; does it degrade like sqlite's WAL or survive like postgres? @@ -129,17 +129,17 @@ were never the bottleneck axis (sqlite was already the fastest get at adding threads only adds internal transaction-lock contention plus the None-durability freelist growth penalty (pages freed between durable commits accumulate until an Immediate commit collects them — redb's -documented "rapid growth" behavior). Contrast POC #4: sqlite's +documented "rapid growth" behavior). Contrast POC #5: sqlite's contentioned WAL plateaued ~1.2k, pg *scaled* to 28k+. redb's None mode is the only arm in three POCs where throughput goes **down** as workers go up. ### Finding C5 (the decisive one): redb write scale-out is perfectly flat — ~40-44 ops/s at 1, 2, 4, and 8 threads -`redb_conc` Immediate mode (same fresh-key discipline as POC #4's +`redb_conc` Immediate mode (same fresh-key discipline as POC #5's `pgdiag6`/`sqlite_conc`): -| workers | redb Immediate | sqlite (same probe) | pg (from POC #4) | +| workers | redb Immediate | sqlite (same probe) | pg (from POC #5) | |---------|---------------|---------------------|------------------| | 1 | ~42-44 ops/s | ~700-800/s | ~2.5-3.2k/s | | 2 | ~40 ops/s | ~1.0-1.1k/s | ~4.6-5.5k/s | @@ -168,7 +168,7 @@ go up. crash-consistent (unpersisted commits can leave the file failing to open until repaired — the freelist bookkeeping itself is lost on crash), `Immediate` pays a full fdatasync per commit. The - durability-posture comparison that POC #4 could tune continuously + durability-posture comparison that POC #5 could tune continuously (`synchronous_commit=off` on pg) doesn't exist here. - redb's `Database` is process-lifetime; concurrency modes (`ExclusiveWriter` default, `SingleWriter`, `MultiWriter`) only matter @@ -208,7 +208,7 @@ go up. - **Anti-recommendation adopted (ADR-007, 2026-10-02): redb stays ruled out** — recorded in the decision so the question does not reopen implicitly. SQLite's engine pin now has four-way triangulated - evidence (POC #3 fs, POC #4/#5 pg, POC #6 redb, plus the lineage + evidence (POC #3 fs, POC #5 pg, POC #6 redb, plus the lineage citation), and ADR-007 ships the kv tier as sqlite + postgres. If a future consumer wanted redb-shaped durability trade-offs (ephemeral cache tier), that's a *different* tier than the kv engine slot. @@ -222,7 +222,7 @@ go up. - clippy `-D warnings` clean, fmt clean. - **Honest caveats**: mdraid single dev box whose fdatasync latency ranged 0.24-30 ms during the session under co-resident load (pg - container churn from POC #4 ran through the first bench; stopped + container churn from POC #5 ran through the first bench; stopped before the final redbdiag re-run — relative conclusions unaffected, absolute redb-Immediate numbers are *device-pressure-dependent* and the honest statement is "1 fdatasync per commit, cost = device @@ -237,4 +237,4 @@ go up. `redbdiag2.rs` (None + periodic-flush group-commit emulation), `redbdiag3.rs` (commit-vs-insert split, per-put file growth), `redbdiag4.rs` (warm distribution), `redb_conc.rs` (scale-out, both - durability modes), `sqlite_conc.rs` (copied head-to-head from #4). \ No newline at end of file + durability modes), `sqlite_conc.rs` (copied head-to-head from #5). \ No newline at end of file