From abf8c5ae9c84a5a0e180e79f2a78c452c51a0035 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Sat, 3 Oct 2026 05:55:20 +0000 Subject: [PATCH] =?UTF-8?q?docs(architecture):=20ADR-010/011=20=E2=80=94?= =?UTF-8?q?=20I/O=20seams,=20GC-state=20home,=20vocabulary;=20pre-decompos?= =?UTF-8?q?ition=20consistency=20round?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ADR-010 (new): the Backend trait's I/O seams pinned before any code — 'get' returns a crate-internal read cursor (kv engines materialize; 'local' pread loop; pg-lo lo_get windows), 'put' has two named forms (whole-value kv / staged-put large); GC state is store-core-owned via a crate-internal contract-tested engine-state seam on the SQL-backed engines (backends never learn liveness — ADR-005 verbatim); the store core holds the joint entry+pin transaction (corrects ADR-009 §2's ownership statement); kv-tier fleet-validity rule added (mirror of ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin; renewal rides have/re-put; pause-past-TTL falls to delete-then-recover); feature graph pinned (pg-lo does not imply postgres). ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs engine (concrete impl, exactly five); store instance / node / fleet split (fleet = pool-sharing, not node count — fixes requirements.md's self-contradiction with REQ-2); mem demoted from 'backend'/'testing tier' to the kv tier's third engine (contract-reference engine); fs tier renamed 'large' including the feature name (last free moment before code exists); constructor modes pinned (dual-tier default, kv-only, mem-only) plus the composite fleet-validity predicate. Consistency round across all specs and ADR-003/004/007/008/009: stale pg-lo passages resolved per ADR-009's supersession; ADR-008 owner 'node id' → store-instance id; threshold wording corrected (conservative edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport shape specced; mem classification unified; 008/009 ADR files renamed to match the tier rename; README deferral-policy recap aligned (third category = decided-but-sequenced work, not a parking kind); POC crate list completed. Verification: two independent architecture review rounds (the first found 4 criticals — unpinned trait I/O shapes, missing fleet-state seam, node/fleet self-contradiction, pin-token/TTL conflict — all resolved; final round: zero criticals); all markdown links resolve; ADR tables complete (11 ADRs). --- docs/architecture/README.md | 76 ++-- docs/architecture/backends-and-dispatch.md | 269 +++++++++------ .../002-canonical-hash-and-key-encoding.md | 4 +- .../003-backend-contract-and-dispatch.md | 42 ++- .../decisions/004-two-backends-no-third.md | 48 +-- .../005-pooled-cas-and-gc-mechanism.md | 2 +- .../007-two-kv-engines-sqlite-and-postgres.md | 12 +- ...-size-probe-fleet-gc-and-large-engines.md} | 123 ++++--- ...gine.md => 009-pg-lo-large-tier-engine.md} | 27 +- ...0-backend-io-seams-and-fleet-state-home.md | 325 ++++++++++++++++++ ...ocabulary-tiers-engines-and-fleet-state.md | 225 ++++++++++++ docs/architecture/gc-and-namespaces.md | 32 +- docs/architecture/hashing-and-keys.md | 2 +- docs/architecture/open-questions.md | 9 +- docs/architecture/ops-surface.md | 46 ++- docs/architecture/overview.md | 83 +++-- docs/architecture/requirements.md | 60 ++-- docs/architecture/store-api.md | 88 +++-- 18 files changed, 1137 insertions(+), 336 deletions(-) rename docs/architecture/decisions/{008-trait-size-probe-fleet-gc-and-fs-engines.md => 008-trait-size-probe-fleet-gc-and-large-engines.md} (70%) rename docs/architecture/decisions/{009-pg-lo-fs-tier-engine.md => 009-pg-lo-large-tier-engine.md} (87%) create mode 100644 docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md create mode 100644 docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index 02da1da..dd75cdf 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7) +last_updated: 2026-10-03 (ADR-010/011 — I/O seams + GC-state home; vocabulary pinned, fs → large, mem joins kv) --- # alkblobs — Architecture @@ -10,8 +10,9 @@ last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7) **Phase 1 (Architecture).** Phase 0 (Exploration) converged — see `docs/research/phase-0.md` for the vision, the evidence trail, and the POC register (all POCs passed or were absorbed; POC code lives in -standalone crates (`/workspace/alkblobs-trait-poc`, -`/workspace/alkblobs-largeblob-poc`); findings live in +standalone crates under `/workspace/` — `alkblobs-trait-poc`, +`alkblobs-largeblob-poc`, `alkblobs-postgres-poc`, `alkblobs-redb-poc`, +`alkblobs-pglo-poc`; findings live in `docs/research/`). No implementation exists yet (`src/lib.rs` is a placeholder). @@ -24,14 +25,14 @@ there — not inherited as hedges: the family pattern, not deviation from it — and the ops surface rides alkcall, feature-gated. - **ADR-004** fixes the "two backends" framing: the crate ships exactly - two backends (kv small / fs large — both required by measured scale + two tiers (kv small / large — both required by measured scale economics); the path→hash mapping layer (the alkfs/alknet "vfs" shape) is a consumer-layer concern and must never become a third backend. - **ADR-007** widens the kv tier's engine set rather than hedging it: sqlite (default) + postgres (feature-gated) behind one trait. The deployment economics this records: a downstream node (git server, vfs) - classically needed three storage systems — kv + fs + relational — and - this design collapses them to **one relational engine + fs**, with the + classically needed three storage systems — kv + large content + relational — and + this design collapses the three to two: **one relational engine + large-tier storage**, with the engine a per-node constructor choice. Resolved from evidence in hand (POCs #3/#5/#6); the would-be "deployment trigger" framing was caught at review as circular hedging (a deployment running alkblobs cannot @@ -42,13 +43,13 @@ there — not inherited as hedges: planning fact predating all POCs): the `Backend` trait gains a `size` probe (before any backend ships data); fleet GC gets DB-backed pins, an advisory-locked single sweeper, and SQL-level - delete arbitration (single-node postures unchanged); and the fs tier + delete arbitration (single-node postures unchanged); and the large tier becomes engine-selectable (`local` default; `pg-lo` named candidate, admission-gated on POC #7). It also adds [requirements.md](requirements.md) — the REQ anchor that ends per-session re-derivation of consumer facts. - **ADR-009** (2026-10-03) admits `pg-lo` — postgres Large Objects as - the fs tier's second engine (feature `pg-lo`, default-off) — on + the large tier's second engine (feature `pg-lo`, default-off) — on POC #7's passed gate (`docs/research/poc-pglo-findings.md`): contract 10/10 exact-count tests, durable put ≈ fs durable put at ≥1 MiB, descriptorless `lo_get` window gets, transactional LO @@ -59,16 +60,44 @@ there — not inherited as hedges: readers), catalog space reused-but-never-returned, autovacuum inherited. The admission door (ADR-003, exercised by ADR-007 and ADR-009) has now shipped both its tier-extending engines. +- **ADR-010** (2026-10-03) pins the `Backend` trait's I/O seams — + the last unpinned signatures before implementation: `get` returns a + crate-internal **read cursor** (kv engines materialize; large-tier + engines stream — `local`'s pread loop, pg-lo's `lo_get` windows), `put` has + two named forms (whole-value for kv, **staged put** for large — + stage-then-commit-rename promoted from behavior to contract); rules + that **GC state is store-core-owned** (backends never learn + liveness — ADR-005 verbatim) hosted by SQL-backed engines via a + crate-internal contract-tested **engine-state companion seam** + (schema/pin/window/lock clauses), rules **the store core holds the + joint entry+pin transaction** (correcting ADR-009 §2's ownership + statement), adds the **kv tier's fleet-validity rule** (mirror of + ADR-008 §3 — postgres is the fleet engine; a sqlite-fleet posture + is not on offer), resolves the **ops pin-token TTL race** (the + token *is* the pin; renewal rides have/re-put; pause-past-TTL falls + to delete-then-recover), and pins the **feature graph** (`pg-lo` + does not imply `postgres`). +- **ADR-011** (2026-10-03) pins the vocabulary: **tier** (contract + + routing class — exactly two) vs **engine** (concrete impl — exactly + five: kv's sqlite/postgres/mem, large's `local`/pg-lo); + **mem demoted from "backend"/"testing tier" to the kv tier's + third engine — the contract-reference engine**; **store instance** / + **node** / **fleet** split so REQ-2's "one node may run multiple + instances" phrasing is no longer self-contradictory (fleet = + pool-sharing, not node count); and the **large tier is renamed + `large`** (feature `fs` → `large`) — its routing criterion is + length, not medium — with constructor modes (dual-tier default, + kv-only, mem-only) pinned in the same ADR's table. ## Architecture Documents | Document | Scope | Status | |---|---|---| | [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft | -| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, node/pool/fleet vocabulary | draft | +| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, pinned vocabulary (tier/engine/instance/node/fleet) | draft | | [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft | | [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft | -| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, shipped backends, size-threshold dispatch | draft | +| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, tiers + engines table, size-threshold dispatch, constructor modes | draft | | [gc-and-namespaces.md](gc-and-namespaces.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | draft | | [ops-surface.md](ops-surface.md) | alkcall-backed have/need + fetch/put ops, ACL mapping | draft | | [open-questions.md](open-questions.md) | Centralized OQ tracker (incl. promoted Phase 0 register) | draft | @@ -84,8 +113,10 @@ there — not inherited as hedges: | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted | | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted | -| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo — postgres Large Objects as the fs tier's second engine (admitted on POC #7) | Accepted | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); large-tier engines | Accepted | +| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo — postgres Large Objects as the large tier's second engine (admitted on POC #7) | Accepted | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Backend I/O seams (read cursor, staged put); GC state store-core-owned via the engine-state seam; kv fleet-validity; pin-token renewal | Accepted | +| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary: tier/engine/instance/node/fleet; mem joins kv; large tier renamed `large`; constructor modes | Accepted | ## Open Questions @@ -94,25 +125,28 @@ Tracked in [open-questions.md](open-questions.md). The Phase 0 register remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping, alkfs intake; carried for visibility, gating nothing here). OQ-09 (namespace-visibility default) resolved closed-by-default; OQ-10 -(second kv engine) resolved by ADR-007. The pg-lo fs-engine question +(second kv engine) resolved by ADR-007. The pg-lo large-engine question resolved to *sequenced work*, not a parked question: REQ-2/ADR-008 -named it the candidate fs-tier engine and POC #7 +named it the candidate large-tier engine and POC #7 (`docs/research/poc-pglo-spec.md`) was its admission evidence — a task, not a wait; that task completed 2026-10-03 (`docs/research/poc-pglo-findings.md`, passed) and ADR-009 shipped the engine. **Deferral policy (the "Schrödinger's code" rule):** a *decision this crate needs before shipping* may not be deferred on a dependency that is itself -waiting for this crate to exist. Two parking kinds remain legitimate: +waiting for this crate to exist. Legitimate parked kinds: `deferred(scope)` on a deciding fact that exists independently of this crate, and externally-owned questions (how a consumer maps onto this crate) that gate no decision here — full definitions in the header of -[open-questions.md](open-questions.md). A corollary worth restating, -because it recurs: a fact this crate must create (its own first -deployment, its own first consumer) is never a deciding input — -decisions stand on evidence in hand, and future needs reopen via named -requirements, not via waiting. (Corollary applications so far: OQ-09 -and OQ-10 both resolved on it.) +[open-questions.md](open-questions.md). A third category is *not* a +parked kind: **decided-but-sequenced work** (a resolved decision whose +admission gate requires evidence — the pg-lo round's POC #7 is the +worked example; a task with an owner, not a parked question). A +corollary worth restating, because it recurs: a fact this crate must +create (its own first deployment, its own first consumer) is never a +deciding input — decisions stand on evidence in hand, and future needs +reopen via named requirements, not via waiting. (Corollary +applications so far: OQ-09 and OQ-10 both resolved on it.) ## Lifecycle diff --git a/docs/architecture/backends-and-dispatch.md b/docs/architecture/backends-and-dispatch.md index 86d545e..03b9833 100644 --- a/docs/architecture/backends-and-dispatch.md +++ b/docs/architecture/backends-and-dispatch.md @@ -1,17 +1,23 @@ --- status: draft -last_updated: 2026-10-03 +last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — fs tier → large; mem is a kv engine) --- -# Backends and dispatch +# Backends, tiers, and dispatch ## What this is -The physical storage layer: the `Backend` trait contract every backend -implements, the two shipped backends (kv, fs), and the size-threshold -dispatch that routes between them. The trait is the crate's second -stable seam (with the key encoding, ADR-002): backends must survive -digest-layer evolution, so the boundary stays dumb. +The physical storage layer: the `Backend` trait contract every engine +implements, the two shipped tiers (kv, large — the tier formerly named +`fs`; renamed in ADR-011), and the size-threshold dispatch that routes +between them. The trait is the crate's second stable seam (with the +key encoding, ADR-002): engines must survive digest-layer evolution, +so the boundary stays dumb. + +Vocabulary is pinned in requirements.md (ADR-011): **tier** = contract ++ routing class (exactly two; ADR-004), **engine** = concrete impl of +one tier's contract (exactly five), and this document's table is the +spec home of the constructor table (ADR-011 §4). ## What the Backend trait is (and is not) @@ -32,28 +38,46 @@ purpose-built option. The wrong turn to rule out is "my data isn't blobs, therefore I implement a Backend for it"; nothing a consumer needs is ever reachable through that door. -## The Backend trait contract (ADR-003) +## The Backend trait contract (ADR-003; I/O seams per ADR-010) -- **Opaque byte keys, opaque byte values.** Backends never learn what a - digest is; typed `Key` converts at the store boundary (ADR-002). This - is also the persistence boundary — a kv row or an fs filename has no - schema-migration story, so the byte layout must be self-describing - (algorithm-tagged keys). +- **Opaque byte keys, opaque byte-valued entries.** Engines never learn + what a digest is; typed `Key` converts at the store boundary + (ADR-002). This is also the persistence boundary — a kv row, a + `local` filename, or an LO companion row has no schema-migration story, so + the byte layout must be self-describing (algorithm-tagged keys). - **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` / - `size` (the length probe added by ADR-008). Streaming put/get shapes - ride the store layer's seams (store-api.md); - backends provide byte- or handle-level primitives underneath - (`get` on the fs tier yields a file handle, not a loaded buffer). - `size(key)` returns the entry's length or `None` if absent — - metadata only, never a content fetch: kv engines probe the indexed - `size` column both row shapes carry (POC #5 B5); the `local` fs - engine stats the file. Store-level `stat` and pool accounting ride - this method. + `size` (the length probe added by ADR-008). `size(key)` returns the + entry's length or `None` if absent — metadata only, never a content + fetch: kv engines probe the indexed `size` column both row shapes + carry (POC #5 B5); `local` stats the file; pg-lo reads the companion + row. Store-level `stat` and pool accounting ride this method. +- **The I/O seams are crate-internal abstractions, not engine shapes** + (ADR-010 §1 — the gap that would otherwise force guessing): + - `get` returns a **read cursor** the store core drives. kv-engine + cursors materialize the whole value (bounded by tier policy); + large-engine cursors stream — `local`'s is a pread loop over the + committed file, pg-lo's drives descriptorless `lo_get(oid, off, + len)` windows (ADR-009 §3, the C4 posture). Both serve + `read_range` (slice + slice digest, ADR-006) at one seam. The + cursor is not public API: store consumers see the facade's + `(len, stream)` shapes with the cursor behind them. + - `put` has **two forms**: the **whole-value put** (kv tier — the + settled CAS row write) and the **staged put** (large tier — + ADR-003's stage-then-commit-rename as a named per-engine + contract: the engine hands out a *staging cursor*; commit + publishes; every failure path converges on discard with zero + residue). Both land in the CAS entry shape; both participate in + pin-before-publish (ADR-005/010 §2 — the store core holds the + joint entry+pin transaction via the engine-state seam). + - Per-engine conformance: an engine is admitted only if its + cursor/stage/list bodies pass the uniform contract suite — the + POC #7 gate (10/10 exact-count sweep-outcome tests) is the norm; + suite details in store-api.md's invariants. - **`list()` complete by contract.** A *malformed* list is as deadly as an incomplete one: list correctness is only observable through GC (POC #1 finding 2 — the redb key-vs-value trap produced a sweep that deleted the wrong blobs, silently). Therefore: - - every backend implementation must prove list correctness through + - every engine implementation must prove list correctness through sweep-outcome tests (the invariant's test gate lives in store-api.md); - an implementation that cannot enumerate (the rudolfs S3 @@ -61,49 +85,60 @@ is ever reachable through that door. optional extra (ADR-005). - **Virgin-store reads are no-ops:** read paths treat a missing table as absent/empty (POC #1 finding 2). -- **Namespace-blind:** backends never see namespaces, tenants, or +- **Namespace-blind:** engines never see namespaces, tenants, or reference structure (ADR-005 — the rudolfs inversion; physical storage is `hash → bytes` flat). +- **GC state never rides this trait** (ADR-010 §2): pins, window rows, + and sweep locks are store-core-owned via a crate-internal + engine-state companion seam on the SQL-backed engines (sqlite, + postgres, pg-lo) — invisible at the public trait; `local` and `mem` + cannot express it and are never fleet engines. -## Shipped backends +## Shipped tiers and engines Two tiers, exactly — this counts *trait implementations this crate -ships* as CAS tiers, the dispatch routes between them; it is the +ships as CAS tiers* (the contracts dispatch routes between); it is the complete set the problem requires, and the "third backend" fear is a -category error fixed in ADR-004. It says nothing about other storage -existing in a deployment: a downstream runs whatever else it needs on -its own media, beside the pool, above the store's seams (see "What the -Backend trait is"). **Tier count is not engine count**: the kv tier -carries two engines (sqlite, postgres) behind one trait — ADR-007; -per-node engine selection is a constructor parameter. +category error fixed in ADR-004 (tier count is not engine count). It +says nothing about other storage existing in a deployment: a downstream +runs whatever else it needs on its own media, beside the pool, above +the store's seams (see "What the Backend trait is"). -### kv backend (feature `kv`, default-on); engines: sqlite (default) + postgres (feature `postgres`, default-off) +| Tier | Engine | Feature | Default | Fleet-valid | +|---|---|---|---|---| +| kv | sqlite | — (tier default) | yes (tier default-on) | no | +| kv | postgres | `postgres` | no | **yes** | +| kv | mem | `mem` | no | no (test/reference engine) | +| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion | +| large | pg-lo | `pg-lo` | no | **yes** | + +### kv tier (tier default-on, feature `kv`); engines sqlite (default) + postgres (feature `postgres`, default-off) + mem (feature `mem`, default-off) Small blobs — **`kv` is the tier name; the engine is a per-node constructor choice** between the shipped engines (ADR-007; ADR-003 carries sqlite's on-disk format trade: file compat is upstream sqlite's guarantee). The shipped trait impls write plain -content-addressed rows into the chosen engine, nothing else. A -*third* engine remains the substitution door as amended: a new ADR -with its own sweep-safety proof (the door is no longer hypothetical — -ADR-007 is its first in-repo exercise); and a downstream's *other* -storage — schemas, manifests, whatever it likes including some other -sqlite file — never comes near this tier at all, per "What the -Backend trait is". +content-addressed rows into the chosen engine, nothing else (ADR-010: +the fleet state rides the companion seam, never the public trait). A +*third* non-mem engine remains the substitution door as amended: a new +ADR with its own sweep-safety proof (the door is exercised — ADR-007 +in-repo); and a downstream's *other* storage — schemas, manifests, +whatever it likes including some other sqlite file — never comes near +this tier at all, per "What the Backend trait is". The collapse ADR-007 buys: the classical downstream stack (git -server, vfs node) needed three storage systems — kv + fs + relational. +server, vfs node) needed three storage systems — kv + large content + relational. Here the kv tier rides the relational engine itself, so a node -provisions **one relational engine + fs**; consumer tables (refs, -manifests, queues) sit beside the pool on the same engine or file by -the deployment's own choice (co-tenancy note: same-file consumer -writes share sqlite's single-writer ceiling — that sharing is the -node's trade; separate files remain available). +provisions **one relational engine + large-tier storage**; consumer +tables (refs, manifests, queues) sit beside the pool on the same +engine or file by the deployment's own choice (co-tenancy note: same- +file consumer writes share sqlite's single-writer ceiling — that +sharing is the node's trade; separate files remain available). -Evidence (POC #3 finding A4, first-party measured): -sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime -— most git objects, workspace files, manifests), with the crossover at -~128-256 KiB where fs stops paying the B-tree row rewrite and wins. +Evidence (POC #3 finding A4, first-party measured): sqlite is ~9-10× +faster than fs at 1-16 KiB (the git small-blob regime — most git +objects, workspace files, manifests), with the crossover at ~128-256 +KiB where fs stops paying the B-tree row rewrite and wins. **sqlite engine** (the default): WAL + `synchronous=NORMAL` as the shipped durability tier (matching what the benchmark measured and what @@ -128,22 +163,31 @@ sweep-outcome CI gate, run against dockerized postgres under near-linearly to ~37k puts/s at 24 conns — the only engine whose throughput *increases* under concurrency. -### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (feature `pg-lo`, default-off) +**mem engine** (feature `mem`, default-off; ADR-011 §2 — formerly +described as the "mem backend"/"testing tier" before the vocabulary +was pinned): a `BTreeMap`-shaped ephemeral implementation of the kv +tier's contract, including `size` (trivially). For tests and +in-process ephemerality — never a production story, never +fleet-valid, and the **contract-reference engine**: every other +engine's conformance tests mirror its suite (the role it already +played in POC #1's miniature). -Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract, -multiple engines** — the medium is a per-node constructor choice -(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing -for fleets): +### large tier (tier default-on, feature `large`; the tier formerly named `fs` — renamed in ADR-011); engines `local` (default) + `pg-lo` (feature `pg-lo`, default-off) + +Large blobs. Like the kv tier (ADR-007), the large tier is **one +contract, multiple engines** — the medium is a per-node constructor +choice (ADR-008; REQ-2 in requirements.md makes the engine concept +load-bearing for fleets): - **`local` engine** (the default; the ADR-003 backend as originally specced): flat sharded layout: `{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's conclusion (limits directory size on huge pools); stage-then-commit- - rename for the two-pass unknown-length path; pread-based range - reads (POC #3 finding A2 — local range serving is sound, e.g. for - packfiles). + rename for the two-pass unknown-length path (the staged-put form, + ADR-010); pread-based range reads (POC #3 finding A2 — local range + serving is sound, e.g. for packfiles). - **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009): - postgres Large Objects as the fs tier's storage — the same + postgres Large Objects as the large tier's storage — the same engine-behind-one-trait move as ADR-007, one tier over, admitted on POC #7's measured evidence (contract 10/10 exact-count tests; durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC @@ -154,66 +198,90 @@ for fleets): operator's shrink path), autovacuum inherited, cached-get 20–50× behind page-cache fs single-stream (~700 MB/s aggregate at 16 readers — the fleet serving picture). -- **Fleet locality contract (ADR-008, extended by ADR-009):** over one - shared pool, the fs tier is one of: shared media (every node's - `local` root on the same fleet-shared media — soundness caveats and - the required media guarantees in ADR-008), re-routed (one storage - node serves pool large-blob content via the ops surface; client - nodes are kv-tier-only over pool content), **or `pg-lo` for the - whole fleet** (ADR-009 — one engine, no shared media, no re-routing; - pool content lives in the same pg instance the kv tier rides). - Mixed per-node-local fs tiers over one pool is the partitioning - failure this contract exists to prevent — a documented deployment - invariant the constructor cannot fully prove but must be declared - against (ADR-008 names the enforceable seam and the detection - symptom). - -### mem backend (feature `mem`, default-off) - -`BTreeMap`-shaped ephemeral backend for tests and in-process -ephemerality. A testing/utility tier, never a production story. -Implements the full trait contract including `size` (trivially) — it -exists partly as the reference impl every other backend's conformance -tests can mirror. +- **Fleet locality contract (ADR-008, extended by ADR-009/010):** over + one shared pool, the large tier is one of: shared media (every + node's `local` root on the same fleet-shared media — soundness + caveats and the required media guarantees in ADR-008), re-routed + (one storage node serves pool large-blob content via the ops + surface; client nodes are kv-only over pool content), **or + `pg-lo` for the whole fleet** (ADR-009 — one engine, no shared + media, no re-routing; pool content lives in the same pg instance + the kv tier rides; the joint entry+pin tx is store-core-held, + ADR-010 §2). Mixed per-node-local large tiers over one pool is the + partitioning failure this contract exists to prevent — a documented + deployment invariant the constructor cannot fully prove but must be + declared against (ADR-008 names the enforceable seam and the + detection symptom). The kv tier has the mirrored rule (ADR-010 §3): + a fleet constructor requires a pool-shared kv engine (postgres); + two sqlite engines are two pools, never one. +- **The composite fleet-validity rule (one sentence, both tiers):** a + constructor configuration is fleet-valid iff **kv = postgres AND + (large = pg-lo OR large = `local`-on-declared-shared-media OR + large = none with the re-routing posture carrying pool large + content)**. Any other combination over one shared pool is invalid — + the constructor requires the declarations that make this predicate + checkable per node; cross-node truth remains the deployment's + verified invariant (ADR-008's seam). This predicate is + backends-and-dispatch.md's and ADR-011 §4's table combined; it is + stated here once so no reader composes it by inference. ## Size-threshold dispatch (ADR-003) - **Routing is a pure function of content length.** Same content ⇒ same - length ⇒ same backend; re-puts are deterministic. No content ever - migrates between backends — the migration question existed only to + length ⇒ same tier; re-puts are deterministic. No content ever + migrates between tiers — the migration question existed only to patch the unknown-length asymmetry, which ADR-003's pre-threshold buffering eliminates by construction. -- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the - measured flat zone (POC #3 A4). Re-tuning per deployment media is a - constructor parameter, not an API change. +- **Default threshold: 128 KiB** (constructor-tunable) — the bottom of + the measured crossover zone (~128-256 KiB, POC #3 A4; chosen at the + zone's conservative edge, not its midpoint). Re-tuning per + deployment media is a constructor parameter, not an API change. - **Get fall-through:** small-tier miss queries the large tier (deterministic, cheap — a stat probe). -- Per-namespace or per-tenant backend configuration: **rejected** +- Per-namespace or per-tenant engine configuration: **rejected** (ADR-003 §Consequences — it would re-weld namespacing into the physical layer, the rudolfs anti-pattern ADR-005 inverts). -## Where a *new* backend or engine could come from +### Constructor modes (ADR-011 §4) + +- **dual-tier** (the default): kv + large, size-threshold dispatch + (ADR-003). +- **kv-only** (ADR-008's re-routing client posture, now named): no + large tier; over-threshold puts are a constructor-time error (the + re-routing posture handles them via the ops surface — the + consumer's composition, not a store mode). +- **mem-only:** the mem engine alone, no dispatch threshold in + effect; a testing/embedder-ephemeral posture, never production. +- Single-tier SQL modes (postgres-only, pg-lo-only *tiers*) do not + exist: both tiers are load-bearing (ADR-004). "Postgres-only" as in + one SQL *instance* serving both tiers exists and is the ADR-009 + consolidation: dual-tier mode with kv=postgres + large=pg-lo over + one pool. + +## Where a *new* engine could come from The trait is open to future implementations (network stores, S3-like tiers), but nothing in the current consumer set requires one, and the contract is deliberately hostile to half-implementations (complete -`list()`, GC-participating `delete`). Any future backend — or a third -kv engine or second fs engine — is a new ADR carrying its own -sweep-safety story (ADR-007 is the precedent for a kv engine -addition; ADR-008 named pg-lo as the candidate second fs engine with -POC #7 as its admission evidence — the door is exercised twice). This -crate's roadmap is not blocked on one (see open-questions.md — alkfs -intake may name needs externally; OQ-08). +`list()`, GC-participating `delete`). Any future engine — per tier — +is a new ADR carrying its own sweep-safety story (ADR-007 is the +precedent for a kv engine addition; ADR-008 named pg-lo and ADR-009 +admitted it with POC #7's evidence — the door is exercised twice). +This crate's roadmap is not blocked on one (see open-questions.md — +alkfs intake may name needs externally; OQ-08). ## Design Decisions | ADR | Decision | Summary | |---|---|---| | [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration | -| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption | -| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only | -| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) | +| [004](decisions/004-two-backends-no-third.md) | Two tiers, no third | scope boundary against manifest-layer absorption | +| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | engines see hashes only | +| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + large tier per node | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; large tier engine-selectable (`local` default) | +| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo admitted | postgres Large Objects as the large tier's second engine, on POC #7 | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; the kv fleet-validity rule | +| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary | tier/engine/instance/node/fleet; mem is a kv engine; fs → large; constructor modes | ## Open Questions @@ -230,10 +298,13 @@ intake may name needs externally; OQ-08). curves + engine-posture deltas (ADR-007's evidence base) - `docs/research/poc-redb-kv-findings.md` — redb ruled out at a durability-tier mismatch (POC #6; recorded in ADR-007) +- `docs/research/poc-pglo-findings.md` — pg-lo's admission evidence + (POC #7; ADR-009) - `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed (sharding, crash ordering, inline thresholds rejected as weld) - rudolfs notes — `list()` anti-lesson, decorator alternative noted and not adopted (threshold dispatch chose the simpler policy; ADR-003 §Context) - ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants - backends must satisfy) \ No newline at end of file + engines must satisfy); [requirements.md](requirements.md) (the pinned + vocabulary) \ No newline at end of file diff --git a/docs/architecture/decisions/002-canonical-hash-and-key-encoding.md b/docs/architecture/decisions/002-canonical-hash-and-key-encoding.md index cd8289c..71e5dfc 100644 --- a/docs/architecture/decisions/002-canonical-hash-and-key-encoding.md +++ b/docs/architecture/decisions/002-canonical-hash-and-key-encoding.md @@ -16,7 +16,7 @@ addresses, not algorithms). The residual was the **key encoding mechanics** — the byte layout of keys inside backends. This is the crate's declared one-way door -(AGENTS convention 5): key bytes persist in kv rows and fs filenames, +(AGENTS convention 5): key bytes persist in kv rows and `local`-engine filenames, with no schema-migration story; and this is the wire-format ADR that must exist before the first consumer (convention 7's gate on publishing wire-relevant decisions). POC #1 validated the mechanics empirically: @@ -26,7 +26,7 @@ both algorithms in one flat pool (finding 5). ## Decision -### Hash policy (three tiers) +### Hash policy (three classes) 1. **Canonical — `git-blob-sha-256`:** `H("blob \0" + content)`, SHA-256. The store's canonical algorithm; git objects are first-class diff --git a/docs/architecture/decisions/003-backend-contract-and-dispatch.md b/docs/architecture/decisions/003-backend-contract-and-dispatch.md index 195b10e..fd2b1d0 100644 --- a/docs/architecture/decisions/003-backend-contract-and-dispatch.md +++ b/docs/architecture/decisions/003-backend-contract-and-dispatch.md @@ -5,10 +5,15 @@ Accepted (engine pin amended by ADR-007 — the kv tier ships two engines, sqlite and postgres, behind this contract; the dispatch layer and trait are unchanged. The trait was extended by ADR-008 with a -`size` length probe — before any backend ships data — and the fs -tier became engine-selectable (`local` default; pg-lo candidate), so -this ADR's "engines"/"backends" counts are read through ADR-004's -tier≠engine scope note) +`size` length probe — before any backend ships data — and the second +tier became engine-selectable (`local` default; pg-lo admitted by +ADR-009), so this ADR's "engines"/"backends" counts are read through +ADR-004's tier≠engine scope note. The I/O shapes behind `get`/`put` +are pinned by ADR-010 (read cursor / staged put). The tier's name +changed: `fs` → `large` (ADR-011 — its routing criterion is length, +not medium); this ADR's per-engine counts also gained the `mem` kv +engine as ADR-011's vocabulary resolves — mem is a kv engine, not a +tier) ## Context @@ -54,28 +59,34 @@ Evidence in hand (no further POC or consumer-waiting needed): - **Virgin-store reads are no-ops** (missing table = absent/empty). - **Namespace-blind** (ADR-005): backends see hashes only. -### Shipped backends (two, default-on; plus a testing tier) +### Shipped tiers (two, default-on; the kv tier also carries the mem engine) - **`kv` (default-on): sqlite** for small blobs — measured ~9-10× faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark), the regime holding most git objects, workspace files, and manifests. WAL + `synchronous=NORMAL` as shipped durability; prepared statements; no-tables-yet tolerance. (The tier's engine set was - widened to include postgres by ADR-007.) -- **`fs` (default-on)** for large blobs — flat - `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving + widened to include postgres by ADR-007, and mem — the ephemeral + reference engine — by ADR-011's vocabulary resolution.) +- **`fs` (default-on; renamed `large` by ADR-011)** for large blobs — + flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving conclusion); stage-then-commit-rename; pread range reads (sound - locally per ADR-006). -- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and - in-process ephemerality. Never a production story. + locally per ADR-006). (The tier's engine set was widened by ADR-008/ + 009.) +- **`mem` (default-off):** ephemeral `BTreeMap` kv engine for tests + and in-process ephemerality — **an engine of the kv tier, not a + third tier** (ADR-011 §2; the pre-ADR-011 "testing tier" phrasing + is superseded). Never a production story; the contract-reference + engine. -Both production backends are default-on (batteries-included, the +Both production tiers are default-on (batteries-included, the alkgit feature-model precedent); `mem` is not. ### Dispatch policy - **Size-threshold routing; default 128 KiB, constructor-tunable** - (midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4). + (chosen at the bottom of the measured ~128-256 KiB crossover zone — + the conservative edge of the flat zone; POC #3 A4). - **Unknown-length puts buffer in memory up to the threshold; the buffer's fate has three pinned semantics** (finding A6's option (d)): 1. **Threshold exceeded mid-stream** (non-restartable sources are @@ -141,9 +152,10 @@ alkgit feature-model precedent); `mem` is not. **Neutral** -- A network-backed backend (S3-like) remains possible via the same +- A network-backed engine (S3-like) remains possible via the same trait but ships nothing here; its list/GC obligations are the - acceptance gate. + acceptance gate; per-tier, via a new ADR (ADR-011's vocabulary: an + engine addition, not a tier addition). ## References diff --git a/docs/architecture/decisions/004-two-backends-no-third.md b/docs/architecture/decisions/004-two-backends-no-third.md index 0f6a5e9..7697db3 100644 --- a/docs/architecture/decisions/004-two-backends-no-third.md +++ b/docs/architecture/decisions/004-two-backends-no-third.md @@ -2,7 +2,12 @@ ## Status -Accepted +Accepted (the count's vocabulary is pinned by ADR-011: the "two" here +are *tiers* — kv + large, the latter renamed from `fs`; "backend" +names the same thing at a different altitude. The `mem` item below is +the kv tier's third engine, per ADR-011 §2, not an extra tier — +the pre-ADR-011 "testing tier" carve-out is superseded. The I/O +seam-level refinements of the trait contract are ADR-010's) ## Context @@ -11,9 +16,10 @@ rounds before being edited away, but the correction it was reaching for arrived only in the Phase 1 review, and needs recording so no future round reabsorbs the concern incorrectly: -- There are **always** two physical backends in this problem: storage - for small content (fast, kv/sqlite) and the filesystem for large - content. This is not ugly, it is the measured shape of the problem +- There are **always** two physical tiers in this problem: storage + for small content (fast, kv/sqlite) and storage for large content + (the large tier / filesystem). This is not ugly, it is the measured + shape of the problem (POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it). - The "two backends is ugly" sentiment was actually about a **third layer** in the old alknet-filesystem research — the *virtual @@ -30,23 +36,25 @@ is the boundary it will build against. ## Decision -**This crate ships exactly two production backends and nothing more; +**This crate ships exactly two production tiers and nothing more; every manifest/namespace/path-tree layer is a consumer, by construction -and by contract.** (A non-production ephemeral `mem` tier exists for -tests — ADR-003; it is not a storage story and does not extend this -boundary.) +and by contract.** (A non-production ephemeral `mem` engine exists for +tests — ADR-003, resolved to a kv-tier engine by ADR-011 §2; it is not +a storage story and does not extend this boundary.) -**Scope note (post-ADR-007 reconciliation): the "two" here counts -*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the -engines behind it to a set (sqlite default + postgres, feature-gated) -so a node's kv rides its relational engine of choice. The count that -matters to this ADR — trait impls *this crate ships* as CAS tiers — -remains two (kv + fs); a second kv engine does not make a third -backend, and nothing in the engine set touches the manifest-layer -boundary below. +**Scope note (post-ADR-007 reconciliation, vocabulary finalized by +ADR-011): the "two" here counts *tiers*, not engines.** The kv tier is +one contract; ADR-007 widened the engines behind it to a set (sqlite +default + postgres, feature-gated; mem the ephemeral reference engine, +ADR-011) so a node's kv rides its relational engine of choice. The +count that matters to this ADR — trait impls *this crate ships* as CAS +tiers — remains two (kv + large, the tier ADR-011 renamed from `fs`); +a second kv engine does not make a third tier, and nothing in the +engine set touches the manifest-layer boundary below. -1. **Two production backends is the complete set.** kv/sqlite (small) - + fs (large) cover the scale economics with measured evidence +1. **Two production tiers is the complete set.** kv/sqlite (small) + + large (the tier formerly named `fs`) cover the scale economics + with measured evidence (ADR-003); no other physical tier is needed by any current or planned consumer's *blob* needs. 2. **The "third backend" is not this crate's layer.** The vfs/appfile @@ -86,8 +94,8 @@ boundary below. **Positive** -- The crate's scope stays testable and small: two backends, one pool, - one hash domain. No schema/versioning surface for manifest formats — +- The crate's scope stays testable and small: two tiers, one pool, + one hash domain. No schema/versioning surface for manifest formats — consumers iterate on theirs freely. - alkgit and alkfs build their (very different) mapping layers on one shared pool without either bending the store; their manifests interop diff --git a/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md b/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md index 348391b..0b336e9 100644 --- a/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md +++ b/docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md @@ -49,7 +49,7 @@ served by range reads). (git refs, alkfs path trees) and hand liveness over per sweep via the live-shared seam. In-memory namespace tables remain available for ephemeral consumers. This also dissolves the hidden wrinkle of a - store-internal root table under fs-only deployments (where would its + store-internal root table under deployments without a SQL engine (where would its durable bytes live?). Ephemeral consumers are covered by pins; a deployment that needs store-durable roots registers them as consumer-owned durable state and sweeps through the seam. diff --git a/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md b/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md index d149857..410a7bf 100644 --- a/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md +++ b/docs/architecture/decisions/007-two-kv-engines-sqlite-and-postgres.md @@ -2,7 +2,9 @@ ## Status -Accepted +Accepted (the engine story's I/O seams and GC-state home are ruled by +ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as +its third engine — is pinned by ADR-011) ## Context @@ -57,9 +59,9 @@ in OQ-09. **The kv tier ships two engines — sqlite (default) and postgres — behind one `Backend` trait; engine selection is a constructor parameter on the backend's configuration.** A node provisions one -relational engine plus the fs tier; the kv tier rides whichever +relational engine plus the large tier; the kv tier rides whichever engine the operator chose. This collapses the classical three-engine -downstream stack (kv + fs + relational) to two, and makes which +downstream stack (kv + large content + relational) to two, and makes which relational engine a per-node choice rather than a crate fork. 1. **sqlite stays the default engine**: zero-ops, no daemon, solo @@ -83,7 +85,7 @@ relational engine a per-node choice rather than a crate fork. (`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's `INSERT OR IGNORE`), complete `list()`, GC-participating `delete`, virgin-store no-op reads, namespace-blindness. The dispatch layer - (size threshold, kv↔fs routing, fall-through) is engine-blind and + (size threshold, kv↔large routing, fall-through) is engine-blind and unchanged. 4. **The pg engine's impl contract carries POC #5 finding B5's deltas as requirements** (they are the difference between a port and an @@ -127,7 +129,7 @@ collapses to its own deployment choice among shipped engines. **Positive** - The classical downstream stack lands at two engines per node (one - relational + fs), eliminating the "sqlite *beside* the real + relational engine + large tier), eliminating the "sqlite *beside* the real database" duplication this crate's consumers would otherwise carry forever. - Per-node engine choice (constructor parameter, per ADR-003/004) diff --git a/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md b/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md similarity index 70% rename from docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md rename to docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md index 3aefa5e..e0a3192 100644 --- a/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md +++ b/docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md @@ -1,10 +1,14 @@ -# ADR-008: Trait length probe, fleet GC, and fs-tier engines +# ADR-008: Trait length probe, fleet GC, and large-tier engines ## Status Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's -passed gate; every other part of this ADR stands as written) +passed gate — the two §Consequences passages that said pg-lo "remains +unbuilt" are resolved by that same supersession; the tier this ADR's +title and text call "fs" is renamed `large` by ADR-011; the fleet GC +mechanism's state home and the joint-tx ownership are ruled by +ADR-010; every other part of this ADR stands as written) ## Context @@ -18,7 +22,7 @@ sharing one pool via postgres; requirements.md): probe"; store-api.md) and pool accounting rides `list` + `stat` (ADR-005), but the trait's methods are `has/get/put/delete/list/ name` with no value-metadata access. The kv engines' row shapes - both carry a `size` column (POC #5 B5), and the fs tier can stat a + both carry a `size` column (POC #5 B5), and the large tier can stat a file — but with no trait method, kv `stat` degenerates to a full value load. The trait is the crate's second stable seam (ADR-003); discovering this during implementation would be a breaking trait @@ -32,8 +36,8 @@ sharing one pool via postgres; requirements.md): node B's in-flight put is invisible. Data loss, structurally: the topology REQ-2 requires does not satisfy the protocol ADR-005 specifies. -3. **The fs tier is node-local, silently partitioning a fleet.** The - fs backend stores to a sharded layout on the node's own filesystem +3. **The large tier is node-local, silently partitioning a fleet.** The + large tier stores to a sharded layout on the node's own filesystem (ADR-003). Under a fleet sharing one kv pool, a >threshold blob put via node A exists only on A's disk; node B's get fall-through queries its own fs and misses, and `list()` unions disagree per @@ -41,7 +45,7 @@ sharing one pool via postgres; requirements.md): hypothetical. These three travel together: the fleet topology forces #2 and #3, and -#3's resolution constrains what the fs tier must support, while #1 is +#3's resolution constrains what the large tier must support, while #1 is a trait-level gap any topology hits. ## Decision @@ -55,7 +59,7 @@ tier: - **kv engines**: an indexed column probe (`SELECT size …WHERE key`), never a value load. The `size` column both POC row shapes carry becomes contract-relevant. -- **fs engine (`local`)**: stat/metadata read, no content fetch. +- **large engine (`local`)**: stat/metadata read, no content fetch. - The store facade's `stat` (store-api.md) and the accounting path (ADR-005) ride this method; the `has` method remains a pure existence probe. @@ -71,18 +75,20 @@ never deleted by a sweep started before it committed** (ADR-005). Under a fleet, the mechanism moves into the shared engine: - **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two - inserts in one transaction on the fleet engine: the entry row and - its pin row commit together (or neither does). The pin row carries - an owner (node id) and expiry: the pin guard renews its row every - TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor - parameters, not API). **Protection = an un-expired pin row**: at - delete-time arbitration, only `expiry > now()` pins count; an - expired pin is abandoned (its holder failed to renew — the same - liveness-failure signal as an in-process guard's drop) and is reaped - by the sweeper's maintenance step (the GC-of-GC cleanup). A node - that pauses past TTL loses its pins; delete-then-recover (ADR-005) - remains the correctness backstop — content is re-put byte-identically - under the same key. + inserts in one transaction on the fleet engine: the entry row and + its pin row commit together (or neither does). The pin row carries + an owner (a store-instance id — ADR-011's vocabulary; "node id" is + this ADR's original wording, superseded because a node may host + several instances) and expiry: the pin guard renews its row every + TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor + parameters, not API). **Protection = an un-expired pin row**: at + delete-time arbitration, only `expiry > now()` pins count; an + expired pin is abandoned (its holder failed to renew — the same + liveness-failure signal as an in-process guard's drop) and is reaped + by the sweeper's maintenance step (the GC-of-GC cleanup). A node + that pauses past TTL loses its pins; delete-then-recover (ADR-005) + remains the correctness backstop — content is re-put byte-identically + under the same key. - **Registered liveness sources have one fleet form: a liveness table.** A fleet embedder registers its liveness as a named table of `(key, …)` rows in the shared engine (schema pinned by the embedder, @@ -131,9 +137,9 @@ Under a fleet, the mechanism moves into the shared engine: is shared machinery. The pin-row expiry/ttl machinery exists only where fleet engines are active. -### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible +### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible -The fs tier is a tier, not a medium. It ships engines the way the kv +The large tier is a tier, not a medium. It ships engines the way the kv tier does (ADR-007), and a deployment's topology determines which are valid: @@ -141,7 +147,7 @@ valid: unchanged (ADR-003). Valid for any single-node topology (REQ-1, REQ-4) and for fleet nodes using **shared media** (below). - **Fleet + shared media (deployment requirement):** multi-node pools - may keep the `local` engine by pointing every node's fs tier at + may keep the `local` engine by pointing every node's large tier at shared/replicated media. The soundness claim is *qualified*, not assumed: immutable-once-committed files plus stage-then-commit- rename are sound on media that provides (a) atomic rename within a @@ -158,43 +164,52 @@ valid: not share media designates **one storage node**; large-blob writes and reads over the pool route through it via the ops surface (verified fetch/put; ADR-001). Client nodes' constructors are - configured with a **kv-tier-only dispatch** (or an fs tier reserved - for node-private, consumer-owned content — which is outside the - pool and outside this crate's GC; it is the consumer's own storage - beside the pool per ADR-004, not pool content). The facade shape on + configured with **kv-only mode** (ADR-011 §4's name for what this + bullet first called "kv-tier-only dispatch") — no local large tier + over pool content at all; a client node's *node-private, + consumer-owned* large-tier content is outside the pool and outside + this crate's GC entirely: it is the consumer's own storage beside + the pool per ADR-004, not a constructor mode over pool content, and + the composite fleet-validity rule (backends-and-dispatch.md) + therefore has no "local on node-private media over one pool" entry. + The facade shape on a client node is its own store for sub-threshold content plus its registered ops-put path for large content — a consumer-side composition, not a store-level mode. -- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects - as the fs tier's storage — the same "engine behind one contract" - move as ADR-007, one tier over. Named here so the door is explicit, - with its admission gated on measured evidence (POC #7: LO write/read +- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects + as the large tier's storage — + the same "engine behind one contract" + move as ADR-007, one tier over. Originally named as a candidate, + admission gated on measured evidence (POC #7: LO write/read curves at the packfile regime, tx-scoped handle cost under pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan - behavior) and its own engine ADR. **Not shipped speculatively;** + behavior) and its own engine ADR. NOT shipped speculatively; REQ-2's immediate fleet answers are shared media or re-routing; pg-lo is the consolidation option if those are unacceptable. *Update 2026-10-03: POC #7 ran and passed its gate — `docs/research/poc-pglo-findings.md` (companion-table authority, `lo_get` window gets, durable put ≈ fs's durable put, cached gets - behind page-cache fs by a measured 20-50x aggregate-multiplied to - ~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture). - The engine ADR's remaining evidence path is open; admission remains - gated on it, not on speculation.* -- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at + behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s + aggregate at 16 readers). Superseded the same day: ADR-009 admitted + the engine — this bullet's original "candidate, not shipped" + posture is historical; the engine is shipped, feature `pg-lo`.* +- **The no-mixed-large-tiers rule is a deployment invariant, enforced at the enforceable seam.** Cross-node configuration cannot be validated by any one constructor (it sees only its own node). What the constructor *can* enforce: a fleet-mode constructor (fleet engines - active) requires an explicit fs-tier declaration — `local` (with the + active) requires an explicit large-tier declaration — `local` (with the operator's assertion that its root is fleet-shared media), routed - (no local fs over pool content), or — when shipped — `pg-lo`. The + (kv-only mode, ADR-011 §4; no local filesystem over pool content), or + `pg-lo`. The cross-node truth ("is node B's `local` actually the same media as node A's?") is a deployment invariant this crate documents and ops verifies, not something a constructor can prove; a partitioned tier's observable signature (get fall-through misses for entries - another node wrote) is the documented detection symptom. + another node wrote) is the documented detection symptom. The + composite fleet-validity predicate over both tiers is pinned in + backends-and-dispatch.md ("The composite fleet-validity rule"). -In sum: a fleet's fs tier over pool content is either shared media (all +In sum: a fleet's large tier over pool content is either shared media (all nodes' `local` roots on the same fleet-shared media) or routed (one storage node); mixing per-node-local media over one pool is the partitioning failure this decision exists to prevent. @@ -211,9 +226,10 @@ partitioning failure this decision exists to prevent. holds the lock, deletes re-arbitrate in-transaction. The ADR-005 invariant extends to fleets as a *stated* mechanism, not an accident of single-process luck. -- The fs tier's fleet problem has named, shipped-code-free answers - (shared media, re-routing) and a named consolidation candidate - (pg-lo) whose admission path is the same POC-first door as ADR-007. +- The large tier's fleet problem has named answers: shipped-code-free + for shared media and re-routing, shipped-code for the consolidation + option (pg-lo — admitted by ADR-009, the same POC-first door as + ADR-007). **Negative** @@ -223,9 +239,9 @@ partitioning failure this decision exists to prevent. surface for orphaned pins at node crash — the expiry is the cleanup handle), advisory-lock coordination, and SQL-level arbitration added on top of the already-intricate delete-window state machine. -- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot - re-route pays for its ADR + POC before shipping large blobs — the - measured, deliberate deferral of the ADR-006 pattern, not a hedge. +- `pg-lo` was built under ADR-009 after POC #7 passed — the + measured, deliberate sequence of the ADR-006 pattern (admission + gate first, engine second), not a hedge; resolved same-day. **Neutral** @@ -245,13 +261,16 @@ partitioning failure this decision exists to prevent. columns; the driver stack pg-lo would ride) - ADR-002 (the one-way-door discipline this trait change applies), ADR-003 (the trait contract and admission gate `size` and pg-lo - amend/enter), ADR-004 (tier≠engine, extended to the fs tier), + amend/enter), ADR-004 (tier≠engine, extended to the large tier), ADR-005 (the GC invariant and single-node protocol extended here, not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission follows), ADR-007 (the engine-behind-one-trait precedent) - [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait - and tier surface this amends); - [store-api.md](../store-api.md) (`stat`'s contract); - [gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC - mechanism); [ops-surface.md](../ops-surface.md) (the re-routing - topology's transport) \ No newline at end of file + and tier surface this amends); + [store-api.md](../store-api.md) (`stat`'s contract); + [gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC + mechanism); [ops-surface.md](../ops-surface.md) (the re-routing + topology's transport); ADR-010 (the fleet mechanism's state home + and the joint-tx ownership this ADR's §2 implied but did not rule); + ADR-011 (the vocabulary — instance/fleet — this ADR's topology + language resolves to; the large tier renamed `large`) \ No newline at end of file diff --git a/docs/architecture/decisions/009-pg-lo-fs-tier-engine.md b/docs/architecture/decisions/009-pg-lo-large-tier-engine.md similarity index 87% rename from docs/architecture/decisions/009-pg-lo-fs-tier-engine.md rename to docs/architecture/decisions/009-pg-lo-large-tier-engine.md index 6ae30e2..b354ffe 100644 --- a/docs/architecture/decisions/009-pg-lo-fs-tier-engine.md +++ b/docs/architecture/decisions/009-pg-lo-large-tier-engine.md @@ -1,14 +1,17 @@ -# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine +# ADR-009: pg-lo — postgres Large Objects as the large tier's second engine ## Status Accepted (admission evidence: POC #7, passed 2026-10-03 — `docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's -gate) +gate. The tier this ADR calls "fs" is renamed `large` by ADR-011); +§2's "put = one transaction" tx-ownership statement is corrected by +ADR-010 §2 (the store core holds the tx; the engine contributes its +staged-put mechanics into it — engine content unchanged)) ## Context -ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's +ADR-008 named `pg-lo` as the candidate large-tier engine for REQ-2's fleet topology (multiple store instances sharing one pg pool, serving large/beyond-threshold pool content) — expressly gated on the ADR-003 admission door: measured evidence first, in the ADR-007 shape. POC #7 @@ -38,7 +41,7 @@ The gate's three legs all passed: ## Decision -**The fs tier ships `pg-lo` as its second engine — postgres Large +**The large tier ships `pg-lo` as its second engine — postgres Large Objects — behind the same `Backend` trait, feature `pg-lo` (default- off), constructor-selected per node.** REQ-2's deployment choice set (ADR-008) extends from {shared media, re-routing} to include the @@ -58,7 +61,7 @@ consolidation option as a shipped engine. into the LO and commit the row on completion; rollback discards both. 3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4): - the fs-tier handle closes over the companion row (oid + length) — + the large-tier handle closes over the companion row (oid + length) — immutable, so the row is authoritative — not over a tx-scoped descriptor. Ranged reads are single stateless statements, pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor @@ -82,7 +85,7 @@ consolidation option as a shipped engine. (committed LOs without companion rows — mixed/legacy stores). The crash story is structurally clean: LO lifecycle is transactional in modern postgres; kill mid-tx leaves zero pages, zero - half-commits (cleaner than fs `local`'s stage-file residue, + half-commits (cleaner than `local`'s stage-file residue, which needs its own recovery sweep). 6. **Performance deltas, as named capacity facts (not hidden, not solved):** single-stream cached get runs 20–50× behind page-cache @@ -105,13 +108,13 @@ consolidation option as a shipped engine. **Positive** - REQ-2's consolidation option is real and shipped: a fleet node runs - *postgres only* — kv (bytea) tier + fs (LO) tier from one engine, + *postgres only* — kv (bytea) tier + large (LO) tier from one engine, no shared-media requirement, no re-routing topology, no consumer-ops dependency for pool content. - The engine adds ~400 lines of impl over an already-fixed contract — the additive shape both ADR-007 and ADR-008 projected, now evidenced. -- Crash hygiene is *better* than fs `local` (transactional LO +- Crash hygiene is *better* than `local` (transactional LO lifecycle vs. stage-file residue); the engine's only recovery sweep targets a class (bypassed-LO) that healthy deployments never produce. @@ -123,9 +126,9 @@ consolidation option as a shipped engine. — the fleet case — scales to ~700 MB/s on the POC box; per-box numbers shift with cores.) - LO catalog space is never returned (monitor + occasional - operator-run `VACUUM FULL`), the inverse of fs `local`'s + operator-run `VACUUM FULL`), the inverse of `local`'s immediate-return. -- Feature/maintenance surface: the fs tier now carries two engines +- Feature/maintenance surface: the large tier now carries two engines (like the kv tier), each with posture deltas to keep honest (ADR-007's negative, inherited). @@ -147,10 +150,10 @@ consolidation option as a shipped engine. - [requirements.md](../requirements.md) — REQ-2 (the fleet topology this engine serves); REQ-1/3/4 unchanged postures - ADR-003 (the trait contract + admission door exercised twice now), - ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC + ADR-004 (tier ≠ engine, large tier edition), ADR-005/008 (the GC invariants this engine participates in — table-level, nothing fs-specific), ADR-007 (the engine-admission precedent and driver stack) -- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs +- [backends-and-dispatch.md](../backends-and-dispatch.md) (the large-tier engine surface this amends); [overview.md](../overview.md); [store-api.md](../store-api.md) \ No newline at end of file diff --git a/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md b/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md new file mode 100644 index 0000000..bf8289b --- /dev/null +++ b/docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md @@ -0,0 +1,325 @@ +# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam + +## Status + +Accepted + +## Context + +The engine story closed at the decision level (ADR-007/008/009), but +three seams that implementation must not guess are unpinned. All three +concentrate where the tier/engine split stretched the original +contract: + +1. **The `get`/`put` value shapes were never unified.** The trait POC's + `get` returns `Option>`; the largeblob POC's fs arm "yields + a file handle, not a loaded buffer"; the pg-lo POC's get is a + `LoGetHandle` driving `lo_get(oid, off, len)` windows — three + structurally different shapes behind one trait, and no document + named the unified contract. Same on the put side: a byte slice + (kv), a stage-then-commit-rename stream (`local`), and a + chunked-`lowrite` transaction (pg-lo) are all "the put". A unified + trait is the crate's second stable seam (ADR-003); its I/O shapes + are decided before code, not discovered per engine. + +2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying + rows committed with the entry; staged delete-window candidates + (including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg + advisory lock for the sweeper. None of that is expressible through + the public trait method set, and ADR-003's "the shipped trait + impls write plain content-addressed rows into the chosen engine, + nothing else" reads as a bar the fleet mechanism violates. Also + unresolved: who owns the joint entry+pin transaction — ADR-009 + says the pg-lo put's single tx carries the pin row, but ADR-005 + says `Pin` is a store-core RAII guard under the arbitration lock. + The trait has no parameter for either, and "the backend learns + about pins" would contradict ADR-005. The ops pin-token contract + (ops-surface.md) inherits the same conflict: a remote putter holds + a token, but an expiry-guarded pin row goes stale if the holder + pauses — a sweep could delete a remotely-in-flight put whose row + expired. + +3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains + the large tier's fleet forms (shared media, re-routing) but pins + nothing for the kv tier, which the fleet story centers on: two + nodes' sqlite engines are trivially partitioned (two files = two + pools, not one), and no document said so. + +The deciding facts are in hand (Schrödinger's-code rule — none of +these may wait on consumers): the POCs fixed every engine's native +shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry, +advisory lock, staged window); ADR-001 §4 fixed that ops is a +store-facade consumer with no backend side-door. What is missing is +composition: how the store core drives all engines through one +contract while owning GC state. + +Alternatives considered for the fleet-state seam, rejected: + +- **Put the fleet machinery on the public trait** (pin/window/table + methods on `Backend`): every engine — including `mem` and `local` — + must then implement SQL-shaped machinery it cannot express. Rejected. +- **Backends grow pin awareness** (the pg-lo put tx carries the pin + row, backend-maintained): contradicts ADR-005's "backends never + learn liveness"; and the arbitration decision (expiry compare, + liveness consult) still belongs to the store, so the store ends up + reaching into the engine's tx anyway. Rejected. +- **A raw-SQL side-channel** (connection escape hatch the store core + may use arbitrarily): turns the engine abstraction off for exactly + the machinery (fleet GC) it most needs to be sound on. Rejected. + The seam below is that escape hatch made honest — named clauses, + contract-tested per engine, instead of raw access. + +## Decision + +### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put + +**The trait stays one trait.** Every engine implements the same method +set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split. +What changes: the trait's I/O types are the crate-internal +abstractions below, which each engine constructs its own way — the +store core programs the abstraction, never the engine shape. + +- **`get` returns a read cursor.** One abstraction, two body families: + - **kv engines** (sqlite, postgres, mem): the cursor materializes + the whole value (the engine's row read; bounded by the small-blob + tier policy). Its range reads are in-memory slices; `len()` reads + the row's `size` column. + - **large engines** (`local`, pg-lo): the cursor streams; the medium + is hidden in the cursor body — `local`'s is a pread loop over the + committed file (immutable once committed); pg-lo's drives + descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4 + posture — pool-friendly, stateless per window; the + tx-scoped-descriptor form is the fallback). `len()` reads the + companion row or stats the file — metadata only, never content + (ADR-008's discipline). + - Both body families serve the ADR-006 range contract (`read_range` + → slice + slice digest) at one seam. The cursor is + **store-internal — not public API**: store consumers see the + facade's `(len, stream)` shapes (store-api.md) with the cursor + staying behind them; the store's stream utility adapts engine + cursor bodies, it does not expose them. +- **`put` exists in two forms, selected by the store core before + content flows; both land in the CAS entry shape and both + participate in pin-before-publish (§2 decides who holds the tx).** + - **Whole-value put** for the kv tier: the settled row write. + POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres + `ON CONFLICT DO NOTHING`, mem `entry().or_insert()`. + - **Staged put** for the large tier — ADR-003's + stage-then-commit-rename promoted from behavior to a **named + per-engine contract**: the engine hands out a *staging cursor* + (`local`: the stage file under the store's stage-hygiene protocol; + pg-lo: chunked `lowrite` statements over a tx the store core + holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's + measured knee). Commit publishes the entry; + every failure path converges on discard (store-api.md stage + hygiene, per engine, exact-count tested — POC #3 A5's shape). +- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list` + walks the engine's companion structure (rows / files / LO + companion); `size` is metadata-only (ADR-008). The exact-count + sweep-outcome gate (ADR-003) runs per engine. + +**Feature-graph consequence** (fills a gap ADR-007/009 left): feature +`pg-lo` does **not** imply feature `postgres`. Each is an engine +feature pulling the shared driver cluster (`tokio-postgres` + +`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite +node is coherent and supported. overview.md's dependency table pins +the edge. + +### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested + +**Ownership: the store core owns all GC state — in-process pin and +liveness maps *and* the fleet state's rows and locks. The public +`Backend` trait stays exactly what ADR-003 says (opaque +`hash → bytes`, namespace-blind; "plain content-addressed rows, +nothing else" remains true of the public trait). Backends never learn +liveness (ADR-005) — verbatim.** + +The store core reaches the fleet machinery through a **crate-internal +(not exported) engine-state companion trait**, implemented by the +SQL-backed engines (sqlite, postgres, pg-lo), with `local` and +`mem` excluded by construction (it cannot be expressed on them and +they are never fleet engines). ADR-008's mechanism fixes *what*; the +seam is *where* the mechanism's SQL lives. Clauses: + + - `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and + `gc_window (key, visible_after)` at store construction (store-core + call, engine-neutral API; dialect-specific SQL bodies). `owner` + is the **store-instance id** (ADR-011's vocabulary; ADR-008's + "node id" wording predates that split). The pin row's identity is + `(key, owner)` — a dedup re-put by a different instance is a + different pin row; protection composes by row (an entry is + protected while any un-expired pin row exists), which is + ADR-005's refcount semantics expressed in rows. The embedder's + liveness table is *not* created by this seam — ADR-008: schema + pinned by the embedder; the sweep reads it and never owns it. + **Per-engine applicability (the test-gate shape):** sqlite — + `ensure_gc_schema` (window table only, since the window staging + is shared machinery per ADR-008 §2) + `window_stage`/ + `window_take` + the sweep-lock clause's single-node body; the + pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite + bodies but are exercised by unit tests only — they are dead + machinery on sqlite because no sqlite fleet is on offer (§3). + postgres and pg-lo — all clause families, contract-tested, + fleet-active. +- `pin_commit(joint_tx, entries)` — the store core **holds the + transaction**; the engine contributes its put mechanics (row + shape, CAS clause, LO writes) and the pin-row inserts, and both + commit together. This is the joint entry+pin transaction's owner, + ruled: **the store holds the tx; the engine contributes mechanics + to it.** The public `put` remains the plain, pin-free path — the + store's normal put path routes through `pin_commit` wherever the + engine is seam-capable and fleet state is active. + - pg-lo correction: ADR-009 §2's "put = one transaction" is + unchanged in content but its ownership statement is hereby + corrected — the tx is held by the store core; the pg-lo engine + contributes its staged-put mechanics and companion-row insert + into it. The engine sees pins only as rows; never as semantics. +- `pin_renew(key, owner, ttl)` — the engine executes the expiry + update. The store's pin guard owns the TTL/3 renewal schedule + (ADR-008); the engine owns only the SQL. +- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine + executes ADR-008's delete-time arbitration (un-expired pin rows + + embedded liveness tables, `SELECT … FOR UPDATE` finger-lock, + keyed, at delete time); the store core receives the answer and + applies the decision. If fleet clauses ever grow, they grow here — + the seam is the named home, with contract tests per clause. +- `window_stage(candidates)` / `window_take(now)` — the staged + delete-window candidates. Per ADR-008 §2 the window table exists + on **every** engine (sqlite included): staged candidates are + engine rows; what differs by topology is where *liveness* lives + (in-process maps for single-node engines, tables for fleets), not + whether staging exists. +- `sweep_lock_acquire()/release()` — the sweep single-flighter: the + pg advisory lock (ADR-008) for the SQL engines it applies to; the + store treats lock contention exactly as ADR-008 says — the second + sweeper's sweep aborts with the typed `GcAborted` (fleet cause), + nothing deleted. (SQLite today has no fleet story — see §3 — so its + clause body exists but is exercised by single-node tests only.) + +Each clause is contract-tested per its applicability (the +per-engine split above): the conformance suite of §1 grows the fleet +clauses for postgres and pg-lo; `local` and +`mem` are excluded by construction. + +### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3) + +**A fleet constructor requires a pool-shared kv engine.** Postgres is +the shipped engine that satisfies this. A sqlite-fleet mechanism does +not exist and is not on offer (per ADR-007's redb precedent this is +recorded as a non-option, not an open question); two nodes' sqlite +engines over "one pool" are two pools — the partitioning failure, the +kv-tier edition of what ADR-008 §3 prevents for the large tier. The +enforceable +seam is the same as ADR-008 §3's: the constructor can require an +explicit declaration; it cannot prove cross-node truth. The +constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md) +and is closed under both tiers. + +### 4. The ops pin-token contract's TTL race, resolved by the ownership rule + +**The token *is* the pin — there is no second pin state.** Under fleet +engines the put's pin is the expiry-carrying row (ADR-008); the token +identifies the putter as the row's owner. The token itself is an +**opaque id minted by the ops layer** over the pin row it just +committed — the ops layer (not the store facade) owns the +token↔(key, owner) mapping, in-memory for the handler's lifetime plus +a lookup by the renewal op; no schema is specced for it beyond that +(it is a handle, not durable state — the durable state is the pin row +the seam wrote). Concretely: + +- **When the server-side `Pin` guard drops:** at response time. The + put lands pinned (the joint tx guarantees the entry and its pin row + commit together); the handler's in-process guard — the thing ADR-005 + pins *in-process* with — is released when the `blobs/put` response + completes and the token is handed out. From that moment the pin + row's expiry is the only protection, which is exactly why the + renewal duty below exists. (If the handler kept its guard for the + token's whole life, the TTL machinery would be dead code on the ops + path — the row's expiry, not the guard's presence, is the fleet + contract's liveness signal.) +- The putter renews the pin by renewing the row — through ops it + already speaks: a `blobs/have` probe whose handler, observing a + valid token presented alongside the digest, calls the + crate-internal `pin_renew(key, owner)` via the ops module's own + store-core access (the ops module is in-crate — it crosses the + crate-internal seam by construction, the same as the put handler + holds the `Pin` guard; what never happens is a *consumer* reaching + the seam. ADR-008's `has` remains a pure probe: the renewal is the + ops layer's token-authenticated act on the pin row, not a `has` + semantic), or a re-put of the same digest (CAS dedup; the re-put's + joint tx upserts the key+owner pin row, renewing it). The renewal + cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 — + 20 s at the default 60 s TTL.** +- A putter pausing past TTL falls to ADR-008's expiry semantics + verbatim: the row is abandoned and reaped; the backstop is + delete-then-recover (ADR-005) — the digest is re-offered, the put + is idempotent under CAS. The race resolves to a re-put, never to + silent loss. +- For the `local` engine the token is the in-process guard as ADR-005 + had it: nothing persists, nothing expires; the process crash is + ADR-005's in-flight-put loss case with the same backstop. (The + cross-wire token under `local` is the ops layer's in-memory handle + to that guard — same mapping discipline, no expiry.) + +ops-surface.md's hand-over contract gains the renewal line and the +token-mapping note; nothing else changes. + +## Consequences + +**Positive** + +- The trait's I/O shapes are decided before code; engine tasks can be + written without guessing signatures. The per-engine + conformance-suite shape (POC #7's 10/10 gate as the norm) gives + decomposition a uniform test gate across all five engines. +- "Backends never learn liveness" survives the fleet mechanism + intact; the store core owns GC state per ADR-005 exactly as + written; ADR-008's mechanism lives in a named, tested seam instead + of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing + else" stays true of the public trait. +- The ops pin-token TTL race is resolved with no new ops; the + backstop is already-validated semantics (delete-then-recover). +- The kv tier's fleet-validity rule closes the constructor table + under both tiers. + +**Negative** + +- A crate-internal companion trait is real machinery: six clause + families on postgres/pg-lo (fleet-active, all contract-tested); + sqlite carries only the window + lock clauses as fleet-relevant + machinery (its pin clauses have unit-test bodies but no fleet mode + on offer — §3). The seam bounds where fleet-GC growth lands; it + does not shrink it. +- Remote putters gain a real protocol duty (renew past 20 s; + pause-past-TTL ⇒ delete-then-recover) — documented, but a duty. +- The sqlite sweep-lock clause body exists but is exercised only by + single-node tests until a sqlite-fleet posture is ever + decided (it is not on offer — §3). + +**Neutral** + +- The `Backend` trait's method set is unchanged from ADR-003/008 — + no public API change beyond what store-api.md already specced; the + cursor/staging abstractions are crate-internal. +- The staged-put contract already existed behaviorally (POC #3 A5); + this ADR names it. +- ADR-009's engine content is unchanged; only its tx-ownership + statement is corrected here. + +## References + +- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004 + (tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the + liveness-ownership rule this ADR preserves), ADR-006 (the range + contract the cursor serves), ADR-008 (the fleet mechanism whose + state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership + correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md) + (vocabulary + constructor table) +- `docs/research/poc-trait-dispatch-findings.md` finding 1; + `docs/research/poc-largeblob-findings.md` findings A1/A5/A6; + `docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the + 512 KiB knee, windowed gets) +- [ops-surface.md](../ops-surface.md) (the pin-token contract §4 + extends); [store-api.md](../store-api.md) (stage hygiene, sweep + semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md) + (REQ-2, the fleet requirement §2/§3 serve) \ No newline at end of file diff --git a/docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md b/docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md new file mode 100644 index 0000000..2b26877 --- /dev/null +++ b/docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md @@ -0,0 +1,225 @@ +# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large` + +## Status + +Accepted + +## Context + +The engine story (ADR-007/008/009) grew the physical layer faster than +its vocabulary grew, and the pre-decomposition review found the drift +is now self-contradictory in the anchor document itself: + +1. **Node vs store instance.** requirements.md defines **Node** as + "one running store instance" — then REQ-2 states "a replicator node + may run multiple store instances (a fleet)" and **Fleet** as "two + or more nodes over one shared pool". Under the written definition, + one process hosting two instances is simultaneously one node (by + definition) and two nodes (by REQ-2), and the GC arbitration + domain ("node is the GC arbitration domain") is undefined for the + exact topology ADR-008 exists for. ADR-010 fixes pin ownership; + the pin's *owner id* and the node/instance distinction it needs + are still unpinned vocabulary. +2. **The "fs" tier name is now wrong.** The tier was named when its + only engine was a filesystem (ADR-003's `local` layout). After + ADR-009, half its engines are postgres Large Objects — the name + describes one engine's medium, not the tier. Worse, "fs" collides + with the dispatch language: "large content goes to fs" never said + *why* — the tier's actual routing criterion is content length + (ADR-003's size-threshold dispatch). The docs already speak the + true name informally: store-api.md says "small tier"/"large tier". +3. **mem's classification drifts.** backends-and-dispatch.md calls it + a "backend" in one place and a "testing/utility tier" in another; + ADR-003 lists it under "shipped backends … plus a testing tier"; + ADR-004 calls it an ephemeral tier and carves it out of the + two-backend count. In truth it is neither a tier nor a separate + backend: it implements exactly the kv tier's contract (including + `size`, trivially), it is the contract-reference engine every + other engine's conformance tests mirror, and "engine" names what + it is (ADR-007 established that an engine is a concrete + implementation of one tier's contract — requirements.md already + defines the word this way). +4. **Tier and Backend were never formally defined** even though + ADR-004's scope note ("the 'two' here counts tiers, not engines") + leans on both words. Also undecided: whether the feature names + follow the tier rename now or after code exists — the last free + moment is now (zero code; after implementation the rename breaks + crates.io/feature ergonomics for every consumer). + +Deciding facts are in hand: REQ-2's fleet fact, the engine set's +final shape (ADR-007/009), the dispatch mechanism (ADR-003), and +ADR-010's GC-state-home table. This is a vocabulary-and-boundary +decision, not an architecture change: ADR-004's tier≠engine note +already established the shape; this ADR gives it final, consistent +names. + +Candidates weighed for the tier rename: + +- **`lo`/`los`** — names the *postgres* engine (Large Objects), so + "the lo tier's `local` engine" would be backwards. Rejected. +- **`lfs`** — workable, echoes git-lfs specifically; but the tier + also serves packfiles and any oversized workspace artifact, and the + echo binds the crate's naming to one consumer's protocol shape. + Rejected. +- **`large`** — the dispatch-true name (size-threshold routing), the + word the specs already use informally ("large tier"), medium-agnostic. + **Chosen.** For symmetry in prose, the kv tier stays `kv` (its + routing criterion — "small" — lives in the prose as "small blobs" + and needs no rename: `kv` names the mechanism class, and `small` + as a feature/tier name would read as an adjective; the pair's + asymmetry is cosmetic, the *load-bearing* correction is that + neither tier is named after a medium). + +Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it +names the engine's mechanism, accurate); `postgres` keeps its name +(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep +their names. + +## Decision + +### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy) + +- **Tier** — one *contract* (a `Backend` trait implementation + boundary) serving one routing class of the pool. The crate ships + exactly two tiers: **kv** (small content) and **large** (over + the size-threshold content). Tier count (2) is ADR-004's count. +- **Engine** — a concrete storage implementation of one tier's + contract (the existing definition, unchanged). Engine count (5) is + this ADR's count. +- **Backend** — *the* contract (the `Backend` trait) plus its shipped + engine set, i.e. the tier as shipped. "Backend" and "tier" name the + same thing at different altitudes; the crate says "tier" for + counting, "the `Backend` trait" for the contract. "Two backends" + reads forever as "two tiers". +- **Store instance** — one running instantiation of the store facade + (one engine set — one or two engines per the constructor mode, §4 — + plus dispatch and GC state). **The GC + arbitration domain** for single-instance topologies; the pin-row + owner id under fleet engines (ADR-010's `owner`; this corrects + ADR-008's "node id" wording, which predates the instance/node + split — a store instance is what owns pins). +- **Node** — one running *process* (an embedder's binary). A node + hosts **one or more store instances**. Node is the process/ops unit: + alkcall connections, ACL grants, sweep cadence, and deployment + identity attach to nodes; GC arbitration and pin ownership attach + to store instances. +- **Fleet** — **two or more store instances over one shared pool** + (via a shared SQL engine). Deliberately not defined by node count: + two instances in one process over one postgres pool are a fleet + under this definition (and REQ-2's phrasing "a replicator node may + run multiple store instances (a fleet)" reads true as written); + two processes on two sqlite files are two pools, never a fleet. +- **Deployment** — unchanged (one operator's whole system). + +### 2. mem joins the kv tier as its third engine + +`mem` is the **kv tier's ephemeral engine** (feature `mem`, +default-off): a `BTreeMap`-shaped implementation of the kv contract, +including `size` (trivially) and list; never fleet-valid (ADR-010 §3's +rule and this ADR §4's constructor table); the **contract-reference +engine** — every other engine's +conformance tests mirror its suite. It is not a tier, not a third +backend, and the ADR-004 count ("two") survives untouched because +tier and engine were never the same count — this ADR is that +distinction's final form. + +### 3. The fs tier is renamed `large` + +In every document, status line, feature name, and constructor +parameter: tier `fs` → tier `large`; feature `fs` → feature `large` +(default-on, as `fs` was); dispatch prose says "large tier"/"small +tier" as store-api.md already does; the `local` engine's name is +kept (its meaning — this node's own filesystem — stays the +constructor's meaning even under the shared-media fleet posture, +where it is explicitly *not* only-node-local; the constructor's +explicit shared-media assertion, ADR-008 §3, is what disambiguates). + +### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here) + +| Tier | Engine | Feature | Default | Fleet-valid | +|---|---|---|---|---| +| kv | sqlite | — (tier default) | yes (tier default-on) | no | +| kv | postgres | `postgres` | no | **yes** | +| kv | mem | `mem` | no | no | +| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion | +| large | pg-lo | `pg-lo` | no | **yes** | + +Constructor modes (dispatch shapes) — pinned here (implied by +ADR-008 §3's re-routing bullet and the mem tests, but never named): +- **dual-tier** (the default): kv + large, size-threshold dispatch + between them (ADR-003). +- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a + constructor mode with no large tier; over-threshold puts are a + constructor-time error (the client-node re-routing posture handles + them via the ops surface instead — its composition, not a store + mode). +- **mem-only** (test/embedder-ephemeral mode): the mem engine alone, + no dispatch threshold in effect (everything is local and small); + a testing posture, explicitly not a production story. +- Single-tier SQL modes (postgres-only, pg-lo-only) are **not** + constructor modes this crate ships: the two-tier shape is + ADR-004's requirement (both tiers are load-bearing by measured + economics); "postgres-only" as in *one SQL instance serving both + tiers* already exists (kv=postgres + large=pg-lo — the ADR-009 + consolidation), which is the dual-tier mode over one engine. + +### 5. Deferral-policy wording (README alignment) + +The deferral policy's categories are: `deferred(scope)`, +externally-owned questions, and decided-but-sequenced work — three +names, defined in open-questions.md's header. README.md's recap says +"two parking kinds" and is corrected to name all three (the third is +not a *parking* kind — it is work, not a parked question — hence +this ADR's wording here rather than a silent edit). + +## Consequences + +**Positive** + +- The anchor document's self-contradiction (node/fleet) becomes a + three-level vocabulary (store instance / node / deployment) whose + levels map one-to-one onto the mechanisms (GC domain / process-ops + unit / operator whole). +- The fs→large rename removes the last medium-named tier; feature + names, tier names, and dispatch prose now agree, before any code + exists to break. +- mem's classification is stated once, as the contract-reference kv + engine; the "plus a testing tier" carve-outs dissolve. +- The constructor table gives task decomposition a closed set: 5 + engine implementations, 3 constructor modes, 1 dispatch policy. + +**Negative** + +- Vocabulary churn in every doc that says "fs" (the rename is + mechanical but touches all specs and ADR-003/004/007/008/009 status + lines and reference sections). Zero code exists — this is the last + free moment; after Phase 1 starts, feature renames break consumers. +- "Backend"/"tier" as interchangeable words risks reader confusion + with ADR-003's "Backend trait" — mitigated by the rule that the + *trait* is always named with "trait" ("the `Backend` trait"), never + bare "the backend". + +**Neutral** + +- No engine's contract, feature flag semantics, or constructor + parameter changes; `mem` gains the word "engine" and loses the + word "backend". +- REQ-2's meaning is unchanged (the fleet definition is re-derived + from the same fact, not re-fact'd). + +## References + +- ADR-003 (the trait contract; its "shipped backends" list is now + read through this ADR's vocabulary), ADR-004 (tier≠engine — the + scope note this ADR gives final names), ADR-007 (kv engines; + constructor selection), ADR-008 (the second tier's engine candidate + and the fleet topology; the vocabulary it assumed is now pinned), + ADR-009 (pg-lo admission; the consolidation option whose + constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md) + (the I/O seams and GC-state home this vocabulary must fit) +- [requirements.md](../requirements.md) (the canonical vocabulary + copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md) + (the tier/engine table's spec home) +- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the + reference-impl role mem already played in the POCs); + `poc-pglo-findings.md` (the gate the engine suite mirrors) \ No newline at end of file diff --git a/docs/architecture/gc-and-namespaces.md b/docs/architecture/gc-and-namespaces.md index 10d78ac..2460c8e 100644 --- a/docs/architecture/gc-and-namespaces.md +++ b/docs/architecture/gc-and-namespaces.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-03 +last_updated: 2026-10-03 (ADR-010 state-home rule + ADR-011 instance/fleet vocabulary) --- # Pooling, namespaces, and GC @@ -16,17 +16,23 @@ price, and this document specifies the mechanism the POCs validated ## The pooled CAS -- **One pool per deployment; a node is one store instance** (the - engine/vocabulary definitions live in requirements.md). Repos, - workspaces, and appfile stores are all +- **One pool per deployment; a store instance is one GC arbitration + domain** (the engine/vocabulary definitions live in requirements.md + / ADR-011: a node — one process — may host several instances; the + instance is where pins and arbitration live for single-instance + topologies). Repos, workspaces, and appfile stores are all *sets of hash references* over the same CAS — never walled-off per-repo stores. Git's `alternates`/object-pool mechanism (GitLab object pools) is the same idea done awkwardly; the pool generalizes it - to unrelated repos and to p2p replication. **A fleet (REQ-2) — two or - more nodes sharing one pool — is the one topology where the GC - mechanism below extends beyond a node: ADR-008 pins, sweeper - locking, and delete arbitration move into the shared engine; - the invariants are unchanged.** + to unrelated repos and to p2p replication. **A fleet (REQ-2) — two + or more store instances sharing one pool — is the one topology where + the GC mechanism below extends beyond one instance: ADR-008 pins, + sweeper locking, and delete arbitration move into the shared engine; + the invariants are unchanged. Where this crate's GC state (fleet + rows and locks) physically lives, and who holds the joint + entry+pin transaction, is ruled by ADR-010 §2: the store core owns + all GC state; SQL-backed engines host it through a crate-internal, + contract-tested seam; backends never learn liveness.** - **One address space.** Under the canonical hash (ADR-002), the same file content has the same address whether it entered via a git oid or a workspace manifest — cross-consumer dedup is free by construction, @@ -46,7 +52,8 @@ price, and this document specifies the mechanism the POCs validated `register_liveness_source`; the POC's copy-semantics failure finding 7 is the pinning reason). Namespace tables may also exist purely in-memory for ephemeral consumers; nothing in the store persists - namespaces — where roots live when a deployment runs fs-only is a + namespaces — where roots live when a deployment has no SQL engine (no + kv engine with rows) is a non-question by design: roots live with the consumer, whose references are as durable as they need to be. - ACL mapping for network ops: the namespace is the resource @@ -112,7 +119,7 @@ existence flips atomically per key). - **Accounting** (per-namespace sizes): computed from reference sets by consumers; the store reports pool-level totals only (`list` + `stat`). Physical-layer accounting would re-introduce namespace - physics — rejected with the per-namespace backend config + physics — rejected with the per-namespace engine config (ADR-003 §Consequences). ## Design Decisions @@ -122,7 +129,8 @@ existence flips atomically per key). | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling | | [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate | | [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | GC-state home | the store core owns all GC state; SQL engines host it via the contract-tested engine-state seam; pin-tx ownership ruled | ## Open Questions diff --git a/docs/architecture/hashing-and-keys.md b/docs/architecture/hashing-and-keys.md index d6919a1..b64a657 100644 --- a/docs/architecture/hashing-and-keys.md +++ b/docs/architecture/hashing-and-keys.md @@ -17,7 +17,7 @@ residual). ## The hash family -Three tiers (ADR-002 §Decision; resolved from OQ-BL-03 after the +Three hash-policy classes (ADR-002 §Decision; resolved from OQ-BL-03 after the "multi-hash" premise dissolved — nothing forces BLAKE3 once iroh-blobs' inheritance is rejected): diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index 4f5f4e9..047b0d6 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -27,7 +27,8 @@ not one. Two kinds of parked question remain legitimate: no decision here; it would be Schrödinger's code if it gated one). Invalid for either kind: a blocker that is a decision this crate must -make before shipping. Phase 0's residual list was swept under this +make before shipping. (A third category — **decided-but-sequenced +work** — is not a parked kind at all; see below.) Phase 0's residual list was swept under this rule: every residual either resolved into an ADR (the evidence was already in hand) or re-owned as below. @@ -38,7 +39,7 @@ OQ-08). Deferring on a consumer's *deployment of this crate* is invalid — that fact is created by this crate (OQ-10's original framing, OQ-09's original framing). And a third thing is neither: **decided-but-sequenced work**. Once a question resolves to a decision -(the pg-lo engine is the named fs-tier candidate under REQ-2/ADR-008), +(the pg-lo engine is the named large-tier candidate under REQ-2/ADR-008), what remains is the evidence pipeline the decision's own admission gate requires (POC #7) — a task with an owner, not a parked question, and not a hedge. A decision is not "unmade" because its verification @@ -246,12 +247,12 @@ a deployment trigger — the same invalid shape, caught at review: the trigger was a fact only this crate could create. It resolved to ADR-007 (two kv engines; see its entry above). -The pg-lo fs-engine question (raised in the ADR-008 review round, +The pg-lo large-engine question (raised in the ADR-008 review round, 2026-10-03) never became a parked question: REQ-2 made the multi-instance-with-large-blobs topology a standing requirement (requirements.md — a planning fact predating all POCs), so there was nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the -candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`) +candidate large-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`) as its admission evidence — sequenced work, per the deferral-policy header's third category. That work completed the same day: POC #7 passed (`docs/research/poc-pglo-findings.md`, gate verdict passed diff --git a/docs/architecture/ops-surface.md b/docs/architecture/ops-surface.md index 245ae58..b96660f 100644 --- a/docs/architecture/ops-surface.md +++ b/docs/architecture/ops-surface.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-03 +last_updated: 2026-10-03 (pin-token renewal contract pinned per ADR-010 §4) --- # Ops surface (alkcall-backed network operations) @@ -96,8 +96,11 @@ the server. The hand-over contract: 1. the put lands pinned (ADR-005) — the entry cannot be swept while the handler holds the pin; -2. the `blobs/put` response returns a **pin token** (opaque handle); - the putter can later confirm its registration landed by observing +2. the `blobs/put` response returns a **pin token** (opaque handle — + minted by the ops layer over the pin row the put committed; the + token↔(key, owner) mapping is ops-layer state, not durable state — + the durable state is the pin row itself, ADR-010 §4); the putter + can later confirm its registration landed by observing the digest present via `blobs/have`; 3. the embedder's registered liveness source is the *actual* root of record: the contract on the embedder (documented on the ops module) @@ -112,6 +115,32 @@ the server. The hand-over contract: conversion on the server side (token → registered liveness source) is the mechanism the embedder plugs its registry into. +**Token renewal (ADR-010 §4 — the fleet-TTL race, resolved): the +token *is* the pin — there is no second pin state.** The server-side +handler's in-process `Pin` guard drops when the `blobs/put` response +completes (if it never dropped, the TTL machinery would be dead on +the ops path — the pin row's expiry, not the guard's presence, is the +fleet liveness signal). Under fleet +engines the put's pin is an expiry-carrying row (ADR-008: TTL 60 s +default, renewed every TTL/3). The remote putter holding a token is +the row's owner and renews by the ops it already speaks: a +`blobs/have` probe **presenting the token alongside the digest** (the +handler verifies the token maps to the digest's pin row and calls the +crate-internal `pin_renew` via the ops module's store-core access — +the ops module is in-crate and crosses the seam by construction, the +same way the put handler holds the `Pin` guard; `has` itself remains +a pure existence probe per ADR-008 — renewal is a token-authenticated +act on the pin row, not a `has` semantic), or a re-put of the same +digest (CAS dedup renews the row). **A putter may pause past the +TTL/3 mark (20 s at the default TTL) only if it renews before expiry; +a putter pausing past TTL lets the row expire** — the sweeper reaps +it (ADR-008) and delete-then-recover (ADR-005) is the backstop: the +putter re-offers the digest; CAS makes the re-put idempotent. The +race resolves to a re-put, never to silent loss. For non-fleet +(`local`-engine) serving nodes the token is the in-process guard — +nothing expires; the process-crash case is ADR-005's in-flight-put +loss with the same backstop. + Placement of the op *registration* (which embedders wire where they want them exposed) matches the alkgit ops pattern (its `git/repo/*` call ops): the module exports spec/handler pairs; the @@ -143,10 +172,11 @@ handlers call `put`; both honor pins and delete windows. There is no ops-internal path around the store facade (ADR-001 §Consequences — this is what makes "re-homing the ops surface later loses nothing" true). One fleet topology additionally rides this module: **re-routing -(ADR-008)** — a fleet whose nodes do not share fs media routes +(ADR-008)** — a fleet whose nodes do not share large-tier media routes large-blob access through a single storage node via these ops, with -client nodes' fs tiers serving only node-private content. The module -supports this shape without change; no fleet-specific op exists. +client nodes' constructors in kv-only mode (ADR-011 §4) for pool +content. The module supports this shape without change; no +fleet-specific op exists. ## Design Decisions @@ -154,7 +184,9 @@ supports this shape without change; no fleet-specific op exists. |---|---|---| | [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Pin-token renewal | the token is the pin; renewal rides have/re-put; TTL expiry falls to delete-then-recover | +| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + modes | instance/node/fleet split; kv-only mode named (the re-routing posture's client shape) | ## Open Questions diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 3083698..2496c16 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-03 +last_updated: 2026-10-03 (ADR-010/011 — I/O seams pinned; vocabulary fs→large, mem joins kv) --- # Overview @@ -9,10 +9,11 @@ last_updated: 2026-10-03 `alkblobs` is content-addressed blob storage in the alk* family: a put/get/verify store keyed under one canonical content hash (the -git-family derivation; legacy SHA-1-tolerant), with pluggable backends -(small blobs in a kv store, large blobs on a filesystem fallback), -split from any wire/ops-protocol layer. Network ops ride the alkcall -substrate behind a feature-gated `ops` module (ADR-001). +git-family derivation; legacy SHA-1-tolerant), with pluggable tier +engines (small blobs in a kv store per ADR-007/011, over-threshold +blobs on the `large` tier per ADR-008/009 — the tier formerly named +`fs`), split from any wire/ops-protocol layer. Network ops ride the +alkcall substrate behind a feature-gated `ops` module (ADR-001). Deployment-relevant consumer facts — what each use case forces on this crate — are pinned in [requirements.md](requirements.md) (REQ-1..4, @@ -63,15 +64,22 @@ or three times. │ store core (always on) │ │ hashing & keys (ADR-002) · put/get/stat/range/pin seams │ │ mark-and-sweep GC + liveness registration (ADR-005) │ -│ size-threshold dispatch (ADR-003) │ +│ size-threshold dispatch (ADR-003); GC state owned │ +│ here, hosted by SQL engines via the seam (ADR-010) │ └───────┬───────────────────┬───────────────────────────────┘ ▼ ▼ -│ kv backend fs backend (ADR-003: exactly two tiers; -│ │ ADR-007: kv = 2 engines; -│ ▼ ADR-008: fs engines too) -│ engine per node: sqlite (default) | postgres (feature "postgres") -│ fs engine per node: local (default) | pg-lo (feature "pg-lo", -│ default-off; ADR-009) +┌──────────────────────┐ ┌─────────────────────────────────┐ +│ kv tier (ADR-003/ │ │ large tier (ADR-003/008/011; │ +│ 007/011) │ │ formerly "fs") │ +│ sqlite (default) │ │ local (default) │ +│ postgres ("postgres") │ pg-lo ("pg-lo"; ADR-009) │ +│ mem ("mem"; tests + │ │ +│ contract reference) │ │ +└──────────────────────┘ └─────────────────────────────────┘ + ▲ ▲ + └── one Backend trait (ADR-003/010: read cursor + + staged put I/O seams); engine = per-node + constructor choice (ADR-011's table) transport: none, anywhere. alkgit/alkfs or the embedder dials; an alkcall Connection is handed in at the ops boundary. @@ -81,36 +89,37 @@ or three times. | Layer | Dependencies | Notes | |---|---|---| -| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks | -| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` | -| `pg-lo` feature (ADR-009) | Large Objects as the fs tier's second engine, default-off | rides the same `tokio-postgres`/`deadpool-postgres` stack as the `postgres` kv engine (SQL `lo_*` functions, no new dependency); same admission/test-gate shape | +| base crate (store core + tier engines) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `large`/`mem` | no alkcall, no serialization frameworks | +| `postgres` engine feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` | +| `pg-lo` engine feature (ADR-009) | Large Objects as the large tier's second engine, default-off | pulls the same `tokio-postgres`/`deadpool-postgres` cluster as the `postgres` kv engine (SQL `lo_*` functions, no additional dependency); **does not imply the `postgres` feature** — engine features pull the driver cluster independently (ADR-010 §1), so a `large`=pg-lo + kv=sqlite node is coherent and supported; same admission/test-gate shape | +| `mem` engine feature (ADR-011) | the kv tier's ephemeral engine, default-off | `BTreeMap`-shaped; the contract-reference engine; no dependencies | | `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 | | consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs | -Backends are feature-gated; both `kv` and `fs` ship default-on -(batteries-included, the alkgit feature-model pattern — default-on -features, opt-out rather than opt-in); the `kv` tier's postgres engine -(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only -kv node (postgres engine without rusqlite) is **not** a supported -configuration this crate optimizes for — feature-graph work for it is -deferred until a named requirement exists (the base `kv` feature -pulling rusqlite is the accepted v1 shape; convention 8's -lean-base principle applies to *default* builds, which stay lean since -`kv` is what the feature model controls, not what default-on means -downstream). Wasm: the backends are not a wasm story; a wasm client is -a consumer of the `ops` surface (which rides alkcall, itself -wasm-clean), never an in-process embedder (ADR-001 §Consequences). +Tiers are feature-gated; both default-on (batteries-included, the +alkgit feature-model pattern — default-on features, opt-out rather +than opt-in); the engine features `postgres` and `pg-lo`, and `mem`, +are default-off, as are `ops`. A pg-only kv node (postgres engine +without rusqlite) is **not** a supported configuration this crate +optimizes for — feature-graph work for it is deferred until a named +requirement exists (the base `kv` feature pulling rusqlite is the +accepted v1 shape; convention 8's lean-base principle applies to +*default* builds, which stay lean since `kv` is what the feature model +controls, not what default-on means downstream). Wasm: the tier +engines are not a wasm story; a wasm client is a consumer of the `ops` +surface (which rides alkcall, itself wasm-clean), never an in-process +embedder (ADR-001 §Consequences). ## Non-goals (each enforced by an ADR) - No wire framing, op dispatch, ACL, or transport inside the store layer (ADR-001). -- No adopted iroh-blobs wire surface — no tickets, no postcard, no - provider protocol (AGENTS convention 7; ADR-001 §Context). +- **No adopted iroh-blobs wire surface — no tickets, no postcard, no + provider protocol (AGENTS convention 7; ADR-001 §Context).** - No namespaces-as-physical-storage; the pool is flat, namespaces are consumer-level reference sets (ADR-005). -- No manifest/path-tree/vfs tables; consumers never implement a Backend - to model paths (ADR-004). +- No manifest/path-tree/vfs tables; consumers never implement a + Backend to model paths (ADR-004). - No chunk-tree/CDC storage encoding; whole-file CAS is the store's only granularity (ADR-006). - No per-repo or per-namespace walled stores — one pooled CAS @@ -126,9 +135,11 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences). | [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side | | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping | -| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable | -| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo fs engine | postgres Large Objects as the fs tier's second engine, admitted on POC #7; a fleet node can run postgres-only | +| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + large tier per node | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; fleet pins/sweeps engine-backed; the large tier (now `large`) engine-selectable | +| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo large engine | postgres Large Objects as the large tier's second engine, admitted on POC #7; a fleet node can run postgres-only | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; kv fleet-validity rule; pin-token renewal | +| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + rename | tier/engine/instance/node/fleet pinned; mem joins the kv tier; large tier renamed `large`; constructor modes | ## Open Questions @@ -144,7 +155,7 @@ crate level: - **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres shipped behind one trait; per-node constructor choice) - **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects - as the fs tier's `pg-lo` engine — **passed** 2026-10-03; admitted as + as the large tier's `pg-lo` engine — **passed** 2026-10-03; admitted as a shipped engine by ADR-009 (REQ-2's consolidation option) ## References diff --git a/docs/architecture/requirements.md b/docs/architecture/requirements.md index a1e2772..3a0fc05 100644 --- a/docs/architecture/requirements.md +++ b/docs/architecture/requirements.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: 2026-10-03 (pg-lo admitted by ADR-009 on POC #7 — REQ-2's consolidation option is shipped) +last_updated: 2026-10-03 (vocabulary pinned by ADR-011 — tiers/engines/instances/fleet; large tier renamed `large`) --- # Requirements @@ -12,24 +12,44 @@ conversation or vision prose. **This document is the anchor: when a consumer fact changes, it changes here first, with a dated note, and the ADRs re-derive from it.** -## Vocabulary (defined once; used everywhere) +## Vocabulary (defined once; used everywhere — ADR-011 is the decision behind this section) +- **Tier** — one contract (a `Backend` trait implementation boundary) + serving one routing class of the pool. Exactly two tiers: **kv** + (small content) and **large** (over the size-threshold content; + the tier formerly named `fs` — renamed in ADR-011 because the tier's + routing criterion is length, and half its engines are not + filesystems). "Backend" names the same thing at a different + altitude (the tier as shipped including its engine set); "two + backends" (ADR-004) reads forever as "two tiers". - **Engine** — a concrete storage implementation of one tier's - contract: the kv tier's engines are sqlite and postgres (ADR-007); - the fs tier's engines are `local` (default) and `pg-lo` (ADR-009). + contract. Five ship: the kv tier's `sqlite` (default), + `postgres` (feature `postgres`), and `mem` (ephemeral; feature + `mem`; the contract-reference engine; ADR-003); the large tier's + `local` (default; ADR-003) and `pg-lo` (ADR-009). - **Pool** — the flat content-addressed namespace of one deployment (`hash → bytes` everywhere; ADR-005). -- **Node** — one running store instance (one process embedder; its fs - tier is its own media). Node is the GC arbitration domain for - single-node topologies (ADR-005's pins, liveness locks, and delete - windows live inside a node); under a fleet, arbitration moves into - the shared engine (ADR-008). -- **Fleet** — two or more nodes over **one shared pool** (e.g. via one - postgres instance). A fleet is the only topology in which - cross-node GC and fs-locality questions exist (ADR-008). +- **Store instance** — one running instantiation of the store facade + (one engine set — one or two engines per the constructor mode, + ADR-011 §4 — plus dispatch and GC state). The **GC arbitration + domain** for single-instance topologies (ADR-005's pins, liveness + locks, and delete windows live inside one); the pin-row owner id + under fleet engines (ADR-008/010 — ADR-008 wrote "node id" before + this split existed; a store instance is what owns pins). +- **Node** — one running *process* (an embedder's binary). **A node + hosts one or more store instances** — this fixes the prior + definition's collapse (which REQ-2's phrasing contradicted). Node + is the process/ops unit: alkcall connections, ACL grants, sweep + cadence, and deployment identity attach to nodes. +- **Fleet** — **two or more store instances over one shared pool** + (via a shared SQL engine). Defined by pool-sharing, not node count: + two instances in one process over one postgres pool are a fleet; + two processes on two sqlite files are two pools, never a fleet. A + fleet is the only topology in which cross-instance GC and + large-tier locality questions exist (ADR-008). - **Deployment** — one operator's whole system: typically one fleet - (multi-node) or one node; consumer tables, op registrations, and - sweep cadence are deployment choices. + or one node; consumer tables, op registrations, and sweep cadence + are deployment choices. ## The use cases and their requirements @@ -37,7 +57,7 @@ the ADRs re-derive from it.** Git repos for a user/team on one machine. Small-object regime dominated by loose objects in the kv tier; packs and large artifacts (git-lfs -shape) on the fs tier. +shape) on the large tier. - **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics: sqlite is the default kv engine; single-writer WAL ceiling (~1.2k @@ -59,9 +79,9 @@ replicators; ACL/ownership above (alkcall layer). discussion; it predates all POCs and architecture docs, recorded here on the operator's authority as the use case's stated shape). Consequences are derived in ADR-008: fleet GC (DB-backed pins, - advisory-locked sweeper, SQL delete arbitration), the fs-tier - engine concept with its locality contract, and the pg-lo candidate - engine (POC #7 — **passed** 2026-10-03, admitted as a shipped + advisory-locked sweeper, SQL delete arbitration), the large-tier + engine concept with its locality contract, and the pg-lo engine + (POC #7 — **passed** 2026-10-03, admitted as a shipped engine by ADR-009: a fleet node can consolidate on postgres-only). - **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out: the fleet's hub-class nodes need put throughput that *increases* @@ -79,7 +99,7 @@ path→hash manifests above. - **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space across workspaces; dedup must be structural (not configured). Small edits mean the small-blob regime dominates: sqlite-class kv - performance is what matters; the fs tier is hit rarely (large + performance is what matters; the large tier is hit rarely (large workspace artifacts). ## What this document is not @@ -100,4 +120,4 @@ path→hash manifests above. | REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 | | REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 | | REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 | -| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-005 | \ No newline at end of file +| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-002, ADR-005 | \ No newline at end of file diff --git a/docs/architecture/store-api.md b/docs/architecture/store-api.md index aa6bdc0..418a8cf 100644 --- a/docs/architecture/store-api.md +++ b/docs/architecture/store-api.md @@ -1,19 +1,20 @@ --- status: draft -last_updated: 2026-10-03 +last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — large tier → large) --- # Store API ## What this is -The store core's public surface: the typed facade above the backends -that consumers (alkgit, alkfs, the ops module) program against. It owns -typed keys and hashing (ADR-002), size-threshold dispatch (ADR-003), -the put-pinning seam, and the liveness/sweep seam (ADR-005) — and -nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). The -facade is **engine-blind** (ADR-007/008): which kv/fs engine a node -runs changes nothing here except fleet-GC mechanism wiring (ADR-008). +The store core's public surface: the typed facade above the tier +engines that consumers (alkgit, alkfs, the ops module) program against. +It owns typed keys and hashing (ADR-002), size-threshold dispatch +(ADR-003), the put-pinning seam, and the liveness/sweep seam (ADR-005) +— and nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). +The facade is **engine-blind** (ADR-007/008/010): which engines a node +runs changes nothing here except fleet-GC mechanism wiring (ADR-008's +mechanism lives in the engine-state seam, ADR-010 §2). ## Public surface @@ -29,13 +30,17 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008). threshold; on commit the buffer is the whole stored artifact (kv tier) — or, on mid-stream threshold overflow, the buffer flushes into a stage file and the stream continues into it - (fs tier — the streaming pass cannot include the preamble, so the - derivation pass restarts over preamble + staged bytes; forced by - the preamble being inside the hashed input, not a design choice). - Verification runs over the eventual stored artifact as a whole; - a stream error on either path leaves no entry and no stage file. - Eliminates the A6 dispatch asymmetry by construction: routing is - a pure function of content length (ADR-003). + (large tier — the streaming pass cannot include the preamble, so + the derivation pass restarts over preamble + staged bytes; forced + by the preamble being inside the hashed input, not a design + choice). Verification runs over the eventual stored artifact as a + whole; a stream error on either path leaves no entry and no stage + file. Eliminates the A6 dispatch asymmetry by construction: + routing is a pure function of content length (ADR-003). The large + tier's stage mechanics are the **staged-put contract** (ADR-010 + §1): the engine hands out a staging cursor (`local`: the stage file; + pg-lo: chunked `lowrite` statements inside its tx), commit + publishes, every failure path discards. - **Stage-file hygiene invariant** (POC #3 finding A5, codified): every failure path converges on stage-discard; every success path converges on commit-rename. No early returns that bypass cleanup; this is an @@ -51,7 +56,10 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008). ### Read path - **`get(key) → Option<(len, stream)>`** — small tier returns a - bounded read; large tier streams a file handle. Dispatch + bounded read; large tier streams (`local`: a pread loop; pg-lo: `lo_get` + windows). The per-engine shapes are hidden behind the crate-internal + read cursor the store core drives (ADR-010 §1 — engines construct + their own cursor bodies; the facade's shape is uniform). Dispatch fall-through: a small-tier miss queries the large tier (POC #1 finding 6). - **`stat(key) → Option`** — cheap length/type probe @@ -60,8 +68,9 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008). carries `len: u64` and the key's algorithm (the "type" from the store's perspective is the hash algorithm — there is no other type at this layer). `stat` rides the `Backend` trait's `size` method - (ADR-008): an indexed-column probe on kv engines, a file stat on the - fs `local` engine — never a value/content load. + (ADR-008): an indexed-column probe on kv engines, a file stat on + `local`, a companion-row probe on pg-lo — never a value/content + load. - **`read_range(key, range) → (slice, slice_digest)`** — slice plus an out-of-band slice digest: SHA-256 over the slice bytes, fixed by convention so callers and ops handlers agree without negotiation @@ -89,12 +98,26 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008). (finding 7) is why the verb/name is pinned here). - **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own the cadence (ADR-005, no ambient timers). A sweep with no registered - liveness sources aborts without deleting — the safe default. Under - a fleet (multiple nodes over one pool — requirements.md REQ-2), - sweep/put/delete semantics are mechanism-extended by ADR-008: pins - are DB-backed rows committed atomically with entries, one sweeper - holds the advisory lock, and delete arbitration is SQL-level — the - facade's invariant text is unchanged. + liveness sources aborts without deleting — the safe default. + `SweepReport`'s shape (specced here, resolving its earlier + name-only status): `candidates_staged`, `deleted`, + `cancelled_by_arbitration` (u64 counts), `aborted: Option` + (the typed cause — protection-source failure (ADR-005) or fleet + sweeper-lock contention (ADR-008) — present iff the sweep deleted + nothing for an abort reason; named `GcAbortCause` to keep it a data + enum, distinct from the `GcAborted` error which carries the + direct-delete refusal path), and `started_at`/`finished_at`. The + report is data, not an error carrier: an aborted sweep returns + `Ok(report)` with `aborted: Some(..)` — the typed error + `GcAborted` is reserved for the direct-delete refusal path. + + Under a fleet (multiple store instances over one pool — + requirements.md REQ-2), sweep/put/delete semantics are + mechanism-extended by ADR-008: pins are DB-backed rows committed + atomically with entries (the store core holds the joint tx, + ADR-010 §2), one sweeper holds the advisory lock, and delete + arbitration is SQL-level — the facade's invariant text is + unchanged. ## Error model @@ -129,8 +152,8 @@ lesson generalizes to any kv engine). 1. Whole-blob put/get round-trips byte-identical under the canonical derivation, both tiers, both put paths, **and every shipped engine - of both tiers** (kv: sqlite + postgres; fs: `local` + pg-lo; - ADR-007/008/009) + of both tiers** (kv: sqlite + postgres + mem; large: `local` + + pg-lo; ADR-007/008/009/011) (interop with real git remains the source of truth, per POC #1's lesson about hardcoded vectors). 2. Dedup: putting identical content twice (same or different path) is @@ -148,8 +171,14 @@ lesson generalizes to any kv engine). tests assert through sweep outcomes (POC #1 finding 2's lesson codified). 6. `size`/`stat` on any engine returns the entry length without - fetching content (asserted per-backend: kv probe, fs stat, pg-lo - companion-row probe — ADR-008/009). + fetching content (asserted per engine: kv row probe, `local` stat, + pg-lo companion-row probe — ADR-008/009). +7. **Engine-conformance uniformity (ADR-010 §1):** every engine's + cursor/stage/list bodies pass the same contract suite mem + (the reference engine) passes — exact-count sweep outcomes, stage + hygiene, virgin-store no-ops, CAS re-put, ranged reads (the POC #7 + gate is the norm; per-engine suite runs in CI, `--all-features` for + the feature-gated engines). ## Design Decisions @@ -159,7 +188,8 @@ lesson generalizes to any kv engine). | [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering | | [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics | | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band | -| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed | +| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed | +| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put behind the facade; the store core holds the joint entry+pin tx | ## Open Questions