docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round
ADR-010 (new): the Backend trait's I/O seams pinned before any code — 'get' returns a crate-internal read cursor (kv engines materialize; 'local' pread loop; pg-lo lo_get windows), 'put' has two named forms (whole-value kv / staged-put large); GC state is store-core-owned via a crate-internal contract-tested engine-state seam on the SQL-backed engines (backends never learn liveness — ADR-005 verbatim); the store core holds the joint entry+pin transaction (corrects ADR-009 §2's ownership statement); kv-tier fleet-validity rule added (mirror of ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin; renewal rides have/re-put; pause-past-TTL falls to delete-then-recover); feature graph pinned (pg-lo does not imply postgres). ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs engine (concrete impl, exactly five); store instance / node / fleet split (fleet = pool-sharing, not node count — fixes requirements.md's self-contradiction with REQ-2); mem demoted from 'backend'/'testing tier' to the kv tier's third engine (contract-reference engine); fs tier renamed 'large' including the feature name (last free moment before code exists); constructor modes pinned (dual-tier default, kv-only, mem-only) plus the composite fleet-validity predicate. Consistency round across all specs and ADR-003/004/007/008/009: stale pg-lo passages resolved per ADR-009's supersession; ADR-008 owner 'node id' → store-instance id; threshold wording corrected (conservative edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport shape specced; mem classification unified; 008/009 ADR files renamed to match the tier rename; README deferral-policy recap aligned (third category = decided-but-sequenced work, not a parking kind); POC crate list completed. Verification: two independent architecture review rounds (the first found 4 criticals — unpinned trait I/O shapes, missing fleet-state seam, node/fleet self-contradiction, pin-token/TTL conflict — all resolved; final round: zero criticals); all markdown links resolve; ADR tables complete (11 ADRs).
This commit is contained in:
1 parent
9bbe6f3ad9
commit
abf8c5ae9c
18 files changed
+1137
-336
No files matched your search
+55
-21
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7)
|
last_updated: 2026-10-03 (ADR-010/011 — I/O seams + GC-state home; vocabulary pinned, fs → large, mem joins kv)
|
||||||
---
|
---
|
||||||
|
|
||||||
# alkblobs — Architecture
|
# alkblobs — Architecture
|
||||||
@@ -10,8 +10,9 @@ last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7)
|
|||||||
**Phase 1 (Architecture).** Phase 0 (Exploration) converged — see
|
**Phase 1 (Architecture).** Phase 0 (Exploration) converged — see
|
||||||
`docs/research/phase-0.md` for the vision, the evidence trail, and the
|
`docs/research/phase-0.md` for the vision, the evidence trail, and the
|
||||||
POC register (all POCs passed or were absorbed; POC code lives in
|
POC register (all POCs passed or were absorbed; POC code lives in
|
||||||
standalone crates (`/workspace/alkblobs-trait-poc`,
|
standalone crates under `/workspace/` — `alkblobs-trait-poc`,
|
||||||
`/workspace/alkblobs-largeblob-poc`); findings live in
|
`alkblobs-largeblob-poc`, `alkblobs-postgres-poc`, `alkblobs-redb-poc`,
|
||||||
|
`alkblobs-pglo-poc`; findings live in
|
||||||
`docs/research/`). No implementation exists yet (`src/lib.rs` is a
|
`docs/research/`). No implementation exists yet (`src/lib.rs` is a
|
||||||
placeholder).
|
placeholder).
|
||||||
|
|
||||||
@@ -24,14 +25,14 @@ there — not inherited as hedges:
|
|||||||
the family pattern, not deviation from it — and the ops surface rides
|
the family pattern, not deviation from it — and the ops surface rides
|
||||||
alkcall, feature-gated.
|
alkcall, feature-gated.
|
||||||
- **ADR-004** fixes the "two backends" framing: the crate ships exactly
|
- **ADR-004** fixes the "two backends" framing: the crate ships exactly
|
||||||
two backends (kv small / fs large — both required by measured scale
|
two tiers (kv small / large — both required by measured scale
|
||||||
economics); the path→hash mapping layer (the alkfs/alknet "vfs" shape)
|
economics); the path→hash mapping layer (the alkfs/alknet "vfs" shape)
|
||||||
is a consumer-layer concern and must never become a third backend.
|
is a consumer-layer concern and must never become a third backend.
|
||||||
- **ADR-007** widens the kv tier's engine set rather than hedging it:
|
- **ADR-007** widens the kv tier's engine set rather than hedging it:
|
||||||
sqlite (default) + postgres (feature-gated) behind one trait. The
|
sqlite (default) + postgres (feature-gated) behind one trait. The
|
||||||
deployment economics this records: a downstream node (git server, vfs)
|
deployment economics this records: a downstream node (git server, vfs)
|
||||||
classically needed three storage systems — kv + fs + relational — and
|
classically needed three storage systems — kv + large content + relational — and
|
||||||
this design collapses them to **one relational engine + fs**, with the
|
this design collapses the three to two: **one relational engine + large-tier storage**, with the
|
||||||
engine a per-node constructor choice. Resolved from evidence in hand
|
engine a per-node constructor choice. Resolved from evidence in hand
|
||||||
(POCs #3/#5/#6); the would-be "deployment trigger" framing was caught
|
(POCs #3/#5/#6); the would-be "deployment trigger" framing was caught
|
||||||
at review as circular hedging (a deployment running alkblobs cannot
|
at review as circular hedging (a deployment running alkblobs cannot
|
||||||
@@ -42,13 +43,13 @@ there — not inherited as hedges:
|
|||||||
planning fact predating all POCs): the `Backend` trait gains a
|
planning fact predating all POCs): the `Backend` trait gains a
|
||||||
`size` probe (before any backend ships data); fleet GC gets
|
`size` probe (before any backend ships data); fleet GC gets
|
||||||
DB-backed pins, an advisory-locked single sweeper, and SQL-level
|
DB-backed pins, an advisory-locked single sweeper, and SQL-level
|
||||||
delete arbitration (single-node postures unchanged); and the fs tier
|
delete arbitration (single-node postures unchanged); and the large tier
|
||||||
becomes engine-selectable (`local` default; `pg-lo` named candidate,
|
becomes engine-selectable (`local` default; `pg-lo` named candidate,
|
||||||
admission-gated on POC #7). It also adds
|
admission-gated on POC #7). It also adds
|
||||||
[requirements.md](requirements.md) — the REQ anchor that ends
|
[requirements.md](requirements.md) — the REQ anchor that ends
|
||||||
per-session re-derivation of consumer facts.
|
per-session re-derivation of consumer facts.
|
||||||
- **ADR-009** (2026-10-03) admits `pg-lo` — postgres Large Objects as
|
- **ADR-009** (2026-10-03) admits `pg-lo` — postgres Large Objects as
|
||||||
the fs tier's second engine (feature `pg-lo`, default-off) — on
|
the large tier's second engine (feature `pg-lo`, default-off) — on
|
||||||
POC #7's passed gate (`docs/research/poc-pglo-findings.md`):
|
POC #7's passed gate (`docs/research/poc-pglo-findings.md`):
|
||||||
contract 10/10 exact-count tests, durable put ≈ fs durable put at
|
contract 10/10 exact-count tests, durable put ≈ fs durable put at
|
||||||
≥1 MiB, descriptorless `lo_get` window gets, transactional LO
|
≥1 MiB, descriptorless `lo_get` window gets, transactional LO
|
||||||
@@ -59,16 +60,44 @@ there — not inherited as hedges:
|
|||||||
readers), catalog space reused-but-never-returned, autovacuum
|
readers), catalog space reused-but-never-returned, autovacuum
|
||||||
inherited. The admission door (ADR-003, exercised by ADR-007 and
|
inherited. The admission door (ADR-003, exercised by ADR-007 and
|
||||||
ADR-009) has now shipped both its tier-extending engines.
|
ADR-009) has now shipped both its tier-extending engines.
|
||||||
|
- **ADR-010** (2026-10-03) pins the `Backend` trait's I/O seams —
|
||||||
|
the last unpinned signatures before implementation: `get` returns a
|
||||||
|
crate-internal **read cursor** (kv engines materialize; large-tier
|
||||||
|
engines stream — `local`'s pread loop, pg-lo's `lo_get` windows), `put` has
|
||||||
|
two named forms (whole-value for kv, **staged put** for large —
|
||||||
|
stage-then-commit-rename promoted from behavior to contract); rules
|
||||||
|
that **GC state is store-core-owned** (backends never learn
|
||||||
|
liveness — ADR-005 verbatim) hosted by SQL-backed engines via a
|
||||||
|
crate-internal contract-tested **engine-state companion seam**
|
||||||
|
(schema/pin/window/lock clauses), rules **the store core holds the
|
||||||
|
joint entry+pin transaction** (correcting ADR-009 §2's ownership
|
||||||
|
statement), adds the **kv tier's fleet-validity rule** (mirror of
|
||||||
|
ADR-008 §3 — postgres is the fleet engine; a sqlite-fleet posture
|
||||||
|
is not on offer), resolves the **ops pin-token TTL race** (the
|
||||||
|
token *is* the pin; renewal rides have/re-put; pause-past-TTL falls
|
||||||
|
to delete-then-recover), and pins the **feature graph** (`pg-lo`
|
||||||
|
does not imply `postgres`).
|
||||||
|
- **ADR-011** (2026-10-03) pins the vocabulary: **tier** (contract +
|
||||||
|
routing class — exactly two) vs **engine** (concrete impl — exactly
|
||||||
|
five: kv's sqlite/postgres/mem, large's `local`/pg-lo);
|
||||||
|
**mem demoted from "backend"/"testing tier" to the kv tier's
|
||||||
|
third engine — the contract-reference engine**; **store instance** /
|
||||||
|
**node** / **fleet** split so REQ-2's "one node may run multiple
|
||||||
|
instances" phrasing is no longer self-contradictory (fleet =
|
||||||
|
pool-sharing, not node count); and the **large tier is renamed
|
||||||
|
`large`** (feature `fs` → `large`) — its routing criterion is
|
||||||
|
length, not medium — with constructor modes (dual-tier default,
|
||||||
|
kv-only, mem-only) pinned in the same ADR's table.
|
||||||
|
|
||||||
## Architecture Documents
|
## Architecture Documents
|
||||||
|
|
||||||
| Document | Scope | Status |
|
| Document | Scope | Status |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft |
|
| [overview.md](overview.md) | Purpose, consumers, layer map, dependency posture | draft |
|
||||||
| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, node/pool/fleet vocabulary | draft |
|
| [requirements.md](requirements.md) | Consumer requirements REQ-1..4, pinned vocabulary (tier/engine/instance/node/fleet) | draft |
|
||||||
| [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft |
|
| [hashing-and-keys.md](hashing-and-keys.md) | Canonical hash, key encoding (the one-way door) | draft |
|
||||||
| [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft |
|
| [store-api.md](store-api.md) | Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants | draft |
|
||||||
| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, shipped backends, size-threshold dispatch | draft |
|
| [backends-and-dispatch.md](backends-and-dispatch.md) | Backend trait contract, tiers + engines table, size-threshold dispatch, constructor modes | draft |
|
||||||
| [gc-and-namespaces.md](gc-and-namespaces.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | draft |
|
| [gc-and-namespaces.md](gc-and-namespaces.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | draft |
|
||||||
| [ops-surface.md](ops-surface.md) | alkcall-backed have/need + fetch/put ops, ACL mapping | draft |
|
| [ops-surface.md](ops-surface.md) | alkcall-backed have/need + fetch/put ops, ACL mapping | draft |
|
||||||
| [open-questions.md](open-questions.md) | Centralized OQ tracker (incl. promoted Phase 0 register) | draft |
|
| [open-questions.md](open-questions.md) | Centralized OQ tracker (incl. promoted Phase 0 register) | draft |
|
||||||
@@ -84,8 +113,10 @@ there — not inherited as hedges:
|
|||||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted |
|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS, liveness seams, mark-and-sweep, delete windows | Accepted |
|
||||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted |
|
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted |
|
||||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted |
|
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); large-tier engines | Accepted |
|
||||||
| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo — postgres Large Objects as the fs tier's second engine (admitted on POC #7) | Accepted |
|
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo — postgres Large Objects as the large tier's second engine (admitted on POC #7) | Accepted |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Backend I/O seams (read cursor, staged put); GC state store-core-owned via the engine-state seam; kv fleet-validity; pin-token renewal | Accepted |
|
||||||
|
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary: tier/engine/instance/node/fleet; mem joins kv; large tier renamed `large`; constructor modes | Accepted |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
@@ -94,25 +125,28 @@ Tracked in [open-questions.md](open-questions.md). The Phase 0 register
|
|||||||
remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping,
|
remain parked — OQ-07/OQ-08 externally-owned (alkgit seam mapping,
|
||||||
alkfs intake; carried for visibility, gating nothing here). OQ-09
|
alkfs intake; carried for visibility, gating nothing here). OQ-09
|
||||||
(namespace-visibility default) resolved closed-by-default; OQ-10
|
(namespace-visibility default) resolved closed-by-default; OQ-10
|
||||||
(second kv engine) resolved by ADR-007. The pg-lo fs-engine question
|
(second kv engine) resolved by ADR-007. The pg-lo large-engine question
|
||||||
resolved to *sequenced work*, not a parked question: REQ-2/ADR-008
|
resolved to *sequenced work*, not a parked question: REQ-2/ADR-008
|
||||||
named it the candidate fs-tier engine and POC #7
|
named it the candidate large-tier engine and POC #7
|
||||||
(`docs/research/poc-pglo-spec.md`) was its admission evidence — a
|
(`docs/research/poc-pglo-spec.md`) was its admission evidence — a
|
||||||
task, not a wait; that task completed 2026-10-03
|
task, not a wait; that task completed 2026-10-03
|
||||||
(`docs/research/poc-pglo-findings.md`, passed) and ADR-009 shipped
|
(`docs/research/poc-pglo-findings.md`, passed) and ADR-009 shipped
|
||||||
the engine. **Deferral
|
the engine. **Deferral
|
||||||
policy (the "Schrödinger's code" rule):** a *decision this crate needs
|
policy (the "Schrödinger's code" rule):** a *decision this crate needs
|
||||||
before shipping* may not be deferred on a dependency that is itself
|
before shipping* may not be deferred on a dependency that is itself
|
||||||
waiting for this crate to exist. Two parking kinds remain legitimate:
|
waiting for this crate to exist. Legitimate parked kinds:
|
||||||
`deferred(scope)` on a deciding fact that exists independently of this
|
`deferred(scope)` on a deciding fact that exists independently of this
|
||||||
crate, and externally-owned questions (how a consumer maps onto this
|
crate, and externally-owned questions (how a consumer maps onto this
|
||||||
crate) that gate no decision here — full definitions in the header of
|
crate) that gate no decision here — full definitions in the header of
|
||||||
[open-questions.md](open-questions.md). A corollary worth restating,
|
[open-questions.md](open-questions.md). A third category is *not* a
|
||||||
because it recurs: a fact this crate must create (its own first
|
parked kind: **decided-but-sequenced work** (a resolved decision whose
|
||||||
deployment, its own first consumer) is never a deciding input —
|
admission gate requires evidence — the pg-lo round's POC #7 is the
|
||||||
decisions stand on evidence in hand, and future needs reopen via named
|
worked example; a task with an owner, not a parked question). A
|
||||||
requirements, not via waiting. (Corollary applications so far: OQ-09
|
corollary worth restating, because it recurs: a fact this crate must
|
||||||
and OQ-10 both resolved on it.)
|
create (its own first deployment, its own first consumer) is never a
|
||||||
|
deciding input — decisions stand on evidence in hand, and future needs
|
||||||
|
reopen via named requirements, not via waiting. (Corollary
|
||||||
|
applications so far: OQ-09 and OQ-10 both resolved on it.)
|
||||||
|
|
||||||
## Lifecycle
|
## Lifecycle
|
||||||
|
|
||||||
|
|||||||
@@ -1,17 +1,23 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03
|
last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — fs tier → large; mem is a kv engine)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Backends and dispatch
|
# Backends, tiers, and dispatch
|
||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
The physical storage layer: the `Backend` trait contract every backend
|
The physical storage layer: the `Backend` trait contract every engine
|
||||||
implements, the two shipped backends (kv, fs), and the size-threshold
|
implements, the two shipped tiers (kv, large — the tier formerly named
|
||||||
dispatch that routes between them. The trait is the crate's second
|
`fs`; renamed in ADR-011), and the size-threshold dispatch that routes
|
||||||
stable seam (with the key encoding, ADR-002): backends must survive
|
between them. The trait is the crate's second stable seam (with the
|
||||||
digest-layer evolution, so the boundary stays dumb.
|
key encoding, ADR-002): engines must survive digest-layer evolution,
|
||||||
|
so the boundary stays dumb.
|
||||||
|
|
||||||
|
Vocabulary is pinned in requirements.md (ADR-011): **tier** = contract
|
||||||
|
+ routing class (exactly two; ADR-004), **engine** = concrete impl of
|
||||||
|
one tier's contract (exactly five), and this document's table is the
|
||||||
|
spec home of the constructor table (ADR-011 §4).
|
||||||
|
|
||||||
## What the Backend trait is (and is not)
|
## What the Backend trait is (and is not)
|
||||||
|
|
||||||
@@ -32,28 +38,46 @@ purpose-built option. The wrong turn to rule out is "my data isn't
|
|||||||
blobs, therefore I implement a Backend for it"; nothing a consumer needs
|
blobs, therefore I implement a Backend for it"; nothing a consumer needs
|
||||||
is ever reachable through that door.
|
is ever reachable through that door.
|
||||||
|
|
||||||
## The Backend trait contract (ADR-003)
|
## The Backend trait contract (ADR-003; I/O seams per ADR-010)
|
||||||
|
|
||||||
- **Opaque byte keys, opaque byte values.** Backends never learn what a
|
- **Opaque byte keys, opaque byte-valued entries.** Engines never learn
|
||||||
digest is; typed `Key` converts at the store boundary (ADR-002). This
|
what a digest is; typed `Key` converts at the store boundary
|
||||||
is also the persistence boundary — a kv row or an fs filename has no
|
(ADR-002). This is also the persistence boundary — a kv row, a
|
||||||
schema-migration story, so the byte layout must be self-describing
|
`local` filename, or an LO companion row has no schema-migration story, so
|
||||||
(algorithm-tagged keys).
|
the byte layout must be self-describing (algorithm-tagged keys).
|
||||||
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` /
|
- **Methods:** `has` / `get` / `put` / `delete` / `list` / `name` /
|
||||||
`size` (the length probe added by ADR-008). Streaming put/get shapes
|
`size` (the length probe added by ADR-008). `size(key)` returns the
|
||||||
ride the store layer's seams (store-api.md);
|
entry's length or `None` if absent — metadata only, never a content
|
||||||
backends provide byte- or handle-level primitives underneath
|
fetch: kv engines probe the indexed `size` column both row shapes
|
||||||
(`get` on the fs tier yields a file handle, not a loaded buffer).
|
carry (POC #5 B5); `local` stats the file; pg-lo reads the companion
|
||||||
`size(key)` returns the entry's length or `None` if absent —
|
row. Store-level `stat` and pool accounting ride this method.
|
||||||
metadata only, never a content fetch: kv engines probe the indexed
|
- **The I/O seams are crate-internal abstractions, not engine shapes**
|
||||||
`size` column both row shapes carry (POC #5 B5); the `local` fs
|
(ADR-010 §1 — the gap that would otherwise force guessing):
|
||||||
engine stats the file. Store-level `stat` and pool accounting ride
|
- `get` returns a **read cursor** the store core drives. kv-engine
|
||||||
this method.
|
cursors materialize the whole value (bounded by tier policy);
|
||||||
|
large-engine cursors stream — `local`'s is a pread loop over the
|
||||||
|
committed file, pg-lo's drives descriptorless `lo_get(oid, off,
|
||||||
|
len)` windows (ADR-009 §3, the C4 posture). Both serve
|
||||||
|
`read_range` (slice + slice digest, ADR-006) at one seam. The
|
||||||
|
cursor is not public API: store consumers see the facade's
|
||||||
|
`(len, stream)` shapes with the cursor behind them.
|
||||||
|
- `put` has **two forms**: the **whole-value put** (kv tier — the
|
||||||
|
settled CAS row write) and the **staged put** (large tier —
|
||||||
|
ADR-003's stage-then-commit-rename as a named per-engine
|
||||||
|
contract: the engine hands out a *staging cursor*; commit
|
||||||
|
publishes; every failure path converges on discard with zero
|
||||||
|
residue). Both land in the CAS entry shape; both participate in
|
||||||
|
pin-before-publish (ADR-005/010 §2 — the store core holds the
|
||||||
|
joint entry+pin transaction via the engine-state seam).
|
||||||
|
- Per-engine conformance: an engine is admitted only if its
|
||||||
|
cursor/stage/list bodies pass the uniform contract suite — the
|
||||||
|
POC #7 gate (10/10 exact-count sweep-outcome tests) is the norm;
|
||||||
|
suite details in store-api.md's invariants.
|
||||||
- **`list()` complete by contract.** A *malformed* list is as deadly as
|
- **`list()` complete by contract.** A *malformed* list is as deadly as
|
||||||
an incomplete one: list correctness is only observable through GC
|
an incomplete one: list correctness is only observable through GC
|
||||||
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
|
(POC #1 finding 2 — the redb key-vs-value trap produced a sweep that
|
||||||
deleted the wrong blobs, silently). Therefore:
|
deleted the wrong blobs, silently). Therefore:
|
||||||
- every backend implementation must prove list correctness through
|
- every engine implementation must prove list correctness through
|
||||||
sweep-outcome tests (the invariant's test gate lives in
|
sweep-outcome tests (the invariant's test gate lives in
|
||||||
store-api.md);
|
store-api.md);
|
||||||
- an implementation that cannot enumerate (the rudolfs S3
|
- an implementation that cannot enumerate (the rudolfs S3
|
||||||
@@ -61,49 +85,60 @@ is ever reachable through that door.
|
|||||||
optional extra (ADR-005).
|
optional extra (ADR-005).
|
||||||
- **Virgin-store reads are no-ops:** read paths treat a missing table
|
- **Virgin-store reads are no-ops:** read paths treat a missing table
|
||||||
as absent/empty (POC #1 finding 2).
|
as absent/empty (POC #1 finding 2).
|
||||||
- **Namespace-blind:** backends never see namespaces, tenants, or
|
- **Namespace-blind:** engines never see namespaces, tenants, or
|
||||||
reference structure (ADR-005 — the rudolfs inversion; physical
|
reference structure (ADR-005 — the rudolfs inversion; physical
|
||||||
storage is `hash → bytes` flat).
|
storage is `hash → bytes` flat).
|
||||||
|
- **GC state never rides this trait** (ADR-010 §2): pins, window rows,
|
||||||
|
and sweep locks are store-core-owned via a crate-internal
|
||||||
|
engine-state companion seam on the SQL-backed engines (sqlite,
|
||||||
|
postgres, pg-lo) — invisible at the public trait; `local` and `mem`
|
||||||
|
cannot express it and are never fleet engines.
|
||||||
|
|
||||||
## Shipped backends
|
## Shipped tiers and engines
|
||||||
|
|
||||||
Two tiers, exactly — this counts *trait implementations this crate
|
Two tiers, exactly — this counts *trait implementations this crate
|
||||||
ships* as CAS tiers, the dispatch routes between them; it is the
|
ships as CAS tiers* (the contracts dispatch routes between); it is the
|
||||||
complete set the problem requires, and the "third backend" fear is a
|
complete set the problem requires, and the "third backend" fear is a
|
||||||
category error fixed in ADR-004. It says nothing about other storage
|
category error fixed in ADR-004 (tier count is not engine count). It
|
||||||
existing in a deployment: a downstream runs whatever else it needs on
|
says nothing about other storage existing in a deployment: a downstream
|
||||||
its own media, beside the pool, above the store's seams (see "What the
|
runs whatever else it needs on its own media, beside the pool, above
|
||||||
Backend trait is"). **Tier count is not engine count**: the kv tier
|
the store's seams (see "What the Backend trait is").
|
||||||
carries two engines (sqlite, postgres) behind one trait — ADR-007;
|
|
||||||
per-node engine selection is a constructor parameter.
|
|
||||||
|
|
||||||
### kv backend (feature `kv`, default-on); engines: sqlite (default) + postgres (feature `postgres`, default-off)
|
| Tier | Engine | Feature | Default | Fleet-valid |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
|
||||||
|
| kv | postgres | `postgres` | no | **yes** |
|
||||||
|
| kv | mem | `mem` | no | no (test/reference engine) |
|
||||||
|
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
|
||||||
|
| large | pg-lo | `pg-lo` | no | **yes** |
|
||||||
|
|
||||||
|
### kv tier (tier default-on, feature `kv`); engines sqlite (default) + postgres (feature `postgres`, default-off) + mem (feature `mem`, default-off)
|
||||||
|
|
||||||
Small blobs — **`kv` is the tier name; the engine is a per-node
|
Small blobs — **`kv` is the tier name; the engine is a per-node
|
||||||
constructor choice** between the shipped engines (ADR-007; ADR-003
|
constructor choice** between the shipped engines (ADR-007; ADR-003
|
||||||
carries sqlite's on-disk format trade: file compat is upstream
|
carries sqlite's on-disk format trade: file compat is upstream
|
||||||
sqlite's guarantee). The shipped trait impls write plain
|
sqlite's guarantee). The shipped trait impls write plain
|
||||||
content-addressed rows into the chosen engine, nothing else. A
|
content-addressed rows into the chosen engine, nothing else (ADR-010:
|
||||||
*third* engine remains the substitution door as amended: a new ADR
|
the fleet state rides the companion seam, never the public trait). A
|
||||||
with its own sweep-safety proof (the door is no longer hypothetical —
|
*third* non-mem engine remains the substitution door as amended: a new
|
||||||
ADR-007 is its first in-repo exercise); and a downstream's *other*
|
ADR with its own sweep-safety proof (the door is exercised — ADR-007
|
||||||
storage — schemas, manifests, whatever it likes including some other
|
in-repo); and a downstream's *other* storage — schemas, manifests,
|
||||||
sqlite file — never comes near this tier at all, per "What the
|
whatever it likes including some other sqlite file — never comes near
|
||||||
Backend trait is".
|
this tier at all, per "What the Backend trait is".
|
||||||
|
|
||||||
The collapse ADR-007 buys: the classical downstream stack (git
|
The collapse ADR-007 buys: the classical downstream stack (git
|
||||||
server, vfs node) needed three storage systems — kv + fs + relational.
|
server, vfs node) needed three storage systems — kv + large content + relational.
|
||||||
Here the kv tier rides the relational engine itself, so a node
|
Here the kv tier rides the relational engine itself, so a node
|
||||||
provisions **one relational engine + fs**; consumer tables (refs,
|
provisions **one relational engine + large-tier storage**; consumer
|
||||||
manifests, queues) sit beside the pool on the same engine or file by
|
tables (refs, manifests, queues) sit beside the pool on the same
|
||||||
the deployment's own choice (co-tenancy note: same-file consumer
|
engine or file by the deployment's own choice (co-tenancy note: same-
|
||||||
writes share sqlite's single-writer ceiling — that sharing is the
|
file consumer writes share sqlite's single-writer ceiling — that
|
||||||
node's trade; separate files remain available).
|
sharing is the node's trade; separate files remain available).
|
||||||
|
|
||||||
Evidence (POC #3 finding A4, first-party measured):
|
Evidence (POC #3 finding A4, first-party measured): sqlite is ~9-10×
|
||||||
sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime
|
faster than fs at 1-16 KiB (the git small-blob regime — most git
|
||||||
— most git objects, workspace files, manifests), with the crossover at
|
objects, workspace files, manifests), with the crossover at ~128-256
|
||||||
~128-256 KiB where fs stops paying the B-tree row rewrite and wins.
|
KiB where fs stops paying the B-tree row rewrite and wins.
|
||||||
|
|
||||||
**sqlite engine** (the default): WAL + `synchronous=NORMAL` as the
|
**sqlite engine** (the default): WAL + `synchronous=NORMAL` as the
|
||||||
shipped durability tier (matching what the benchmark measured and what
|
shipped durability tier (matching what the benchmark measured and what
|
||||||
@@ -128,22 +163,31 @@ sweep-outcome CI gate, run against dockerized postgres under
|
|||||||
near-linearly to ~37k puts/s at 24 conns — the only engine whose
|
near-linearly to ~37k puts/s at 24 conns — the only engine whose
|
||||||
throughput *increases* under concurrency.
|
throughput *increases* under concurrency.
|
||||||
|
|
||||||
### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
|
**mem engine** (feature `mem`, default-off; ADR-011 §2 — formerly
|
||||||
|
described as the "mem backend"/"testing tier" before the vocabulary
|
||||||
|
was pinned): a `BTreeMap`-shaped ephemeral implementation of the kv
|
||||||
|
tier's contract, including `size` (trivially). For tests and
|
||||||
|
in-process ephemerality — never a production story, never
|
||||||
|
fleet-valid, and the **contract-reference engine**: every other
|
||||||
|
engine's conformance tests mirror its suite (the role it already
|
||||||
|
played in POC #1's miniature).
|
||||||
|
|
||||||
Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract,
|
### large tier (tier default-on, feature `large`; the tier formerly named `fs` — renamed in ADR-011); engines `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
|
||||||
multiple engines** — the medium is a per-node constructor choice
|
|
||||||
(ADR-008; REQ-2 in requirements.md makes the engine concept load-bearing
|
Large blobs. Like the kv tier (ADR-007), the large tier is **one
|
||||||
for fleets):
|
contract, multiple engines** — the medium is a per-node constructor
|
||||||
|
choice (ADR-008; REQ-2 in requirements.md makes the engine concept
|
||||||
|
load-bearing for fleets):
|
||||||
|
|
||||||
- **`local` engine** (the default; the ADR-003 backend as originally
|
- **`local` engine** (the default; the ADR-003 backend as originally
|
||||||
specced): flat sharded layout:
|
specced): flat sharded layout:
|
||||||
`{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's
|
`{hex-prefix}/{hex-prefix}/{hash}` sharding survives from iroh's
|
||||||
conclusion (limits directory size on huge pools); stage-then-commit-
|
conclusion (limits directory size on huge pools); stage-then-commit-
|
||||||
rename for the two-pass unknown-length path; pread-based range
|
rename for the two-pass unknown-length path (the staged-put form,
|
||||||
reads (POC #3 finding A2 — local range serving is sound, e.g. for
|
ADR-010); pread-based range reads (POC #3 finding A2 — local range
|
||||||
packfiles).
|
serving is sound, e.g. for packfiles).
|
||||||
- **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009):
|
- **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009):
|
||||||
postgres Large Objects as the fs tier's storage — the same
|
postgres Large Objects as the large tier's storage — the same
|
||||||
engine-behind-one-trait move as ADR-007, one tier over, admitted on
|
engine-behind-one-trait move as ADR-007, one tier over, admitted on
|
||||||
POC #7's measured evidence (contract 10/10 exact-count tests;
|
POC #7's measured evidence (contract 10/10 exact-count tests;
|
||||||
durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC
|
durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC
|
||||||
@@ -154,66 +198,90 @@ for fleets):
|
|||||||
operator's shrink path), autovacuum inherited, cached-get 20–50×
|
operator's shrink path), autovacuum inherited, cached-get 20–50×
|
||||||
behind page-cache fs single-stream (~700 MB/s aggregate at 16
|
behind page-cache fs single-stream (~700 MB/s aggregate at 16
|
||||||
readers — the fleet serving picture).
|
readers — the fleet serving picture).
|
||||||
- **Fleet locality contract (ADR-008, extended by ADR-009):** over one
|
- **Fleet locality contract (ADR-008, extended by ADR-009/010):** over
|
||||||
shared pool, the fs tier is one of: shared media (every node's
|
one shared pool, the large tier is one of: shared media (every
|
||||||
`local` root on the same fleet-shared media — soundness caveats and
|
node's `local` root on the same fleet-shared media — soundness
|
||||||
the required media guarantees in ADR-008), re-routed (one storage
|
caveats and the required media guarantees in ADR-008), re-routed
|
||||||
node serves pool large-blob content via the ops surface; client
|
(one storage node serves pool large-blob content via the ops
|
||||||
nodes are kv-tier-only over pool content), **or `pg-lo` for the
|
surface; client nodes are kv-only over pool content), **or
|
||||||
whole fleet** (ADR-009 — one engine, no shared media, no re-routing;
|
`pg-lo` for the whole fleet** (ADR-009 — one engine, no shared
|
||||||
pool content lives in the same pg instance the kv tier rides).
|
media, no re-routing; pool content lives in the same pg instance
|
||||||
Mixed per-node-local fs tiers over one pool is the partitioning
|
the kv tier rides; the joint entry+pin tx is store-core-held,
|
||||||
failure this contract exists to prevent — a documented deployment
|
ADR-010 §2). Mixed per-node-local large tiers over one pool is the
|
||||||
invariant the constructor cannot fully prove but must be declared
|
partitioning failure this contract exists to prevent — a documented
|
||||||
against (ADR-008 names the enforceable seam and the detection
|
deployment invariant the constructor cannot fully prove but must be
|
||||||
symptom).
|
declared against (ADR-008 names the enforceable seam and the
|
||||||
|
detection symptom). The kv tier has the mirrored rule (ADR-010 §3):
|
||||||
### mem backend (feature `mem`, default-off)
|
a fleet constructor requires a pool-shared kv engine (postgres);
|
||||||
|
two sqlite engines are two pools, never one.
|
||||||
`BTreeMap`-shaped ephemeral backend for tests and in-process
|
- **The composite fleet-validity rule (one sentence, both tiers):** a
|
||||||
ephemerality. A testing/utility tier, never a production story.
|
constructor configuration is fleet-valid iff **kv = postgres AND
|
||||||
Implements the full trait contract including `size` (trivially) — it
|
(large = pg-lo OR large = `local`-on-declared-shared-media OR
|
||||||
exists partly as the reference impl every other backend's conformance
|
large = none with the re-routing posture carrying pool large
|
||||||
tests can mirror.
|
content)**. Any other combination over one shared pool is invalid —
|
||||||
|
the constructor requires the declarations that make this predicate
|
||||||
|
checkable per node; cross-node truth remains the deployment's
|
||||||
|
verified invariant (ADR-008's seam). This predicate is
|
||||||
|
backends-and-dispatch.md's and ADR-011 §4's table combined; it is
|
||||||
|
stated here once so no reader composes it by inference.
|
||||||
|
|
||||||
## Size-threshold dispatch (ADR-003)
|
## Size-threshold dispatch (ADR-003)
|
||||||
|
|
||||||
- **Routing is a pure function of content length.** Same content ⇒ same
|
- **Routing is a pure function of content length.** Same content ⇒ same
|
||||||
length ⇒ same backend; re-puts are deterministic. No content ever
|
length ⇒ same tier; re-puts are deterministic. No content ever
|
||||||
migrates between backends — the migration question existed only to
|
migrates between tiers — the migration question existed only to
|
||||||
patch the unknown-length asymmetry, which ADR-003's pre-threshold
|
patch the unknown-length asymmetry, which ADR-003's pre-threshold
|
||||||
buffering eliminates by construction.
|
buffering eliminates by construction.
|
||||||
- **Default threshold: 128 KiB** (constructor-tunable). Midpoint of the
|
- **Default threshold: 128 KiB** (constructor-tunable) — the bottom of
|
||||||
measured flat zone (POC #3 A4). Re-tuning per deployment media is a
|
the measured crossover zone (~128-256 KiB, POC #3 A4; chosen at the
|
||||||
constructor parameter, not an API change.
|
zone's conservative edge, not its midpoint). Re-tuning per
|
||||||
|
deployment media is a constructor parameter, not an API change.
|
||||||
- **Get fall-through:** small-tier miss queries the large tier
|
- **Get fall-through:** small-tier miss queries the large tier
|
||||||
(deterministic, cheap — a stat probe).
|
(deterministic, cheap — a stat probe).
|
||||||
- Per-namespace or per-tenant backend configuration: **rejected**
|
- Per-namespace or per-tenant engine configuration: **rejected**
|
||||||
(ADR-003 §Consequences — it would re-weld namespacing into the
|
(ADR-003 §Consequences — it would re-weld namespacing into the
|
||||||
physical layer, the rudolfs anti-pattern ADR-005 inverts).
|
physical layer, the rudolfs anti-pattern ADR-005 inverts).
|
||||||
|
|
||||||
## Where a *new* backend or engine could come from
|
### Constructor modes (ADR-011 §4)
|
||||||
|
|
||||||
|
- **dual-tier** (the default): kv + large, size-threshold dispatch
|
||||||
|
(ADR-003).
|
||||||
|
- **kv-only** (ADR-008's re-routing client posture, now named): no
|
||||||
|
large tier; over-threshold puts are a constructor-time error (the
|
||||||
|
re-routing posture handles them via the ops surface — the
|
||||||
|
consumer's composition, not a store mode).
|
||||||
|
- **mem-only:** the mem engine alone, no dispatch threshold in
|
||||||
|
effect; a testing/embedder-ephemeral posture, never production.
|
||||||
|
- Single-tier SQL modes (postgres-only, pg-lo-only *tiers*) do not
|
||||||
|
exist: both tiers are load-bearing (ADR-004). "Postgres-only" as in
|
||||||
|
one SQL *instance* serving both tiers exists and is the ADR-009
|
||||||
|
consolidation: dual-tier mode with kv=postgres + large=pg-lo over
|
||||||
|
one pool.
|
||||||
|
|
||||||
|
## Where a *new* engine could come from
|
||||||
|
|
||||||
The trait is open to future implementations (network stores, S3-like
|
The trait is open to future implementations (network stores, S3-like
|
||||||
tiers), but nothing in the current consumer set requires one, and the
|
tiers), but nothing in the current consumer set requires one, and the
|
||||||
contract is deliberately hostile to half-implementations (complete
|
contract is deliberately hostile to half-implementations (complete
|
||||||
`list()`, GC-participating `delete`). Any future backend — or a third
|
`list()`, GC-participating `delete`). Any future engine — per tier —
|
||||||
kv engine or second fs engine — is a new ADR carrying its own
|
is a new ADR carrying its own sweep-safety story (ADR-007 is the
|
||||||
sweep-safety story (ADR-007 is the precedent for a kv engine
|
precedent for a kv engine addition; ADR-008 named pg-lo and ADR-009
|
||||||
addition; ADR-008 named pg-lo as the candidate second fs engine with
|
admitted it with POC #7's evidence — the door is exercised twice).
|
||||||
POC #7 as its admission evidence — the door is exercised twice). This
|
This crate's roadmap is not blocked on one (see open-questions.md —
|
||||||
crate's roadmap is not blocked on one (see open-questions.md — alkfs
|
alkfs intake may name needs externally; OQ-08).
|
||||||
intake may name needs externally; OQ-08).
|
|
||||||
|
|
||||||
## Design Decisions
|
## Design Decisions
|
||||||
|
|
||||||
| ADR | Decision | Summary |
|
| ADR | Decision | Summary |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
|
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | opaque keys, complete list, pure-function routing, no migration |
|
||||||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | scope boundary against manifest-layer absorption |
|
| [004](decisions/004-two-backends-no-third.md) | Two tiers, no third | scope boundary against manifest-layer absorption |
|
||||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | backends see hashes only |
|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Namespace-blindness | engines see hashes only |
|
||||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + fs per node |
|
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite (default) + postgres behind one trait; one relational engine + large tier per node |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; fs tier engine-selectable (`local` default, pg-lo candidate) |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; DB-backed pins/advisory-locked sweeper for fleets; large tier engine-selectable (`local` default) |
|
||||||
|
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo admitted | postgres Large Objects as the large tier's second engine, on POC #7 |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; the kv fleet-validity rule |
|
||||||
|
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary | tier/engine/instance/node/fleet; mem is a kv engine; fs → large; constructor modes |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
@@ -230,10 +298,13 @@ intake may name needs externally; OQ-08).
|
|||||||
curves + engine-posture deltas (ADR-007's evidence base)
|
curves + engine-posture deltas (ADR-007's evidence base)
|
||||||
- `docs/research/poc-redb-kv-findings.md` — redb ruled out at a
|
- `docs/research/poc-redb-kv-findings.md` — redb ruled out at a
|
||||||
durability-tier mismatch (POC #6; recorded in ADR-007)
|
durability-tier mismatch (POC #6; recorded in ADR-007)
|
||||||
|
- `docs/research/poc-pglo-findings.md` — pg-lo's admission evidence
|
||||||
|
(POC #7; ADR-009)
|
||||||
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
|
- `docs/research/iroh-blobs-eval.md` — the fs-layout conclusions borrowed
|
||||||
(sharding, crash ordering, inline thresholds rejected as weld)
|
(sharding, crash ordering, inline thresholds rejected as weld)
|
||||||
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
|
- rudolfs notes — `list()` anti-lesson, decorator alternative noted and
|
||||||
not adopted (threshold dispatch chose the simpler policy;
|
not adopted (threshold dispatch chose the simpler policy;
|
||||||
ADR-003 §Context)
|
ADR-003 §Context)
|
||||||
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
|
- ADR-003, ADR-004, ADR-005; [store-api.md](store-api.md) (the invariants
|
||||||
backends must satisfy)
|
engines must satisfy); [requirements.md](requirements.md) (the pinned
|
||||||
|
vocabulary)
|
||||||
@@ -16,7 +16,7 @@ addresses, not algorithms).
|
|||||||
|
|
||||||
The residual was the **key encoding mechanics** — the byte layout of
|
The residual was the **key encoding mechanics** — the byte layout of
|
||||||
keys inside backends. This is the crate's declared one-way door
|
keys inside backends. This is the crate's declared one-way door
|
||||||
(AGENTS convention 5): key bytes persist in kv rows and fs filenames,
|
(AGENTS convention 5): key bytes persist in kv rows and `local`-engine filenames,
|
||||||
with no schema-migration story; and this is the wire-format ADR that
|
with no schema-migration story; and this is the wire-format ADR that
|
||||||
must exist before the first consumer (convention 7's gate on publishing
|
must exist before the first consumer (convention 7's gate on publishing
|
||||||
wire-relevant decisions). POC #1 validated the mechanics empirically:
|
wire-relevant decisions). POC #1 validated the mechanics empirically:
|
||||||
@@ -26,7 +26,7 @@ both algorithms in one flat pool (finding 5).
|
|||||||
|
|
||||||
## Decision
|
## Decision
|
||||||
|
|
||||||
### Hash policy (three tiers)
|
### Hash policy (three classes)
|
||||||
|
|
||||||
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
|
1. **Canonical — `git-blob-sha-256`:** `H("blob <len>\0" + content)`,
|
||||||
SHA-256. The store's canonical algorithm; git objects are first-class
|
SHA-256. The store's canonical algorithm; git objects are first-class
|
||||||
|
|||||||
@@ -5,10 +5,15 @@
|
|||||||
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
||||||
engines, sqlite and postgres, behind this contract; the dispatch layer
|
engines, sqlite and postgres, behind this contract; the dispatch layer
|
||||||
and trait are unchanged. The trait was extended by ADR-008 with a
|
and trait are unchanged. The trait was extended by ADR-008 with a
|
||||||
`size` length probe — before any backend ships data — and the fs
|
`size` length probe — before any backend ships data — and the second
|
||||||
tier became engine-selectable (`local` default; pg-lo candidate), so
|
tier became engine-selectable (`local` default; pg-lo admitted by
|
||||||
this ADR's "engines"/"backends" counts are read through ADR-004's
|
ADR-009), so this ADR's "engines"/"backends" counts are read through
|
||||||
tier≠engine scope note)
|
ADR-004's tier≠engine scope note. The I/O shapes behind `get`/`put`
|
||||||
|
are pinned by ADR-010 (read cursor / staged put). The tier's name
|
||||||
|
changed: `fs` → `large` (ADR-011 — its routing criterion is length,
|
||||||
|
not medium); this ADR's per-engine counts also gained the `mem` kv
|
||||||
|
engine as ADR-011's vocabulary resolves — mem is a kv engine, not a
|
||||||
|
tier)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -54,28 +59,34 @@ Evidence in hand (no further POC or consumer-waiting needed):
|
|||||||
- **Virgin-store reads are no-ops** (missing table = absent/empty).
|
- **Virgin-store reads are no-ops** (missing table = absent/empty).
|
||||||
- **Namespace-blind** (ADR-005): backends see hashes only.
|
- **Namespace-blind** (ADR-005): backends see hashes only.
|
||||||
|
|
||||||
### Shipped backends (two, default-on; plus a testing tier)
|
### Shipped tiers (two, default-on; the kv tier also carries the mem engine)
|
||||||
|
|
||||||
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
|
- **`kv` (default-on): sqlite** for small blobs — measured ~9-10×
|
||||||
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
|
faster than fs at 1-16 KiB (POC #3 A4, first-party benchmark),
|
||||||
the regime holding most git objects, workspace files, and manifests.
|
the regime holding most git objects, workspace files, and manifests.
|
||||||
WAL + `synchronous=NORMAL` as shipped durability; prepared
|
WAL + `synchronous=NORMAL` as shipped durability; prepared
|
||||||
statements; no-tables-yet tolerance. (The tier's engine set was
|
statements; no-tables-yet tolerance. (The tier's engine set was
|
||||||
widened to include postgres by ADR-007.)
|
widened to include postgres by ADR-007, and mem — the ephemeral
|
||||||
- **`fs` (default-on)** for large blobs — flat
|
reference engine — by ADR-011's vocabulary resolution.)
|
||||||
`{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
|
- **`fs` (default-on; renamed `large` by ADR-011)** for large blobs —
|
||||||
|
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's surviving
|
||||||
conclusion); stage-then-commit-rename; pread range reads (sound
|
conclusion); stage-then-commit-rename; pread range reads (sound
|
||||||
locally per ADR-006).
|
locally per ADR-006). (The tier's engine set was widened by ADR-008/
|
||||||
- **`mem` (default-off):** ephemeral `BTreeMap` tier for tests and
|
009.)
|
||||||
in-process ephemerality. Never a production story.
|
- **`mem` (default-off):** ephemeral `BTreeMap` kv engine for tests
|
||||||
|
and in-process ephemerality — **an engine of the kv tier, not a
|
||||||
|
third tier** (ADR-011 §2; the pre-ADR-011 "testing tier" phrasing
|
||||||
|
is superseded). Never a production story; the contract-reference
|
||||||
|
engine.
|
||||||
|
|
||||||
Both production backends are default-on (batteries-included, the
|
Both production tiers are default-on (batteries-included, the
|
||||||
alkgit feature-model precedent); `mem` is not.
|
alkgit feature-model precedent); `mem` is not.
|
||||||
|
|
||||||
### Dispatch policy
|
### Dispatch policy
|
||||||
|
|
||||||
- **Size-threshold routing; default 128 KiB, constructor-tunable**
|
- **Size-threshold routing; default 128 KiB, constructor-tunable**
|
||||||
(midpoint of the measured ~128-256 KiB crossover zone; POC #3 A4).
|
(chosen at the bottom of the measured ~128-256 KiB crossover zone —
|
||||||
|
the conservative edge of the flat zone; POC #3 A4).
|
||||||
- **Unknown-length puts buffer in memory up to the threshold; the
|
- **Unknown-length puts buffer in memory up to the threshold; the
|
||||||
buffer's fate has three pinned semantics** (finding A6's option (d)):
|
buffer's fate has three pinned semantics** (finding A6's option (d)):
|
||||||
1. **Threshold exceeded mid-stream** (non-restartable sources are
|
1. **Threshold exceeded mid-stream** (non-restartable sources are
|
||||||
@@ -141,9 +152,10 @@ alkgit feature-model precedent); `mem` is not.
|
|||||||
|
|
||||||
**Neutral**
|
**Neutral**
|
||||||
|
|
||||||
- A network-backed backend (S3-like) remains possible via the same
|
- A network-backed engine (S3-like) remains possible via the same
|
||||||
trait but ships nothing here; its list/GC obligations are the
|
trait but ships nothing here; its list/GC obligations are the
|
||||||
acceptance gate.
|
acceptance gate; per-tier, via a new ADR (ADR-011's vocabulary: an
|
||||||
|
engine addition, not a tier addition).
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|
||||||
|
|||||||
@@ -2,7 +2,12 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Accepted
|
Accepted (the count's vocabulary is pinned by ADR-011: the "two" here
|
||||||
|
are *tiers* — kv + large, the latter renamed from `fs`; "backend"
|
||||||
|
names the same thing at a different altitude. The `mem` item below is
|
||||||
|
the kv tier's third engine, per ADR-011 §2, not an extra tier —
|
||||||
|
the pre-ADR-011 "testing tier" carve-out is superseded. The I/O
|
||||||
|
seam-level refinements of the trait contract are ADR-010's)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -11,9 +16,10 @@ rounds before being edited away, but the correction it was reaching for
|
|||||||
arrived only in the Phase 1 review, and needs recording so no future
|
arrived only in the Phase 1 review, and needs recording so no future
|
||||||
round reabsorbs the concern incorrectly:
|
round reabsorbs the concern incorrectly:
|
||||||
|
|
||||||
- There are **always** two physical backends in this problem: storage
|
- There are **always** two physical tiers in this problem: storage
|
||||||
for small content (fast, kv/sqlite) and the filesystem for large
|
for small content (fast, kv/sqlite) and storage for large content
|
||||||
content. This is not ugly, it is the measured shape of the problem
|
(the large tier / filesystem). This is not ugly, it is the measured
|
||||||
|
shape of the problem
|
||||||
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
|
(POC #3 A4: crossover ~128-256 KiB; SQLite ~9-10× faster below it).
|
||||||
- The "two backends is ugly" sentiment was actually about a **third
|
- The "two backends is ugly" sentiment was actually about a **third
|
||||||
layer** in the old alknet-filesystem research — the *virtual
|
layer** in the old alknet-filesystem research — the *virtual
|
||||||
@@ -30,23 +36,25 @@ is the boundary it will build against.
|
|||||||
|
|
||||||
## Decision
|
## Decision
|
||||||
|
|
||||||
**This crate ships exactly two production backends and nothing more;
|
**This crate ships exactly two production tiers and nothing more;
|
||||||
every manifest/namespace/path-tree layer is a consumer, by construction
|
every manifest/namespace/path-tree layer is a consumer, by construction
|
||||||
and by contract.** (A non-production ephemeral `mem` tier exists for
|
and by contract.** (A non-production ephemeral `mem` engine exists for
|
||||||
tests — ADR-003; it is not a storage story and does not extend this
|
tests — ADR-003, resolved to a kv-tier engine by ADR-011 §2; it is not
|
||||||
boundary.)
|
a storage story and does not extend this boundary.)
|
||||||
|
|
||||||
**Scope note (post-ADR-007 reconciliation): the "two" here counts
|
**Scope note (post-ADR-007 reconciliation, vocabulary finalized by
|
||||||
*tiers*, not engines.** The kv tier is one backend; ADR-007 widened the
|
ADR-011): the "two" here counts *tiers*, not engines.** The kv tier is
|
||||||
engines behind it to a set (sqlite default + postgres, feature-gated)
|
one contract; ADR-007 widened the engines behind it to a set (sqlite
|
||||||
so a node's kv rides its relational engine of choice. The count that
|
default + postgres, feature-gated; mem the ephemeral reference engine,
|
||||||
matters to this ADR — trait impls *this crate ships* as CAS tiers —
|
ADR-011) so a node's kv rides its relational engine of choice. The
|
||||||
remains two (kv + fs); a second kv engine does not make a third
|
count that matters to this ADR — trait impls *this crate ships* as CAS
|
||||||
backend, and nothing in the engine set touches the manifest-layer
|
tiers — remains two (kv + large, the tier ADR-011 renamed from `fs`);
|
||||||
boundary below.
|
a second kv engine does not make a third tier, and nothing in the
|
||||||
|
engine set touches the manifest-layer boundary below.
|
||||||
|
|
||||||
1. **Two production backends is the complete set.** kv/sqlite (small)
|
1. **Two production tiers is the complete set.** kv/sqlite (small)
|
||||||
+ fs (large) cover the scale economics with measured evidence
|
+ large (the tier formerly named `fs`) cover the scale economics
|
||||||
|
with measured evidence
|
||||||
(ADR-003); no other physical tier is needed by any current or
|
(ADR-003); no other physical tier is needed by any current or
|
||||||
planned consumer's *blob* needs.
|
planned consumer's *blob* needs.
|
||||||
2. **The "third backend" is not this crate's layer.** The vfs/appfile
|
2. **The "third backend" is not this crate's layer.** The vfs/appfile
|
||||||
@@ -86,8 +94,8 @@ boundary below.
|
|||||||
|
|
||||||
**Positive**
|
**Positive**
|
||||||
|
|
||||||
- The crate's scope stays testable and small: two backends, one pool,
|
- The crate's scope stays testable and small: two tiers, one pool,
|
||||||
one hash domain. No schema/versioning surface for manifest formats —
|
one hash domain. No schema/versioning surface for manifest formats —
|
||||||
consumers iterate on theirs freely.
|
consumers iterate on theirs freely.
|
||||||
- alkgit and alkfs build their (very different) mapping layers on one
|
- alkgit and alkfs build their (very different) mapping layers on one
|
||||||
shared pool without either bending the store; their manifests interop
|
shared pool without either bending the store; their manifests interop
|
||||||
|
|||||||
@@ -49,7 +49,7 @@ served by range reads).
|
|||||||
(git refs, alkfs path trees) and hand liveness over per sweep via the
|
(git refs, alkfs path trees) and hand liveness over per sweep via the
|
||||||
live-shared seam. In-memory namespace tables remain available for
|
live-shared seam. In-memory namespace tables remain available for
|
||||||
ephemeral consumers. This also dissolves the hidden wrinkle of a
|
ephemeral consumers. This also dissolves the hidden wrinkle of a
|
||||||
store-internal root table under fs-only deployments (where would its
|
store-internal root table under deployments without a SQL engine (where would its
|
||||||
durable bytes live?). Ephemeral consumers are covered by pins; a
|
durable bytes live?). Ephemeral consumers are covered by pins; a
|
||||||
deployment that needs store-durable roots registers them as
|
deployment that needs store-durable roots registers them as
|
||||||
consumer-owned durable state and sweeps through the seam.
|
consumer-owned durable state and sweeps through the seam.
|
||||||
|
|||||||
@@ -2,7 +2,9 @@
|
|||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Accepted
|
Accepted (the engine story's I/O seams and GC-state home are ruled by
|
||||||
|
ADR-010; the vocabulary — tier/engine, and mem joining the kv tier as
|
||||||
|
its third engine — is pinned by ADR-011)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -57,9 +59,9 @@ in OQ-09.
|
|||||||
**The kv tier ships two engines — sqlite (default) and postgres —
|
**The kv tier ships two engines — sqlite (default) and postgres —
|
||||||
behind one `Backend` trait; engine selection is a constructor
|
behind one `Backend` trait; engine selection is a constructor
|
||||||
parameter on the backend's configuration.** A node provisions one
|
parameter on the backend's configuration.** A node provisions one
|
||||||
relational engine plus the fs tier; the kv tier rides whichever
|
relational engine plus the large tier; the kv tier rides whichever
|
||||||
engine the operator chose. This collapses the classical three-engine
|
engine the operator chose. This collapses the classical three-engine
|
||||||
downstream stack (kv + fs + relational) to two, and makes which
|
downstream stack (kv + large content + relational) to two, and makes which
|
||||||
relational engine a per-node choice rather than a crate fork.
|
relational engine a per-node choice rather than a crate fork.
|
||||||
|
|
||||||
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
||||||
@@ -83,7 +85,7 @@ relational engine a per-node choice rather than a crate fork.
|
|||||||
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
|
(`INSERT ... ON CONFLICT DO NOTHING` maps sqlite's
|
||||||
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
|
`INSERT OR IGNORE`), complete `list()`, GC-participating `delete`,
|
||||||
virgin-store no-op reads, namespace-blindness. The dispatch layer
|
virgin-store no-op reads, namespace-blindness. The dispatch layer
|
||||||
(size threshold, kv↔fs routing, fall-through) is engine-blind and
|
(size threshold, kv↔large routing, fall-through) is engine-blind and
|
||||||
unchanged.
|
unchanged.
|
||||||
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
|
4. **The pg engine's impl contract carries POC #5 finding B5's deltas
|
||||||
as requirements** (they are the difference between a port and an
|
as requirements** (they are the difference between a port and an
|
||||||
@@ -127,7 +129,7 @@ collapses to its own deployment choice among shipped engines.
|
|||||||
**Positive**
|
**Positive**
|
||||||
|
|
||||||
- The classical downstream stack lands at two engines per node (one
|
- The classical downstream stack lands at two engines per node (one
|
||||||
relational + fs), eliminating the "sqlite *beside* the real
|
relational engine + large tier), eliminating the "sqlite *beside* the real
|
||||||
database" duplication this crate's consumers would otherwise carry
|
database" duplication this crate's consumers would otherwise carry
|
||||||
forever.
|
forever.
|
||||||
- Per-node engine choice (constructor parameter, per ADR-003/004)
|
- Per-node engine choice (constructor parameter, per ADR-003/004)
|
||||||
|
|||||||
+71
-52
@@ -1,10 +1,14 @@
|
|||||||
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
|
# ADR-008: Trait length probe, fleet GC, and large-tier engines
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
|
Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
|
||||||
superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
|
superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
|
||||||
passed gate; every other part of this ADR stands as written)
|
passed gate — the two §Consequences passages that said pg-lo "remains
|
||||||
|
unbuilt" are resolved by that same supersession; the tier this ADR's
|
||||||
|
title and text call "fs" is renamed `large` by ADR-011; the fleet GC
|
||||||
|
mechanism's state home and the joint-tx ownership are ruled by
|
||||||
|
ADR-010; every other part of this ADR stands as written)
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
@@ -18,7 +22,7 @@ sharing one pool via postgres; requirements.md):
|
|||||||
probe"; store-api.md) and pool accounting rides `list` + `stat`
|
probe"; store-api.md) and pool accounting rides `list` + `stat`
|
||||||
(ADR-005), but the trait's methods are `has/get/put/delete/list/
|
(ADR-005), but the trait's methods are `has/get/put/delete/list/
|
||||||
name` with no value-metadata access. The kv engines' row shapes
|
name` with no value-metadata access. The kv engines' row shapes
|
||||||
both carry a `size` column (POC #5 B5), and the fs tier can stat a
|
both carry a `size` column (POC #5 B5), and the large tier can stat a
|
||||||
file — but with no trait method, kv `stat` degenerates to a full
|
file — but with no trait method, kv `stat` degenerates to a full
|
||||||
value load. The trait is the crate's second stable seam (ADR-003);
|
value load. The trait is the crate's second stable seam (ADR-003);
|
||||||
discovering this during implementation would be a breaking trait
|
discovering this during implementation would be a breaking trait
|
||||||
@@ -32,8 +36,8 @@ sharing one pool via postgres; requirements.md):
|
|||||||
node B's in-flight put is invisible. Data loss, structurally: the
|
node B's in-flight put is invisible. Data loss, structurally: the
|
||||||
topology REQ-2 requires does not satisfy the protocol ADR-005
|
topology REQ-2 requires does not satisfy the protocol ADR-005
|
||||||
specifies.
|
specifies.
|
||||||
3. **The fs tier is node-local, silently partitioning a fleet.** The
|
3. **The large tier is node-local, silently partitioning a fleet.** The
|
||||||
fs backend stores to a sharded layout on the node's own filesystem
|
large tier stores to a sharded layout on the node's own filesystem
|
||||||
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
|
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
|
||||||
put via node A exists only on A's disk; node B's get fall-through
|
put via node A exists only on A's disk; node B's get fall-through
|
||||||
queries its own fs and misses, and `list()` unions disagree per
|
queries its own fs and misses, and `list()` unions disagree per
|
||||||
@@ -41,7 +45,7 @@ sharing one pool via postgres; requirements.md):
|
|||||||
hypothetical.
|
hypothetical.
|
||||||
|
|
||||||
These three travel together: the fleet topology forces #2 and #3, and
|
These three travel together: the fleet topology forces #2 and #3, and
|
||||||
#3's resolution constrains what the fs tier must support, while #1 is
|
#3's resolution constrains what the large tier must support, while #1 is
|
||||||
a trait-level gap any topology hits.
|
a trait-level gap any topology hits.
|
||||||
|
|
||||||
## Decision
|
## Decision
|
||||||
@@ -55,7 +59,7 @@ tier:
|
|||||||
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
|
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
|
||||||
never a value load. The `size` column both POC row shapes carry
|
never a value load. The `size` column both POC row shapes carry
|
||||||
becomes contract-relevant.
|
becomes contract-relevant.
|
||||||
- **fs engine (`local`)**: stat/metadata read, no content fetch.
|
- **large engine (`local`)**: stat/metadata read, no content fetch.
|
||||||
- The store facade's `stat` (store-api.md) and the accounting path
|
- The store facade's `stat` (store-api.md) and the accounting path
|
||||||
(ADR-005) ride this method; the `has` method remains a pure
|
(ADR-005) ride this method; the `has` method remains a pure
|
||||||
existence probe.
|
existence probe.
|
||||||
@@ -71,18 +75,20 @@ never deleted by a sweep started before it committed** (ADR-005).
|
|||||||
Under a fleet, the mechanism moves into the shared engine:
|
Under a fleet, the mechanism moves into the shared engine:
|
||||||
|
|
||||||
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
|
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
|
||||||
inserts in one transaction on the fleet engine: the entry row and
|
inserts in one transaction on the fleet engine: the entry row and
|
||||||
its pin row commit together (or neither does). The pin row carries
|
its pin row commit together (or neither does). The pin row carries
|
||||||
an owner (node id) and expiry: the pin guard renews its row every
|
an owner (a store-instance id — ADR-011's vocabulary; "node id" is
|
||||||
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
this ADR's original wording, superseded because a node may host
|
||||||
parameters, not API). **Protection = an un-expired pin row**: at
|
several instances) and expiry: the pin guard renews its row every
|
||||||
delete-time arbitration, only `expiry > now()` pins count; an
|
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
||||||
expired pin is abandoned (its holder failed to renew — the same
|
parameters, not API). **Protection = an un-expired pin row**: at
|
||||||
liveness-failure signal as an in-process guard's drop) and is reaped
|
delete-time arbitration, only `expiry > now()` pins count; an
|
||||||
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
expired pin is abandoned (its holder failed to renew — the same
|
||||||
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
liveness-failure signal as an in-process guard's drop) and is reaped
|
||||||
remains the correctness backstop — content is re-put byte-identically
|
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
||||||
under the same key.
|
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
||||||
|
remains the correctness backstop — content is re-put byte-identically
|
||||||
|
under the same key.
|
||||||
- **Registered liveness sources have one fleet form: a liveness
|
- **Registered liveness sources have one fleet form: a liveness
|
||||||
table.** A fleet embedder registers its liveness as a named table of
|
table.** A fleet embedder registers its liveness as a named table of
|
||||||
`(key, …)` rows in the shared engine (schema pinned by the embedder,
|
`(key, …)` rows in the shared engine (schema pinned by the embedder,
|
||||||
@@ -131,9 +137,9 @@ Under a fleet, the mechanism moves into the shared engine:
|
|||||||
is shared machinery. The pin-row expiry/ttl machinery exists only
|
is shared machinery. The pin-row expiry/ttl machinery exists only
|
||||||
where fleet engines are active.
|
where fleet engines are active.
|
||||||
|
|
||||||
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
|
### 3. The large tier is engine-selectable; fleet topology dictates which engines are permissible
|
||||||
|
|
||||||
The fs tier is a tier, not a medium. It ships engines the way the kv
|
The large tier is a tier, not a medium. It ships engines the way the kv
|
||||||
tier does (ADR-007), and a deployment's topology determines which are
|
tier does (ADR-007), and a deployment's topology determines which are
|
||||||
valid:
|
valid:
|
||||||
|
|
||||||
@@ -141,7 +147,7 @@ valid:
|
|||||||
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
|
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
|
||||||
REQ-4) and for fleet nodes using **shared media** (below).
|
REQ-4) and for fleet nodes using **shared media** (below).
|
||||||
- **Fleet + shared media (deployment requirement):** multi-node pools
|
- **Fleet + shared media (deployment requirement):** multi-node pools
|
||||||
may keep the `local` engine by pointing every node's fs tier at
|
may keep the `local` engine by pointing every node's large tier at
|
||||||
shared/replicated media. The soundness claim is *qualified*, not
|
shared/replicated media. The soundness claim is *qualified*, not
|
||||||
assumed: immutable-once-committed files plus stage-then-commit-
|
assumed: immutable-once-committed files plus stage-then-commit-
|
||||||
rename are sound on media that provides (a) atomic rename within a
|
rename are sound on media that provides (a) atomic rename within a
|
||||||
@@ -158,43 +164,52 @@ valid:
|
|||||||
not share media designates **one storage node**; large-blob writes
|
not share media designates **one storage node**; large-blob writes
|
||||||
and reads over the pool route through it via the ops surface
|
and reads over the pool route through it via the ops surface
|
||||||
(verified fetch/put; ADR-001). Client nodes' constructors are
|
(verified fetch/put; ADR-001). Client nodes' constructors are
|
||||||
configured with a **kv-tier-only dispatch** (or an fs tier reserved
|
configured with **kv-only mode** (ADR-011 §4's name for what this
|
||||||
for node-private, consumer-owned content — which is outside the
|
bullet first called "kv-tier-only dispatch") — no local large tier
|
||||||
pool and outside this crate's GC; it is the consumer's own storage
|
over pool content at all; a client node's *node-private,
|
||||||
beside the pool per ADR-004, not pool content). The facade shape on
|
consumer-owned* large-tier content is outside the pool and outside
|
||||||
|
this crate's GC entirely: it is the consumer's own storage beside
|
||||||
|
the pool per ADR-004, not a constructor mode over pool content, and
|
||||||
|
the composite fleet-validity rule (backends-and-dispatch.md)
|
||||||
|
therefore has no "local on node-private media over one pool" entry.
|
||||||
|
The facade shape on
|
||||||
a client node is its own store for sub-threshold content plus its
|
a client node is its own store for sub-threshold content plus its
|
||||||
registered ops-put path for large content — a consumer-side
|
registered ops-put path for large content — a consumer-side
|
||||||
composition, not a store-level mode.
|
composition, not a store-level mode.
|
||||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
- `pg-lo` (admitted by ADR-009; the gate): postgres Large Objects
|
||||||
as the fs tier's storage — the same "engine behind one contract"
|
as the large tier's storage —
|
||||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
the same "engine behind one contract"
|
||||||
with its admission gated on measured evidence (POC #7: LO write/read
|
move as ADR-007, one tier over. Originally named as a candidate,
|
||||||
|
admission gated on measured evidence (POC #7: LO write/read
|
||||||
curves at the packfile regime, tx-scoped handle cost under
|
curves at the packfile regime, tx-scoped handle cost under
|
||||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
behavior) and its own engine ADR. NOT shipped speculatively;
|
||||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||||
pg-lo is the consolidation option if those are unacceptable. *Update
|
pg-lo is the consolidation option if those are unacceptable. *Update
|
||||||
2026-10-03: POC #7 ran and passed its gate —
|
2026-10-03: POC #7 ran and passed its gate —
|
||||||
`docs/research/poc-pglo-findings.md` (companion-table authority,
|
`docs/research/poc-pglo-findings.md` (companion-table authority,
|
||||||
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
|
`lo_get` window gets, durable put ≈ fs's durable put, cached gets
|
||||||
behind page-cache fs by a measured 20-50x aggregate-multiplied to
|
behind page-cache fs by a measured 20-50x single-stream, ~700 MB/s
|
||||||
~0.7 GB/s, clean crash-orphan behavior, bounded vacuum posture).
|
aggregate at 16 readers). Superseded the same day: ADR-009 admitted
|
||||||
The engine ADR's remaining evidence path is open; admission remains
|
the engine — this bullet's original "candidate, not shipped"
|
||||||
gated on it, not on speculation.*
|
posture is historical; the engine is shipped, feature `pg-lo`.*
|
||||||
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
|
- **The no-mixed-large-tiers rule is a deployment invariant, enforced at
|
||||||
the enforceable seam.** Cross-node configuration cannot be validated
|
the enforceable seam.** Cross-node configuration cannot be validated
|
||||||
by any one constructor (it sees only its own node). What the
|
by any one constructor (it sees only its own node). What the
|
||||||
constructor *can* enforce: a fleet-mode constructor (fleet engines
|
constructor *can* enforce: a fleet-mode constructor (fleet engines
|
||||||
active) requires an explicit fs-tier declaration — `local` (with the
|
active) requires an explicit large-tier declaration — `local` (with the
|
||||||
operator's assertion that its root is fleet-shared media), routed
|
operator's assertion that its root is fleet-shared media), routed
|
||||||
(no local fs over pool content), or — when shipped — `pg-lo`. The
|
(kv-only mode, ADR-011 §4; no local filesystem over pool content), or
|
||||||
|
`pg-lo`. The
|
||||||
cross-node truth ("is node B's `local` actually the same media as
|
cross-node truth ("is node B's `local` actually the same media as
|
||||||
node A's?") is a deployment invariant this crate documents and ops
|
node A's?") is a deployment invariant this crate documents and ops
|
||||||
verifies, not something a constructor can prove; a partitioned
|
verifies, not something a constructor can prove; a partitioned
|
||||||
tier's observable signature (get fall-through misses for entries
|
tier's observable signature (get fall-through misses for entries
|
||||||
another node wrote) is the documented detection symptom.
|
another node wrote) is the documented detection symptom. The
|
||||||
|
composite fleet-validity predicate over both tiers is pinned in
|
||||||
|
backends-and-dispatch.md ("The composite fleet-validity rule").
|
||||||
|
|
||||||
In sum: a fleet's fs tier over pool content is either shared media (all
|
In sum: a fleet's large tier over pool content is either shared media (all
|
||||||
nodes' `local` roots on the same fleet-shared media) or routed (one
|
nodes' `local` roots on the same fleet-shared media) or routed (one
|
||||||
storage node); mixing per-node-local media over one pool is the
|
storage node); mixing per-node-local media over one pool is the
|
||||||
partitioning failure this decision exists to prevent.
|
partitioning failure this decision exists to prevent.
|
||||||
@@ -211,9 +226,10 @@ partitioning failure this decision exists to prevent.
|
|||||||
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
|
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
|
||||||
invariant extends to fleets as a *stated* mechanism, not an
|
invariant extends to fleets as a *stated* mechanism, not an
|
||||||
accident of single-process luck.
|
accident of single-process luck.
|
||||||
- The fs tier's fleet problem has named, shipped-code-free answers
|
- The large tier's fleet problem has named answers: shipped-code-free
|
||||||
(shared media, re-routing) and a named consolidation candidate
|
for shared media and re-routing, shipped-code for the consolidation
|
||||||
(pg-lo) whose admission path is the same POC-first door as ADR-007.
|
option (pg-lo — admitted by ADR-009, the same POC-first door as
|
||||||
|
ADR-007).
|
||||||
|
|
||||||
**Negative**
|
**Negative**
|
||||||
|
|
||||||
@@ -223,9 +239,9 @@ partitioning failure this decision exists to prevent.
|
|||||||
surface for orphaned pins at node crash — the expiry is the cleanup
|
surface for orphaned pins at node crash — the expiry is the cleanup
|
||||||
handle), advisory-lock coordination, and SQL-level arbitration
|
handle), advisory-lock coordination, and SQL-level arbitration
|
||||||
added on top of the already-intricate delete-window state machine.
|
added on top of the already-intricate delete-window state machine.
|
||||||
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
|
- `pg-lo` was built under ADR-009 after POC #7 passed — the
|
||||||
re-route pays for its ADR + POC before shipping large blobs — the
|
measured, deliberate sequence of the ADR-006 pattern (admission
|
||||||
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
|
gate first, engine second), not a hedge; resolved same-day.
|
||||||
|
|
||||||
**Neutral**
|
**Neutral**
|
||||||
|
|
||||||
@@ -245,13 +261,16 @@ partitioning failure this decision exists to prevent.
|
|||||||
columns; the driver stack pg-lo would ride)
|
columns; the driver stack pg-lo would ride)
|
||||||
- ADR-002 (the one-way-door discipline this trait change applies),
|
- ADR-002 (the one-way-door discipline this trait change applies),
|
||||||
ADR-003 (the trait contract and admission gate `size` and pg-lo
|
ADR-003 (the trait contract and admission gate `size` and pg-lo
|
||||||
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
|
amend/enter), ADR-004 (tier≠engine, extended to the large tier),
|
||||||
ADR-005 (the GC invariant and single-node protocol extended here,
|
ADR-005 (the GC invariant and single-node protocol extended here,
|
||||||
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
|
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
|
||||||
follows), ADR-007 (the engine-behind-one-trait precedent)
|
follows), ADR-007 (the engine-behind-one-trait precedent)
|
||||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
|
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
|
||||||
and tier surface this amends);
|
and tier surface this amends);
|
||||||
[store-api.md](../store-api.md) (`stat`'s contract);
|
[store-api.md](../store-api.md) (`stat`'s contract);
|
||||||
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
||||||
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
||||||
topology's transport)
|
topology's transport); ADR-010 (the fleet mechanism's state home
|
||||||
|
and the joint-tx ownership this ADR's §2 implied but did not rule);
|
||||||
|
ADR-011 (the vocabulary — instance/fleet — this ADR's topology
|
||||||
|
language resolves to; the large tier renamed `large`)
|
||||||
+15
-12
@@ -1,14 +1,17 @@
|
|||||||
# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine
|
# ADR-009: pg-lo — postgres Large Objects as the large tier's second engine
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Accepted (admission evidence: POC #7, passed 2026-10-03 —
|
Accepted (admission evidence: POC #7, passed 2026-10-03 —
|
||||||
`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
|
`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
|
||||||
gate)
|
gate. The tier this ADR calls "fs" is renamed `large` by ADR-011);
|
||||||
|
§2's "put = one transaction" tx-ownership statement is corrected by
|
||||||
|
ADR-010 §2 (the store core holds the tx; the engine contributes its
|
||||||
|
staged-put mechanics into it — engine content unchanged))
|
||||||
|
|
||||||
## Context
|
## Context
|
||||||
|
|
||||||
ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's
|
ADR-008 named `pg-lo` as the candidate large-tier engine for REQ-2's
|
||||||
fleet topology (multiple store instances sharing one pg pool, serving
|
fleet topology (multiple store instances sharing one pg pool, serving
|
||||||
large/beyond-threshold pool content) — expressly gated on the ADR-003
|
large/beyond-threshold pool content) — expressly gated on the ADR-003
|
||||||
admission door: measured evidence first, in the ADR-007 shape. POC #7
|
admission door: measured evidence first, in the ADR-007 shape. POC #7
|
||||||
@@ -38,7 +41,7 @@ The gate's three legs all passed:
|
|||||||
|
|
||||||
## Decision
|
## Decision
|
||||||
|
|
||||||
**The fs tier ships `pg-lo` as its second engine — postgres Large
|
**The large tier ships `pg-lo` as its second engine — postgres Large
|
||||||
Objects — behind the same `Backend` trait, feature `pg-lo` (default-
|
Objects — behind the same `Backend` trait, feature `pg-lo` (default-
|
||||||
off), constructor-selected per node.** REQ-2's deployment choice set
|
off), constructor-selected per node.** REQ-2's deployment choice set
|
||||||
(ADR-008) extends from {shared media, re-routing} to include the
|
(ADR-008) extends from {shared media, re-routing} to include the
|
||||||
@@ -58,7 +61,7 @@ consolidation option as a shipped engine.
|
|||||||
into the LO and commit the row on completion; rollback discards
|
into the LO and commit the row on completion; rollback discards
|
||||||
both.
|
both.
|
||||||
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
|
3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
|
||||||
the fs-tier handle closes over the companion row (oid + length) —
|
the large-tier handle closes over the companion row (oid + length) —
|
||||||
immutable, so the row is authoritative — not over a tx-scoped
|
immutable, so the row is authoritative — not over a tx-scoped
|
||||||
descriptor. Ranged reads are single stateless statements,
|
descriptor. Ranged reads are single stateless statements,
|
||||||
pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
|
pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
|
||||||
@@ -82,7 +85,7 @@ consolidation option as a shipped engine.
|
|||||||
(committed LOs without companion rows — mixed/legacy stores). The
|
(committed LOs without companion rows — mixed/legacy stores). The
|
||||||
crash story is structurally clean: LO lifecycle is transactional
|
crash story is structurally clean: LO lifecycle is transactional
|
||||||
in modern postgres; kill mid-tx leaves zero pages, zero
|
in modern postgres; kill mid-tx leaves zero pages, zero
|
||||||
half-commits (cleaner than fs `local`'s stage-file residue,
|
half-commits (cleaner than `local`'s stage-file residue,
|
||||||
which needs its own recovery sweep).
|
which needs its own recovery sweep).
|
||||||
6. **Performance deltas, as named capacity facts (not hidden, not
|
6. **Performance deltas, as named capacity facts (not hidden, not
|
||||||
solved):** single-stream cached get runs 20–50× behind page-cache
|
solved):** single-stream cached get runs 20–50× behind page-cache
|
||||||
@@ -105,13 +108,13 @@ consolidation option as a shipped engine.
|
|||||||
**Positive**
|
**Positive**
|
||||||
|
|
||||||
- REQ-2's consolidation option is real and shipped: a fleet node runs
|
- REQ-2's consolidation option is real and shipped: a fleet node runs
|
||||||
*postgres only* — kv (bytea) tier + fs (LO) tier from one engine,
|
*postgres only* — kv (bytea) tier + large (LO) tier from one engine,
|
||||||
no shared-media requirement, no re-routing topology, no consumer-ops
|
no shared-media requirement, no re-routing topology, no consumer-ops
|
||||||
dependency for pool content.
|
dependency for pool content.
|
||||||
- The engine adds ~400 lines of impl over an already-fixed contract —
|
- The engine adds ~400 lines of impl over an already-fixed contract —
|
||||||
the additive shape both ADR-007 and ADR-008 projected, now
|
the additive shape both ADR-007 and ADR-008 projected, now
|
||||||
evidenced.
|
evidenced.
|
||||||
- Crash hygiene is *better* than fs `local` (transactional LO
|
- Crash hygiene is *better* than `local` (transactional LO
|
||||||
lifecycle vs. stage-file residue); the engine's only recovery sweep
|
lifecycle vs. stage-file residue); the engine's only recovery sweep
|
||||||
targets a class (bypassed-LO) that healthy deployments never produce.
|
targets a class (bypassed-LO) that healthy deployments never produce.
|
||||||
|
|
||||||
@@ -123,9 +126,9 @@ consolidation option as a shipped engine.
|
|||||||
— the fleet case — scales to ~700 MB/s on the POC box; per-box
|
— the fleet case — scales to ~700 MB/s on the POC box; per-box
|
||||||
numbers shift with cores.)
|
numbers shift with cores.)
|
||||||
- LO catalog space is never returned (monitor + occasional
|
- LO catalog space is never returned (monitor + occasional
|
||||||
operator-run `VACUUM FULL`), the inverse of fs `local`'s
|
operator-run `VACUUM FULL`), the inverse of `local`'s
|
||||||
immediate-return.
|
immediate-return.
|
||||||
- Feature/maintenance surface: the fs tier now carries two engines
|
- Feature/maintenance surface: the large tier now carries two engines
|
||||||
(like the kv tier), each with posture deltas to keep honest
|
(like the kv tier), each with posture deltas to keep honest
|
||||||
(ADR-007's negative, inherited).
|
(ADR-007's negative, inherited).
|
||||||
|
|
||||||
@@ -147,10 +150,10 @@ consolidation option as a shipped engine.
|
|||||||
- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
|
- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
|
||||||
this engine serves); REQ-1/3/4 unchanged postures
|
this engine serves); REQ-1/3/4 unchanged postures
|
||||||
- ADR-003 (the trait contract + admission door exercised twice now),
|
- ADR-003 (the trait contract + admission door exercised twice now),
|
||||||
ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC
|
ADR-004 (tier ≠ engine, large tier edition), ADR-005/008 (the GC
|
||||||
invariants this engine participates in — table-level, nothing
|
invariants this engine participates in — table-level, nothing
|
||||||
fs-specific), ADR-007 (the engine-admission precedent and driver
|
fs-specific), ADR-007 (the engine-admission precedent and driver
|
||||||
stack)
|
stack)
|
||||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs
|
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the large-tier
|
||||||
engine surface this amends); [overview.md](../overview.md);
|
engine surface this amends); [overview.md](../overview.md);
|
||||||
[store-api.md](../store-api.md)
|
[store-api.md](../store-api.md)
|
||||||
@@ -0,0 +1,325 @@
|
|||||||
|
# ADR-010: The Backend trait's I/O seams — read cursor, staged put, and the engine-state companion seam
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The engine story closed at the decision level (ADR-007/008/009), but
|
||||||
|
three seams that implementation must not guess are unpinned. All three
|
||||||
|
concentrate where the tier/engine split stretched the original
|
||||||
|
contract:
|
||||||
|
|
||||||
|
1. **The `get`/`put` value shapes were never unified.** The trait POC's
|
||||||
|
`get` returns `Option<Vec<u8>>`; the largeblob POC's fs arm "yields
|
||||||
|
a file handle, not a loaded buffer"; the pg-lo POC's get is a
|
||||||
|
`LoGetHandle` driving `lo_get(oid, off, len)` windows — three
|
||||||
|
structurally different shapes behind one trait, and no document
|
||||||
|
named the unified contract. Same on the put side: a byte slice
|
||||||
|
(kv), a stage-then-commit-rename stream (`local`), and a
|
||||||
|
chunked-`lowrite` transaction (pg-lo) are all "the put". A unified
|
||||||
|
trait is the crate's second stable seam (ADR-003); its I/O shapes
|
||||||
|
are decided before code, not discovered per engine.
|
||||||
|
|
||||||
|
2. **ADR-008's fleet state has no seam.** Pins become expiry-carrying
|
||||||
|
rows committed with the entry; staged delete-window candidates
|
||||||
|
(including on sqlite); `SELECT … FOR UPDATE` arbitration; a pg
|
||||||
|
advisory lock for the sweeper. None of that is expressible through
|
||||||
|
the public trait method set, and ADR-003's "the shipped trait
|
||||||
|
impls write plain content-addressed rows into the chosen engine,
|
||||||
|
nothing else" reads as a bar the fleet mechanism violates. Also
|
||||||
|
unresolved: who owns the joint entry+pin transaction — ADR-009
|
||||||
|
says the pg-lo put's single tx carries the pin row, but ADR-005
|
||||||
|
says `Pin` is a store-core RAII guard under the arbitration lock.
|
||||||
|
The trait has no parameter for either, and "the backend learns
|
||||||
|
about pins" would contradict ADR-005. The ops pin-token contract
|
||||||
|
(ops-surface.md) inherits the same conflict: a remote putter holds
|
||||||
|
a token, but an expiry-guarded pin row goes stale if the holder
|
||||||
|
pauses — a sweep could delete a remotely-in-flight put whose row
|
||||||
|
expired.
|
||||||
|
|
||||||
|
3. **The kv tier had no fleet-validity rule.** ADR-008 §3 constrains
|
||||||
|
the large tier's fleet forms (shared media, re-routing) but pins
|
||||||
|
nothing for the kv tier, which the fleet story centers on: two
|
||||||
|
nodes' sqlite engines are trivially partitioned (two files = two
|
||||||
|
pools, not one), and no document said so.
|
||||||
|
|
||||||
|
The deciding facts are in hand (Schrödinger's-code rule — none of
|
||||||
|
these may wait on consumers): the POCs fixed every engine's native
|
||||||
|
shape; ADR-008 fixed the fleet mechanism's shape (rows, expiry,
|
||||||
|
advisory lock, staged window); ADR-001 §4 fixed that ops is a
|
||||||
|
store-facade consumer with no backend side-door. What is missing is
|
||||||
|
composition: how the store core drives all engines through one
|
||||||
|
contract while owning GC state.
|
||||||
|
|
||||||
|
Alternatives considered for the fleet-state seam, rejected:
|
||||||
|
|
||||||
|
- **Put the fleet machinery on the public trait** (pin/window/table
|
||||||
|
methods on `Backend`): every engine — including `mem` and `local` —
|
||||||
|
must then implement SQL-shaped machinery it cannot express. Rejected.
|
||||||
|
- **Backends grow pin awareness** (the pg-lo put tx carries the pin
|
||||||
|
row, backend-maintained): contradicts ADR-005's "backends never
|
||||||
|
learn liveness"; and the arbitration decision (expiry compare,
|
||||||
|
liveness consult) still belongs to the store, so the store ends up
|
||||||
|
reaching into the engine's tx anyway. Rejected.
|
||||||
|
- **A raw-SQL side-channel** (connection escape hatch the store core
|
||||||
|
may use arbitrarily): turns the engine abstraction off for exactly
|
||||||
|
the machinery (fleet GC) it most needs to be sound on. Rejected.
|
||||||
|
The seam below is that escape hatch made honest — named clauses,
|
||||||
|
contract-tested per engine, instead of raw access.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### 1. One `Backend` trait; two I/O abstractions per tier — read cursor and staged put
|
||||||
|
|
||||||
|
**The trait stays one trait.** Every engine implements the same method
|
||||||
|
set (ADR-003's methods + ADR-008's `size`) — no per-tier trait split.
|
||||||
|
What changes: the trait's I/O types are the crate-internal
|
||||||
|
abstractions below, which each engine constructs its own way — the
|
||||||
|
store core programs the abstraction, never the engine shape.
|
||||||
|
|
||||||
|
- **`get` returns a read cursor.** One abstraction, two body families:
|
||||||
|
- **kv engines** (sqlite, postgres, mem): the cursor materializes
|
||||||
|
the whole value (the engine's row read; bounded by the small-blob
|
||||||
|
tier policy). Its range reads are in-memory slices; `len()` reads
|
||||||
|
the row's `size` column.
|
||||||
|
- **large engines** (`local`, pg-lo): the cursor streams; the medium
|
||||||
|
is hidden in the cursor body — `local`'s is a pread loop over the
|
||||||
|
committed file (immutable once committed); pg-lo's drives
|
||||||
|
descriptorless `lo_get(oid, off, len)` windows (ADR-009 §3's C4
|
||||||
|
posture — pool-friendly, stateless per window; the
|
||||||
|
tx-scoped-descriptor form is the fallback). `len()` reads the
|
||||||
|
companion row or stats the file — metadata only, never content
|
||||||
|
(ADR-008's discipline).
|
||||||
|
- Both body families serve the ADR-006 range contract (`read_range`
|
||||||
|
→ slice + slice digest) at one seam. The cursor is
|
||||||
|
**store-internal — not public API**: store consumers see the
|
||||||
|
facade's `(len, stream)` shapes (store-api.md) with the cursor
|
||||||
|
staying behind them; the store's stream utility adapts engine
|
||||||
|
cursor bodies, it does not expose them.
|
||||||
|
- **`put` exists in two forms, selected by the store core before
|
||||||
|
content flows; both land in the CAS entry shape and both
|
||||||
|
participate in pin-before-publish (§2 decides who holds the tx).**
|
||||||
|
- **Whole-value put** for the kv tier: the settled row write.
|
||||||
|
POC-shaped bodies: sqlite `INSERT OR IGNORE`, postgres
|
||||||
|
`ON CONFLICT DO NOTHING`, mem `entry().or_insert()`.
|
||||||
|
- **Staged put** for the large tier — ADR-003's
|
||||||
|
stage-then-commit-rename promoted from behavior to a **named
|
||||||
|
per-engine contract**: the engine hands out a *staging cursor*
|
||||||
|
(`local`: the stage file under the store's stage-hygiene protocol;
|
||||||
|
pg-lo: chunked `lowrite` statements over a tx the store core
|
||||||
|
holds — §2's `pin_commit` is that holder; 512 KiB per POC #7's
|
||||||
|
measured knee). Commit publishes the entry;
|
||||||
|
every failure path converges on discard (store-api.md stage
|
||||||
|
hygiene, per engine, exact-count tested — POC #3 A5's shape).
|
||||||
|
- **`list`/`delete`/`has`/`size`/`name` are tier-agnostic.** `list`
|
||||||
|
walks the engine's companion structure (rows / files / LO
|
||||||
|
companion); `size` is metadata-only (ADR-008). The exact-count
|
||||||
|
sweep-outcome gate (ADR-003) runs per engine.
|
||||||
|
|
||||||
|
**Feature-graph consequence** (fills a gap ADR-007/009 left): feature
|
||||||
|
`pg-lo` does **not** imply feature `postgres`. Each is an engine
|
||||||
|
feature pulling the shared driver cluster (`tokio-postgres` +
|
||||||
|
`deadpool-postgres`) independently — a `large`=pg-lo + kv=sqlite
|
||||||
|
node is coherent and supported. overview.md's dependency table pins
|
||||||
|
the edge.
|
||||||
|
|
||||||
|
### 2. GC state is store-core-owned; per-engine state seams are crate-internal and contract-tested
|
||||||
|
|
||||||
|
**Ownership: the store core owns all GC state — in-process pin and
|
||||||
|
liveness maps *and* the fleet state's rows and locks. The public
|
||||||
|
`Backend` trait stays exactly what ADR-003 says (opaque
|
||||||
|
`hash → bytes`, namespace-blind; "plain content-addressed rows,
|
||||||
|
nothing else" remains true of the public trait). Backends never learn
|
||||||
|
liveness (ADR-005) — verbatim.**
|
||||||
|
|
||||||
|
The store core reaches the fleet machinery through a **crate-internal
|
||||||
|
(not exported) engine-state companion trait**, implemented by the
|
||||||
|
SQL-backed engines (sqlite, postgres, pg-lo), with `local` and
|
||||||
|
`mem` excluded by construction (it cannot be expressed on them and
|
||||||
|
they are never fleet engines). ADR-008's mechanism fixes *what*; the
|
||||||
|
seam is *where* the mechanism's SQL lives. Clauses:
|
||||||
|
|
||||||
|
- `ensure_gc_schema()` — create `gc_pins (key, owner, expiry)` and
|
||||||
|
`gc_window (key, visible_after)` at store construction (store-core
|
||||||
|
call, engine-neutral API; dialect-specific SQL bodies). `owner`
|
||||||
|
is the **store-instance id** (ADR-011's vocabulary; ADR-008's
|
||||||
|
"node id" wording predates that split). The pin row's identity is
|
||||||
|
`(key, owner)` — a dedup re-put by a different instance is a
|
||||||
|
different pin row; protection composes by row (an entry is
|
||||||
|
protected while any un-expired pin row exists), which is
|
||||||
|
ADR-005's refcount semantics expressed in rows. The embedder's
|
||||||
|
liveness table is *not* created by this seam — ADR-008: schema
|
||||||
|
pinned by the embedder; the sweep reads it and never owns it.
|
||||||
|
**Per-engine applicability (the test-gate shape):** sqlite —
|
||||||
|
`ensure_gc_schema` (window table only, since the window staging
|
||||||
|
is shared machinery per ADR-008 §2) + `window_stage`/
|
||||||
|
`window_take` + the sweep-lock clause's single-node body; the
|
||||||
|
pin clauses (`pin_commit`/`pin_renew`/`pin_check`) have sqlite
|
||||||
|
bodies but are exercised by unit tests only — they are dead
|
||||||
|
machinery on sqlite because no sqlite fleet is on offer (§3).
|
||||||
|
postgres and pg-lo — all clause families, contract-tested,
|
||||||
|
fleet-active.
|
||||||
|
- `pin_commit(joint_tx, entries)` — the store core **holds the
|
||||||
|
transaction**; the engine contributes its put mechanics (row
|
||||||
|
shape, CAS clause, LO writes) and the pin-row inserts, and both
|
||||||
|
commit together. This is the joint entry+pin transaction's owner,
|
||||||
|
ruled: **the store holds the tx; the engine contributes mechanics
|
||||||
|
to it.** The public `put` remains the plain, pin-free path — the
|
||||||
|
store's normal put path routes through `pin_commit` wherever the
|
||||||
|
engine is seam-capable and fleet state is active.
|
||||||
|
- pg-lo correction: ADR-009 §2's "put = one transaction" is
|
||||||
|
unchanged in content but its ownership statement is hereby
|
||||||
|
corrected — the tx is held by the store core; the pg-lo engine
|
||||||
|
contributes its staged-put mechanics and companion-row insert
|
||||||
|
into it. The engine sees pins only as rows; never as semantics.
|
||||||
|
- `pin_renew(key, owner, ttl)` — the engine executes the expiry
|
||||||
|
update. The store's pin guard owns the TTL/3 renewal schedule
|
||||||
|
(ADR-008); the engine owns only the SQL.
|
||||||
|
- `pin_check(key) → PinState` / `arbitrate(key) → bool` — the engine
|
||||||
|
executes ADR-008's delete-time arbitration (un-expired pin rows +
|
||||||
|
embedded liveness tables, `SELECT … FOR UPDATE` finger-lock,
|
||||||
|
keyed, at delete time); the store core receives the answer and
|
||||||
|
applies the decision. If fleet clauses ever grow, they grow here —
|
||||||
|
the seam is the named home, with contract tests per clause.
|
||||||
|
- `window_stage(candidates)` / `window_take(now)` — the staged
|
||||||
|
delete-window candidates. Per ADR-008 §2 the window table exists
|
||||||
|
on **every** engine (sqlite included): staged candidates are
|
||||||
|
engine rows; what differs by topology is where *liveness* lives
|
||||||
|
(in-process maps for single-node engines, tables for fleets), not
|
||||||
|
whether staging exists.
|
||||||
|
- `sweep_lock_acquire()/release()` — the sweep single-flighter: the
|
||||||
|
pg advisory lock (ADR-008) for the SQL engines it applies to; the
|
||||||
|
store treats lock contention exactly as ADR-008 says — the second
|
||||||
|
sweeper's sweep aborts with the typed `GcAborted` (fleet cause),
|
||||||
|
nothing deleted. (SQLite today has no fleet story — see §3 — so its
|
||||||
|
clause body exists but is exercised by single-node tests only.)
|
||||||
|
|
||||||
|
Each clause is contract-tested per its applicability (the
|
||||||
|
per-engine split above): the conformance suite of §1 grows the fleet
|
||||||
|
clauses for postgres and pg-lo; `local` and
|
||||||
|
`mem` are excluded by construction.
|
||||||
|
|
||||||
|
### 3. The kv tier's fleet-validity rule (the mirror of ADR-008 §3)
|
||||||
|
|
||||||
|
**A fleet constructor requires a pool-shared kv engine.** Postgres is
|
||||||
|
the shipped engine that satisfies this. A sqlite-fleet mechanism does
|
||||||
|
not exist and is not on offer (per ADR-007's redb precedent this is
|
||||||
|
recorded as a non-option, not an open question); two nodes' sqlite
|
||||||
|
engines over "one pool" are two pools — the partitioning failure, the
|
||||||
|
kv-tier edition of what ADR-008 §3 prevents for the large tier. The
|
||||||
|
enforceable
|
||||||
|
seam is the same as ADR-008 §3's: the constructor can require an
|
||||||
|
explicit declaration; it cannot prove cross-node truth. The
|
||||||
|
constructor table is pinned in [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
|
||||||
|
and is closed under both tiers.
|
||||||
|
|
||||||
|
### 4. The ops pin-token contract's TTL race, resolved by the ownership rule
|
||||||
|
|
||||||
|
**The token *is* the pin — there is no second pin state.** Under fleet
|
||||||
|
engines the put's pin is the expiry-carrying row (ADR-008); the token
|
||||||
|
identifies the putter as the row's owner. The token itself is an
|
||||||
|
**opaque id minted by the ops layer** over the pin row it just
|
||||||
|
committed — the ops layer (not the store facade) owns the
|
||||||
|
token↔(key, owner) mapping, in-memory for the handler's lifetime plus
|
||||||
|
a lookup by the renewal op; no schema is specced for it beyond that
|
||||||
|
(it is a handle, not durable state — the durable state is the pin row
|
||||||
|
the seam wrote). Concretely:
|
||||||
|
|
||||||
|
- **When the server-side `Pin` guard drops:** at response time. The
|
||||||
|
put lands pinned (the joint tx guarantees the entry and its pin row
|
||||||
|
commit together); the handler's in-process guard — the thing ADR-005
|
||||||
|
pins *in-process* with — is released when the `blobs/put` response
|
||||||
|
completes and the token is handed out. From that moment the pin
|
||||||
|
row's expiry is the only protection, which is exactly why the
|
||||||
|
renewal duty below exists. (If the handler kept its guard for the
|
||||||
|
token's whole life, the TTL machinery would be dead code on the ops
|
||||||
|
path — the row's expiry, not the guard's presence, is the fleet
|
||||||
|
contract's liveness signal.)
|
||||||
|
- The putter renews the pin by renewing the row — through ops it
|
||||||
|
already speaks: a `blobs/have` probe whose handler, observing a
|
||||||
|
valid token presented alongside the digest, calls the
|
||||||
|
crate-internal `pin_renew(key, owner)` via the ops module's own
|
||||||
|
store-core access (the ops module is in-crate — it crosses the
|
||||||
|
crate-internal seam by construction, the same as the put handler
|
||||||
|
holds the `Pin` guard; what never happens is a *consumer* reaching
|
||||||
|
the seam. ADR-008's `has` remains a pure probe: the renewal is the
|
||||||
|
ops layer's token-authenticated act on the pin row, not a `has`
|
||||||
|
semantic), or a re-put of the same digest (CAS dedup; the re-put's
|
||||||
|
joint tx upserts the key+owner pin row, renewing it). The renewal
|
||||||
|
cadence is the one ADR-008 fixed on the pin guard: **every TTL/3 —
|
||||||
|
20 s at the default 60 s TTL.**
|
||||||
|
- A putter pausing past TTL falls to ADR-008's expiry semantics
|
||||||
|
verbatim: the row is abandoned and reaped; the backstop is
|
||||||
|
delete-then-recover (ADR-005) — the digest is re-offered, the put
|
||||||
|
is idempotent under CAS. The race resolves to a re-put, never to
|
||||||
|
silent loss.
|
||||||
|
- For the `local` engine the token is the in-process guard as ADR-005
|
||||||
|
had it: nothing persists, nothing expires; the process crash is
|
||||||
|
ADR-005's in-flight-put loss case with the same backstop. (The
|
||||||
|
cross-wire token under `local` is the ops layer's in-memory handle
|
||||||
|
to that guard — same mapping discipline, no expiry.)
|
||||||
|
|
||||||
|
ops-surface.md's hand-over contract gains the renewal line and the
|
||||||
|
token-mapping note; nothing else changes.
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- The trait's I/O shapes are decided before code; engine tasks can be
|
||||||
|
written without guessing signatures. The per-engine
|
||||||
|
conformance-suite shape (POC #7's 10/10 gate as the norm) gives
|
||||||
|
decomposition a uniform test gate across all five engines.
|
||||||
|
- "Backends never learn liveness" survives the fleet mechanism
|
||||||
|
intact; the store core owns GC state per ADR-005 exactly as
|
||||||
|
written; ADR-008's mechanism lives in a named, tested seam instead
|
||||||
|
of an implied raw-SQL escape hatch; ADR-003's "plain rows, nothing
|
||||||
|
else" stays true of the public trait.
|
||||||
|
- The ops pin-token TTL race is resolved with no new ops; the
|
||||||
|
backstop is already-validated semantics (delete-then-recover).
|
||||||
|
- The kv tier's fleet-validity rule closes the constructor table
|
||||||
|
under both tiers.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- A crate-internal companion trait is real machinery: six clause
|
||||||
|
families on postgres/pg-lo (fleet-active, all contract-tested);
|
||||||
|
sqlite carries only the window + lock clauses as fleet-relevant
|
||||||
|
machinery (its pin clauses have unit-test bodies but no fleet mode
|
||||||
|
on offer — §3). The seam bounds where fleet-GC growth lands; it
|
||||||
|
does not shrink it.
|
||||||
|
- Remote putters gain a real protocol duty (renew past 20 s;
|
||||||
|
pause-past-TTL ⇒ delete-then-recover) — documented, but a duty.
|
||||||
|
- The sqlite sweep-lock clause body exists but is exercised only by
|
||||||
|
single-node tests until a sqlite-fleet posture is ever
|
||||||
|
decided (it is not on offer — §3).
|
||||||
|
|
||||||
|
**Neutral**
|
||||||
|
|
||||||
|
- The `Backend` trait's method set is unchanged from ADR-003/008 —
|
||||||
|
no public API change beyond what store-api.md already specced; the
|
||||||
|
cursor/staging abstractions are crate-internal.
|
||||||
|
- The staged-put contract already existed behaviorally (POC #3 A5);
|
||||||
|
this ADR names it.
|
||||||
|
- ADR-009's engine content is unchanged; only its tx-ownership
|
||||||
|
statement is corrected here.
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- ADR-003 (the trait contract this ADR's I/O shapes refine), ADR-004
|
||||||
|
(tier ≠ engine — the vocabulary is ADR-011's), ADR-005 (the
|
||||||
|
liveness-ownership rule this ADR preserves), ADR-006 (the range
|
||||||
|
contract the cursor serves), ADR-008 (the fleet mechanism whose
|
||||||
|
state home this ADR is), ADR-009 (the pg-lo engine; tx-ownership
|
||||||
|
correction), [ADR-011](011-vocabulary-tiers-engines-and-fleet-state.md)
|
||||||
|
(vocabulary + constructor table)
|
||||||
|
- `docs/research/poc-trait-dispatch-findings.md` finding 1;
|
||||||
|
`docs/research/poc-largeblob-findings.md` findings A1/A5/A6;
|
||||||
|
`docs/research/poc-pglo-findings.md` (C1-C7 — the handle shape, the
|
||||||
|
512 KiB knee, windowed gets)
|
||||||
|
- [ops-surface.md](../ops-surface.md) (the pin-token contract §4
|
||||||
|
extends); [store-api.md](../store-api.md) (stage hygiene, sweep
|
||||||
|
semantics, cursor-adjacent facade shapes); [requirements.md](../requirements.md)
|
||||||
|
(REQ-2, the fleet requirement §2/§3 serve)
|
||||||
@@ -0,0 +1,225 @@
|
|||||||
|
# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large`
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Accepted
|
||||||
|
|
||||||
|
## Context
|
||||||
|
|
||||||
|
The engine story (ADR-007/008/009) grew the physical layer faster than
|
||||||
|
its vocabulary grew, and the pre-decomposition review found the drift
|
||||||
|
is now self-contradictory in the anchor document itself:
|
||||||
|
|
||||||
|
1. **Node vs store instance.** requirements.md defines **Node** as
|
||||||
|
"one running store instance" — then REQ-2 states "a replicator node
|
||||||
|
may run multiple store instances (a fleet)" and **Fleet** as "two
|
||||||
|
or more nodes over one shared pool". Under the written definition,
|
||||||
|
one process hosting two instances is simultaneously one node (by
|
||||||
|
definition) and two nodes (by REQ-2), and the GC arbitration
|
||||||
|
domain ("node is the GC arbitration domain") is undefined for the
|
||||||
|
exact topology ADR-008 exists for. ADR-010 fixes pin ownership;
|
||||||
|
the pin's *owner id* and the node/instance distinction it needs
|
||||||
|
are still unpinned vocabulary.
|
||||||
|
2. **The "fs" tier name is now wrong.** The tier was named when its
|
||||||
|
only engine was a filesystem (ADR-003's `local` layout). After
|
||||||
|
ADR-009, half its engines are postgres Large Objects — the name
|
||||||
|
describes one engine's medium, not the tier. Worse, "fs" collides
|
||||||
|
with the dispatch language: "large content goes to fs" never said
|
||||||
|
*why* — the tier's actual routing criterion is content length
|
||||||
|
(ADR-003's size-threshold dispatch). The docs already speak the
|
||||||
|
true name informally: store-api.md says "small tier"/"large tier".
|
||||||
|
3. **mem's classification drifts.** backends-and-dispatch.md calls it
|
||||||
|
a "backend" in one place and a "testing/utility tier" in another;
|
||||||
|
ADR-003 lists it under "shipped backends … plus a testing tier";
|
||||||
|
ADR-004 calls it an ephemeral tier and carves it out of the
|
||||||
|
two-backend count. In truth it is neither a tier nor a separate
|
||||||
|
backend: it implements exactly the kv tier's contract (including
|
||||||
|
`size`, trivially), it is the contract-reference engine every
|
||||||
|
other engine's conformance tests mirror, and "engine" names what
|
||||||
|
it is (ADR-007 established that an engine is a concrete
|
||||||
|
implementation of one tier's contract — requirements.md already
|
||||||
|
defines the word this way).
|
||||||
|
4. **Tier and Backend were never formally defined** even though
|
||||||
|
ADR-004's scope note ("the 'two' here counts tiers, not engines")
|
||||||
|
leans on both words. Also undecided: whether the feature names
|
||||||
|
follow the tier rename now or after code exists — the last free
|
||||||
|
moment is now (zero code; after implementation the rename breaks
|
||||||
|
crates.io/feature ergonomics for every consumer).
|
||||||
|
|
||||||
|
Deciding facts are in hand: REQ-2's fleet fact, the engine set's
|
||||||
|
final shape (ADR-007/009), the dispatch mechanism (ADR-003), and
|
||||||
|
ADR-010's GC-state-home table. This is a vocabulary-and-boundary
|
||||||
|
decision, not an architecture change: ADR-004's tier≠engine note
|
||||||
|
already established the shape; this ADR gives it final, consistent
|
||||||
|
names.
|
||||||
|
|
||||||
|
Candidates weighed for the tier rename:
|
||||||
|
|
||||||
|
- **`lo`/`los`** — names the *postgres* engine (Large Objects), so
|
||||||
|
"the lo tier's `local` engine" would be backwards. Rejected.
|
||||||
|
- **`lfs`** — workable, echoes git-lfs specifically; but the tier
|
||||||
|
also serves packfiles and any oversized workspace artifact, and the
|
||||||
|
echo binds the crate's naming to one consumer's protocol shape.
|
||||||
|
Rejected.
|
||||||
|
- **`large`** — the dispatch-true name (size-threshold routing), the
|
||||||
|
word the specs already use informally ("large tier"), medium-agnostic.
|
||||||
|
**Chosen.** For symmetry in prose, the kv tier stays `kv` (its
|
||||||
|
routing criterion — "small" — lives in the prose as "small blobs"
|
||||||
|
and needs no rename: `kv` names the mechanism class, and `small`
|
||||||
|
as a feature/tier name would read as an adjective; the pair's
|
||||||
|
asymmetry is cosmetic, the *load-bearing* correction is that
|
||||||
|
neither tier is named after a medium).
|
||||||
|
|
||||||
|
Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it
|
||||||
|
names the engine's mechanism, accurate); `postgres` keeps its name
|
||||||
|
(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep
|
||||||
|
their names.
|
||||||
|
|
||||||
|
## Decision
|
||||||
|
|
||||||
|
### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy)
|
||||||
|
|
||||||
|
- **Tier** — one *contract* (a `Backend` trait implementation
|
||||||
|
boundary) serving one routing class of the pool. The crate ships
|
||||||
|
exactly two tiers: **kv** (small content) and **large** (over
|
||||||
|
the size-threshold content). Tier count (2) is ADR-004's count.
|
||||||
|
- **Engine** — a concrete storage implementation of one tier's
|
||||||
|
contract (the existing definition, unchanged). Engine count (5) is
|
||||||
|
this ADR's count.
|
||||||
|
- **Backend** — *the* contract (the `Backend` trait) plus its shipped
|
||||||
|
engine set, i.e. the tier as shipped. "Backend" and "tier" name the
|
||||||
|
same thing at different altitudes; the crate says "tier" for
|
||||||
|
counting, "the `Backend` trait" for the contract. "Two backends"
|
||||||
|
reads forever as "two tiers".
|
||||||
|
- **Store instance** — one running instantiation of the store facade
|
||||||
|
(one engine set — one or two engines per the constructor mode, §4 —
|
||||||
|
plus dispatch and GC state). **The GC
|
||||||
|
arbitration domain** for single-instance topologies; the pin-row
|
||||||
|
owner id under fleet engines (ADR-010's `owner`; this corrects
|
||||||
|
ADR-008's "node id" wording, which predates the instance/node
|
||||||
|
split — a store instance is what owns pins).
|
||||||
|
- **Node** — one running *process* (an embedder's binary). A node
|
||||||
|
hosts **one or more store instances**. Node is the process/ops unit:
|
||||||
|
alkcall connections, ACL grants, sweep cadence, and deployment
|
||||||
|
identity attach to nodes; GC arbitration and pin ownership attach
|
||||||
|
to store instances.
|
||||||
|
- **Fleet** — **two or more store instances over one shared pool**
|
||||||
|
(via a shared SQL engine). Deliberately not defined by node count:
|
||||||
|
two instances in one process over one postgres pool are a fleet
|
||||||
|
under this definition (and REQ-2's phrasing "a replicator node may
|
||||||
|
run multiple store instances (a fleet)" reads true as written);
|
||||||
|
two processes on two sqlite files are two pools, never a fleet.
|
||||||
|
- **Deployment** — unchanged (one operator's whole system).
|
||||||
|
|
||||||
|
### 2. mem joins the kv tier as its third engine
|
||||||
|
|
||||||
|
`mem` is the **kv tier's ephemeral engine** (feature `mem`,
|
||||||
|
default-off): a `BTreeMap`-shaped implementation of the kv contract,
|
||||||
|
including `size` (trivially) and list; never fleet-valid (ADR-010 §3's
|
||||||
|
rule and this ADR §4's constructor table); the **contract-reference
|
||||||
|
engine** — every other engine's
|
||||||
|
conformance tests mirror its suite. It is not a tier, not a third
|
||||||
|
backend, and the ADR-004 count ("two") survives untouched because
|
||||||
|
tier and engine were never the same count — this ADR is that
|
||||||
|
distinction's final form.
|
||||||
|
|
||||||
|
### 3. The fs tier is renamed `large`
|
||||||
|
|
||||||
|
In every document, status line, feature name, and constructor
|
||||||
|
parameter: tier `fs` → tier `large`; feature `fs` → feature `large`
|
||||||
|
(default-on, as `fs` was); dispatch prose says "large tier"/"small
|
||||||
|
tier" as store-api.md already does; the `local` engine's name is
|
||||||
|
kept (its meaning — this node's own filesystem — stays the
|
||||||
|
constructor's meaning even under the shared-media fleet posture,
|
||||||
|
where it is explicitly *not* only-node-local; the constructor's
|
||||||
|
explicit shared-media assertion, ADR-008 §3, is what disambiguates).
|
||||||
|
|
||||||
|
### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here)
|
||||||
|
|
||||||
|
| Tier | Engine | Feature | Default | Fleet-valid |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
|
||||||
|
| kv | postgres | `postgres` | no | **yes** |
|
||||||
|
| kv | mem | `mem` | no | no |
|
||||||
|
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
|
||||||
|
| large | pg-lo | `pg-lo` | no | **yes** |
|
||||||
|
|
||||||
|
Constructor modes (dispatch shapes) — pinned here (implied by
|
||||||
|
ADR-008 §3's re-routing bullet and the mem tests, but never named):
|
||||||
|
- **dual-tier** (the default): kv + large, size-threshold dispatch
|
||||||
|
between them (ADR-003).
|
||||||
|
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
|
||||||
|
constructor mode with no large tier; over-threshold puts are a
|
||||||
|
constructor-time error (the client-node re-routing posture handles
|
||||||
|
them via the ops surface instead — its composition, not a store
|
||||||
|
mode).
|
||||||
|
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
|
||||||
|
no dispatch threshold in effect (everything is local and small);
|
||||||
|
a testing posture, explicitly not a production story.
|
||||||
|
- Single-tier SQL modes (postgres-only, pg-lo-only) are **not**
|
||||||
|
constructor modes this crate ships: the two-tier shape is
|
||||||
|
ADR-004's requirement (both tiers are load-bearing by measured
|
||||||
|
economics); "postgres-only" as in *one SQL instance serving both
|
||||||
|
tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
|
||||||
|
consolidation), which is the dual-tier mode over one engine.
|
||||||
|
|
||||||
|
### 5. Deferral-policy wording (README alignment)
|
||||||
|
|
||||||
|
The deferral policy's categories are: `deferred(scope)`,
|
||||||
|
externally-owned questions, and decided-but-sequenced work — three
|
||||||
|
names, defined in open-questions.md's header. README.md's recap says
|
||||||
|
"two parking kinds" and is corrected to name all three (the third is
|
||||||
|
not a *parking* kind — it is work, not a parked question — hence
|
||||||
|
this ADR's wording here rather than a silent edit).
|
||||||
|
|
||||||
|
## Consequences
|
||||||
|
|
||||||
|
**Positive**
|
||||||
|
|
||||||
|
- The anchor document's self-contradiction (node/fleet) becomes a
|
||||||
|
three-level vocabulary (store instance / node / deployment) whose
|
||||||
|
levels map one-to-one onto the mechanisms (GC domain / process-ops
|
||||||
|
unit / operator whole).
|
||||||
|
- The fs→large rename removes the last medium-named tier; feature
|
||||||
|
names, tier names, and dispatch prose now agree, before any code
|
||||||
|
exists to break.
|
||||||
|
- mem's classification is stated once, as the contract-reference kv
|
||||||
|
engine; the "plus a testing tier" carve-outs dissolve.
|
||||||
|
- The constructor table gives task decomposition a closed set: 5
|
||||||
|
engine implementations, 3 constructor modes, 1 dispatch policy.
|
||||||
|
|
||||||
|
**Negative**
|
||||||
|
|
||||||
|
- Vocabulary churn in every doc that says "fs" (the rename is
|
||||||
|
mechanical but touches all specs and ADR-003/004/007/008/009 status
|
||||||
|
lines and reference sections). Zero code exists — this is the last
|
||||||
|
free moment; after Phase 1 starts, feature renames break consumers.
|
||||||
|
- "Backend"/"tier" as interchangeable words risks reader confusion
|
||||||
|
with ADR-003's "Backend trait" — mitigated by the rule that the
|
||||||
|
*trait* is always named with "trait" ("the `Backend` trait"), never
|
||||||
|
bare "the backend".
|
||||||
|
|
||||||
|
**Neutral**
|
||||||
|
|
||||||
|
- No engine's contract, feature flag semantics, or constructor
|
||||||
|
parameter changes; `mem` gains the word "engine" and loses the
|
||||||
|
word "backend".
|
||||||
|
- REQ-2's meaning is unchanged (the fleet definition is re-derived
|
||||||
|
from the same fact, not re-fact'd).
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- ADR-003 (the trait contract; its "shipped backends" list is now
|
||||||
|
read through this ADR's vocabulary), ADR-004 (tier≠engine — the
|
||||||
|
scope note this ADR gives final names), ADR-007 (kv engines;
|
||||||
|
constructor selection), ADR-008 (the second tier's engine candidate
|
||||||
|
and the fleet topology; the vocabulary it assumed is now pinned),
|
||||||
|
ADR-009 (pg-lo admission; the consolidation option whose
|
||||||
|
constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
|
||||||
|
(the I/O seams and GC-state home this vocabulary must fit)
|
||||||
|
- [requirements.md](../requirements.md) (the canonical vocabulary
|
||||||
|
copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
|
||||||
|
(the tier/engine table's spec home)
|
||||||
|
- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the
|
||||||
|
reference-impl role mem already played in the POCs);
|
||||||
|
`poc-pglo-findings.md` (the gate the engine suite mirrors)
|
||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03
|
last_updated: 2026-10-03 (ADR-010 state-home rule + ADR-011 instance/fleet vocabulary)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Pooling, namespaces, and GC
|
# Pooling, namespaces, and GC
|
||||||
@@ -16,17 +16,23 @@ price, and this document specifies the mechanism the POCs validated
|
|||||||
|
|
||||||
## The pooled CAS
|
## The pooled CAS
|
||||||
|
|
||||||
- **One pool per deployment; a node is one store instance** (the
|
- **One pool per deployment; a store instance is one GC arbitration
|
||||||
engine/vocabulary definitions live in requirements.md). Repos,
|
domain** (the engine/vocabulary definitions live in requirements.md
|
||||||
workspaces, and appfile stores are all
|
/ ADR-011: a node — one process — may host several instances; the
|
||||||
|
instance is where pins and arbitration live for single-instance
|
||||||
|
topologies). Repos, workspaces, and appfile stores are all
|
||||||
*sets of hash references* over the same CAS — never walled-off
|
*sets of hash references* over the same CAS — never walled-off
|
||||||
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
|
per-repo stores. Git's `alternates`/object-pool mechanism (GitLab
|
||||||
object pools) is the same idea done awkwardly; the pool generalizes it
|
object pools) is the same idea done awkwardly; the pool generalizes it
|
||||||
to unrelated repos and to p2p replication. **A fleet (REQ-2) — two or
|
to unrelated repos and to p2p replication. **A fleet (REQ-2) — two
|
||||||
more nodes sharing one pool — is the one topology where the GC
|
or more store instances sharing one pool — is the one topology where
|
||||||
mechanism below extends beyond a node: ADR-008 pins, sweeper
|
the GC mechanism below extends beyond one instance: ADR-008 pins,
|
||||||
locking, and delete arbitration move into the shared engine;
|
sweeper locking, and delete arbitration move into the shared engine;
|
||||||
the invariants are unchanged.**
|
the invariants are unchanged. Where this crate's GC state (fleet
|
||||||
|
rows and locks) physically lives, and who holds the joint
|
||||||
|
entry+pin transaction, is ruled by ADR-010 §2: the store core owns
|
||||||
|
all GC state; SQL-backed engines host it through a crate-internal,
|
||||||
|
contract-tested seam; backends never learn liveness.**
|
||||||
- **One address space.** Under the canonical hash (ADR-002), the same
|
- **One address space.** Under the canonical hash (ADR-002), the same
|
||||||
file content has the same address whether it entered via a git oid or
|
file content has the same address whether it entered via a git oid or
|
||||||
a workspace manifest — cross-consumer dedup is free by construction,
|
a workspace manifest — cross-consumer dedup is free by construction,
|
||||||
@@ -46,7 +52,8 @@ price, and this document specifies the mechanism the POCs validated
|
|||||||
`register_liveness_source`; the POC's copy-semantics failure finding 7
|
`register_liveness_source`; the POC's copy-semantics failure finding 7
|
||||||
is the pinning reason). Namespace tables may also exist purely
|
is the pinning reason). Namespace tables may also exist purely
|
||||||
in-memory for ephemeral consumers; nothing in the store persists
|
in-memory for ephemeral consumers; nothing in the store persists
|
||||||
namespaces — where roots live when a deployment runs fs-only is a
|
namespaces — where roots live when a deployment has no SQL engine (no
|
||||||
|
kv engine with rows) is a
|
||||||
non-question by design: roots live with the consumer, whose references
|
non-question by design: roots live with the consumer, whose references
|
||||||
are as durable as they need to be.
|
are as durable as they need to be.
|
||||||
- ACL mapping for network ops: the namespace is the resource
|
- ACL mapping for network ops: the namespace is the resource
|
||||||
@@ -112,7 +119,7 @@ existence flips atomically per key).
|
|||||||
- **Accounting** (per-namespace sizes): computed from reference sets by
|
- **Accounting** (per-namespace sizes): computed from reference sets by
|
||||||
consumers; the store reports pool-level totals only (`list` +
|
consumers; the store reports pool-level totals only (`list` +
|
||||||
`stat`). Physical-layer accounting would re-introduce namespace
|
`stat`). Physical-layer accounting would re-introduce namespace
|
||||||
physics — rejected with the per-namespace backend config
|
physics — rejected with the per-namespace engine config
|
||||||
(ADR-003 §Consequences).
|
(ADR-003 §Consequences).
|
||||||
|
|
||||||
## Design Decisions
|
## Design Decisions
|
||||||
@@ -122,7 +129,8 @@ existence flips atomically per key).
|
|||||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | one pool, registered liveness, mark-and-sweep, delete windows, no ambient scheduling |
|
||||||
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
|
| [003](decisions/003-backend-contract-and-dispatch.md) | list() contract | complete lists are the GC substrate |
|
||||||
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
|
| [004](decisions/004-two-backends-no-third.md) | Structure-blindness | manifests/refs never sink into the store |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet GC extension | DB-backed pins, advisory-locked single sweeper, SQL delete arbitration — for shared-pool (fleet) topologies only; the in-process protocol is unchanged elsewhere |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | GC-state home | the store core owns all GC state; SQL engines host it via the contract-tested engine-state seam; pin-tx ownership ruled |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
|
|||||||
@@ -17,7 +17,7 @@ residual).
|
|||||||
|
|
||||||
## The hash family
|
## The hash family
|
||||||
|
|
||||||
Three tiers (ADR-002 §Decision; resolved from OQ-BL-03 after the
|
Three hash-policy classes (ADR-002 §Decision; resolved from OQ-BL-03 after the
|
||||||
"multi-hash" premise dissolved — nothing forces BLAKE3 once iroh-blobs'
|
"multi-hash" premise dissolved — nothing forces BLAKE3 once iroh-blobs'
|
||||||
inheritance is rejected):
|
inheritance is rejected):
|
||||||
|
|
||||||
|
|||||||
@@ -27,7 +27,8 @@ not one. Two kinds of parked question remain legitimate:
|
|||||||
no decision here; it would be Schrödinger's code if it gated one).
|
no decision here; it would be Schrödinger's code if it gated one).
|
||||||
|
|
||||||
Invalid for either kind: a blocker that is a decision this crate must
|
Invalid for either kind: a blocker that is a decision this crate must
|
||||||
make before shipping. Phase 0's residual list was swept under this
|
make before shipping. (A third category — **decided-but-sequenced
|
||||||
|
work** — is not a parked kind at all; see below.) Phase 0's residual list was swept under this
|
||||||
rule: every residual either resolved into an ADR (the evidence was
|
rule: every residual either resolved into an ADR (the evidence was
|
||||||
already in hand) or re-owned as below.
|
already in hand) or re-owned as below.
|
||||||
|
|
||||||
@@ -38,7 +39,7 @@ OQ-08). Deferring on a consumer's *deployment of this crate* is
|
|||||||
invalid — that fact is created by this crate (OQ-10's original
|
invalid — that fact is created by this crate (OQ-10's original
|
||||||
framing, OQ-09's original framing). And a third thing is neither:
|
framing, OQ-09's original framing). And a third thing is neither:
|
||||||
**decided-but-sequenced work**. Once a question resolves to a decision
|
**decided-but-sequenced work**. Once a question resolves to a decision
|
||||||
(the pg-lo engine is the named fs-tier candidate under REQ-2/ADR-008),
|
(the pg-lo engine is the named large-tier candidate under REQ-2/ADR-008),
|
||||||
what remains is the evidence pipeline the decision's own admission
|
what remains is the evidence pipeline the decision's own admission
|
||||||
gate requires (POC #7) — a task with an owner, not a parked question,
|
gate requires (POC #7) — a task with an owner, not a parked question,
|
||||||
and not a hedge. A decision is not "unmade" because its verification
|
and not a hedge. A decision is not "unmade" because its verification
|
||||||
@@ -246,12 +247,12 @@ a deployment trigger — the same invalid shape, caught at review: the
|
|||||||
trigger was a fact only this crate could create. It resolved to
|
trigger was a fact only this crate could create. It resolved to
|
||||||
ADR-007 (two kv engines; see its entry above).
|
ADR-007 (two kv engines; see its entry above).
|
||||||
|
|
||||||
The pg-lo fs-engine question (raised in the ADR-008 review round,
|
The pg-lo large-engine question (raised in the ADR-008 review round,
|
||||||
2026-10-03) never became a parked question: REQ-2 made the
|
2026-10-03) never became a parked question: REQ-2 made the
|
||||||
multi-instance-with-large-blobs topology a standing requirement
|
multi-instance-with-large-blobs topology a standing requirement
|
||||||
(requirements.md — a planning fact predating all POCs), so there was
|
(requirements.md — a planning fact predating all POCs), so there was
|
||||||
nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the
|
nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the
|
||||||
candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
|
candidate large-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
|
||||||
as its admission evidence — sequenced work, per the deferral-policy
|
as its admission evidence — sequenced work, per the deferral-policy
|
||||||
header's third category. That work completed the same day: POC #7
|
header's third category. That work completed the same day: POC #7
|
||||||
passed (`docs/research/poc-pglo-findings.md`, gate verdict passed
|
passed (`docs/research/poc-pglo-findings.md`, gate verdict passed
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03
|
last_updated: 2026-10-03 (pin-token renewal contract pinned per ADR-010 §4)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Ops surface (alkcall-backed network operations)
|
# Ops surface (alkcall-backed network operations)
|
||||||
@@ -96,8 +96,11 @@ the server. The hand-over contract:
|
|||||||
|
|
||||||
1. the put lands pinned (ADR-005) — the entry cannot be swept while
|
1. the put lands pinned (ADR-005) — the entry cannot be swept while
|
||||||
the handler holds the pin;
|
the handler holds the pin;
|
||||||
2. the `blobs/put` response returns a **pin token** (opaque handle);
|
2. the `blobs/put` response returns a **pin token** (opaque handle —
|
||||||
the putter can later confirm its registration landed by observing
|
minted by the ops layer over the pin row the put committed; the
|
||||||
|
token↔(key, owner) mapping is ops-layer state, not durable state —
|
||||||
|
the durable state is the pin row itself, ADR-010 §4); the putter
|
||||||
|
can later confirm its registration landed by observing
|
||||||
the digest present via `blobs/have`;
|
the digest present via `blobs/have`;
|
||||||
3. the embedder's registered liveness source is the *actual* root of
|
3. the embedder's registered liveness source is the *actual* root of
|
||||||
record: the contract on the embedder (documented on the ops module)
|
record: the contract on the embedder (documented on the ops module)
|
||||||
@@ -112,6 +115,32 @@ the server. The hand-over contract:
|
|||||||
conversion on the server side (token → registered liveness source)
|
conversion on the server side (token → registered liveness source)
|
||||||
is the mechanism the embedder plugs its registry into.
|
is the mechanism the embedder plugs its registry into.
|
||||||
|
|
||||||
|
**Token renewal (ADR-010 §4 — the fleet-TTL race, resolved): the
|
||||||
|
token *is* the pin — there is no second pin state.** The server-side
|
||||||
|
handler's in-process `Pin` guard drops when the `blobs/put` response
|
||||||
|
completes (if it never dropped, the TTL machinery would be dead on
|
||||||
|
the ops path — the pin row's expiry, not the guard's presence, is the
|
||||||
|
fleet liveness signal). Under fleet
|
||||||
|
engines the put's pin is an expiry-carrying row (ADR-008: TTL 60 s
|
||||||
|
default, renewed every TTL/3). The remote putter holding a token is
|
||||||
|
the row's owner and renews by the ops it already speaks: a
|
||||||
|
`blobs/have` probe **presenting the token alongside the digest** (the
|
||||||
|
handler verifies the token maps to the digest's pin row and calls the
|
||||||
|
crate-internal `pin_renew` via the ops module's store-core access —
|
||||||
|
the ops module is in-crate and crosses the seam by construction, the
|
||||||
|
same way the put handler holds the `Pin` guard; `has` itself remains
|
||||||
|
a pure existence probe per ADR-008 — renewal is a token-authenticated
|
||||||
|
act on the pin row, not a `has` semantic), or a re-put of the same
|
||||||
|
digest (CAS dedup renews the row). **A putter may pause past the
|
||||||
|
TTL/3 mark (20 s at the default TTL) only if it renews before expiry;
|
||||||
|
a putter pausing past TTL lets the row expire** — the sweeper reaps
|
||||||
|
it (ADR-008) and delete-then-recover (ADR-005) is the backstop: the
|
||||||
|
putter re-offers the digest; CAS makes the re-put idempotent. The
|
||||||
|
race resolves to a re-put, never to silent loss. For non-fleet
|
||||||
|
(`local`-engine) serving nodes the token is the in-process guard —
|
||||||
|
nothing expires; the process-crash case is ADR-005's in-flight-put
|
||||||
|
loss with the same backstop.
|
||||||
|
|
||||||
Placement of the op *registration* (which embedders wire where they
|
Placement of the op *registration* (which embedders wire where they
|
||||||
want them exposed) matches the alkgit ops pattern (its
|
want them exposed) matches the alkgit ops pattern (its
|
||||||
`git/repo/*` call ops): the module exports spec/handler pairs; the
|
`git/repo/*` call ops): the module exports spec/handler pairs; the
|
||||||
@@ -143,10 +172,11 @@ handlers call `put`; both honor pins and delete windows. There is no
|
|||||||
ops-internal path around the store facade (ADR-001 §Consequences —
|
ops-internal path around the store facade (ADR-001 §Consequences —
|
||||||
this is what makes "re-homing the ops surface later loses nothing"
|
this is what makes "re-homing the ops surface later loses nothing"
|
||||||
true). One fleet topology additionally rides this module: **re-routing
|
true). One fleet topology additionally rides this module: **re-routing
|
||||||
(ADR-008)** — a fleet whose nodes do not share fs media routes
|
(ADR-008)** — a fleet whose nodes do not share large-tier media routes
|
||||||
large-blob access through a single storage node via these ops, with
|
large-blob access through a single storage node via these ops, with
|
||||||
client nodes' fs tiers serving only node-private content. The module
|
client nodes' constructors in kv-only mode (ADR-011 §4) for pool
|
||||||
supports this shape without change; no fleet-specific op exists.
|
content. The module supports this shape without change; no
|
||||||
|
fleet-specific op exists.
|
||||||
|
|
||||||
## Design Decisions
|
## Design Decisions
|
||||||
|
|
||||||
@@ -154,7 +184,9 @@ supports this shape without change; no fleet-specific op exists.
|
|||||||
|---|---|---|
|
|---|---|---|
|
||||||
| [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale |
|
| [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & placement | alkcall-backed, feature-gated ops module; conformance-not-deviation rationale |
|
||||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees |
|
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | verified-fetch via carried digests; no chunk trees |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Fleet re-routing topology | non-shared-media fleets route large-blob access through a storage node via these ops; no new ops |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | Pin-token renewal | the token is the pin; renewal rides have/re-put; TTL expiry falls to delete-then-recover |
|
||||||
|
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + modes | instance/node/fleet split; kv-only mode named (the re-routing posture's client shape) |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03
|
last_updated: 2026-10-03 (ADR-010/011 — I/O seams pinned; vocabulary fs→large, mem joins kv)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Overview
|
# Overview
|
||||||
@@ -9,10 +9,11 @@ last_updated: 2026-10-03
|
|||||||
|
|
||||||
`alkblobs` is content-addressed blob storage in the alk* family: a
|
`alkblobs` is content-addressed blob storage in the alk* family: a
|
||||||
put/get/verify store keyed under one canonical content hash (the
|
put/get/verify store keyed under one canonical content hash (the
|
||||||
git-family derivation; legacy SHA-1-tolerant), with pluggable backends
|
git-family derivation; legacy SHA-1-tolerant), with pluggable tier
|
||||||
(small blobs in a kv store, large blobs on a filesystem fallback),
|
engines (small blobs in a kv store per ADR-007/011, over-threshold
|
||||||
split from any wire/ops-protocol layer. Network ops ride the alkcall
|
blobs on the `large` tier per ADR-008/009 — the tier formerly named
|
||||||
substrate behind a feature-gated `ops` module (ADR-001).
|
`fs`), split from any wire/ops-protocol layer. Network ops ride the
|
||||||
|
alkcall substrate behind a feature-gated `ops` module (ADR-001).
|
||||||
|
|
||||||
Deployment-relevant consumer facts — what each use case forces on this
|
Deployment-relevant consumer facts — what each use case forces on this
|
||||||
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
|
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
|
||||||
@@ -63,15 +64,22 @@ or three times.
|
|||||||
│ store core (always on) │
|
│ store core (always on) │
|
||||||
│ hashing & keys (ADR-002) · put/get/stat/range/pin seams │
|
│ hashing & keys (ADR-002) · put/get/stat/range/pin seams │
|
||||||
│ mark-and-sweep GC + liveness registration (ADR-005) │
|
│ mark-and-sweep GC + liveness registration (ADR-005) │
|
||||||
│ size-threshold dispatch (ADR-003) │
|
│ size-threshold dispatch (ADR-003); GC state owned │
|
||||||
|
│ here, hosted by SQL engines via the seam (ADR-010) │
|
||||||
└───────┬───────────────────┬───────────────────────────────┘
|
└───────┬───────────────────┬───────────────────────────────┘
|
||||||
▼ ▼
|
▼ ▼
|
||||||
│ kv backend fs backend (ADR-003: exactly two tiers;
|
┌──────────────────────┐ ┌─────────────────────────────────┐
|
||||||
│ │ ADR-007: kv = 2 engines;
|
│ kv tier (ADR-003/ │ │ large tier (ADR-003/008/011; │
|
||||||
│ ▼ ADR-008: fs engines too)
|
│ 007/011) │ │ formerly "fs") │
|
||||||
│ engine per node: sqlite (default) | postgres (feature "postgres")
|
│ sqlite (default) │ │ local (default) │
|
||||||
│ fs engine per node: local (default) | pg-lo (feature "pg-lo",
|
│ postgres ("postgres") │ pg-lo ("pg-lo"; ADR-009) │
|
||||||
│ default-off; ADR-009)
|
│ mem ("mem"; tests + │ │
|
||||||
|
│ contract reference) │ │
|
||||||
|
└──────────────────────┘ └─────────────────────────────────┘
|
||||||
|
▲ ▲
|
||||||
|
└── one Backend trait (ADR-003/010: read cursor +
|
||||||
|
staged put I/O seams); engine = per-node
|
||||||
|
constructor choice (ADR-011's table)
|
||||||
|
|
||||||
transport: none, anywhere. alkgit/alkfs or the embedder dials;
|
transport: none, anywhere. alkgit/alkfs or the embedder dials;
|
||||||
an alkcall Connection is handed in at the ops boundary.
|
an alkcall Connection is handed in at the ops boundary.
|
||||||
@@ -81,36 +89,37 @@ or three times.
|
|||||||
|
|
||||||
| Layer | Dependencies | Notes |
|
| Layer | Dependencies | Notes |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
|
| base crate (store core + tier engines) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `large`/`mem` | no alkcall, no serialization frameworks |
|
||||||
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
| `postgres` engine feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||||||
| `pg-lo` feature (ADR-009) | Large Objects as the fs tier's second engine, default-off | rides the same `tokio-postgres`/`deadpool-postgres` stack as the `postgres` kv engine (SQL `lo_*` functions, no new dependency); same admission/test-gate shape |
|
| `pg-lo` engine feature (ADR-009) | Large Objects as the large tier's second engine, default-off | pulls the same `tokio-postgres`/`deadpool-postgres` cluster as the `postgres` kv engine (SQL `lo_*` functions, no additional dependency); **does not imply the `postgres` feature** — engine features pull the driver cluster independently (ADR-010 §1), so a `large`=pg-lo + kv=sqlite node is coherent and supported; same admission/test-gate shape |
|
||||||
|
| `mem` engine feature (ADR-011) | the kv tier's ephemeral engine, default-off | `BTreeMap`-shaped; the contract-reference engine; no dependencies |
|
||||||
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
|
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
|
||||||
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
|
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
|
||||||
|
|
||||||
Backends are feature-gated; both `kv` and `fs` ship default-on
|
Tiers are feature-gated; both default-on (batteries-included, the
|
||||||
(batteries-included, the alkgit feature-model pattern — default-on
|
alkgit feature-model pattern — default-on features, opt-out rather
|
||||||
features, opt-out rather than opt-in); the `kv` tier's postgres engine
|
than opt-in); the engine features `postgres` and `pg-lo`, and `mem`,
|
||||||
(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only
|
are default-off, as are `ops`. A pg-only kv node (postgres engine
|
||||||
kv node (postgres engine without rusqlite) is **not** a supported
|
without rusqlite) is **not** a supported configuration this crate
|
||||||
configuration this crate optimizes for — feature-graph work for it is
|
optimizes for — feature-graph work for it is deferred until a named
|
||||||
deferred until a named requirement exists (the base `kv` feature
|
requirement exists (the base `kv` feature pulling rusqlite is the
|
||||||
pulling rusqlite is the accepted v1 shape; convention 8's
|
accepted v1 shape; convention 8's lean-base principle applies to
|
||||||
lean-base principle applies to *default* builds, which stay lean since
|
*default* builds, which stay lean since `kv` is what the feature model
|
||||||
`kv` is what the feature model controls, not what default-on means
|
controls, not what default-on means downstream). Wasm: the tier
|
||||||
downstream). Wasm: the backends are not a wasm story; a wasm client is
|
engines are not a wasm story; a wasm client is a consumer of the `ops`
|
||||||
a consumer of the `ops` surface (which rides alkcall, itself
|
surface (which rides alkcall, itself wasm-clean), never an in-process
|
||||||
wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
embedder (ADR-001 §Consequences).
|
||||||
|
|
||||||
## Non-goals (each enforced by an ADR)
|
## Non-goals (each enforced by an ADR)
|
||||||
|
|
||||||
- No wire framing, op dispatch, ACL, or transport inside the store layer
|
- No wire framing, op dispatch, ACL, or transport inside the store layer
|
||||||
(ADR-001).
|
(ADR-001).
|
||||||
- No adopted iroh-blobs wire surface — no tickets, no postcard, no
|
- **No adopted iroh-blobs wire surface — no tickets, no postcard, no
|
||||||
provider protocol (AGENTS convention 7; ADR-001 §Context).
|
provider protocol (AGENTS convention 7; ADR-001 §Context).**
|
||||||
- No namespaces-as-physical-storage; the pool is flat, namespaces are
|
- No namespaces-as-physical-storage; the pool is flat, namespaces are
|
||||||
consumer-level reference sets (ADR-005).
|
consumer-level reference sets (ADR-005).
|
||||||
- No manifest/path-tree/vfs tables; consumers never implement a Backend
|
- No manifest/path-tree/vfs tables; consumers never implement a
|
||||||
to model paths (ADR-004).
|
Backend to model paths (ADR-004).
|
||||||
- No chunk-tree/CDC storage encoding; whole-file CAS is the store's only
|
- No chunk-tree/CDC storage encoding; whole-file CAS is the store's only
|
||||||
granularity (ADR-006).
|
granularity (ADR-006).
|
||||||
- No per-repo or per-namespace walled stores — one pooled CAS
|
- No per-repo or per-namespace walled stores — one pooled CAS
|
||||||
@@ -126,9 +135,11 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
|||||||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side |
|
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side |
|
||||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
|
||||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
|
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
|
||||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
|
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + large tier per node |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; fleet pins/sweeps engine-backed; the large tier (now `large`) engine-selectable |
|
||||||
| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo fs engine | postgres Large Objects as the fs tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
|
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo large engine | postgres Large Objects as the large tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; kv fleet-validity rule; pin-token renewal |
|
||||||
|
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + rename | tier/engine/instance/node/fleet pinned; mem joins the kv tier; large tier renamed `large`; constructor modes |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
@@ -144,7 +155,7 @@ crate level:
|
|||||||
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
|
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
|
||||||
shipped behind one trait; per-node constructor choice)
|
shipped behind one trait; per-node constructor choice)
|
||||||
- **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects
|
- **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects
|
||||||
as the fs tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
|
as the large tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
|
||||||
a shipped engine by ADR-009 (REQ-2's consolidation option)
|
a shipped engine by ADR-009 (REQ-2's consolidation option)
|
||||||
|
|
||||||
## References
|
## References
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03 (pg-lo admitted by ADR-009 on POC #7 — REQ-2's consolidation option is shipped)
|
last_updated: 2026-10-03 (vocabulary pinned by ADR-011 — tiers/engines/instances/fleet; large tier renamed `large`)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Requirements
|
# Requirements
|
||||||
@@ -12,24 +12,44 @@ conversation or vision prose. **This document is the anchor: when a
|
|||||||
consumer fact changes, it changes here first, with a dated note, and
|
consumer fact changes, it changes here first, with a dated note, and
|
||||||
the ADRs re-derive from it.**
|
the ADRs re-derive from it.**
|
||||||
|
|
||||||
## Vocabulary (defined once; used everywhere)
|
## Vocabulary (defined once; used everywhere — ADR-011 is the decision behind this section)
|
||||||
|
|
||||||
|
- **Tier** — one contract (a `Backend` trait implementation boundary)
|
||||||
|
serving one routing class of the pool. Exactly two tiers: **kv**
|
||||||
|
(small content) and **large** (over the size-threshold content;
|
||||||
|
the tier formerly named `fs` — renamed in ADR-011 because the tier's
|
||||||
|
routing criterion is length, and half its engines are not
|
||||||
|
filesystems). "Backend" names the same thing at a different
|
||||||
|
altitude (the tier as shipped including its engine set); "two
|
||||||
|
backends" (ADR-004) reads forever as "two tiers".
|
||||||
- **Engine** — a concrete storage implementation of one tier's
|
- **Engine** — a concrete storage implementation of one tier's
|
||||||
contract: the kv tier's engines are sqlite and postgres (ADR-007);
|
contract. Five ship: the kv tier's `sqlite` (default),
|
||||||
the fs tier's engines are `local` (default) and `pg-lo` (ADR-009).
|
`postgres` (feature `postgres`), and `mem` (ephemeral; feature
|
||||||
|
`mem`; the contract-reference engine; ADR-003); the large tier's
|
||||||
|
`local` (default; ADR-003) and `pg-lo` (ADR-009).
|
||||||
- **Pool** — the flat content-addressed namespace of one deployment
|
- **Pool** — the flat content-addressed namespace of one deployment
|
||||||
(`hash → bytes` everywhere; ADR-005).
|
(`hash → bytes` everywhere; ADR-005).
|
||||||
- **Node** — one running store instance (one process embedder; its fs
|
- **Store instance** — one running instantiation of the store facade
|
||||||
tier is its own media). Node is the GC arbitration domain for
|
(one engine set — one or two engines per the constructor mode,
|
||||||
single-node topologies (ADR-005's pins, liveness locks, and delete
|
ADR-011 §4 — plus dispatch and GC state). The **GC arbitration
|
||||||
windows live inside a node); under a fleet, arbitration moves into
|
domain** for single-instance topologies (ADR-005's pins, liveness
|
||||||
the shared engine (ADR-008).
|
locks, and delete windows live inside one); the pin-row owner id
|
||||||
- **Fleet** — two or more nodes over **one shared pool** (e.g. via one
|
under fleet engines (ADR-008/010 — ADR-008 wrote "node id" before
|
||||||
postgres instance). A fleet is the only topology in which
|
this split existed; a store instance is what owns pins).
|
||||||
cross-node GC and fs-locality questions exist (ADR-008).
|
- **Node** — one running *process* (an embedder's binary). **A node
|
||||||
|
hosts one or more store instances** — this fixes the prior
|
||||||
|
definition's collapse (which REQ-2's phrasing contradicted). Node
|
||||||
|
is the process/ops unit: alkcall connections, ACL grants, sweep
|
||||||
|
cadence, and deployment identity attach to nodes.
|
||||||
|
- **Fleet** — **two or more store instances over one shared pool**
|
||||||
|
(via a shared SQL engine). Defined by pool-sharing, not node count:
|
||||||
|
two instances in one process over one postgres pool are a fleet;
|
||||||
|
two processes on two sqlite files are two pools, never a fleet. A
|
||||||
|
fleet is the only topology in which cross-instance GC and
|
||||||
|
large-tier locality questions exist (ADR-008).
|
||||||
- **Deployment** — one operator's whole system: typically one fleet
|
- **Deployment** — one operator's whole system: typically one fleet
|
||||||
(multi-node) or one node; consumer tables, op registrations, and
|
or one node; consumer tables, op registrations, and sweep cadence
|
||||||
sweep cadence are deployment choices.
|
are deployment choices.
|
||||||
|
|
||||||
## The use cases and their requirements
|
## The use cases and their requirements
|
||||||
|
|
||||||
@@ -37,7 +57,7 @@ the ADRs re-derive from it.**
|
|||||||
|
|
||||||
Git repos for a user/team on one machine. Small-object regime dominated
|
Git repos for a user/team on one machine. Small-object regime dominated
|
||||||
by loose objects in the kv tier; packs and large artifacts (git-lfs
|
by loose objects in the kv tier; packs and large artifacts (git-lfs
|
||||||
shape) on the fs tier.
|
shape) on the large tier.
|
||||||
|
|
||||||
- **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics:
|
- **REQ-1 (traces: ADR-007, POC #3 A4)** — single-machine economics:
|
||||||
sqlite is the default kv engine; single-writer WAL ceiling (~1.2k
|
sqlite is the default kv engine; single-writer WAL ceiling (~1.2k
|
||||||
@@ -59,9 +79,9 @@ replicators; ACL/ownership above (alkcall layer).
|
|||||||
discussion; it predates all POCs and architecture docs, recorded
|
discussion; it predates all POCs and architecture docs, recorded
|
||||||
here on the operator's authority as the use case's stated shape).
|
here on the operator's authority as the use case's stated shape).
|
||||||
Consequences are derived in ADR-008: fleet GC (DB-backed pins,
|
Consequences are derived in ADR-008: fleet GC (DB-backed pins,
|
||||||
advisory-locked sweeper, SQL delete arbitration), the fs-tier
|
advisory-locked sweeper, SQL delete arbitration), the large-tier
|
||||||
engine concept with its locality contract, and the pg-lo candidate
|
engine concept with its locality contract, and the pg-lo engine
|
||||||
engine (POC #7 — **passed** 2026-10-03, admitted as a shipped
|
(POC #7 — **passed** 2026-10-03, admitted as a shipped
|
||||||
engine by ADR-009: a fleet node can consolidate on postgres-only).
|
engine by ADR-009: a fleet node can consolidate on postgres-only).
|
||||||
- **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out:
|
- **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out:
|
||||||
the fleet's hub-class nodes need put throughput that *increases*
|
the fleet's hub-class nodes need put throughput that *increases*
|
||||||
@@ -79,7 +99,7 @@ path→hash manifests above.
|
|||||||
- **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space
|
- **REQ-4 (traces: ADR-005, ADR-002)** — pooling and one address space
|
||||||
across workspaces; dedup must be structural (not configured). Small
|
across workspaces; dedup must be structural (not configured). Small
|
||||||
edits mean the small-blob regime dominates: sqlite-class kv
|
edits mean the small-blob regime dominates: sqlite-class kv
|
||||||
performance is what matters; the fs tier is hit rarely (large
|
performance is what matters; the large tier is hit rarely (large
|
||||||
workspace artifacts).
|
workspace artifacts).
|
||||||
|
|
||||||
## What this document is not
|
## What this document is not
|
||||||
@@ -100,4 +120,4 @@ path→hash manifests above.
|
|||||||
| REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 |
|
| REQ-1 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #3 A4 | ADR-007 |
|
||||||
| REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 |
|
| REQ-2 | 2026-10-03 (fact: use case origin discussion, pre-POC) | user/planning record | ADR-007, ADR-008 |
|
||||||
| REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 |
|
| REQ-3 | 2026-10-03 (fact: phase-0 §deployment posture) | POC #5 B3/B4 | ADR-007 |
|
||||||
| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-005 |
|
| REQ-4 | 2026-10-03 (fact: phase-0 §vision) | POC #1, ADR-002 | ADR-002, ADR-005 |
|
||||||
@@ -1,19 +1,20 @@
|
|||||||
---
|
---
|
||||||
status: draft
|
status: draft
|
||||||
last_updated: 2026-10-03
|
last_updated: 2026-10-03 (ADR-010 I/O seams + ADR-011 rename — large tier → large)
|
||||||
---
|
---
|
||||||
|
|
||||||
# Store API
|
# Store API
|
||||||
|
|
||||||
## What this is
|
## What this is
|
||||||
|
|
||||||
The store core's public surface: the typed facade above the backends
|
The store core's public surface: the typed facade above the tier
|
||||||
that consumers (alkgit, alkfs, the ops module) program against. It owns
|
engines that consumers (alkgit, alkfs, the ops module) program against.
|
||||||
typed keys and hashing (ADR-002), size-threshold dispatch (ADR-003),
|
It owns typed keys and hashing (ADR-002), size-threshold dispatch
|
||||||
the put-pinning seam, and the liveness/sweep seam (ADR-005) — and
|
(ADR-003), the put-pinning seam, and the liveness/sweep seam (ADR-005)
|
||||||
nothing else: no paths, no manifests, no wire (ADR-004, ADR-001). The
|
— and nothing else: no paths, no manifests, no wire (ADR-004, ADR-001).
|
||||||
facade is **engine-blind** (ADR-007/008): which kv/fs engine a node
|
The facade is **engine-blind** (ADR-007/008/010): which engines a node
|
||||||
runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
runs changes nothing here except fleet-GC mechanism wiring (ADR-008's
|
||||||
|
mechanism lives in the engine-state seam, ADR-010 §2).
|
||||||
|
|
||||||
## Public surface
|
## Public surface
|
||||||
|
|
||||||
@@ -29,13 +30,17 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
|||||||
threshold; on commit the buffer is the whole stored artifact
|
threshold; on commit the buffer is the whole stored artifact
|
||||||
(kv tier) — or, on mid-stream threshold overflow, the buffer
|
(kv tier) — or, on mid-stream threshold overflow, the buffer
|
||||||
flushes into a stage file and the stream continues into it
|
flushes into a stage file and the stream continues into it
|
||||||
(fs tier — the streaming pass cannot include the preamble, so the
|
(large tier — the streaming pass cannot include the preamble, so
|
||||||
derivation pass restarts over preamble + staged bytes; forced by
|
the derivation pass restarts over preamble + staged bytes; forced
|
||||||
the preamble being inside the hashed input, not a design choice).
|
by the preamble being inside the hashed input, not a design
|
||||||
Verification runs over the eventual stored artifact as a whole;
|
choice). Verification runs over the eventual stored artifact as a
|
||||||
a stream error on either path leaves no entry and no stage file.
|
whole; a stream error on either path leaves no entry and no stage
|
||||||
Eliminates the A6 dispatch asymmetry by construction: routing is
|
file. Eliminates the A6 dispatch asymmetry by construction:
|
||||||
a pure function of content length (ADR-003).
|
routing is a pure function of content length (ADR-003). The large
|
||||||
|
tier's stage mechanics are the **staged-put contract** (ADR-010
|
||||||
|
§1): the engine hands out a staging cursor (`local`: the stage file;
|
||||||
|
pg-lo: chunked `lowrite` statements inside its tx), commit
|
||||||
|
publishes, every failure path discards.
|
||||||
- **Stage-file hygiene invariant** (POC #3 finding A5, codified): every
|
- **Stage-file hygiene invariant** (POC #3 finding A5, codified): every
|
||||||
failure path converges on stage-discard; every success path converges
|
failure path converges on stage-discard; every success path converges
|
||||||
on commit-rename. No early returns that bypass cleanup; this is an
|
on commit-rename. No early returns that bypass cleanup; this is an
|
||||||
@@ -51,7 +56,10 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
|||||||
### Read path
|
### Read path
|
||||||
|
|
||||||
- **`get(key) → Option<(len, stream)>`** — small tier returns a
|
- **`get(key) → Option<(len, stream)>`** — small tier returns a
|
||||||
bounded read; large tier streams a file handle. Dispatch
|
bounded read; large tier streams (`local`: a pread loop; pg-lo: `lo_get`
|
||||||
|
windows). The per-engine shapes are hidden behind the crate-internal
|
||||||
|
read cursor the store core drives (ADR-010 §1 — engines construct
|
||||||
|
their own cursor bodies; the facade's shape is uniform). Dispatch
|
||||||
fall-through: a small-tier miss queries the large tier (POC #1
|
fall-through: a small-tier miss queries the large tier (POC #1
|
||||||
finding 6).
|
finding 6).
|
||||||
- **`stat(key) → Option<EntryMeta>`** — cheap length/type probe
|
- **`stat(key) → Option<EntryMeta>`** — cheap length/type probe
|
||||||
@@ -60,8 +68,9 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
|||||||
carries `len: u64` and the key's algorithm (the "type" from the
|
carries `len: u64` and the key's algorithm (the "type" from the
|
||||||
store's perspective is the hash algorithm — there is no other type at
|
store's perspective is the hash algorithm — there is no other type at
|
||||||
this layer). `stat` rides the `Backend` trait's `size` method
|
this layer). `stat` rides the `Backend` trait's `size` method
|
||||||
(ADR-008): an indexed-column probe on kv engines, a file stat on the
|
(ADR-008): an indexed-column probe on kv engines, a file stat on
|
||||||
fs `local` engine — never a value/content load.
|
`local`, a companion-row probe on pg-lo — never a value/content
|
||||||
|
load.
|
||||||
- **`read_range(key, range) → (slice, slice_digest)`** — slice plus an
|
- **`read_range(key, range) → (slice, slice_digest)`** — slice plus an
|
||||||
out-of-band slice digest: SHA-256 over the slice bytes, fixed by
|
out-of-band slice digest: SHA-256 over the slice bytes, fixed by
|
||||||
convention so callers and ops handlers agree without negotiation
|
convention so callers and ops handlers agree without negotiation
|
||||||
@@ -89,12 +98,26 @@ runs changes nothing here except fleet-GC mechanism wiring (ADR-008).
|
|||||||
(finding 7) is why the verb/name is pinned here).
|
(finding 7) is why the verb/name is pinned here).
|
||||||
- **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own
|
- **`sweep() → SweepReport`** — explicit mark-and-sweep; embedders own
|
||||||
the cadence (ADR-005, no ambient timers). A sweep with no registered
|
the cadence (ADR-005, no ambient timers). A sweep with no registered
|
||||||
liveness sources aborts without deleting — the safe default. Under
|
liveness sources aborts without deleting — the safe default.
|
||||||
a fleet (multiple nodes over one pool — requirements.md REQ-2),
|
`SweepReport`'s shape (specced here, resolving its earlier
|
||||||
sweep/put/delete semantics are mechanism-extended by ADR-008: pins
|
name-only status): `candidates_staged`, `deleted`,
|
||||||
are DB-backed rows committed atomically with entries, one sweeper
|
`cancelled_by_arbitration` (u64 counts), `aborted: Option<GcAbortCause>`
|
||||||
holds the advisory lock, and delete arbitration is SQL-level — the
|
(the typed cause — protection-source failure (ADR-005) or fleet
|
||||||
facade's invariant text is unchanged.
|
sweeper-lock contention (ADR-008) — present iff the sweep deleted
|
||||||
|
nothing for an abort reason; named `GcAbortCause` to keep it a data
|
||||||
|
enum, distinct from the `GcAborted` error which carries the
|
||||||
|
direct-delete refusal path), and `started_at`/`finished_at`. The
|
||||||
|
report is data, not an error carrier: an aborted sweep returns
|
||||||
|
`Ok(report)` with `aborted: Some(..)` — the typed error
|
||||||
|
`GcAborted` is reserved for the direct-delete refusal path.
|
||||||
|
|
||||||
|
Under a fleet (multiple store instances over one pool —
|
||||||
|
requirements.md REQ-2), sweep/put/delete semantics are
|
||||||
|
mechanism-extended by ADR-008: pins are DB-backed rows committed
|
||||||
|
atomically with entries (the store core holds the joint tx,
|
||||||
|
ADR-010 §2), one sweeper holds the advisory lock, and delete
|
||||||
|
arbitration is SQL-level — the facade's invariant text is
|
||||||
|
unchanged.
|
||||||
|
|
||||||
## Error model
|
## Error model
|
||||||
|
|
||||||
@@ -129,8 +152,8 @@ lesson generalizes to any kv engine).
|
|||||||
|
|
||||||
1. Whole-blob put/get round-trips byte-identical under the canonical
|
1. Whole-blob put/get round-trips byte-identical under the canonical
|
||||||
derivation, both tiers, both put paths, **and every shipped engine
|
derivation, both tiers, both put paths, **and every shipped engine
|
||||||
of both tiers** (kv: sqlite + postgres; fs: `local` + pg-lo;
|
of both tiers** (kv: sqlite + postgres + mem; large: `local` +
|
||||||
ADR-007/008/009)
|
pg-lo; ADR-007/008/009/011)
|
||||||
(interop with real git remains the source of truth, per POC #1's
|
(interop with real git remains the source of truth, per POC #1's
|
||||||
lesson about hardcoded vectors).
|
lesson about hardcoded vectors).
|
||||||
2. Dedup: putting identical content twice (same or different path) is
|
2. Dedup: putting identical content twice (same or different path) is
|
||||||
@@ -148,8 +171,14 @@ lesson generalizes to any kv engine).
|
|||||||
tests assert through sweep outcomes (POC #1 finding 2's lesson
|
tests assert through sweep outcomes (POC #1 finding 2's lesson
|
||||||
codified).
|
codified).
|
||||||
6. `size`/`stat` on any engine returns the entry length without
|
6. `size`/`stat` on any engine returns the entry length without
|
||||||
fetching content (asserted per-backend: kv probe, fs stat, pg-lo
|
fetching content (asserted per engine: kv row probe, `local` stat,
|
||||||
companion-row probe — ADR-008/009).
|
pg-lo companion-row probe — ADR-008/009).
|
||||||
|
7. **Engine-conformance uniformity (ADR-010 §1):** every engine's
|
||||||
|
cursor/stage/list bodies pass the same contract suite mem
|
||||||
|
(the reference engine) passes — exact-count sweep outcomes, stage
|
||||||
|
hygiene, virgin-store no-ops, CAS re-put, ranged reads (the POC #7
|
||||||
|
gate is the norm; per-engine suite runs in CI, `--all-features` for
|
||||||
|
the feature-gated engines).
|
||||||
|
|
||||||
## Design Decisions
|
## Design Decisions
|
||||||
|
|
||||||
@@ -159,7 +188,8 @@ lesson generalizes to any kv engine).
|
|||||||
| [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering |
|
| [003](decisions/003-backend-contract-and-dispatch.md) | Dispatch | routing is a pure function of length; pre-threshold buffering |
|
||||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics |
|
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pools & GC | pins, liveness seams, delete windows, sweep semantics |
|
||||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band |
|
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification | whole-blob checks; slice digests out-of-band |
|
||||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed |
|
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | Trait `size` + fleet GC | stat/accounting ride the trait's `size` probe; fleet pins/sweeps are engine-backed |
|
||||||
|
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put behind the facade; the store core holds the joint entry+pin tx |
|
||||||
|
|
||||||
## Open Questions
|
## Open Questions
|
||||||
|
|
||||||
|
|||||||
Reference in new issue
Block a user