docs(architecture): ADR-008 — trait size probe, fleet GC, fs-tier engines; requirements anchor
- requirements.md (new): the three use cases pinned as REQ-1..4 with the node/pool/fleet/engine vocabulary defined once — ends per-session re-derivation of consumer facts. REQ-2 (replicator fleet over one shared pg pool, incl. large-blob serving) is recorded as a planning fact predating all POCs, on the operator's authority. - ADR-008: Backend trait gains size(key) length probe (before any backend ships data — ADR-002 one-way-door discipline); fleet GC mechanism (DB-backed pin rows committed atomically with entries, TTL+renewal semantics, liveness = embedder-owned table, protect callback is single-node-only, advisory-locked single sweeper, staged re-arbitrated delete window on both engines); fs tier becomes engine-selectable (local default; pg-lo named candidate) with the fleet locality contract (shared media or re-routing; mixed tiers are a documented deployment invariant, not a constructor-provable one). - poc-pglo-spec.md (new): POC #7 spec — pg Large Objects as the fs tier's pg-lo engine; instruments, decision gate, registered in phase-0.md OQ-BL-06. - ADR-003/005 status amendments point to ADR-008's extensions; specs ripple (backends/store-api/gc/ops/overview/README). - open-questions.md: deferral-policy header gains the decisions-vs- sequenced-work distinction; pg-lo's why-not-parked audit trail recorded. - research fixes: postgres POC renumbered #4->#5 to the canonical register (phase-0 OQ-BL-06), redb cross-refs fixed, thinking- artifact sentence in B1 replaced with the honest reading. Verification: docs-only change; reference-integrity sweep across the tree (ADR/REQ/POC refs resolve); architecture-reviewer pass on the delta — original 3 criticals addressed, its follow-up (fleet liveness form, pin TTL, staged-delete semantics, enforceability, shared-media caveats) fixed in this commit.
This commit is contained in:
glm-5.3-flash committed
1 parent
395d58cf3f
commit
281c37876e
16 files changed
+689
-88
No files matched your search
@@ -4,7 +4,11 @@
|
||||
|
||||
Accepted (engine pin amended by ADR-007 — the kv tier ships two
|
||||
engines, sqlite and postgres, behind this contract; the dispatch layer
|
||||
and trait are unchanged)
|
||||
and trait are unchanged. The trait was extended by ADR-008 with a
|
||||
`size` length probe — before any backend ships data — and the fs
|
||||
tier became engine-selectable (`local` default; pg-lo candidate), so
|
||||
this ADR's "engines"/"backends" counts are read through ADR-004's
|
||||
tier≠engine scope note)
|
||||
|
||||
## Context
|
||||
|
||||
@@ -38,9 +42,10 @@ Evidence in hand (no further POC or consumer-waiting needed):
|
||||
generic-over-`K` alternative was considered and rejected in POC #1
|
||||
finding 1: with one key type, generics buy nothing and break the
|
||||
persistence boundary.)
|
||||
- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`. Streaming and
|
||||
range shapes ride the store layer above (fs `get` yields a file
|
||||
handle, not a loaded buffer).
|
||||
- **Methods:** `has`/`get`/`put`/`delete`/`list`/`name`/`size` (the
|
||||
length probe added by ADR-008 before any backend shipped data).
|
||||
Streaming and range shapes ride the store layer above (fs `get`
|
||||
yields a file handle, not a loaded buffer).
|
||||
- **`list()` complete by contract** — and *well-formed*: a malformed
|
||||
list (wrong destructure, values-as-keys) is as deadly as an
|
||||
incomplete one, and silent until a sweep deletes the wrong things
|
||||
|
||||
@@ -2,7 +2,13 @@
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
Accepted (the sweep-safety protocol here is the single-node posture;
|
||||
ADR-008 extends the mechanism to fleet topologies — DB-backed pins,
|
||||
advisory-locked single sweeper, SQL-level delete arbitration under
|
||||
shared-pool engines (REQ-2) — while keeping this ADR's invariants
|
||||
and the in-process protocol unchanged everywhere else. The "one pool
|
||||
per node" phrasing below reads as "per deployment" for fleets; the
|
||||
vocabulary is pinned in requirements.md)
|
||||
|
||||
## Context
|
||||
|
||||
|
||||
@@ -64,7 +64,9 @@ relational engine a per-node choice rather than a crate fork.
|
||||
|
||||
1. **sqlite stays the default engine**: zero-ops, no daemon, solo
|
||||
economics (44k solo puts/s at 1-16 KiB), and consumer co-tenancy
|
||||
rides the same file (honker's WAL-NORMAL defaults are literally the
|
||||
rides the same file (honker — the alk* family's SQLite
|
||||
notify/queue extension, `/workspace/honker` — ships WAL + the
|
||||
NORMAL-synchronous posture, so its defaults are literally the
|
||||
shipped tier). Single-machine nodes are sqlite by measurement, and
|
||||
remain so even at sustained rates well above observed push traffic
|
||||
(~1.2k/s WAL ceiling vs. single-node push rates an order below it).
|
||||
|
||||
@@ -0,0 +1,256 @@
|
||||
# ADR-008: Trait length probe, fleet GC, and fs-tier engines
|
||||
|
||||
## Status
|
||||
|
||||
Accepted
|
||||
|
||||
## Context
|
||||
|
||||
Three defects surfaced in the post-ADR-007 architecture review, all
|
||||
concentrated at the seams ADR-007 touched and all load-bearing for
|
||||
REQ-2 (a replicator node may run multiple store instances — a fleet —
|
||||
sharing one pool via postgres; requirements.md):
|
||||
|
||||
1. **The `Backend` trait has no length probe.** The store surface
|
||||
promises `stat(key) → Option<EntryMeta>` ("cheap length/type
|
||||
probe"; store-api.md) and pool accounting rides `list` + `stat`
|
||||
(ADR-005), but the trait's methods are `has/get/put/delete/list/
|
||||
name` with no value-metadata access. The kv engines' row shapes
|
||||
both carry a `size` column (POC #5 B5), and the fs tier can stat a
|
||||
file — but with no trait method, kv `stat` degenerates to a full
|
||||
value load. The trait is the crate's second stable seam (ADR-003);
|
||||
discovering this during implementation would be a breaking trait
|
||||
change after backends exist.
|
||||
2. **ADR-005's sweep-safety protocol is per-node by construction.**
|
||||
Pins, the pin/liveness arbitration lock, and delete-window
|
||||
arbitration all live inside one process (ADR-005 §Decision). That
|
||||
is sound for sqlite and single-instance postgres (REQ-1, REQ-3
|
||||
topologies) — but under a fleet (REQ-2), node A's sweep enumerates
|
||||
the *shared* pool and arbitrates against A's in-process pin map;
|
||||
node B's in-flight put is invisible. Data loss, structurally: the
|
||||
topology REQ-2 requires does not satisfy the protocol ADR-005
|
||||
specifies.
|
||||
3. **The fs tier is node-local, silently partitioning a fleet.** The
|
||||
fs backend stores to a sharded layout on the node's own filesystem
|
||||
(ADR-003). Under a fleet sharing one kv pool, a >threshold blob
|
||||
put via node A exists only on A's disk; node B's get fall-through
|
||||
queries its own fs and misses, and `list()` unions disagree per
|
||||
node. REQ-2 makes this topology a shipping requirement, not a
|
||||
hypothetical.
|
||||
|
||||
These three travel together: the fleet topology forces #2 and #3, and
|
||||
#3's resolution constrains what the fs tier must support, while #1 is
|
||||
a trait-level gap any topology hits.
|
||||
|
||||
## Decision
|
||||
|
||||
### 1. `size(key) → Option<u64>` joins the `Backend` trait
|
||||
|
||||
The trait gains a length probe: `size(key)` returns the entry's byte
|
||||
length or `None` if absent. Required of all engines; semantics per
|
||||
tier:
|
||||
|
||||
- **kv engines**: an indexed column probe (`SELECT size …WHERE key`),
|
||||
never a value load. The `size` column both POC row shapes carry
|
||||
becomes contract-relevant.
|
||||
- **fs engine (`local`)**: stat/metadata read, no content fetch.
|
||||
- The store facade's `stat` (store-api.md) and the accounting path
|
||||
(ADR-005) ride this method; the `has` method remains a pure
|
||||
existence probe.
|
||||
|
||||
`size` is a trait addition *before* any backend ships data — the
|
||||
one-way-door discipline of ADR-002/003 applied to the trait itself.
|
||||
|
||||
### 2. Fleet GC: the delete window is durable and re-arbitrated in the database; pins are DB-backed under fleet engines
|
||||
|
||||
The store facade's GC invariant is unchanged — **a visible pool entry
|
||||
is never deleted while liveness protects it, and an in-flight put is
|
||||
never deleted by a sweep started before it committed** (ADR-005).
|
||||
Under a fleet, the mechanism moves into the shared engine:
|
||||
|
||||
- **Pins are rows.** `put`'s pin-before-publish (ADR-005) becomes two
|
||||
inserts in one transaction on the fleet engine: the entry row and
|
||||
its pin row commit together (or neither does). The pin row carries
|
||||
an owner (node id) and expiry: the pin guard renews its row every
|
||||
TTL/3 (default TTL 60 s, constructor-tunable — fleet constructor
|
||||
parameters, not API). **Protection = an un-expired pin row**: at
|
||||
delete-time arbitration, only `expiry > now()` pins count; an
|
||||
expired pin is abandoned (its holder failed to renew — the same
|
||||
liveness-failure signal as an in-process guard's drop) and is reaped
|
||||
by the sweeper's maintenance step (the GC-of-GC cleanup). A node
|
||||
that pauses past TTL loses its pins; delete-then-recover (ADR-005)
|
||||
remains the correctness backstop — content is re-put byte-identically
|
||||
under the same key.
|
||||
- **Registered liveness sources have one fleet form: a liveness
|
||||
table.** A fleet embedder registers its liveness as a named table of
|
||||
`(key, …)` rows in the shared engine (schema pinned by the embedder,
|
||||
shape contractually `(key bytea, …)` — any columns it needs beyond
|
||||
the key). In-process callback *views* are not a fleet liveness form:
|
||||
delete-time arbitration queries/finger-locks tables per key, so a
|
||||
source must be a table for the SQL protocol to see it. Single-node
|
||||
embedders keep callback sources unchanged (ADR-005).
|
||||
- **The protect callback is single-node-only.** A fleet sweeper cannot
|
||||
consult another node's in-process callback, so under fleet engines
|
||||
the callback's fleet expression is: the embedder materializes its
|
||||
protect set into its liveness table (or pin rows) *before* invoking
|
||||
sweep — the callback's abort semantics remain for
|
||||
single-node/topology-local sweeps that cannot express their
|
||||
protection as rows.
|
||||
- **One sweeper per pool.** A fleet node acquires a pg advisory lock
|
||||
before sweeping and holds it for the whole sweep. Two nodes
|
||||
sweeping concurrently is not a state this crate tolerates — the
|
||||
lock is the admission gate, and a second-node sweep returns the
|
||||
typed `GcAborted` (no deletion) rather than racing. `GcAborted`
|
||||
thus carries two cause classes (protection-source failure —
|
||||
ADR-005; sweeper lock contention — this ADR); the typed error's
|
||||
fleet variant identifies which.
|
||||
- **Delete-time arbitration is SQL-level.** The delete window's
|
||||
re-check (ADR-005: "arbitrate under the pin/liveness lock at delete
|
||||
time") executes as `SELECT … FOR UPDATE` against pin rows and
|
||||
liveness tables at delete time, then batch-deletes. The **staged
|
||||
re-arbitrated delete**: during mark, candidate-dead keys are staged
|
||||
as rows carrying `visible_after = mark_end + window` (the delete
|
||||
window's delay — default 5 min, constructor-tunable per
|
||||
deployment); the executor (the sweeper, or the next sweep's owner)
|
||||
re-arbitrates each staged candidate *at execution time* against
|
||||
pins and liveness tables before deleting. This is the durable
|
||||
delete window on every engine — the window table is engine rows on
|
||||
sqlite as well; what differs by topology is only where the liveness
|
||||
state lives (in-process maps and callbacks for single nodes;
|
||||
tables and SQL arbitration for fleets), not whether staging exists.
|
||||
No external queue dependency is required (the pattern is
|
||||
engine-native transactions; consumer-side queues like honker or
|
||||
pg-boss-lineage tools remain consumer-layer options, per ADR-004's
|
||||
boundary).
|
||||
- **Single-node postures are unchanged.** sqlite and single-instance
|
||||
postgres keep ADR-005's in-process protocol verbatim: the pin map,
|
||||
RAII guards, and arbitration lock stay in-process; only the
|
||||
*persistence* of the delete window (the staged-candidate table)
|
||||
is shared machinery. The pin-row expiry/ttl machinery exists only
|
||||
where fleet engines are active.
|
||||
|
||||
### 3. The fs tier is engine-selectable; fleet topology dictates which engines are permissible
|
||||
|
||||
The fs tier is a tier, not a medium. It ships engines the way the kv
|
||||
tier does (ADR-007), and a deployment's topology determines which are
|
||||
valid:
|
||||
|
||||
- **`local` (default):** the sharded-layout filesystem backend,
|
||||
unchanged (ADR-003). Valid for any single-node topology (REQ-1,
|
||||
REQ-4) and for fleet nodes using **shared media** (below).
|
||||
- **Fleet + shared media (deployment requirement):** multi-node pools
|
||||
may keep the `local` engine by pointing every node's fs tier at
|
||||
shared/replicated media. The soundness claim is *qualified*, not
|
||||
assumed: immutable-once-committed files plus stage-then-commit-
|
||||
rename are sound on media that provides (a) atomic rename within a
|
||||
directory, (b) close-to-open consistency (a written+closed file is
|
||||
complete to any later reader — NFSv3+ with the close-to-open
|
||||
discipline; NFSv4 and EFS-class stores qualify), and (c) no
|
||||
cross-node POSIX locking reliance (this store takes none — its
|
||||
cross-node coordination is the fleet engine's locks, never file
|
||||
locks; honker's NFS warning is about sqlite's lock protocol, not
|
||||
this shape). Verifying (a)-(c) on the deployment's actual media is
|
||||
the deployment's responsibility; a verification pass is recorded
|
||||
ops work, not crate code.
|
||||
- **Fleet re-routing (topology alternative):** a fleet whose nodes do
|
||||
not share media designates **one storage node**; large-blob writes
|
||||
and reads over the pool route through it via the ops surface
|
||||
(verified fetch/put; ADR-001). Client nodes' constructors are
|
||||
configured with a **kv-tier-only dispatch** (or an fs tier reserved
|
||||
for node-private, consumer-owned content — which is outside the
|
||||
pool and outside this crate's GC; it is the consumer's own storage
|
||||
beside the pool per ADR-004, not pool content). The facade shape on
|
||||
a client node is its own store for sub-threshold content plus its
|
||||
registered ops-put path for large content — a consumer-side
|
||||
composition, not a store-level mode.
|
||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
||||
as the fs tier's storage — the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
||||
with its admission gated on measured evidence (POC #7: LO write/read
|
||||
curves at the packfile regime, tx-scoped handle cost under
|
||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||
pg-lo is the consolidation option if those are unacceptable.
|
||||
- **The no-mixed-fs-tiers rule is a deployment invariant, enforced at
|
||||
the enforceable seam.** Cross-node configuration cannot be validated
|
||||
by any one constructor (it sees only its own node). What the
|
||||
constructor *can* enforce: a fleet-mode constructor (fleet engines
|
||||
active) requires an explicit fs-tier declaration — `local` (with the
|
||||
operator's assertion that its root is fleet-shared media), routed
|
||||
(no local fs over pool content), or — when shipped — `pg-lo`. The
|
||||
cross-node truth ("is node B's `local` actually the same media as
|
||||
node A's?") is a deployment invariant this crate documents and ops
|
||||
verifies, not something a constructor can prove; a partitioned
|
||||
tier's observable signature (get fall-through misses for entries
|
||||
another node wrote) is the documented detection symptom.
|
||||
- **`pg-lo` (candidate engine, not shipped):** postgres Large Objects
|
||||
as the fs tier's storage — the same "engine behind one contract"
|
||||
move as ADR-007, one tier over. Named here so the door is explicit,
|
||||
with its admission gated on measured evidence (POC #7: LO write/read
|
||||
curves at the packfile regime, tx-scoped handle cost under
|
||||
pooling, `pg_largeobject`/vacuum posture under churn, crash-orphan
|
||||
behavior) and its own engine ADR. **Not shipped speculatively;**
|
||||
REQ-2's immediate fleet answers are shared media or re-routing;
|
||||
pg-lo is the consolidation option if those are unacceptable.
|
||||
|
||||
In sum: a fleet's fs tier over pool content is either shared media (all
|
||||
nodes' `local` roots on the same fleet-shared media) or routed (one
|
||||
storage node); mixing per-node-local media over one pool is the
|
||||
partitioning failure this decision exists to prevent.
|
||||
|
||||
## Consequences
|
||||
|
||||
**Positive**
|
||||
|
||||
- The trait now expressibly supports `stat` and pool accounting on
|
||||
every engine; the "cheap probe" claim of store-api.md is true rather
|
||||
than aspirational.
|
||||
- REQ-2 (fleet over one pool) has a complete, specced GC story: pins
|
||||
move to the shared engine atomically with put-commits, one sweeper
|
||||
holds the lock, deletes re-arbitrate in-transaction. The ADR-005
|
||||
invariant extends to fleets as a *stated* mechanism, not an
|
||||
accident of single-process luck.
|
||||
- The fs tier's fleet problem has named, shipped-code-free answers
|
||||
(shared media, re-routing) and a named consolidation candidate
|
||||
(pg-lo) whose admission path is the same POC-first door as ADR-007.
|
||||
|
||||
**Negative**
|
||||
|
||||
- The `Backend` trait gains a method before any backend ships — the
|
||||
correct moment, but every engine's task list grows by one method.
|
||||
- Fleet GC is real state machinery: pin rows with expiry (GC-of-GC
|
||||
surface for orphaned pins at node crash — the expiry is the cleanup
|
||||
handle), advisory-lock coordination, and SQL-level arbitration
|
||||
added on top of the already-intricate delete-window state machine.
|
||||
- `pg-lo` remains unbuilt: a fleet that cannot share media and cannot
|
||||
re-route pays for its ADR + POC before shipping large blobs — the
|
||||
measured, deliberate deferral of the ADR-006 pattern, not a hedge.
|
||||
|
||||
**Neutral**
|
||||
|
||||
- The sqlite and single-node-postgres postures are unchanged; this ADR
|
||||
costs existing paths nothing at runtime.
|
||||
- `pg-lo`, if admitted, rides the same driver stack ADR-007 already
|
||||
ships (tokio-postgres + deadpool) via SQL `lo_*` functions — no
|
||||
new driver dependency (the `postgres_large_object` crate is a dead
|
||||
0.15-era io-trait glue; unnecessary).
|
||||
|
||||
## References
|
||||
|
||||
- [requirements.md](../requirements.md) — REQ-2 (the fleet fact this
|
||||
ADR binds), REQ-1/3/4 (the topologies left unchanged)
|
||||
- POC #5 (`poc-postgres-kv-findings.md` B5 — row shapes with `size`
|
||||
columns; the driver stack pg-lo would ride)
|
||||
- ADR-002 (the one-way-door discipline this trait change applies),
|
||||
ADR-003 (the trait contract and admission gate `size` and pg-lo
|
||||
amend/enter), ADR-004 (tier≠engine, extended to the fs tier),
|
||||
ADR-005 (the GC invariant and single-node protocol extended here,
|
||||
not replaced), ADR-006 (the deferred-cost pattern pg-lo's admission
|
||||
follows), ADR-007 (the engine-behind-one-trait precedent)
|
||||
- [backends-and-dispatch.md](../backends-and-dispatch.md) (the trait
|
||||
and tier surface this amends);
|
||||
[store-api.md](../store-api.md) (`stat`'s contract);
|
||||
[gc-and-namespaces.md](../gc-and-namespaces.md) (the fleet GC
|
||||
mechanism); [ops-surface.md](../ops-surface.md) (the re-routing
|
||||
topology's transport)
|
||||
Reference in new issue
Block a user