docs(tasks): decompose wave 2 — mem + local engines, facade, in-process GC, first suite columns
Nine tasks against the wave-1 contract surface as reviewed (no amendments from the gate): - engine-mem / engine-local (parallel, dependency-free): the two engines; local owns the stage-hygiene exact-count tests and the Drop-cancellation story the wave-1 review recorded - store-construction: the Store type, constructor modes, size-threshold dispatch; binds the conformance factory's Store associated type - store-read-paths ∥ gc-protocol: read ops with the ADR-012 §6.1 fall-through rule; the ADR-005 in-process protocol (pins, liveness, delete windows, abort-as-data, GcRefuse with the full protection set) - store-put-paths: the two put paths behind one signature, ADR-003's three numbered rules, pin-before-publish, Batch, kv-only rejection - suite-column-mem → suite-column-local: the reference column, then the dual-tier dispatch-coverage shape mirroring it - review-wave-2: the gate (facade/dispatch/GC vs ADR-003/005/012) Verification: dependency graph validated (all ids resolve, no cycles, topological order clean); docs-only change. Refs: docs/plans/implementation.md §Wave 2
This commit is contained in:
glm-5.3-flash committed
1 parent
c1d9494c11
commit
6bce301c55
10 files changed
+834
-2
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: "2026-10-11 (plan created; wave 1 decomposed into tasks/)"
|
||||
last_updated: "2026-10-11 (wave 2 decomposed into tasks/)"
|
||||
---
|
||||
|
||||
# alkblobs — Implementation plan
|
||||
@@ -49,7 +49,7 @@ ops payload surfaces; release readiness is last.
|
||||
| Wave | Contents | Depends on | Status |
|
||||
|---|---|---|---|
|
||||
| [1](#wave-1--foundations) | Crate/feature scaffold; hashing & keys (the one-way door); error model; Backend trait + I/O seams + engine-state seam shape; facade value types; conformance-suite convention | — | decomposed (2026-10-11) |
|
||||
| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | pending |
|
||||
| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | decomposed (2026-10-11) |
|
||||
| [3](#wave-3--sqlite-engine) | sqlite kv engine (default): WAL posture, prepared statements, engine-state window family, suite column | wave 2 | pending |
|
||||
| [4](#wave-4--postgres-kv-engine) | postgres kv engine: schema, pool, CAS rows, stable-cursor list, full engine-state clause families, suite column | wave 2 | pending |
|
||||
| [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | pending |
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
---
|
||||
id: engine-local
|
||||
name: local large engine — sharded files, stage-then-commit-rename, pread cursors
|
||||
status: pending
|
||||
depends_on: []
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-2, engines, local]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the `local` large-tier engine (ungated — dependency-free,
|
||||
always present; ADR-003/008) in `src/local.rs`: the ADR-003 backend as
|
||||
specced.
|
||||
|
||||
- **Constructor**: takes a root directory (tests use fresh temp dirs).
|
||||
Layout: flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's
|
||||
surviving conclusion — limits directory size on huge pools).
|
||||
**Named design point**: the filename encoding — lowercase hex of the
|
||||
full key bytes (`Key::as_bytes`, alg-tagged and self-describing per
|
||||
ADR-002's persistence-boundary ruling), prefixes taken from the hex
|
||||
string; record the decision (and the list()-decode round-trip) for
|
||||
the review gate.
|
||||
- **Staged put** (the staged-put contract realized, ADR-010 §1):
|
||||
stage-then-commit-rename. `put_staged` hands out a staging cursor
|
||||
backed by a stage file in a staging area (e.g. `{root}/staging/`,
|
||||
unique per stage); `append` writes through; `commit` renames the
|
||||
stage file into its sharded committed path (creating parent dirs) —
|
||||
publish is the rename; **every failure path converges on discard with
|
||||
zero residue** (commit failure self-discards; there is no
|
||||
post-failure discard to drive). `Drop` is best-effort residue cleanup
|
||||
— and per the wave-1 review's recorded observation, this engine's
|
||||
hygiene tests must cover its own cancellation story (a staging cursor
|
||||
dropped without commit/discard leaves no residue).
|
||||
- **Read cursor**: the **streaming family** — a pread loop over the
|
||||
committed file (immutable once committed), `read_at` clamping at EOF
|
||||
pread-style, the provided `read_range` composing the slice-digest
|
||||
convention. **Blocking file work lives in `spawn_blocking`** inside
|
||||
the impl (store-api.md §Concurrency posture; the store never blocks
|
||||
the executor).
|
||||
- **Whole-value `put`**: the large tier's settled form for direct
|
||||
whole-value writes — write-then-rename into the sharded path, CAS
|
||||
first-writer-wins (an existing file for the key is never overwritten).
|
||||
- **`size`**: file stat — metadata-only, never a content load.
|
||||
- **`list`**: walk the sharded tree, decode filenames to key bytes —
|
||||
complete and well-formed by contract (a malformed list is as deadly
|
||||
as an incomplete one, ADR-003).
|
||||
- **`delete`**: remove the committed file; `Err(Error::Missing)` when
|
||||
absent. Virgin store (fresh root): reads are no-ops, `list` empty.
|
||||
- **Exact-count stage-hygiene tests** (POC #3 finding A5's shape,
|
||||
store-api.md invariant 3): N failed puts leave zero stage files and
|
||||
zero committed entries; N successful puts leave exactly N committed
|
||||
entries and zero stage files — over both failure modes (explicit
|
||||
discard, failed commit) plus the cancellation/Drop form.
|
||||
|
||||
The conformance column itself is `suite-column-local` (separate task);
|
||||
this task ships the engine and its in-module unit tests only.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `src/local.rs` implements `Backend` completely (ungated), with
|
||||
the sharded layout, the streaming pread cursor, and the
|
||||
stage-then-commit-rename staging cursor
|
||||
- [ ] Filename encoding decided and recorded (self-describing key
|
||||
bytes; list()-decode round-trips)
|
||||
- [ ] All file I/O off the async path (`spawn_blocking`); no blocking
|
||||
on the executor
|
||||
- [ ] Exact-count stage-hygiene tests green over discard, failed
|
||||
commit, and Drop-cancellation forms — zero residue on every
|
||||
failure path, exactly one committed entry per success
|
||||
- [ ] Virgin-store reads are no-ops on a fresh root; `size` is a stat,
|
||||
never a content load; `list` is complete and well-formed
|
||||
- [ ] `cargo test` green (local is ungated — in the default build);
|
||||
`cargo clippy --all-targets -- -D warnings`, `cargo fmt --check`
|
||||
green
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/backends-and-dispatch.md (§large tier — local engine)
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md (stage-then-commit-rename, sharding)
|
||||
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract)
|
||||
- docs/architecture/store-api.md (§Concurrency posture; §Invariants 3)
|
||||
- docs/research/poc-largeblob-findings.md (findings A2/A5 — pread soundness, stage hygiene)
|
||||
- tasks/review-wave-1.md (Notes — the cancellation-residue observation this task owns)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,76 @@
|
||||
---
|
||||
id: engine-mem
|
||||
name: mem kv engine — the contract-reference engine
|
||||
status: pending
|
||||
depends_on: []
|
||||
scope: narrow
|
||||
risk: low
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-2, engines, mem]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the `mem` kv-tier engine (feature `mem`; ADR-011 §2): a
|
||||
`BTreeMap`-shaped ephemeral implementation of the kv tier's contract —
|
||||
the first real engine, and the **contract-reference engine**: every
|
||||
other engine's conformance rows mirror its column (store-api.md
|
||||
invariant 7), so its bodies are the shapes the suite holds the rest to.
|
||||
|
||||
- **Backend impl** (`src/mem.rs`, behind `#[cfg(feature = "mem")]`)
|
||||
against the wave-1 trait (`src/backend.rs`): the seven methods plus
|
||||
`put_staged`, opaque byte keys/values, `name() -> "mem"`.
|
||||
- `put` — CAS first-writer-wins (`entry().or_insert()`); a re-put of
|
||||
an existing key is a no-op, never an overwrite.
|
||||
- `get` — `None` on absence (never `Error::Missing`); the cursor body
|
||||
is the **materialized family** (the whole value; `len()` from the
|
||||
stored value).
|
||||
- `put_staged` — the **degenerate in-memory stage** (ADR-010 §1: kv
|
||||
bodies stage in memory and commit through the whole-value row
|
||||
write); `append` extends the buffer, `commit` publishes via the
|
||||
same CAS clause and self-discards on failure, `discard` drops the
|
||||
buffer with zero residue.
|
||||
- `delete` — `Err(Error::Missing)` when absent; `list` — complete and
|
||||
well-formed (the map's keys); `size` — trivially metadata-only (the
|
||||
stored value's length, no content fetch semantics to violate).
|
||||
- Virgin-store reads are no-ops (an empty map is the virgin store).
|
||||
- **Locking posture**: `parking_lot` for the short-held map lock
|
||||
(AGENTS convention 3; `unwrap_or_else(|e| e.into_inner())` on
|
||||
poisoning). No blocking I/O exists here.
|
||||
- **Unit tests** in-module: the method contract (virgin-store no-ops,
|
||||
CAS re-put, Missing discipline, size exactness), the staged-put happy
|
||||
path and both hygiene forms (discard → zero residue; commit failure →
|
||||
self-discard) — mirroring the wave-1 test-double's assertions, now
|
||||
against the real engine.
|
||||
|
||||
The conformance column itself is `suite-column-mem` (separate task);
|
||||
this task ships the engine and its in-module unit tests only.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `src/mem.rs` implements `Backend` completely (seven methods +
|
||||
`put_staged`) behind feature `mem`, with the materialized cursor
|
||||
body and the degenerate staging body
|
||||
- [ ] CAS first-writer-wins on both put forms; re-put never overwrites
|
||||
- [ ] Stage hygiene holds in both forms (discard → zero residue;
|
||||
failed commit → self-discard), unit-tested
|
||||
- [ ] Virgin-store reads are no-ops; `delete`/absent-key discipline per
|
||||
the trait contract; `size` metadata-only
|
||||
- [ ] `cargo test --features mem` green; `cargo clippy --all-features
|
||||
--all-targets -- -D warnings` and `cargo fmt --check` green
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/backends-and-dispatch.md (§kv tier — mem engine)
|
||||
- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §2
|
||||
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1
|
||||
- src/backend.rs (the trait + seam contracts this implements)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,120 @@
|
||||
---
|
||||
id: gc-protocol
|
||||
name: In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal
|
||||
status: pending
|
||||
depends_on: [store-construction]
|
||||
scope: broad
|
||||
risk: high
|
||||
impact: phase
|
||||
level: implementation
|
||||
tags: [wave-2, gc, facade]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the in-process GC protocol (ADR-005; abort-as-data per
|
||||
ADR-012 §1; non-SQL window staging per ADR-012 §5) on the `Store` type
|
||||
— the architecture's most intricate state machine, in its single-node
|
||||
form. The fleet extension (DB-backed pins, sweep lease, SQL
|
||||
arbitration) is wave 6; this task is the protocol the fleet form
|
||||
extends without changing its invariants.
|
||||
|
||||
- **Pin registry** (ADR-005 §Marks 2): per-key refcounts under the
|
||||
arbitration lock the sweep consults. `Pin` guards minted by the put
|
||||
path (`store-put-paths` wires the call sites; the registry and
|
||||
minting land here); a dedup re-put refcounts the existing entry — a
|
||||
dedup put's guard protects exactly as a fresh put's does. `Pin` drop
|
||||
decrements. **Named design point**: the guard's back-reference shape
|
||||
(how the public `Pin`/`Batch` types reach the registry — the wave-1
|
||||
types carry only the key today) and the lock granularity
|
||||
(`parking_lot` short-held per AGENTS convention 3, poisoning via
|
||||
`unwrap_or_else(|e| e.into_inner())`).
|
||||
- **`Batch` machinery**: a batch's pins drop together on batch commit
|
||||
or batch drop, in one ordered step under the arbitration lock; no
|
||||
early single-pin release. The **conversion ordering rule** (ADR-005,
|
||||
pinned in facade.rs docs): a pin's drop and its replacement's
|
||||
registration into a liveness source are one ordered step under the
|
||||
same lock — the registry needs the API shape that makes the ordered
|
||||
step expressible.
|
||||
- **Registered liveness sources** (`register_liveness_source`, the
|
||||
live-shared seam — ADR-005 §Decision): **named design point** — the
|
||||
source trait/callback shape. Live-shared means consulted at mark
|
||||
time and at delete-time arbitration, never snapshotted at
|
||||
registration (POC #1 finding 7's install-implies-copy failure is the
|
||||
pinning reason).
|
||||
- **Protect callback**: consulted pre-sweep; may add externally-known
|
||||
hashes or abort the run — abort is report data
|
||||
(`GcAbortCause::ProtectFailed`), never an error (ADR-012 §1).
|
||||
- **`sweep() → Result<SweepReport, Error>`** — the delete-window
|
||||
protocol (ADR-005 §Sweep):
|
||||
1. **mark**: enumerate the whole pool (both tiers' complete `list()`
|
||||
unioned); compute the live set from all three sources (registered
|
||||
sources, pins, protect callback); stage the candidate-dead set.
|
||||
2. **delete window**: per candidate, **arbitrate under the pin/
|
||||
liveness lock at delete time** (not once upfront) — pinned or
|
||||
re-observed-live cancels the candidate; otherwise proceed.
|
||||
3. **commit**: remaining candidates deleted in batches (~100/batch,
|
||||
iroh's proven shape); window closed.
|
||||
- **`Ok(SweepReport)` always** — sweeps are never errors; every abort
|
||||
is data in `aborted: Option<GcAbortCause>` (exactly three causes;
|
||||
`NoLivenessSources` is the safe default when zero sources are
|
||||
registered — nothing deleted). Exact counts in the report fields.
|
||||
- **Non-SQL window staging** (ADR-012 §5): `local`/`mem` stage
|
||||
candidates in-process and execute the window protocol **inline in
|
||||
the same sweep** — there is no durable window table on non-SQL
|
||||
engines, and the "executor is the next sweep" clause (ADR-012
|
||||
§6.7) does not apply to them.
|
||||
- The test-asserted property (ADR-005): a visible pool entry is never
|
||||
deleted while liveness protects it, and an in-flight put is never
|
||||
deleted by a sweep started before it committed — concurrency tests
|
||||
(sweep racing a pinned put) assert it.
|
||||
- **Direct `delete(key)`** (wired through the fall-through dispatch —
|
||||
arbitration-then-delete where the entry lives): refuses **any
|
||||
protected key** — pinned, registered-source-live, or
|
||||
protect-callback-named (the full protection set, ADR-012 §6.4) —
|
||||
with the typed `Error::GcRefuse`, nothing deleted; unprotected keys
|
||||
delete; absent keys carry `Err(Missing)`. The refusal is decided at
|
||||
the facade (the altitude discipline: `GcRefuse` never fires below
|
||||
the facade).
|
||||
- **`has`/`get` during a window observe no torn state** — existence
|
||||
flips atomically per key (immutable entries); document the posture
|
||||
the engines' CAS semantics already give.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Pin registry with per-key refcounts under the arbitration lock;
|
||||
dedup re-puts refcount; `Pin`/`Batch` drops release through it
|
||||
(batch drops in one ordered step)
|
||||
- [ ] `register_liveness_source` live-shared (consulted at mark and at
|
||||
delete-time arbitration, never snapshotted); source shape decided
|
||||
and recorded
|
||||
- [ ] Protect callback with abort-as-data (`ProtectFailed`);
|
||||
`NoLivenessSources` safe default; `sweep()` returns
|
||||
`Ok(SweepReport)` on every abort — never an abort error
|
||||
- [ ] Delete-window protocol: mark → arbitrate-at-delete-time →
|
||||
batched commit (~100/batch), with exact-count `SweepReport`
|
||||
fields (`candidates_staged`/`deleted`/`cancelled_by_arbitration`)
|
||||
- [ ] Direct `delete` refuses the full protection set with `GcRefuse`
|
||||
and deletes unprotected keys through the fall-through dispatch
|
||||
- [ ] Concurrency tests: sweep vs in-flight pinned put (the
|
||||
pin-before-publish property from the sweep side), exact-count
|
||||
outcomes, abort shapes
|
||||
- [ ] `cargo test` green; `cargo test --features mem` and `cargo test
|
||||
--all-features` green; clippy/fmt gates green
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (the protocol + invariants)
|
||||
- docs/architecture/gc-and-namespaces.md (§Mark-and-sweep GC; §Delete windows)
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (non-SQL staging), §6.4 (full protection set)
|
||||
- docs/architecture/store-api.md (§Lifecycle — sweep shape; §Invariants 4/5)
|
||||
- src/facade.rs (the pinned Pin/Batch/SweepReport lifecycle contracts this wires)
|
||||
- docs/research/poc-trait-dispatch-findings.md (findings 3/7/8)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,93 @@
|
||||
---
|
||||
id: review-wave-2
|
||||
name: Wave 2 review gate — facade, dispatch, GC, and the first columns
|
||||
status: pending
|
||||
depends_on: [suite-column-local]
|
||||
scope: narrow
|
||||
risk: low
|
||||
impact: phase
|
||||
level: review
|
||||
tags: [wave-2, review]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
The wave-2 review gate, before wave 3 decomposes. Wave 2 made the
|
||||
store work end-to-end dependency-free; the gate reads the
|
||||
implementation against the pinned ADR text line by line (the wave-1
|
||||
gate's method).
|
||||
|
||||
Gate checks:
|
||||
|
||||
- **Facade + dispatch + GC vs ADR-003/005/012**:
|
||||
- the **three numbered unknown-length rules** (ADR-003 §Dispatch
|
||||
policy) each read against their tests — mid-stream flush with no
|
||||
restart requirement; verification over the eventual stored artifact
|
||||
as a whole (buffer form and staged-bytes form); stream error leaves
|
||||
no entry and no stage file;
|
||||
- the **delete-window ordering invariants** (ADR-005 §Sweep):
|
||||
pin-before-publish on both put paths and both tiers; arbitration
|
||||
under the pin/liveness lock at delete time; the test-asserted
|
||||
property (a visible entry is never deleted while liveness protects
|
||||
it; an in-flight put is never deleted by a sweep started before it
|
||||
committed);
|
||||
- **abort-as-data** (ADR-012 §1): `sweep()` returns
|
||||
`Ok(SweepReport)` on every abort; `GcRefuse` is the direct-delete
|
||||
refusal only; the full protection set per ADR-012 §6.4;
|
||||
- dispatch as a pure function of length (no migration); fall-through
|
||||
general rule per ADR-012 §6.1; constructor modes per ADR-011 §4 /
|
||||
ADR-012 §6.2/§6.3; kv-only put-time rejection.
|
||||
- **Stage-hygiene exact-count tests**: present and exact on both
|
||||
engines (mem's degenerate stage; local's real stage form incl. the
|
||||
Drop-cancellation story per the wave-1 review's observation).
|
||||
- **The suite's first columns green on mem + local**: rows
|
||||
ADR-stamped, traceable to the backlog, the factory binding recorded
|
||||
(factory.rs + README if corrected).
|
||||
- **The dispatch-coverage shape exercised**: the `kv = mem` +
|
||||
`large = local` column's dispatch rows (pure-function routing,
|
||||
fall-through, threshold boundaries).
|
||||
- **Named design points audited as resolved** (each recorded in its
|
||||
task's Notes): engine representation (enum vs generic); the public
|
||||
stream type; list return shape; verification-on-get posture;
|
||||
liveness-source shape; Pin registry back-reference + lock
|
||||
granularity; put stream parameter shape; len-mismatch error.
|
||||
- **Family conventions**: no comments in code (doc comments on public
|
||||
API fine), thiserror, tokio-only async, `spawn_blocking` for blocking
|
||||
file work, no unwrap/expect outside tests, naming, module structure.
|
||||
- **Gates green**: `cargo test`, `cargo test --features mem`, `cargo
|
||||
test --all-features`, `cargo clippy --all-targets -- -D warnings`,
|
||||
`cargo clippy --all-features --all-targets -- -D warnings`, `cargo
|
||||
fmt --check`.
|
||||
|
||||
Findings are fixed inline where cheap, or recorded as wave-3
|
||||
amendments where they shape the next decomposition. The gate's
|
||||
completion is what unblocks wave 3's decomposition.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Every gate check above read and verified against the cited ADR
|
||||
text (not from memory of it)
|
||||
- [ ] Named design points each decided, recorded, and consistent with
|
||||
the specs
|
||||
- [ ] Findings fixed inline or recorded as wave-3 amendments with
|
||||
owners
|
||||
- [ ] All gates re-run green after any fixes
|
||||
- [ ] Wave 3 decomposition unblocked (recorded in this task's Summary)
|
||||
|
||||
## References
|
||||
|
||||
- docs/plans/implementation.md (§Wave 2, §Review gates)
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md
|
||||
- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md
|
||||
- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md
|
||||
- docs/architecture/store-api.md
|
||||
- tasks/review-wave-1.md (the gate method; the observations wave 2 owned)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,94 @@
|
||||
---
|
||||
id: store-construction
|
||||
name: Store facade type — constructor modes and size-threshold dispatch
|
||||
status: pending
|
||||
depends_on: [engine-mem, engine-local]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: project
|
||||
level: implementation
|
||||
tags: [wave-2, facade, dispatch]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Introduce the `Store` facade type (store-api.md §What this is) — the
|
||||
public surface consumers program against — with its constructor modes
|
||||
and the size-threshold dispatch. This task builds the *skeleton and
|
||||
routing*: the ops methods land in `store-read-paths`, `gc-protocol`,
|
||||
and `store-put-paths` against this type.
|
||||
|
||||
- **The `Store` type** (public, `src/facade.rs` or a new module —
|
||||
implementer's call, one module per file): holds the kv-tier engine
|
||||
and the large-tier engine (or none, per mode) plus the GC state home
|
||||
the `gc-protocol` task fills. **Named design point — engine
|
||||
representation**: a concrete `Store` with a crate-internal enum
|
||||
dispatching over the closed five-engine set, vs a generic
|
||||
`Store<Kv: Backend, Large: Backend>`. The conformance factory's
|
||||
binding note (tests/conformance/factory.rs) anticipates the concrete
|
||||
type ("most plausibly the associated type collapsing to the concrete
|
||||
facade type across all factories"); engines are statically known and
|
||||
in-crate (no `dyn` anywhere, wave-1 design point 3). Decide, record,
|
||||
and keep the public surface engine-blind (ADR-007/008/010: which
|
||||
engines a node runs changes nothing here).
|
||||
- **Constructor modes** (ADR-011 §4; ADR-012 §6.3):
|
||||
- **dual-tier** (the default): kv + large, size-threshold dispatch.
|
||||
Wave-2 constructible shape: `kv = mem` + `large = local` — the
|
||||
dispatch-coverage test shape ADR-012 §6.3 names (both tiers
|
||||
load-bearing).
|
||||
- **kv-only**: no large tier; over-threshold puts reject at put time
|
||||
(ADR-012 §6.2 — the rejection behavior itself lands in
|
||||
`store-put-paths`; this task carries the mode and its validation).
|
||||
- **mem-only**: the mem engine alone, no dispatch threshold in
|
||||
effect; never production.
|
||||
- Mode machinery is built so the wave-3–5 engine constructors slot in
|
||||
without reshaping (sqlite/postgres/pg-lo constructors are later
|
||||
waves; only feature-gated constructors exist when their feature is
|
||||
on).
|
||||
- **Size-threshold dispatch** (ADR-003): routing as a **pure function
|
||||
of content length** — same content ⇒ same length ⇒ same tier, no
|
||||
content ever migrates. Default threshold **128 KiB**
|
||||
(constructor-tunable; the conservative edge of the measured
|
||||
~128–256 KiB crossover zone, POC #3 A4). The dispatch function is
|
||||
pure and unit-tested (boundary values: exactly-at-threshold routing).
|
||||
- **The dispatch seam** the ops tasks program through: tier selection
|
||||
for a given length, and the fall-through order (kv first, then large)
|
||||
the read paths and delete consume (ADR-012 §6.1's general rule).
|
||||
- **Conformance factory binding**: bind `StoreFactory::Store` to the
|
||||
real facade type (the skeleton's anticipated collapse to the concrete
|
||||
type); if binding reveals a shape correction, make it and record it
|
||||
in factory.rs + tests/conformance/README.md per the skeleton's note.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] Public `Store` type with the three constructor modes; engine
|
||||
representation decided and recorded (enum-dispatch vs generic)
|
||||
with the public surface engine-blind
|
||||
- [ ] Size-threshold dispatch: pure function of length, default 128 KiB,
|
||||
constructor-tunable; boundary-value unit tests (at/below/above
|
||||
threshold route deterministically)
|
||||
- [ ] `kv = mem` + `large = local` dual-tier construction works (the
|
||||
ADR-012 §6.3 dispatch-coverage shape); kv-only and mem-only
|
||||
modes construct and validate
|
||||
- [ ] `StoreFactory::Store` bound to the real facade; any shape
|
||||
correction recorded in factory.rs + README
|
||||
- [ ] `cargo test` green (default build: local ungated); `cargo test
|
||||
--features mem` and `cargo test --all-features` green;
|
||||
clippy/fmt gates green in all three feature shapes
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/store-api.md (§What this is; §Public surface)
|
||||
- docs/architecture/backends-and-dispatch.md (§Size-threshold dispatch; §Constructor modes)
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy)
|
||||
- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §4
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.2/§6.3
|
||||
- tests/conformance/factory.rs (the binding note)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,103 @@
|
||||
---
|
||||
id: store-put-paths
|
||||
name: Store put seam — two internal paths, the three numbered rules, Batch, kv-only rejection
|
||||
status: pending
|
||||
depends_on: [gc-protocol]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-2, facade, write-path]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the facade's put seam on the `Store` type (store-api.md
|
||||
§Write path): **`put(len: Option<u64>, stream) → (Key, Pin)`** — two
|
||||
internal paths behind one signature, with pin-before-publish from the
|
||||
`gc-protocol` registry.
|
||||
|
||||
- **Known-length path** (encouraged, documented as such on the API):
|
||||
the preamble (`"blob <len>\0"`) is known before content flows, so
|
||||
the hash initializes with it and one pass suffices — no staging
|
||||
needed for hashing. Routing by the dispatch seam: ≤ threshold → the
|
||||
kv tier's whole-value put; > threshold → the large tier's staged put
|
||||
(content streams into the staging cursor while hashing; the
|
||||
derivation completes at stream end; commit-rename publishes).
|
||||
- **Unknown-length path**: content buffers in memory up to the
|
||||
dispatch threshold. ADR-003's **three numbered rules** (option (d)),
|
||||
each a test-asserted behavior:
|
||||
1. **Threshold exceeded mid-stream** (non-restartable sources are
|
||||
normal): the buffered prefix flushes *into a fresh stage file*
|
||||
and the stream continues appending into it — there is no
|
||||
"restart the stream" requirement anywhere.
|
||||
2. **Verification runs over the eventual stored artifact as a
|
||||
whole**: the memory buffer for the sub-threshold kv commit; the
|
||||
complete staged bytes for the large commit — the derivation pass
|
||||
restarts over preamble + staged bytes (forced by the preamble
|
||||
being inside the hashed input, ADR-002), reading the staged
|
||||
artifact back through the `StagingCursor: ReadCursor` seam (the
|
||||
wave-1 design-led extension). There is no stitched-input case
|
||||
left ambiguous.
|
||||
3. **Stream error before commit**: no entry lands and no stage file
|
||||
remains — buffer discard on the failure path, exactly like stage
|
||||
discard (the stage-hygiene invariant extends to the buffered
|
||||
case).
|
||||
- **Pin-before-publish** (ADR-005): the pin refcount is acquired under
|
||||
the arbitration lock *before* the entry becomes visible — publish
|
||||
and protection are one step; an entry cannot appear in a sweep's
|
||||
`list()` without already carrying its pin.
|
||||
- **Dedup**: a re-put of existing content acquires no new pool entry —
|
||||
the returned `Pin` refcounts the existing one under the same lock.
|
||||
- **`Batch`** (store-api.md §Write path): the batching put seam —
|
||||
multi-put writes group under one batch-scoped pin; member puts
|
||||
publish pin-before-publish exactly as lone puts do; the batch's pins
|
||||
drop together on commit or drop (the `gc-protocol` machinery drives).
|
||||
- **kv-only mode rejection** (ADR-012 §6.2): known-length puts
|
||||
exceeding the threshold reject **immediately**; unknown-length puts
|
||||
reject **at mid-stream threshold overflow** — lengths are not known
|
||||
at construction, so put time is the only possible rejection point.
|
||||
Zero residue on either rejection.
|
||||
- **Named design points**: (a) the stream parameter shape (`impl
|
||||
AsyncRead + Unpin + Send` vs a boxed trait object — the family's
|
||||
no-dyn posture vs ergonomics); (b) the declared-len mismatch case
|
||||
(a known-length stream that yields fewer/more bytes than `len` —
|
||||
which error family member carries it, and zero residue either way).
|
||||
Record both for the review gate.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `put(len, stream) → (Key, Pin)` with both internal paths behind
|
||||
the one signature; routing is the pure length function (no
|
||||
content ever migrates tiers)
|
||||
- [ ] The three numbered rules each behavior-tested: mid-stream
|
||||
overflow flushes into a fresh stage and the stream continues;
|
||||
the derivation pass runs over the eventual stored artifact as a
|
||||
whole (buffer form and staged-bytes form); stream error leaves
|
||||
no entry and no stage file
|
||||
- [ ] Pin-before-publish holds on both paths and both tiers; dedup
|
||||
re-puts refcount the existing entry
|
||||
- [ ] `Batch` groups multi-put writes; member pins drop together
|
||||
- [ ] kv-only mode: known-length over-threshold rejects immediately;
|
||||
unknown-length rejects at mid-stream overflow; zero residue on
|
||||
both
|
||||
- [ ] Stream-shape and len-mismatch design points decided and recorded
|
||||
- [ ] `cargo test` green; `cargo test --features mem` and `cargo test
|
||||
--all-features` green; clippy/fmt gates green
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/store-api.md (§Write path — the put seam, stage hygiene invariant)
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy — the three numbered rules, option (d))
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.2 (kv-only put-time rejection)
|
||||
- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (pin-before-publish, dedup refcount)
|
||||
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract)
|
||||
- src/backend.rs (StagingCursor: ReadCursor — the verification-pass seam)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,88 @@
|
||||
---
|
||||
id: store-read-paths
|
||||
name: Store read paths — get/stat/read_range/has/list with dispatch fall-through
|
||||
status: pending
|
||||
depends_on: [store-construction]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: component
|
||||
level: implementation
|
||||
tags: [wave-2, facade, read-path]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Implement the facade's read-side ops on the `Store` type (store-api.md
|
||||
§Read path, §Lifecycle), behind the dispatch seam from
|
||||
`store-construction`. Direct `delete` is *not* here — it lands in
|
||||
`gc-protocol` with its arbitration.
|
||||
|
||||
- **The fall-through general rule** (ADR-012 §6.1; POC #1 finding 6):
|
||||
every key-addressed op consults the kv tier first, then the large
|
||||
tier. `get`/`stat`/`has`/`read_range`/`list` all follow it.
|
||||
- **`get(key) → Option<(len, stream)>`**: small tier returns a bounded
|
||||
read; large tier streams. The per-engine cursor shapes stay hidden
|
||||
behind the crate-internal `ReadCursor` the store core drives
|
||||
(ADR-010 §1) — **named design point**: the public stream type that
|
||||
wraps the cursor (the facade's `(len, stream)` shape is uniform
|
||||
across tiers; the cursor is never exposed). A miss on both tiers is
|
||||
`None` — `get` never carries `Error::Missing` (ADR-012 §6.5 ruling 5;
|
||||
the error module's behavioral-enforcement note lands here).
|
||||
- **`stat(key) → Option<EntryMeta>`**: the cheap probe riding the
|
||||
trait's `size` method — length plus algorithm, never a content load
|
||||
(ADR-008's size discipline). Fall-through per the general rule.
|
||||
- **`read_range(key, range) → (slice, slice_digest)`**: slice plus the
|
||||
out-of-band SHA-256 slice digest (ADR-006, fixed by convention —
|
||||
composed through the cursor's `read_range` seam). `Err(Missing)` on a
|
||||
both-tier miss. Range reads fall through tiers exactly as `get` does.
|
||||
- **`has(key) → bool`**: the pure existence probe (ADR-008's `has`
|
||||
discipline) — a size-probe under the fall-through rule; absent on
|
||||
both tiers is `false`; virgin-store reads are no-ops.
|
||||
- **`list() → stream of keys`**: whole-pool enumeration, complete by
|
||||
contract. **Named design point**: the return shape — store-api.md
|
||||
says "stream of keys"; decide Vec vs stream (the sweep consumes the
|
||||
same enumeration crate-internally) and record it.
|
||||
- **Verification posture on get** (named design point): ADR-006's
|
||||
whole-blob verification vs the slice-digest-only reading — where the
|
||||
store core's hash-check actually fires on the read path (kv
|
||||
materialized reads are verifiable cheaply; large streaming reads ride
|
||||
per-range slice digests). Pin the reading against ADR-006's text and
|
||||
record it for the review gate.
|
||||
- **`KeyInvalid` at the boundary**: typed `Key` conversion happens at
|
||||
the store boundary only (ADR-002/003) — malformed key bytes on any
|
||||
read op reject with `Error::KeyInvalid`, never a guess; engines never
|
||||
see typed keys.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `get`/`stat`/`read_range`/`has`/`list` implemented with the
|
||||
fall-through general rule (kv first, then large), unit-tested
|
||||
including the both-tier-miss shapes (`None` vs `Err(Missing)` per
|
||||
op)
|
||||
- [ ] The public stream type decided and recorded; engine cursors never
|
||||
leak into the public surface
|
||||
- [ ] `stat` never fetches content (rides `size`); `EntryMeta` carries
|
||||
len + algorithm
|
||||
- [ ] `read_range` returns slice + SHA-256 slice digest per the ADR-006
|
||||
convention; verification-on-get posture decided and recorded
|
||||
- [ ] Malformed key bytes reject with `KeyInvalid` at the facade
|
||||
boundary on every key-addressed op
|
||||
- [ ] `cargo test` green; `cargo test --features mem` and `cargo test
|
||||
--all-features` green; clippy/fmt gates green
|
||||
|
||||
## References
|
||||
|
||||
- docs/architecture/store-api.md (§Read path, §Lifecycle, §Error model)
|
||||
- docs/architecture/backends-and-dispatch.md (§Size-threshold dispatch — get fall-through)
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.1/§6.5
|
||||
- docs/architecture/decisions/006-verification-posture-and-transfer-encoding.md
|
||||
- docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md (size discipline)
|
||||
- src/backend.rs (the ReadCursor seam this drives)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,80 @@
|
||||
---
|
||||
id: suite-column-local
|
||||
name: Conformance suite — the local column and the dispatch-coverage shape
|
||||
status: pending
|
||||
depends_on: [suite-column-mem]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: phase
|
||||
level: implementation
|
||||
tags: [wave-2, suite, local, dispatch]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Grow the **local column** — the suite's first dual-tier column, run
|
||||
against the `kv = mem` + `large = local` factory: the
|
||||
**dispatch-coverage test shape** ADR-012 §6.3 names (both tiers
|
||||
load-bearing). Rows mirror the mem column's set (INV-7: every engine's
|
||||
rows mirror the reference column), extended with the dual-tier and
|
||||
local-specific properties.
|
||||
|
||||
- **The factory**: fresh, isolated temp-root dual-tier store per call
|
||||
(`kv = mem` + `large = local`), idempotent teardown (temp dir
|
||||
removal); never skips.
|
||||
- **Mirrored rows** (the mem column's set, against the dual-tier
|
||||
store):
|
||||
- **INV-1 round-trips**: both tiers, both put paths — the content
|
||||
matrix crosses the threshold so both tiers' bodies are exercised
|
||||
(sub-threshold kv rows; over-threshold local stage-commit rows).
|
||||
- **INV-3 stage hygiene**: exact counts **on disk** — zero stage
|
||||
files after every failure form, exactly one committed entry per
|
||||
success, over both put paths (the real stage form, not the
|
||||
degenerate kv stage).
|
||||
- **INV-4 sweep safety**: exact-count sweep outcomes over the
|
||||
dual-tier pool (candidates from both tiers; abort shapes; the
|
||||
sweep-vs-pinned-put race).
|
||||
- **INV-2 dedup, INV-5 list-through-sweep, INV-6 stat** mirrored.
|
||||
- **Dispatch rows** (the coverage shape's reason to exist):
|
||||
- Routing is a pure function of length: same content ⇒ same tier
|
||||
across mixed put paths and re-puts; no content ever migrates.
|
||||
- **Fall-through** (ADR-012 §6.1): a kv-tier miss queries the large
|
||||
tier — `get`/`stat`/`has`/`read_range` each served from the large
|
||||
tier when the kv tier misses; both-tier miss shapes per op.
|
||||
- Threshold boundary values (at/below/above the default; a
|
||||
constructor-tuned threshold re-routes deterministically).
|
||||
- **Local-specific rows**: pread range reads over committed files
|
||||
(mid-file slices, EOF clamping); virgin-store no-ops on a fresh root
|
||||
(empty `list`, absent reads); `size` as a stat (never a content
|
||||
load).
|
||||
- Every row ADR-stamped per the convention, tracing to the backlog.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] The dual-tier `kv = mem` + `large = local` factory green with
|
||||
isolation + idempotent teardown
|
||||
- [ ] The local column green: mem's row set mirrored (INV-1/2/3/4/5/6)
|
||||
plus dispatch rows (pure-function routing, fall-through per op,
|
||||
threshold boundaries) and local-specific rows (pread ranges,
|
||||
virgin-store no-ops, stat-as-stat)
|
||||
- [ ] Stage-hygiene exact counts asserted on disk (the real stage form)
|
||||
- [ ] Every row ADR-stamped and traceable to the backlog
|
||||
- [ ] `cargo test` green (local is ungated — the column runs in the
|
||||
default build); clippy/fmt gates green
|
||||
|
||||
## References
|
||||
|
||||
- tests/conformance/README.md + backlog.rs (the conventions; INV first-column plan)
|
||||
- docs/architecture/backends-and-dispatch.md (§Constructor modes — the dispatch-coverage shape; §Size-threshold dispatch)
|
||||
- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.1/§6.3
|
||||
- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy)
|
||||
- docs/architecture/store-api.md (§Invariants)
|
||||
- docs/research/poc-largeblob-findings.md (findings A2/A5)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
@@ -0,0 +1,83 @@
|
||||
---
|
||||
id: suite-column-mem
|
||||
name: Conformance suite — factory binding and the mem column (the reference column)
|
||||
status: pending
|
||||
depends_on: [store-read-paths, store-put-paths, gc-protocol]
|
||||
scope: moderate
|
||||
risk: medium
|
||||
impact: phase
|
||||
level: implementation
|
||||
tags: [wave-2, suite, mem]
|
||||
---
|
||||
|
||||
## Description
|
||||
|
||||
Bind the conformance harness to the real facade and grow the **mem
|
||||
column** — the contract-reference column that defines the required set
|
||||
every other engine's rows mirror (store-api.md invariant 7; ADR-011 §2).
|
||||
|
||||
- **The factory binding** (the skeleton's anticipated wave-2 moment):
|
||||
bind `StoreFactory::Store` to the real facade type — most plausibly
|
||||
the associated type collapsing to the concrete `Store` across all
|
||||
factories. If binding reveals a shape correction, make it and record
|
||||
it in `tests/conformance/factory.rs` + `README.md` per the
|
||||
skeleton's note. The mem factory vends fresh, isolated in-memory
|
||||
stores (no shared state between rows) with idempotent teardown; mem
|
||||
never skips (README §SKIP posture — SKIP machinery stays unneeded
|
||||
until the SQL waves).
|
||||
- **The mem column's rows**, each ADR-stamped per the convention
|
||||
(`Contract stamp:` citing the pinning spec §/ADR §), tracing to the
|
||||
backlog entries:
|
||||
- **INV-1 round-trips**: whole-blob put/get byte-identical under the
|
||||
canonical derivation, over a content matrix looping both put paths
|
||||
(known-length, unknown-length) in-body; the git-interop test
|
||||
remains the derivation's source of truth.
|
||||
- **INV-2 dedup**: identical content twice (same and different put
|
||||
path) is one pool entry; the re-put's `Pin` protects exactly as a
|
||||
fresh put's.
|
||||
- **INV-3 stage hygiene**: exact counts over both put paths and both
|
||||
failure modes (explicit discard, failed commit) — zero residue on
|
||||
failure, exactly one committed entry on success (the degenerate
|
||||
kv stage form).
|
||||
- **INV-4 sweep safety**: single-node exact-count family — happy
|
||||
shape (exact `SweepReport` counts), abort shapes
|
||||
(`ProtectFailed`, `NoLivenessSources` — nothing deleted,
|
||||
`Ok(SweepReport)` always), and the sweep-vs-pinned-put race.
|
||||
- **INV-5 list correctness**: asserted through sweep outcomes (exact
|
||||
counts), never by comparing enumerated lists directly.
|
||||
- **INV-6 stat discipline**: `stat` returns length + algorithm
|
||||
without fetching content.
|
||||
- **Constructor-mode rows** (facade contract, ADR-011 §4 / ADR-012
|
||||
§6.2/§6.3): mem-only mode; kv-only(mem) over-threshold put-time
|
||||
rejection (known-length immediate, unknown-length mid-stream).
|
||||
- Rows take factories, never stores (factory.rs contract); a failing
|
||||
assertion panics the row; teardown is idempotent against that path.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
- [ ] `StoreFactory::Store` bound to the real facade; any shape
|
||||
correction recorded in factory.rs + README
|
||||
- [ ] The mem column green: INV-1/2/3/4/5/6 rows plus constructor-mode
|
||||
rows, every row ADR-stamped and traceable to the backlog
|
||||
- [ ] The mem column's row set is the reference set (INV-7's basis) —
|
||||
recorded in README as what other columns mirror
|
||||
- [ ] Factory isolation + idempotent teardown hold (fresh store per
|
||||
call; teardown safe against a panicking row)
|
||||
- [ ] `cargo test --features mem` green (the column runs under its
|
||||
feature); clippy/fmt gates green
|
||||
|
||||
## References
|
||||
|
||||
- tests/conformance/README.md (the conventions this grows against)
|
||||
- tests/conformance/backlog.rs (INV-1…7 entries and their first-column plan)
|
||||
- docs/architecture/store-api.md (§Invariants — the test gate)
|
||||
- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §2 (mem's reference role)
|
||||
- docs/plans/implementation.md (Decided points — suite home; §Wave 2 — suite columns)
|
||||
|
||||
## Notes
|
||||
|
||||
> To be filled by implementation agent
|
||||
|
||||
## Summary
|
||||
|
||||
> To be filled on completion
|
||||
Reference in new issue
Block a user