diff --git a/docs/plans/implementation.md b/docs/plans/implementation.md index c5255f1..a49b294 100644 --- a/docs/plans/implementation.md +++ b/docs/plans/implementation.md @@ -1,6 +1,6 @@ --- status: draft -last_updated: "2026-10-11 (plan created; wave 1 decomposed into tasks/)" +last_updated: "2026-10-11 (wave 2 decomposed into tasks/)" --- # alkblobs — Implementation plan @@ -49,7 +49,7 @@ ops payload surfaces; release readiness is last. | Wave | Contents | Depends on | Status | |---|---|---|---| | [1](#wave-1--foundations) | Crate/feature scaffold; hashing & keys (the one-way door); error model; Backend trait + I/O seams + engine-state seam shape; facade value types; conformance-suite convention | — | decomposed (2026-10-11) | -| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | pending | +| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | decomposed (2026-10-11) | | [3](#wave-3--sqlite-engine) | sqlite kv engine (default): WAL posture, prepared statements, engine-state window family, suite column | wave 2 | pending | | [4](#wave-4--postgres-kv-engine) | postgres kv engine: schema, pool, CAS rows, stable-cursor list, full engine-state clause families, suite column | wave 2 | pending | | [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | pending | diff --git a/tasks/engine-local.md b/tasks/engine-local.md new file mode 100644 index 0000000..88c00e9 --- /dev/null +++ b/tasks/engine-local.md @@ -0,0 +1,95 @@ +--- +id: engine-local +name: local large engine — sharded files, stage-then-commit-rename, pread cursors +status: pending +depends_on: [] +scope: moderate +risk: medium +impact: component +level: implementation +tags: [wave-2, engines, local] +--- + +## Description + +Implement the `local` large-tier engine (ungated — dependency-free, +always present; ADR-003/008) in `src/local.rs`: the ADR-003 backend as +specced. + +- **Constructor**: takes a root directory (tests use fresh temp dirs). + Layout: flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's + surviving conclusion — limits directory size on huge pools). + **Named design point**: the filename encoding — lowercase hex of the + full key bytes (`Key::as_bytes`, alg-tagged and self-describing per + ADR-002's persistence-boundary ruling), prefixes taken from the hex + string; record the decision (and the list()-decode round-trip) for + the review gate. +- **Staged put** (the staged-put contract realized, ADR-010 §1): + stage-then-commit-rename. `put_staged` hands out a staging cursor + backed by a stage file in a staging area (e.g. `{root}/staging/`, + unique per stage); `append` writes through; `commit` renames the + stage file into its sharded committed path (creating parent dirs) — + publish is the rename; **every failure path converges on discard with + zero residue** (commit failure self-discards; there is no + post-failure discard to drive). `Drop` is best-effort residue cleanup + — and per the wave-1 review's recorded observation, this engine's + hygiene tests must cover its own cancellation story (a staging cursor + dropped without commit/discard leaves no residue). +- **Read cursor**: the **streaming family** — a pread loop over the + committed file (immutable once committed), `read_at` clamping at EOF + pread-style, the provided `read_range` composing the slice-digest + convention. **Blocking file work lives in `spawn_blocking`** inside + the impl (store-api.md §Concurrency posture; the store never blocks + the executor). +- **Whole-value `put`**: the large tier's settled form for direct + whole-value writes — write-then-rename into the sharded path, CAS + first-writer-wins (an existing file for the key is never overwritten). +- **`size`**: file stat — metadata-only, never a content load. +- **`list`**: walk the sharded tree, decode filenames to key bytes — + complete and well-formed by contract (a malformed list is as deadly + as an incomplete one, ADR-003). +- **`delete`**: remove the committed file; `Err(Error::Missing)` when + absent. Virgin store (fresh root): reads are no-ops, `list` empty. +- **Exact-count stage-hygiene tests** (POC #3 finding A5's shape, + store-api.md invariant 3): N failed puts leave zero stage files and + zero committed entries; N successful puts leave exactly N committed + entries and zero stage files — over both failure modes (explicit + discard, failed commit) plus the cancellation/Drop form. + +The conformance column itself is `suite-column-local` (separate task); +this task ships the engine and its in-module unit tests only. + +## Acceptance Criteria + +- [ ] `src/local.rs` implements `Backend` completely (ungated), with + the sharded layout, the streaming pread cursor, and the + stage-then-commit-rename staging cursor +- [ ] Filename encoding decided and recorded (self-describing key + bytes; list()-decode round-trips) +- [ ] All file I/O off the async path (`spawn_blocking`); no blocking + on the executor +- [ ] Exact-count stage-hygiene tests green over discard, failed + commit, and Drop-cancellation forms — zero residue on every + failure path, exactly one committed entry per success +- [ ] Virgin-store reads are no-ops on a fresh root; `size` is a stat, + never a content load; `list` is complete and well-formed +- [ ] `cargo test` green (local is ungated — in the default build); + `cargo clippy --all-targets -- -D warnings`, `cargo fmt --check` + green + +## References + +- docs/architecture/backends-and-dispatch.md (§large tier — local engine) +- docs/architecture/decisions/003-backend-contract-and-dispatch.md (stage-then-commit-rename, sharding) +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract) +- docs/architecture/store-api.md (§Concurrency posture; §Invariants 3) +- docs/research/poc-largeblob-findings.md (findings A2/A5 — pread soundness, stage hygiene) +- tasks/review-wave-1.md (Notes — the cancellation-residue observation this task owns) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/engine-mem.md b/tasks/engine-mem.md new file mode 100644 index 0000000..7793c52 --- /dev/null +++ b/tasks/engine-mem.md @@ -0,0 +1,76 @@ +--- +id: engine-mem +name: mem kv engine — the contract-reference engine +status: pending +depends_on: [] +scope: narrow +risk: low +impact: component +level: implementation +tags: [wave-2, engines, mem] +--- + +## Description + +Implement the `mem` kv-tier engine (feature `mem`; ADR-011 §2): a +`BTreeMap`-shaped ephemeral implementation of the kv tier's contract — +the first real engine, and the **contract-reference engine**: every +other engine's conformance rows mirror its column (store-api.md +invariant 7), so its bodies are the shapes the suite holds the rest to. + +- **Backend impl** (`src/mem.rs`, behind `#[cfg(feature = "mem")]`) + against the wave-1 trait (`src/backend.rs`): the seven methods plus + `put_staged`, opaque byte keys/values, `name() -> "mem"`. + - `put` — CAS first-writer-wins (`entry().or_insert()`); a re-put of + an existing key is a no-op, never an overwrite. + - `get` — `None` on absence (never `Error::Missing`); the cursor body + is the **materialized family** (the whole value; `len()` from the + stored value). + - `put_staged` — the **degenerate in-memory stage** (ADR-010 §1: kv + bodies stage in memory and commit through the whole-value row + write); `append` extends the buffer, `commit` publishes via the + same CAS clause and self-discards on failure, `discard` drops the + buffer with zero residue. + - `delete` — `Err(Error::Missing)` when absent; `list` — complete and + well-formed (the map's keys); `size` — trivially metadata-only (the + stored value's length, no content fetch semantics to violate). + - Virgin-store reads are no-ops (an empty map is the virgin store). +- **Locking posture**: `parking_lot` for the short-held map lock + (AGENTS convention 3; `unwrap_or_else(|e| e.into_inner())` on + poisoning). No blocking I/O exists here. +- **Unit tests** in-module: the method contract (virgin-store no-ops, + CAS re-put, Missing discipline, size exactness), the staged-put happy + path and both hygiene forms (discard → zero residue; commit failure → + self-discard) — mirroring the wave-1 test-double's assertions, now + against the real engine. + +The conformance column itself is `suite-column-mem` (separate task); +this task ships the engine and its in-module unit tests only. + +## Acceptance Criteria + +- [ ] `src/mem.rs` implements `Backend` completely (seven methods + + `put_staged`) behind feature `mem`, with the materialized cursor + body and the degenerate staging body +- [ ] CAS first-writer-wins on both put forms; re-put never overwrites +- [ ] Stage hygiene holds in both forms (discard → zero residue; + failed commit → self-discard), unit-tested +- [ ] Virgin-store reads are no-ops; `delete`/absent-key discipline per + the trait contract; `size` metadata-only +- [ ] `cargo test --features mem` green; `cargo clippy --all-features + --all-targets -- -D warnings` and `cargo fmt --check` green + +## References + +- docs/architecture/backends-and-dispatch.md (§kv tier — mem engine) +- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §2 +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 +- src/backend.rs (the trait + seam contracts this implements) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/gc-protocol.md b/tasks/gc-protocol.md new file mode 100644 index 0000000..d43c70f --- /dev/null +++ b/tasks/gc-protocol.md @@ -0,0 +1,120 @@ +--- +id: gc-protocol +name: In-process GC protocol — pins, liveness, delete windows, sweep, direct-delete refusal +status: pending +depends_on: [store-construction] +scope: broad +risk: high +impact: phase +level: implementation +tags: [wave-2, gc, facade] +--- + +## Description + +Implement the in-process GC protocol (ADR-005; abort-as-data per +ADR-012 §1; non-SQL window staging per ADR-012 §5) on the `Store` type +— the architecture's most intricate state machine, in its single-node +form. The fleet extension (DB-backed pins, sweep lease, SQL +arbitration) is wave 6; this task is the protocol the fleet form +extends without changing its invariants. + +- **Pin registry** (ADR-005 §Marks 2): per-key refcounts under the + arbitration lock the sweep consults. `Pin` guards minted by the put + path (`store-put-paths` wires the call sites; the registry and + minting land here); a dedup re-put refcounts the existing entry — a + dedup put's guard protects exactly as a fresh put's does. `Pin` drop + decrements. **Named design point**: the guard's back-reference shape + (how the public `Pin`/`Batch` types reach the registry — the wave-1 + types carry only the key today) and the lock granularity + (`parking_lot` short-held per AGENTS convention 3, poisoning via + `unwrap_or_else(|e| e.into_inner())`). +- **`Batch` machinery**: a batch's pins drop together on batch commit + or batch drop, in one ordered step under the arbitration lock; no + early single-pin release. The **conversion ordering rule** (ADR-005, + pinned in facade.rs docs): a pin's drop and its replacement's + registration into a liveness source are one ordered step under the + same lock — the registry needs the API shape that makes the ordered + step expressible. +- **Registered liveness sources** (`register_liveness_source`, the + live-shared seam — ADR-005 §Decision): **named design point** — the + source trait/callback shape. Live-shared means consulted at mark + time and at delete-time arbitration, never snapshotted at + registration (POC #1 finding 7's install-implies-copy failure is the + pinning reason). +- **Protect callback**: consulted pre-sweep; may add externally-known + hashes or abort the run — abort is report data + (`GcAbortCause::ProtectFailed`), never an error (ADR-012 §1). +- **`sweep() → Result`** — the delete-window + protocol (ADR-005 §Sweep): + 1. **mark**: enumerate the whole pool (both tiers' complete `list()` + unioned); compute the live set from all three sources (registered + sources, pins, protect callback); stage the candidate-dead set. + 2. **delete window**: per candidate, **arbitrate under the pin/ + liveness lock at delete time** (not once upfront) — pinned or + re-observed-live cancels the candidate; otherwise proceed. + 3. **commit**: remaining candidates deleted in batches (~100/batch, + iroh's proven shape); window closed. + - **`Ok(SweepReport)` always** — sweeps are never errors; every abort + is data in `aborted: Option` (exactly three causes; + `NoLivenessSources` is the safe default when zero sources are + registered — nothing deleted). Exact counts in the report fields. + - **Non-SQL window staging** (ADR-012 §5): `local`/`mem` stage + candidates in-process and execute the window protocol **inline in + the same sweep** — there is no durable window table on non-SQL + engines, and the "executor is the next sweep" clause (ADR-012 + §6.7) does not apply to them. + - The test-asserted property (ADR-005): a visible pool entry is never + deleted while liveness protects it, and an in-flight put is never + deleted by a sweep started before it committed — concurrency tests + (sweep racing a pinned put) assert it. +- **Direct `delete(key)`** (wired through the fall-through dispatch — + arbitration-then-delete where the entry lives): refuses **any + protected key** — pinned, registered-source-live, or + protect-callback-named (the full protection set, ADR-012 §6.4) — + with the typed `Error::GcRefuse`, nothing deleted; unprotected keys + delete; absent keys carry `Err(Missing)`. The refusal is decided at + the facade (the altitude discipline: `GcRefuse` never fires below + the facade). +- **`has`/`get` during a window observe no torn state** — existence + flips atomically per key (immutable entries); document the posture + the engines' CAS semantics already give. + +## Acceptance Criteria + +- [ ] Pin registry with per-key refcounts under the arbitration lock; + dedup re-puts refcount; `Pin`/`Batch` drops release through it + (batch drops in one ordered step) +- [ ] `register_liveness_source` live-shared (consulted at mark and at + delete-time arbitration, never snapshotted); source shape decided + and recorded +- [ ] Protect callback with abort-as-data (`ProtectFailed`); + `NoLivenessSources` safe default; `sweep()` returns + `Ok(SweepReport)` on every abort — never an abort error +- [ ] Delete-window protocol: mark → arbitrate-at-delete-time → + batched commit (~100/batch), with exact-count `SweepReport` + fields (`candidates_staged`/`deleted`/`cancelled_by_arbitration`) +- [ ] Direct `delete` refuses the full protection set with `GcRefuse` + and deletes unprotected keys through the fall-through dispatch +- [ ] Concurrency tests: sweep vs in-flight pinned put (the + pin-before-publish property from the sweep side), exact-count + outcomes, abort shapes +- [ ] `cargo test` green; `cargo test --features mem` and `cargo test + --all-features` green; clippy/fmt gates green + +## References + +- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (the protocol + invariants) +- docs/architecture/gc-and-namespaces.md (§Mark-and-sweep GC; §Delete windows) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §1 (abort-as-data), §5 (non-SQL staging), §6.4 (full protection set) +- docs/architecture/store-api.md (§Lifecycle — sweep shape; §Invariants 4/5) +- src/facade.rs (the pinned Pin/Batch/SweepReport lifecycle contracts this wires) +- docs/research/poc-trait-dispatch-findings.md (findings 3/7/8) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/review-wave-2.md b/tasks/review-wave-2.md new file mode 100644 index 0000000..ff66545 --- /dev/null +++ b/tasks/review-wave-2.md @@ -0,0 +1,93 @@ +--- +id: review-wave-2 +name: Wave 2 review gate — facade, dispatch, GC, and the first columns +status: pending +depends_on: [suite-column-local] +scope: narrow +risk: low +impact: phase +level: review +tags: [wave-2, review] +--- + +## Description + +The wave-2 review gate, before wave 3 decomposes. Wave 2 made the +store work end-to-end dependency-free; the gate reads the +implementation against the pinned ADR text line by line (the wave-1 +gate's method). + +Gate checks: + +- **Facade + dispatch + GC vs ADR-003/005/012**: + - the **three numbered unknown-length rules** (ADR-003 §Dispatch + policy) each read against their tests — mid-stream flush with no + restart requirement; verification over the eventual stored artifact + as a whole (buffer form and staged-bytes form); stream error leaves + no entry and no stage file; + - the **delete-window ordering invariants** (ADR-005 §Sweep): + pin-before-publish on both put paths and both tiers; arbitration + under the pin/liveness lock at delete time; the test-asserted + property (a visible entry is never deleted while liveness protects + it; an in-flight put is never deleted by a sweep started before it + committed); + - **abort-as-data** (ADR-012 §1): `sweep()` returns + `Ok(SweepReport)` on every abort; `GcRefuse` is the direct-delete + refusal only; the full protection set per ADR-012 §6.4; + - dispatch as a pure function of length (no migration); fall-through + general rule per ADR-012 §6.1; constructor modes per ADR-011 §4 / + ADR-012 §6.2/§6.3; kv-only put-time rejection. +- **Stage-hygiene exact-count tests**: present and exact on both + engines (mem's degenerate stage; local's real stage form incl. the + Drop-cancellation story per the wave-1 review's observation). +- **The suite's first columns green on mem + local**: rows + ADR-stamped, traceable to the backlog, the factory binding recorded + (factory.rs + README if corrected). +- **The dispatch-coverage shape exercised**: the `kv = mem` + + `large = local` column's dispatch rows (pure-function routing, + fall-through, threshold boundaries). +- **Named design points audited as resolved** (each recorded in its + task's Notes): engine representation (enum vs generic); the public + stream type; list return shape; verification-on-get posture; + liveness-source shape; Pin registry back-reference + lock + granularity; put stream parameter shape; len-mismatch error. +- **Family conventions**: no comments in code (doc comments on public + API fine), thiserror, tokio-only async, `spawn_blocking` for blocking + file work, no unwrap/expect outside tests, naming, module structure. +- **Gates green**: `cargo test`, `cargo test --features mem`, `cargo + test --all-features`, `cargo clippy --all-targets -- -D warnings`, + `cargo clippy --all-features --all-targets -- -D warnings`, `cargo + fmt --check`. + +Findings are fixed inline where cheap, or recorded as wave-3 +amendments where they shape the next decomposition. The gate's +completion is what unblocks wave 3's decomposition. + +## Acceptance Criteria + +- [ ] Every gate check above read and verified against the cited ADR + text (not from memory of it) +- [ ] Named design points each decided, recorded, and consistent with + the specs +- [ ] Findings fixed inline or recorded as wave-3 amendments with + owners +- [ ] All gates re-run green after any fixes +- [ ] Wave 3 decomposition unblocked (recorded in this task's Summary) + +## References + +- docs/plans/implementation.md (§Wave 2, §Review gates) +- docs/architecture/decisions/003-backend-contract-and-dispatch.md +- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md +- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md +- docs/architecture/store-api.md +- tasks/review-wave-1.md (the gate method; the observations wave 2 owned) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/store-construction.md b/tasks/store-construction.md new file mode 100644 index 0000000..c5a4b9c --- /dev/null +++ b/tasks/store-construction.md @@ -0,0 +1,94 @@ +--- +id: store-construction +name: Store facade type — constructor modes and size-threshold dispatch +status: pending +depends_on: [engine-mem, engine-local] +scope: moderate +risk: medium +impact: project +level: implementation +tags: [wave-2, facade, dispatch] +--- + +## Description + +Introduce the `Store` facade type (store-api.md §What this is) — the +public surface consumers program against — with its constructor modes +and the size-threshold dispatch. This task builds the *skeleton and +routing*: the ops methods land in `store-read-paths`, `gc-protocol`, +and `store-put-paths` against this type. + +- **The `Store` type** (public, `src/facade.rs` or a new module — + implementer's call, one module per file): holds the kv-tier engine + and the large-tier engine (or none, per mode) plus the GC state home + the `gc-protocol` task fills. **Named design point — engine + representation**: a concrete `Store` with a crate-internal enum + dispatching over the closed five-engine set, vs a generic + `Store`. The conformance factory's + binding note (tests/conformance/factory.rs) anticipates the concrete + type ("most plausibly the associated type collapsing to the concrete + facade type across all factories"); engines are statically known and + in-crate (no `dyn` anywhere, wave-1 design point 3). Decide, record, + and keep the public surface engine-blind (ADR-007/008/010: which + engines a node runs changes nothing here). +- **Constructor modes** (ADR-011 §4; ADR-012 §6.3): + - **dual-tier** (the default): kv + large, size-threshold dispatch. + Wave-2 constructible shape: `kv = mem` + `large = local` — the + dispatch-coverage test shape ADR-012 §6.3 names (both tiers + load-bearing). + - **kv-only**: no large tier; over-threshold puts reject at put time + (ADR-012 §6.2 — the rejection behavior itself lands in + `store-put-paths`; this task carries the mode and its validation). + - **mem-only**: the mem engine alone, no dispatch threshold in + effect; never production. + - Mode machinery is built so the wave-3–5 engine constructors slot in + without reshaping (sqlite/postgres/pg-lo constructors are later + waves; only feature-gated constructors exist when their feature is + on). +- **Size-threshold dispatch** (ADR-003): routing as a **pure function + of content length** — same content ⇒ same length ⇒ same tier, no + content ever migrates. Default threshold **128 KiB** + (constructor-tunable; the conservative edge of the measured + ~128–256 KiB crossover zone, POC #3 A4). The dispatch function is + pure and unit-tested (boundary values: exactly-at-threshold routing). +- **The dispatch seam** the ops tasks program through: tier selection + for a given length, and the fall-through order (kv first, then large) + the read paths and delete consume (ADR-012 §6.1's general rule). +- **Conformance factory binding**: bind `StoreFactory::Store` to the + real facade type (the skeleton's anticipated collapse to the concrete + type); if binding reveals a shape correction, make it and record it + in factory.rs + tests/conformance/README.md per the skeleton's note. + +## Acceptance Criteria + +- [ ] Public `Store` type with the three constructor modes; engine + representation decided and recorded (enum-dispatch vs generic) + with the public surface engine-blind +- [ ] Size-threshold dispatch: pure function of length, default 128 KiB, + constructor-tunable; boundary-value unit tests (at/below/above + threshold route deterministically) +- [ ] `kv = mem` + `large = local` dual-tier construction works (the + ADR-012 §6.3 dispatch-coverage shape); kv-only and mem-only + modes construct and validate +- [ ] `StoreFactory::Store` bound to the real facade; any shape + correction recorded in factory.rs + README +- [ ] `cargo test` green (default build: local ungated); `cargo test + --features mem` and `cargo test --all-features` green; + clippy/fmt gates green in all three feature shapes + +## References + +- docs/architecture/store-api.md (§What this is; §Public surface) +- docs/architecture/backends-and-dispatch.md (§Size-threshold dispatch; §Constructor modes) +- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy) +- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §4 +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.2/§6.3 +- tests/conformance/factory.rs (the binding note) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/store-put-paths.md b/tasks/store-put-paths.md new file mode 100644 index 0000000..7ce31a7 --- /dev/null +++ b/tasks/store-put-paths.md @@ -0,0 +1,103 @@ +--- +id: store-put-paths +name: Store put seam — two internal paths, the three numbered rules, Batch, kv-only rejection +status: pending +depends_on: [gc-protocol] +scope: moderate +risk: medium +impact: component +level: implementation +tags: [wave-2, facade, write-path] +--- + +## Description + +Implement the facade's put seam on the `Store` type (store-api.md +§Write path): **`put(len: Option, stream) → (Key, Pin)`** — two +internal paths behind one signature, with pin-before-publish from the +`gc-protocol` registry. + +- **Known-length path** (encouraged, documented as such on the API): + the preamble (`"blob \0"`) is known before content flows, so + the hash initializes with it and one pass suffices — no staging + needed for hashing. Routing by the dispatch seam: ≤ threshold → the + kv tier's whole-value put; > threshold → the large tier's staged put + (content streams into the staging cursor while hashing; the + derivation completes at stream end; commit-rename publishes). +- **Unknown-length path**: content buffers in memory up to the + dispatch threshold. ADR-003's **three numbered rules** (option (d)), + each a test-asserted behavior: + 1. **Threshold exceeded mid-stream** (non-restartable sources are + normal): the buffered prefix flushes *into a fresh stage file* + and the stream continues appending into it — there is no + "restart the stream" requirement anywhere. + 2. **Verification runs over the eventual stored artifact as a + whole**: the memory buffer for the sub-threshold kv commit; the + complete staged bytes for the large commit — the derivation pass + restarts over preamble + staged bytes (forced by the preamble + being inside the hashed input, ADR-002), reading the staged + artifact back through the `StagingCursor: ReadCursor` seam (the + wave-1 design-led extension). There is no stitched-input case + left ambiguous. + 3. **Stream error before commit**: no entry lands and no stage file + remains — buffer discard on the failure path, exactly like stage + discard (the stage-hygiene invariant extends to the buffered + case). +- **Pin-before-publish** (ADR-005): the pin refcount is acquired under + the arbitration lock *before* the entry becomes visible — publish + and protection are one step; an entry cannot appear in a sweep's + `list()` without already carrying its pin. +- **Dedup**: a re-put of existing content acquires no new pool entry — + the returned `Pin` refcounts the existing one under the same lock. +- **`Batch`** (store-api.md §Write path): the batching put seam — + multi-put writes group under one batch-scoped pin; member puts + publish pin-before-publish exactly as lone puts do; the batch's pins + drop together on commit or drop (the `gc-protocol` machinery drives). +- **kv-only mode rejection** (ADR-012 §6.2): known-length puts + exceeding the threshold reject **immediately**; unknown-length puts + reject **at mid-stream threshold overflow** — lengths are not known + at construction, so put time is the only possible rejection point. + Zero residue on either rejection. +- **Named design points**: (a) the stream parameter shape (`impl + AsyncRead + Unpin + Send` vs a boxed trait object — the family's + no-dyn posture vs ergonomics); (b) the declared-len mismatch case + (a known-length stream that yields fewer/more bytes than `len` — + which error family member carries it, and zero residue either way). + Record both for the review gate. + +## Acceptance Criteria + +- [ ] `put(len, stream) → (Key, Pin)` with both internal paths behind + the one signature; routing is the pure length function (no + content ever migrates tiers) +- [ ] The three numbered rules each behavior-tested: mid-stream + overflow flushes into a fresh stage and the stream continues; + the derivation pass runs over the eventual stored artifact as a + whole (buffer form and staged-bytes form); stream error leaves + no entry and no stage file +- [ ] Pin-before-publish holds on both paths and both tiers; dedup + re-puts refcount the existing entry +- [ ] `Batch` groups multi-put writes; member pins drop together +- [ ] kv-only mode: known-length over-threshold rejects immediately; + unknown-length rejects at mid-stream overflow; zero residue on + both +- [ ] Stream-shape and len-mismatch design points decided and recorded +- [ ] `cargo test` green; `cargo test --features mem` and `cargo test + --all-features` green; clippy/fmt gates green + +## References + +- docs/architecture/store-api.md (§Write path — the put seam, stage hygiene invariant) +- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy — the three numbered rules, option (d)) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.2 (kv-only put-time rejection) +- docs/architecture/decisions/005-pooled-cas-and-gc-mechanism.md (pin-before-publish, dedup refcount) +- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract) +- src/backend.rs (StagingCursor: ReadCursor — the verification-pass seam) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/store-read-paths.md b/tasks/store-read-paths.md new file mode 100644 index 0000000..23fb036 --- /dev/null +++ b/tasks/store-read-paths.md @@ -0,0 +1,88 @@ +--- +id: store-read-paths +name: Store read paths — get/stat/read_range/has/list with dispatch fall-through +status: pending +depends_on: [store-construction] +scope: moderate +risk: medium +impact: component +level: implementation +tags: [wave-2, facade, read-path] +--- + +## Description + +Implement the facade's read-side ops on the `Store` type (store-api.md +§Read path, §Lifecycle), behind the dispatch seam from +`store-construction`. Direct `delete` is *not* here — it lands in +`gc-protocol` with its arbitration. + +- **The fall-through general rule** (ADR-012 §6.1; POC #1 finding 6): + every key-addressed op consults the kv tier first, then the large + tier. `get`/`stat`/`has`/`read_range`/`list` all follow it. +- **`get(key) → Option<(len, stream)>`**: small tier returns a bounded + read; large tier streams. The per-engine cursor shapes stay hidden + behind the crate-internal `ReadCursor` the store core drives + (ADR-010 §1) — **named design point**: the public stream type that + wraps the cursor (the facade's `(len, stream)` shape is uniform + across tiers; the cursor is never exposed). A miss on both tiers is + `None` — `get` never carries `Error::Missing` (ADR-012 §6.5 ruling 5; + the error module's behavioral-enforcement note lands here). +- **`stat(key) → Option`**: the cheap probe riding the + trait's `size` method — length plus algorithm, never a content load + (ADR-008's size discipline). Fall-through per the general rule. +- **`read_range(key, range) → (slice, slice_digest)`**: slice plus the + out-of-band SHA-256 slice digest (ADR-006, fixed by convention — + composed through the cursor's `read_range` seam). `Err(Missing)` on a + both-tier miss. Range reads fall through tiers exactly as `get` does. +- **`has(key) → bool`**: the pure existence probe (ADR-008's `has` + discipline) — a size-probe under the fall-through rule; absent on + both tiers is `false`; virgin-store reads are no-ops. +- **`list() → stream of keys`**: whole-pool enumeration, complete by + contract. **Named design point**: the return shape — store-api.md + says "stream of keys"; decide Vec vs stream (the sweep consumes the + same enumeration crate-internally) and record it. +- **Verification posture on get** (named design point): ADR-006's + whole-blob verification vs the slice-digest-only reading — where the + store core's hash-check actually fires on the read path (kv + materialized reads are verifiable cheaply; large streaming reads ride + per-range slice digests). Pin the reading against ADR-006's text and + record it for the review gate. +- **`KeyInvalid` at the boundary**: typed `Key` conversion happens at + the store boundary only (ADR-002/003) — malformed key bytes on any + read op reject with `Error::KeyInvalid`, never a guess; engines never + see typed keys. + +## Acceptance Criteria + +- [ ] `get`/`stat`/`read_range`/`has`/`list` implemented with the + fall-through general rule (kv first, then large), unit-tested + including the both-tier-miss shapes (`None` vs `Err(Missing)` per + op) +- [ ] The public stream type decided and recorded; engine cursors never + leak into the public surface +- [ ] `stat` never fetches content (rides `size`); `EntryMeta` carries + len + algorithm +- [ ] `read_range` returns slice + SHA-256 slice digest per the ADR-006 + convention; verification-on-get posture decided and recorded +- [ ] Malformed key bytes reject with `KeyInvalid` at the facade + boundary on every key-addressed op +- [ ] `cargo test` green; `cargo test --features mem` and `cargo test + --all-features` green; clippy/fmt gates green + +## References + +- docs/architecture/store-api.md (§Read path, §Lifecycle, §Error model) +- docs/architecture/backends-and-dispatch.md (§Size-threshold dispatch — get fall-through) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.1/§6.5 +- docs/architecture/decisions/006-verification-posture-and-transfer-encoding.md +- docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md (size discipline) +- src/backend.rs (the ReadCursor seam this drives) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/suite-column-local.md b/tasks/suite-column-local.md new file mode 100644 index 0000000..91e88f2 --- /dev/null +++ b/tasks/suite-column-local.md @@ -0,0 +1,80 @@ +--- +id: suite-column-local +name: Conformance suite — the local column and the dispatch-coverage shape +status: pending +depends_on: [suite-column-mem] +scope: moderate +risk: medium +impact: phase +level: implementation +tags: [wave-2, suite, local, dispatch] +--- + +## Description + +Grow the **local column** — the suite's first dual-tier column, run +against the `kv = mem` + `large = local` factory: the +**dispatch-coverage test shape** ADR-012 §6.3 names (both tiers +load-bearing). Rows mirror the mem column's set (INV-7: every engine's +rows mirror the reference column), extended with the dual-tier and +local-specific properties. + +- **The factory**: fresh, isolated temp-root dual-tier store per call + (`kv = mem` + `large = local`), idempotent teardown (temp dir + removal); never skips. +- **Mirrored rows** (the mem column's set, against the dual-tier + store): + - **INV-1 round-trips**: both tiers, both put paths — the content + matrix crosses the threshold so both tiers' bodies are exercised + (sub-threshold kv rows; over-threshold local stage-commit rows). + - **INV-3 stage hygiene**: exact counts **on disk** — zero stage + files after every failure form, exactly one committed entry per + success, over both put paths (the real stage form, not the + degenerate kv stage). + - **INV-4 sweep safety**: exact-count sweep outcomes over the + dual-tier pool (candidates from both tiers; abort shapes; the + sweep-vs-pinned-put race). + - **INV-2 dedup, INV-5 list-through-sweep, INV-6 stat** mirrored. +- **Dispatch rows** (the coverage shape's reason to exist): + - Routing is a pure function of length: same content ⇒ same tier + across mixed put paths and re-puts; no content ever migrates. + - **Fall-through** (ADR-012 §6.1): a kv-tier miss queries the large + tier — `get`/`stat`/`has`/`read_range` each served from the large + tier when the kv tier misses; both-tier miss shapes per op. + - Threshold boundary values (at/below/above the default; a + constructor-tuned threshold re-routes deterministically). +- **Local-specific rows**: pread range reads over committed files + (mid-file slices, EOF clamping); virgin-store no-ops on a fresh root + (empty `list`, absent reads); `size` as a stat (never a content + load). +- Every row ADR-stamped per the convention, tracing to the backlog. + +## Acceptance Criteria + +- [ ] The dual-tier `kv = mem` + `large = local` factory green with + isolation + idempotent teardown +- [ ] The local column green: mem's row set mirrored (INV-1/2/3/4/5/6) + plus dispatch rows (pure-function routing, fall-through per op, + threshold boundaries) and local-specific rows (pread ranges, + virgin-store no-ops, stat-as-stat) +- [ ] Stage-hygiene exact counts asserted on disk (the real stage form) +- [ ] Every row ADR-stamped and traceable to the backlog +- [ ] `cargo test` green (local is ungated — the column runs in the + default build); clippy/fmt gates green + +## References + +- tests/conformance/README.md + backlog.rs (the conventions; INV first-column plan) +- docs/architecture/backends-and-dispatch.md (§Constructor modes — the dispatch-coverage shape; §Size-threshold dispatch) +- docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §6.1/§6.3 +- docs/architecture/decisions/003-backend-contract-and-dispatch.md (§Dispatch policy) +- docs/architecture/store-api.md (§Invariants) +- docs/research/poc-largeblob-findings.md (findings A2/A5) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file diff --git a/tasks/suite-column-mem.md b/tasks/suite-column-mem.md new file mode 100644 index 0000000..9f18db2 --- /dev/null +++ b/tasks/suite-column-mem.md @@ -0,0 +1,83 @@ +--- +id: suite-column-mem +name: Conformance suite — factory binding and the mem column (the reference column) +status: pending +depends_on: [store-read-paths, store-put-paths, gc-protocol] +scope: moderate +risk: medium +impact: phase +level: implementation +tags: [wave-2, suite, mem] +--- + +## Description + +Bind the conformance harness to the real facade and grow the **mem +column** — the contract-reference column that defines the required set +every other engine's rows mirror (store-api.md invariant 7; ADR-011 §2). + +- **The factory binding** (the skeleton's anticipated wave-2 moment): + bind `StoreFactory::Store` to the real facade type — most plausibly + the associated type collapsing to the concrete `Store` across all + factories. If binding reveals a shape correction, make it and record + it in `tests/conformance/factory.rs` + `README.md` per the + skeleton's note. The mem factory vends fresh, isolated in-memory + stores (no shared state between rows) with idempotent teardown; mem + never skips (README §SKIP posture — SKIP machinery stays unneeded + until the SQL waves). +- **The mem column's rows**, each ADR-stamped per the convention + (`Contract stamp:` citing the pinning spec §/ADR §), tracing to the + backlog entries: + - **INV-1 round-trips**: whole-blob put/get byte-identical under the + canonical derivation, over a content matrix looping both put paths + (known-length, unknown-length) in-body; the git-interop test + remains the derivation's source of truth. + - **INV-2 dedup**: identical content twice (same and different put + path) is one pool entry; the re-put's `Pin` protects exactly as a + fresh put's. + - **INV-3 stage hygiene**: exact counts over both put paths and both + failure modes (explicit discard, failed commit) — zero residue on + failure, exactly one committed entry on success (the degenerate + kv stage form). + - **INV-4 sweep safety**: single-node exact-count family — happy + shape (exact `SweepReport` counts), abort shapes + (`ProtectFailed`, `NoLivenessSources` — nothing deleted, + `Ok(SweepReport)` always), and the sweep-vs-pinned-put race. + - **INV-5 list correctness**: asserted through sweep outcomes (exact + counts), never by comparing enumerated lists directly. + - **INV-6 stat discipline**: `stat` returns length + algorithm + without fetching content. + - **Constructor-mode rows** (facade contract, ADR-011 §4 / ADR-012 + §6.2/§6.3): mem-only mode; kv-only(mem) over-threshold put-time + rejection (known-length immediate, unknown-length mid-stream). +- Rows take factories, never stores (factory.rs contract); a failing + assertion panics the row; teardown is idempotent against that path. + +## Acceptance Criteria + +- [ ] `StoreFactory::Store` bound to the real facade; any shape + correction recorded in factory.rs + README +- [ ] The mem column green: INV-1/2/3/4/5/6 rows plus constructor-mode + rows, every row ADR-stamped and traceable to the backlog +- [ ] The mem column's row set is the reference set (INV-7's basis) — + recorded in README as what other columns mirror +- [ ] Factory isolation + idempotent teardown hold (fresh store per + call; teardown safe against a panicking row) +- [ ] `cargo test --features mem` green (the column runs under its + feature); clippy/fmt gates green + +## References + +- tests/conformance/README.md (the conventions this grows against) +- tests/conformance/backlog.rs (INV-1…7 entries and their first-column plan) +- docs/architecture/store-api.md (§Invariants — the test gate) +- docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md §2 (mem's reference role) +- docs/plans/implementation.md (Decided points — suite home; §Wave 2 — suite columns) + +## Notes + +> To be filled by implementation agent + +## Summary + +> To be filled on completion \ No newline at end of file