docs(plans): implementation plan — wave-based decomposition; wave 1 decomposed into tasks/
- docs/plans/implementation.md: the meta plan (alkstore pattern) — nine waves, dependency logic, decided points, review gates. Scope authority: v0.1.0 ships the full surface (all five engines, fleet GC, ops). - Wave 1 decomposed: scaffold-crate-features, core-error-model, core-hashing-keys, core-trait-and-types, suite-scaffold, review-wave-1 (the one-way-door gate). - Waves 2-9 shaped in the plan; each decomposes after the prior wave's review gate. Verification: cargo test / clippy -D warnings / fmt --check green (docs-only change).
This commit is contained in:
glm-5.3-flash committed
1 parent
6f6ab19b8e
commit
8a2df4fa57
7 files changed
+829
No files matched your search
@@ -0,0 +1,316 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: "2026-10-11 (plan created; wave 1 decomposed into tasks/)"
|
||||
---
|
||||
|
||||
# alkblobs — Implementation plan
|
||||
|
||||
Wave-based decomposition of the architecture (`docs/architecture/`)
|
||||
into units of implementable work. This is a deliberate deviation from
|
||||
the SDD process's decompose-everything-upfront step, adopting the
|
||||
sibling pattern proven on alkstore: the architecture is large, early
|
||||
waves change the shape of later ones (the seam shapes wave 1 pins are
|
||||
what every engine task implements against; the conformance suite's
|
||||
factory contract feeds every engine's verification work), and
|
||||
decomposing only the next wave at a time keeps each session's task set
|
||||
reviewable and lets later decompositions absorb earlier waves' course
|
||||
corrections.
|
||||
|
||||
Rhythm: decompose a wave → implement it → review gate → decompose the
|
||||
next wave. Task files live in `tasks/` (frontmatter carries the
|
||||
categorical estimates). Wave boundaries are also review boundaries.
|
||||
|
||||
Scope authority: **v0.1.0 ships the full surface** — all five engines
|
||||
(kv: sqlite, postgres, mem; large: local, pg-lo), fleet GC, and the
|
||||
feature-gated `ops` module — per operator decision (2026-10-11): all
|
||||
downstream consumers need all of it, and the base crate must be fully
|
||||
usable and composable across mem/sqlite/postgres from the default
|
||||
build onward.
|
||||
|
||||
## The waves
|
||||
|
||||
Dependency logic in one line: core contract → (mem ∥ local) → sqlite
|
||||
→ postgres → pg-lo → fleet GC → ops → fuzzing → release readiness.
|
||||
The two dependency-free engines (mem, local) prove the entire facade +
|
||||
dispatch + in-process GC story before any SQL driver exists; sqlite
|
||||
and postgres are structurally independent of each other (both need
|
||||
only wave 2's trait + suite) and could run in either order — postgres
|
||||
is placed first of the two SQL engines because its wave establishes
|
||||
the full engine-state seam clause families that pg-lo (window family)
|
||||
and the fleet wave (the host) build on. pg-lo shares postgres's
|
||||
driver cluster and seam pattern. The fleet wave needs both SQL tiers
|
||||
landed: its canonical testable shape is kv=postgres + large=pg-lo on
|
||||
one instance (the composite fleet-validity predicate's joint
|
||||
entry+pin tx is expressible exactly there). The ops wave rides
|
||||
alkcall (published, 0.8.1) behind the default-off feature. Fuzzing
|
||||
follows the ops wave so its targets cover the key encoding *and* the
|
||||
ops payload surfaces; release readiness is last.
|
||||
|
||||
| Wave | Contents | Depends on | Status |
|
||||
|---|---|---|---|
|
||||
| [1](#wave-1--foundations) | Crate/feature scaffold; hashing & keys (the one-way door); error model; Backend trait + I/O seams + engine-state seam shape; facade value types; conformance-suite convention | — | decomposed (2026-10-11) |
|
||||
| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | pending |
|
||||
| [3](#wave-3--sqlite-engine) | sqlite kv engine (default): WAL posture, prepared statements, engine-state window family, suite column | wave 2 | pending |
|
||||
| [4](#wave-4--postgres-kv-engine) | postgres kv engine: schema, pool, CAS rows, stable-cursor list, full engine-state clause families, suite column | wave 2 | pending |
|
||||
| [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | pending |
|
||||
| [6](#wave-6--fleet-gc) | Fleet GC mechanism: DB-backed pins, joint entry+pin tx, sweep lease, SQL delete arbitration, fleet-validity predicate + constructor declarations | waves 4 + 5 | pending |
|
||||
| [7](#wave-7--ops-surface) | `ops` feature: alkcall-backed op family, ACL mapping, pin tokens + renewal, fanout seam | wave 6 | pending |
|
||||
| [8](#wave-8--fuzzing) | Fuzzing: family pattern targets (key encoding, ops payloads), corpus-replay gate | wave 7 | pending |
|
||||
| [9](#wave-9--release-readiness) | Crate docs, feature-graph verification, README, publish prep | wave 8 | pending |
|
||||
|
||||
## Wave 1 — Foundations
|
||||
|
||||
The contract surface in code, engine-blind: everything later waves
|
||||
implement *against*, and nothing that needs an engine to test.
|
||||
|
||||
- **Crate/feature scaffold** — the ADR-013 §4 feature graph declared
|
||||
in the manifest from the first commit: `sqlite` default-on
|
||||
(rusqlite), `postgres` (tokio-postgres + deadpool-postgres), `mem`
|
||||
(dependency-free), `pg-lo` (the shared pg cluster; does **not**
|
||||
imply `postgres` — ADR-010 §1's consequence), `ops` (alkcall,
|
||||
default-off); `local` ungated. Engine modules ride `#[cfg(feature)]`
|
||||
from the start.
|
||||
- **Hashing & keys** (ADR-002 — the one-way door): `Digest`/`Key`
|
||||
types, the canonical `git-blob-sha-256` derivation
|
||||
(`H("blob <len>\0" + content)`), tolerated `git-blob-sha-1`, the
|
||||
algorithm-tagged byte encoding (1 alg byte + digest; 0x01/0x02
|
||||
assigned, 0x00 reserved), rejects-never-guesses validation, and the
|
||||
byte-exact `git hash-object` interop test (POC #1 findings 4/5's
|
||||
standard — validated against the git CLI, not just vectors).
|
||||
- **Error model** (store-api.md; ADR-012 §6.5): the one-enum error
|
||||
family with its altitude discipline — the trait's effective members
|
||||
(`Missing`/`Verification`/`Io`/`KeyInvalid`) and the facade's
|
||||
`GcRefuse` addition.
|
||||
- **Backend trait + I/O seams** (ADR-003/010): the trait method set
|
||||
(`has`/`get`/`put`/`delete`/`list`/`name`/`size`) with opaque byte
|
||||
keys; the read-cursor abstraction (kv materializing vs large
|
||||
streaming bodies; serves `read_range`'s slice + slice-digest
|
||||
contract at one seam); the staged-put contract as a named per-engine
|
||||
shape (staging cursor handout, commit publishes, every failure path
|
||||
discards); and the crate-internal engine-state companion trait with
|
||||
its clause families and per-engine applicability (ADR-010 §2 as
|
||||
reduced by ADR-012 §5) — SQL engines implement it in waves 3–5;
|
||||
`local`/`mem` are excluded by construction.
|
||||
- **Facade value types**: `EntryMeta`, `Pin`/`Batch` (types + pinned
|
||||
lifecycle contract; the arbitration machinery is wave 2),
|
||||
`SweepReport`/`GcAbortCause` (abort-as-data, exactly three causes),
|
||||
`PinState`.
|
||||
- **Conformance-suite convention** — `tests/conformance/` established
|
||||
with the row/stamp convention (ADR-§-stamped rows, the alkstore
|
||||
ADR-017 pattern adapted) and the factory contract (isolation +
|
||||
idempotent teardown); the invariant backlog from store-api.md
|
||||
registered as rows-to-grow. The compile-able harness lands with
|
||||
wave 2's first column (mem — the contract-reference engine).
|
||||
|
||||
The wave-1 review gate is the cheap-place gate: the key encoding is
|
||||
the declared one-way door (AGENTS convention 5) — a shape error found
|
||||
at this gate costs a task revision; found after an engine ships data
|
||||
it costs a migration ADR. The gate reads the encoding, the trait, and
|
||||
the seam shapes against the pinned ADR text line by line.
|
||||
|
||||
## Wave 2 — The store works dependency-free (mem + local)
|
||||
|
||||
The first engines and the facade behavior, chosen so the whole store
|
||||
story is provable with zero driver dependencies:
|
||||
|
||||
- **mem kv engine** (feature `mem`) — the contract-reference engine
|
||||
(ADR-011 §2): a `BTreeMap`-shaped implementation of the kv tier's
|
||||
contract including `size`; every other engine's conformance rows
|
||||
mirror its column.
|
||||
- **local large engine** (ungated) — the ADR-003 backend as specced:
|
||||
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding, stage-then-
|
||||
commit-rename (the staged-put contract realized with its exact-count
|
||||
hygiene tests, POC #3 A5's shape), pread-loop read cursor, pread
|
||||
range reads.
|
||||
- **Facade behavior** — the put seam's two internal paths behind one
|
||||
signature (known-length one-pass; unknown-length pre-threshold
|
||||
buffering with the mid-stream stage flush and the restart-over-
|
||||
staged-bytes derivation pass, ADR-003's option (d) semantics pinned
|
||||
in three numbered rules); `get`/`stat`/`read_range`/`has`/`delete`/
|
||||
`list` with the dispatch fall-through general rule (ADR-012 §6.1);
|
||||
size-threshold dispatch (pure function of length, default 128 KiB,
|
||||
constructor-tunable); constructor modes (dual-tier / kv-only with
|
||||
put-time rejection per ADR-012 §6.2 / mem-only) — `kv=mem +
|
||||
large=local` is the dispatch-coverage test shape ADR-012 §6.3 names.
|
||||
- **In-process GC protocol** (ADR-005) — pins with pin-before-publish
|
||||
ordering, registered liveness sources (live-shared), the protect
|
||||
callback with abort-as-data, delete windows (mark → arbitrate-at-
|
||||
delete-time → batched commit), `sweep()` returning `Ok(SweepReport)`
|
||||
always, direct-delete refusal (`GcRefuse`, the full protection set
|
||||
per ADR-012 §6.4), `local`/`mem` staging window candidates
|
||||
in-process and executing them inline in the same sweep (ADR-012 §5).
|
||||
- **Suite columns** — the harness compile-able and green on the mem
|
||||
and local columns: round-trips, dedup, stage hygiene, sweep-safety
|
||||
exact counts, virgin-store no-ops, `stat`-without-content.
|
||||
|
||||
Wave 2 decomposes after the wave-1 review gate, absorbing whatever the
|
||||
gate corrects.
|
||||
|
||||
## Wave 3 — sqlite engine
|
||||
|
||||
The default kv engine (feature `sqlite`, default-on; REQ-1): WAL +
|
||||
`synchronous=NORMAL` as the shipped durability tier, prepared
|
||||
statements, `has` as an EXISTS probe, the indexed `size` column probe
|
||||
(POC #5 B5's row shape), virgin-store no-tables-yet tolerance, and the
|
||||
engine-state seam's **window family only** (ADR-012 §5's reduction —
|
||||
durable single-node window staging; no pin rows, no sweep-lock body).
|
||||
Suite column green server-less. The single-writer ceiling is a
|
||||
documented posture, not a defect (REQ-1's economics).
|
||||
|
||||
## Wave 4 — postgres kv engine
|
||||
|
||||
The fleet-capable kv engine (feature `postgres`; REQ-2/REQ-3): schema
|
||||
bootstrap, pooled connections (tokio-postgres + deadpool-postgres)
|
||||
with per-connection prepared-statement discipline, `ON CONFLICT DO
|
||||
NOTHING` CAS rows, complete `list()` as a stable cursor over a
|
||||
vacuuming table, batch-delete via `key = ANY($1)`, the
|
||||
`synchronous_commit` parity knob, and the engine-state seam's **full
|
||||
clause families** — postgres is the fleet host (ADR-012 §4):
|
||||
`ensure_gc_schema` (pins + window), `pin_commit`, `pin_renew`,
|
||||
`pin_check`/`arbitrate`, `window_stage`/`window_take`,
|
||||
`sweep_lock_acquire`/`release` (the pg advisory lock, the sweep
|
||||
lease's shipped provider). Suite column green against dockerized
|
||||
postgres (the alkstore harness convention: connection settings ride
|
||||
environment, tests skip cleanly server-less, acceptance requires green
|
||||
against the server). This wave establishes the seam implementation
|
||||
pattern pg-lo reuses.
|
||||
|
||||
## Wave 5 — pg-lo large engine
|
||||
|
||||
The large tier's second engine (feature `pg-lo`; ADR-009, admitted on
|
||||
POC #7): Large Objects as storage — staged put as chunked `lowrite`
|
||||
statements (512 KiB per the POC's measured knee) inside a tx the store
|
||||
core holds, descriptorless `lo_get(oid, off, len)` window reads over
|
||||
the companion table (the C4 posture), companion-row `size` probes,
|
||||
transactional LO creation (zero crash orphans), and the seam's window
|
||||
family (durable window staging where the kv engine is not SQL-backed —
|
||||
the one-window-host rule). Suite column green against dockerized
|
||||
postgres. Named deltas documented (catalog space never returned;
|
||||
`VACUUM FULL` is the operator's shrink path).
|
||||
|
||||
## Wave 6 — Fleet GC
|
||||
|
||||
The mechanism wave (ADR-008/010/012/013; REQ-2): DB-backed pins as
|
||||
expiry-carrying rows committed atomically with entries — the store
|
||||
core holds the joint entry+pin tx and engines contribute mechanics
|
||||
(`pin_commit`); the named exclusive sweep lease (one sweeper per pool,
|
||||
atomic test-and-set, holdership ends at release/owner death/session
|
||||
loss; contention is the `SweeperLock` report abort); SQL-level delete
|
||||
arbitration (`SELECT … FOR UPDATE` finger-lock at delete time); TTL/3
|
||||
pin renewal on the guard; the composite fleet-validity predicate and
|
||||
the explicit `fleet: true` constructor declaration (never inferred);
|
||||
kv-only re-routing posture; the one-window-host rule under dual-SQL
|
||||
instances. Exact-count fleet sweep outcomes against live postgres —
|
||||
the wave's gate is the architecture's most intricate state machine
|
||||
proven under concurrency.
|
||||
|
||||
## Wave 7 — Ops surface
|
||||
|
||||
The `ops` feature (ADR-001; alkcall 0.8.1): the op family —
|
||||
`blobs/stat`, `blobs/have` (with the token-renewal form),
|
||||
`blobs/delete` (internal authority), `blobs/fetch` (Sub), `blobs/put`
|
||||
(Sink, response `{token, digest}`) — JSON control plane via alkcall
|
||||
`OperationSpec` + JSON Schema validation, binary data plane via
|
||||
`channel_open`-allocated Sub/Sink channels; namespace-as-resource ACL
|
||||
mapping (closed-by-default per OQ-09); the pin hand-over + token
|
||||
renewal contract (the token is the pin; TTL/3 renewal via have/re-put;
|
||||
validity domain = minting node); the fetch fanout seam (one reader,
|
||||
store arm + subscriber arms; late joiners degrade to verified `get`).
|
||||
The module is a store-facade consumer with no backend side-door.
|
||||
|
||||
## Wave 8 — Fuzzing
|
||||
|
||||
The family pattern (alkcall/alktunnels/alksocks/alktty/alkstore), per
|
||||
operator standing decision that anything worth fuzzing gets fuzzed:
|
||||
the `fuzz/` sidecar workspace (nightly pinned inside `fuzz/` only, the
|
||||
stable `fuzz/shared` corpus-replay gate, the detached runner), targets
|
||||
over the key-encoding parser (`Key::from_bytes` — the one-way door's
|
||||
reject-never-guess discipline), the ops payload surfaces (JSON specs +
|
||||
binary channel framing, if wave 7's shapes warrant it), and any
|
||||
put-path staging logic the surface review names. Committed crash-input
|
||||
seeds; grown corpora gitignored; the corpus-replay gate joins the
|
||||
verification list when the wave lands. Targets are chosen at
|
||||
decomposition time against the then-current surface — the wave exists
|
||||
in the plan as a commitment, not yet a target inventory.
|
||||
|
||||
## Wave 9 — Release readiness
|
||||
|
||||
Crate docs (public-API rustdoc warning-free; the facade's docs are
|
||||
contract text), the feature-graph verification pass (every coherent
|
||||
engine-feature combination compiles and passes its gates — pg-pure
|
||||
builds, `pg-lo` without `postgres`, `--all-features`), the root README
|
||||
(written last, honestly), publish prep (package metadata, `cargo
|
||||
package` verification per the exclude list, path-dep/version bounds),
|
||||
and the pre-release review gate flipping the release-facing specs to
|
||||
`stable`.
|
||||
|
||||
## Decided points
|
||||
|
||||
- **Wave-based decomposition** (operator-endorsed, 2026-10-11): the
|
||||
SDD decompose-everything-upfront step is replaced by the
|
||||
alkstore-proven rhythm — decompose the next wave after the previous
|
||||
wave's review gate; later decompositions absorb earlier outcomes.
|
||||
- **v0.1.0 scope — full surface** (operator, 2026-10-11): all five
|
||||
engines, fleet GC, and the `ops` module ship in the first release;
|
||||
the base crate is fully usable and composable across
|
||||
mem/sqlite/postgres by default.
|
||||
- **Engine order**: mem + local first (the contract-reference engine
|
||||
plus the ungated engine — the dependency-free pair proves facade,
|
||||
dispatch, and in-process GC end-to-end, and `kv=mem + large=local`
|
||||
is the ADR-012 §6.3 dispatch-coverage shape); then sqlite (default),
|
||||
postgres (establishes the full seam), pg-lo (shares the pg cluster
|
||||
+ seam pattern), fleet GC last of the storage waves (its canonical
|
||||
testable shape needs both SQL tiers).
|
||||
- **Contract-suite home**: in-crate `tests/conformance/` (operator,
|
||||
2026-10-11) — engines are features of one crate, not sibling crates,
|
||||
so a separate suite crate buys nothing; the harness is
|
||||
factory-parameterized over engine constructors, rows ADR-stamped
|
||||
(the alkstore ADR-017 convention adapted), mem the reference column.
|
||||
- **Fuzzing**: adopted family-wide as wave 8, after ops and before
|
||||
release (operator standing rule: fuzz anything worth fuzzing; it
|
||||
almost always pays for itself). Target inventory is fixed at that
|
||||
wave's decomposition against the matured surface.
|
||||
- **Single crate, engine features** (ADR-013 §4): no engine-crate
|
||||
workspace split; the feature graph is declared in the manifest from
|
||||
wave 1; feature-gated engines' gates run under `--all-features`.
|
||||
- **CI**: none, deliberately (family posture): CI and publishing run
|
||||
manually; the merge gates (`cargo test`, `cargo clippy --all-targets
|
||||
-- -D warnings`, `cargo fmt --check`, plus `--all-features` variants
|
||||
once features carry code) are run coordinator-side.
|
||||
|
||||
## Review gates
|
||||
|
||||
Each wave ends in a review task (`review-wave-N`) before the next wave
|
||||
decomposes. Specific gates:
|
||||
|
||||
- **Wave 1 review** — the one-way-door gate: the key encoding read
|
||||
against ADR-002 line by line (a shape error here is a task revision;
|
||||
after engines ship data it is a migration ADR); the trait + seam
|
||||
shapes against ADR-003/010; the error model's altitude split against
|
||||
store-api.md/ADR-012 §6.5; the feature graph against ADR-013 §4;
|
||||
the named design points (single-enum error reading, Backend
|
||||
visibility, trait put-form shapes, oid hex policy, async-trait
|
||||
shape) audited as resolved.
|
||||
- **Wave 2 review** — facade + dispatch + GC vs ADR-003/005/012 (the
|
||||
three numbered unknown-length rules, the delete-window ordering
|
||||
invariants, abort-as-data); stage-hygiene exact-count tests; the
|
||||
suite's first columns green on mem + local; the dispatch-coverage
|
||||
shape exercised.
|
||||
- **Engine wave reviews (3/4/5)** — engine-vs-contract conformance:
|
||||
the suite column green (server-less for sqlite; vs dockerized
|
||||
postgres for the pg engines), the engine-state clause families
|
||||
contract-tested per applicability, virgin-store semantics, `size`
|
||||
metadata-only discipline.
|
||||
- **Wave 6 review** — the fleet gate: exact-count sweep outcomes under
|
||||
concurrency vs live postgres, the joint entry+pin tx atomicity, the
|
||||
sweep lease semantics vs ADR-013 §2, the composite fleet-validity
|
||||
predicate enforced at the constructor, kv-only put-time rejection.
|
||||
- **Wave 7 review** — ops-vs-alkcall conformance (the machinery ridden,
|
||||
not re-invented); ACL mapping vs OQ-09's closed-by-default; the pin
|
||||
token contract vs ADR-010 §4/ADR-012 §2; no backend side-door.
|
||||
- **Wave 8 review** — campaigns run clean (or findings fixed + promoted
|
||||
to seeds), the corpus-replay gate green on stable, seeds committed
|
||||
per the corpus policy.
|
||||
- **Wave 9 review** — the pre-release gate: docs honesty audit, the
|
||||
feature-graph verification re-run, all gates green (including the
|
||||
fuzz replay), release-facing specs flipped `stable`.
|
||||
Reference in new issue
Block a user