glm-5.3-flash
758ccfb359
Six tasks against the wave-2 contract surface as reviewed (no amendments from the gate): - engine-sqlite: the default kv engine — WAL + synchronous=NORMAL, prepared statements, the indexed size column probe (B5's row shape), has-as-EXISTS, virgin-store no-tables-yet tolerance, spawn_blocking discipline - engine-sqlite-window: the engine-state seam's window family only (ADR-012 §5's reduction — gc_window, stage/take; no pin rows, no sweep-lock body) - store-sqlite-construction: KvEngine::Sqlite slots in, the dual-tier/kv-only constructors with the constructor-tunable window delay (default 5 min), ensure_gc_schema at construction, the one-window-host posture - store-sweep-durable-window: the sweep's SQL-backed form — overdue rows execute with re-arbitration before/alongside marking, candidates stage at mark_end + window, counts exact relative to the schedule (ADR-012 §6.7); the inline form preserved verbatim - suite-column-sqlite: the first SQL-backed column, server-less — form-independent reference rows mirror directly; the sweep rows are schedule-relative (the recorded INV-4/5 mirror amendment) - review-wave-3: the engine-wave gate (engine-vs-contract conformance) Verification: dependency graph validated (all ids resolve, no cycles, topological order clean); docs-only change. Refs: docs/plans/implementation.md §Wave 3
316 lines
18 KiB
Markdown
316 lines
18 KiB
Markdown
---
|
||
status: draft
|
||
last_updated: "2026-10-11 (wave 3 decomposed into tasks/)"
|
||
---
|
||
|
||
# alkblobs — Implementation plan
|
||
|
||
Wave-based decomposition of the architecture (`docs/architecture/`)
|
||
into units of implementable work. This is a deliberate deviation from
|
||
the SDD process's decompose-everything-upfront step, adopting the
|
||
sibling pattern proven on alkstore: the architecture is large, early
|
||
waves change the shape of later ones (the seam shapes wave 1 pins are
|
||
what every engine task implements against; the conformance suite's
|
||
factory contract feeds every engine's verification work), and
|
||
decomposing only the next wave at a time keeps each session's task set
|
||
reviewable and lets later decompositions absorb earlier waves' course
|
||
corrections.
|
||
|
||
Rhythm: decompose a wave → implement it → review gate → decompose the
|
||
next wave. Task files live in `tasks/` (frontmatter carries the
|
||
categorical estimates). Wave boundaries are also review boundaries.
|
||
|
||
Scope authority: **v0.1.0 ships the full surface** — all five engines
|
||
(kv: sqlite, postgres, mem; large: local, pg-lo), fleet GC, and the
|
||
feature-gated `ops` module — per operator decision (2026-10-11): all
|
||
downstream consumers need all of it, and the base crate must be fully
|
||
usable and composable across mem/sqlite/postgres from the default
|
||
build onward.
|
||
|
||
## The waves
|
||
|
||
Dependency logic in one line: core contract → (mem ∥ local) → sqlite
|
||
→ postgres → pg-lo → fleet GC → ops → fuzzing → release readiness.
|
||
The two dependency-free engines (mem, local) prove the entire facade +
|
||
dispatch + in-process GC story before any SQL driver exists; sqlite
|
||
and postgres are structurally independent of each other (both need
|
||
only wave 2's trait + suite) and could run in either order — postgres
|
||
is placed first of the two SQL engines because its wave establishes
|
||
the full engine-state seam clause families that pg-lo (window family)
|
||
and the fleet wave (the host) build on. pg-lo shares postgres's
|
||
driver cluster and seam pattern. The fleet wave needs both SQL tiers
|
||
landed: its canonical testable shape is kv=postgres + large=pg-lo on
|
||
one instance (the composite fleet-validity predicate's joint
|
||
entry+pin tx is expressible exactly there). The ops wave rides
|
||
alkcall (published, 0.8.1) behind the default-off feature. Fuzzing
|
||
follows the ops wave so its targets cover the key encoding *and* the
|
||
ops payload surfaces; release readiness is last.
|
||
|
||
| Wave | Contents | Depends on | Status |
|
||
|---|---|---|---|
|
||
| [1](#wave-1--foundations) | Crate/feature scaffold; hashing & keys (the one-way door); error model; Backend trait + I/O seams + engine-state seam shape; facade value types; conformance-suite convention | — | decomposed (2026-10-11) |
|
||
| [2](#wave-2--the-store-works-dependency-free-mem--local) | mem + local engines; facade put/read paths + dispatch + constructor modes; in-process GC protocol; suite's first columns | wave 1 | decomposed (2026-10-11) |
|
||
| [3](#wave-3--sqlite-engine) | sqlite kv engine (default): WAL posture, prepared statements, engine-state window family, suite column | wave 2 | decomposed (2026-10-11) |
|
||
| [4](#wave-4--postgres-kv-engine) | postgres kv engine: schema, pool, CAS rows, stable-cursor list, full engine-state clause families, suite column | wave 2 | pending |
|
||
| [5](#wave-5--pg-lo-large-engine) | pg-lo large engine: LO staged put, descriptorless lo_get windows, companion rows, window-family seam clauses, suite column | wave 4 | pending |
|
||
| [6](#wave-6--fleet-gc) | Fleet GC mechanism: DB-backed pins, joint entry+pin tx, sweep lease, SQL delete arbitration, fleet-validity predicate + constructor declarations | waves 4 + 5 | pending |
|
||
| [7](#wave-7--ops-surface) | `ops` feature: alkcall-backed op family, ACL mapping, pin tokens + renewal, fanout seam | wave 6 | pending |
|
||
| [8](#wave-8--fuzzing) | Fuzzing: family pattern targets (key encoding, ops payloads), corpus-replay gate | wave 7 | pending |
|
||
| [9](#wave-9--release-readiness) | Crate docs, feature-graph verification, README, publish prep | wave 8 | pending |
|
||
|
||
## Wave 1 — Foundations
|
||
|
||
The contract surface in code, engine-blind: everything later waves
|
||
implement *against*, and nothing that needs an engine to test.
|
||
|
||
- **Crate/feature scaffold** — the ADR-013 §4 feature graph declared
|
||
in the manifest from the first commit: `sqlite` default-on
|
||
(rusqlite), `postgres` (tokio-postgres + deadpool-postgres), `mem`
|
||
(dependency-free), `pg-lo` (the shared pg cluster; does **not**
|
||
imply `postgres` — ADR-010 §1's consequence), `ops` (alkcall,
|
||
default-off); `local` ungated. Engine modules ride `#[cfg(feature)]`
|
||
from the start.
|
||
- **Hashing & keys** (ADR-002 — the one-way door): `Digest`/`Key`
|
||
types, the canonical `git-blob-sha-256` derivation
|
||
(`H("blob <len>\0" + content)`), tolerated `git-blob-sha-1`, the
|
||
algorithm-tagged byte encoding (1 alg byte + digest; 0x01/0x02
|
||
assigned, 0x00 reserved), rejects-never-guesses validation, and the
|
||
byte-exact `git hash-object` interop test (POC #1 findings 4/5's
|
||
standard — validated against the git CLI, not just vectors).
|
||
- **Error model** (store-api.md; ADR-012 §6.5): the one-enum error
|
||
family with its altitude discipline — the trait's effective members
|
||
(`Missing`/`Verification`/`Io`/`KeyInvalid`) and the facade's
|
||
`GcRefuse` addition.
|
||
- **Backend trait + I/O seams** (ADR-003/010): the trait method set
|
||
(`has`/`get`/`put`/`delete`/`list`/`name`/`size`) with opaque byte
|
||
keys; the read-cursor abstraction (kv materializing vs large
|
||
streaming bodies; serves `read_range`'s slice + slice-digest
|
||
contract at one seam); the staged-put contract as a named per-engine
|
||
shape (staging cursor handout, commit publishes, every failure path
|
||
discards); and the crate-internal engine-state companion trait with
|
||
its clause families and per-engine applicability (ADR-010 §2 as
|
||
reduced by ADR-012 §5) — SQL engines implement it in waves 3–5;
|
||
`local`/`mem` are excluded by construction.
|
||
- **Facade value types**: `EntryMeta`, `Pin`/`Batch` (types + pinned
|
||
lifecycle contract; the arbitration machinery is wave 2),
|
||
`SweepReport`/`GcAbortCause` (abort-as-data, exactly three causes),
|
||
`PinState`.
|
||
- **Conformance-suite convention** — `tests/conformance/` established
|
||
with the row/stamp convention (ADR-§-stamped rows, the alkstore
|
||
ADR-017 pattern adapted) and the factory contract (isolation +
|
||
idempotent teardown); the invariant backlog from store-api.md
|
||
registered as rows-to-grow. The compile-able harness lands with
|
||
wave 2's first column (mem — the contract-reference engine).
|
||
|
||
The wave-1 review gate is the cheap-place gate: the key encoding is
|
||
the declared one-way door (AGENTS convention 5) — a shape error found
|
||
at this gate costs a task revision; found after an engine ships data
|
||
it costs a migration ADR. The gate reads the encoding, the trait, and
|
||
the seam shapes against the pinned ADR text line by line.
|
||
|
||
## Wave 2 — The store works dependency-free (mem + local)
|
||
|
||
The first engines and the facade behavior, chosen so the whole store
|
||
story is provable with zero driver dependencies:
|
||
|
||
- **mem kv engine** (feature `mem`) — the contract-reference engine
|
||
(ADR-011 §2): a `BTreeMap`-shaped implementation of the kv tier's
|
||
contract including `size`; every other engine's conformance rows
|
||
mirror its column.
|
||
- **local large engine** (ungated) — the ADR-003 backend as specced:
|
||
flat `{hex-prefix}/{hex-prefix}/{hash}` sharding, stage-then-
|
||
commit-rename (the staged-put contract realized with its exact-count
|
||
hygiene tests, POC #3 A5's shape), pread-loop read cursor, pread
|
||
range reads.
|
||
- **Facade behavior** — the put seam's two internal paths behind one
|
||
signature (known-length one-pass; unknown-length pre-threshold
|
||
buffering with the mid-stream stage flush and the restart-over-
|
||
staged-bytes derivation pass, ADR-003's option (d) semantics pinned
|
||
in three numbered rules); `get`/`stat`/`read_range`/`has`/`delete`/
|
||
`list` with the dispatch fall-through general rule (ADR-012 §6.1);
|
||
size-threshold dispatch (pure function of length, default 128 KiB,
|
||
constructor-tunable); constructor modes (dual-tier / kv-only with
|
||
put-time rejection per ADR-012 §6.2 / mem-only) — `kv=mem +
|
||
large=local` is the dispatch-coverage test shape ADR-012 §6.3 names.
|
||
- **In-process GC protocol** (ADR-005) — pins with pin-before-publish
|
||
ordering, registered liveness sources (live-shared), the protect
|
||
callback with abort-as-data, delete windows (mark → arbitrate-at-
|
||
delete-time → batched commit), `sweep()` returning `Ok(SweepReport)`
|
||
always, direct-delete refusal (`GcRefuse`, the full protection set
|
||
per ADR-012 §6.4), `local`/`mem` staging window candidates
|
||
in-process and executing them inline in the same sweep (ADR-012 §5).
|
||
- **Suite columns** — the harness compile-able and green on the mem
|
||
and local columns: round-trips, dedup, stage hygiene, sweep-safety
|
||
exact counts, virgin-store no-ops, `stat`-without-content.
|
||
|
||
Wave 2 decomposes after the wave-1 review gate, absorbing whatever the
|
||
gate corrects.
|
||
|
||
## Wave 3 — sqlite engine
|
||
|
||
The default kv engine (feature `sqlite`, default-on; REQ-1): WAL +
|
||
`synchronous=NORMAL` as the shipped durability tier, prepared
|
||
statements, `has` as an EXISTS probe, the indexed `size` column probe
|
||
(POC #5 B5's row shape), virgin-store no-tables-yet tolerance, and the
|
||
engine-state seam's **window family only** (ADR-012 §5's reduction —
|
||
durable single-node window staging; no pin rows, no sweep-lock body).
|
||
Suite column green server-less. The single-writer ceiling is a
|
||
documented posture, not a defect (REQ-1's economics).
|
||
|
||
## Wave 4 — postgres kv engine
|
||
|
||
The fleet-capable kv engine (feature `postgres`; REQ-2/REQ-3): schema
|
||
bootstrap, pooled connections (tokio-postgres + deadpool-postgres)
|
||
with per-connection prepared-statement discipline, `ON CONFLICT DO
|
||
NOTHING` CAS rows, complete `list()` as a stable cursor over a
|
||
vacuuming table, batch-delete via `key = ANY($1)`, the
|
||
`synchronous_commit` parity knob, and the engine-state seam's **full
|
||
clause families** — postgres is the fleet host (ADR-012 §4):
|
||
`ensure_gc_schema` (pins + window), `pin_commit`, `pin_renew`,
|
||
`pin_check`/`arbitrate`, `window_stage`/`window_take`,
|
||
`sweep_lock_acquire`/`release` (the pg advisory lock, the sweep
|
||
lease's shipped provider). Suite column green against dockerized
|
||
postgres (the alkstore harness convention: connection settings ride
|
||
environment, tests skip cleanly server-less, acceptance requires green
|
||
against the server). This wave establishes the seam implementation
|
||
pattern pg-lo reuses.
|
||
|
||
## Wave 5 — pg-lo large engine
|
||
|
||
The large tier's second engine (feature `pg-lo`; ADR-009, admitted on
|
||
POC #7): Large Objects as storage — staged put as chunked `lowrite`
|
||
statements (512 KiB per the POC's measured knee) inside a tx the store
|
||
core holds, descriptorless `lo_get(oid, off, len)` window reads over
|
||
the companion table (the C4 posture), companion-row `size` probes,
|
||
transactional LO creation (zero crash orphans), and the seam's window
|
||
family (durable window staging where the kv engine is not SQL-backed —
|
||
the one-window-host rule). Suite column green against dockerized
|
||
postgres. Named deltas documented (catalog space never returned;
|
||
`VACUUM FULL` is the operator's shrink path).
|
||
|
||
## Wave 6 — Fleet GC
|
||
|
||
The mechanism wave (ADR-008/010/012/013; REQ-2): DB-backed pins as
|
||
expiry-carrying rows committed atomically with entries — the store
|
||
core holds the joint entry+pin tx and engines contribute mechanics
|
||
(`pin_commit`); the named exclusive sweep lease (one sweeper per pool,
|
||
atomic test-and-set, holdership ends at release/owner death/session
|
||
loss; contention is the `SweeperLock` report abort); SQL-level delete
|
||
arbitration (`SELECT … FOR UPDATE` finger-lock at delete time); TTL/3
|
||
pin renewal on the guard; the composite fleet-validity predicate and
|
||
the explicit `fleet: true` constructor declaration (never inferred);
|
||
kv-only re-routing posture; the one-window-host rule under dual-SQL
|
||
instances. Exact-count fleet sweep outcomes against live postgres —
|
||
the wave's gate is the architecture's most intricate state machine
|
||
proven under concurrency.
|
||
|
||
## Wave 7 — Ops surface
|
||
|
||
The `ops` feature (ADR-001; alkcall 0.8.1): the op family —
|
||
`blobs/stat`, `blobs/have` (with the token-renewal form),
|
||
`blobs/delete` (internal authority), `blobs/fetch` (Sub), `blobs/put`
|
||
(Sink, response `{token, digest}`) — JSON control plane via alkcall
|
||
`OperationSpec` + JSON Schema validation, binary data plane via
|
||
`channel_open`-allocated Sub/Sink channels; namespace-as-resource ACL
|
||
mapping (closed-by-default per OQ-09); the pin hand-over + token
|
||
renewal contract (the token is the pin; TTL/3 renewal via have/re-put;
|
||
validity domain = minting node); the fetch fanout seam (one reader,
|
||
store arm + subscriber arms; late joiners degrade to verified `get`).
|
||
The module is a store-facade consumer with no backend side-door.
|
||
|
||
## Wave 8 — Fuzzing
|
||
|
||
The family pattern (alkcall/alktunnels/alksocks/alktty/alkstore), per
|
||
operator standing decision that anything worth fuzzing gets fuzzed:
|
||
the `fuzz/` sidecar workspace (nightly pinned inside `fuzz/` only, the
|
||
stable `fuzz/shared` corpus-replay gate, the detached runner), targets
|
||
over the key-encoding parser (`Key::from_bytes` — the one-way door's
|
||
reject-never-guess discipline), the ops payload surfaces (JSON specs +
|
||
binary channel framing, if wave 7's shapes warrant it), and any
|
||
put-path staging logic the surface review names. Committed crash-input
|
||
seeds; grown corpora gitignored; the corpus-replay gate joins the
|
||
verification list when the wave lands. Targets are chosen at
|
||
decomposition time against the then-current surface — the wave exists
|
||
in the plan as a commitment, not yet a target inventory.
|
||
|
||
## Wave 9 — Release readiness
|
||
|
||
Crate docs (public-API rustdoc warning-free; the facade's docs are
|
||
contract text), the feature-graph verification pass (every coherent
|
||
engine-feature combination compiles and passes its gates — pg-pure
|
||
builds, `pg-lo` without `postgres`, `--all-features`), the root README
|
||
(written last, honestly), publish prep (package metadata, `cargo
|
||
package` verification per the exclude list, path-dep/version bounds),
|
||
and the pre-release review gate flipping the release-facing specs to
|
||
`stable`.
|
||
|
||
## Decided points
|
||
|
||
- **Wave-based decomposition** (operator-endorsed, 2026-10-11): the
|
||
SDD decompose-everything-upfront step is replaced by the
|
||
alkstore-proven rhythm — decompose the next wave after the previous
|
||
wave's review gate; later decompositions absorb earlier outcomes.
|
||
- **v0.1.0 scope — full surface** (operator, 2026-10-11): all five
|
||
engines, fleet GC, and the `ops` module ship in the first release;
|
||
the base crate is fully usable and composable across
|
||
mem/sqlite/postgres by default.
|
||
- **Engine order**: mem + local first (the contract-reference engine
|
||
plus the ungated engine — the dependency-free pair proves facade,
|
||
dispatch, and in-process GC end-to-end, and `kv=mem + large=local`
|
||
is the ADR-012 §6.3 dispatch-coverage shape); then sqlite (default),
|
||
postgres (establishes the full seam), pg-lo (shares the pg cluster
|
||
+ seam pattern), fleet GC last of the storage waves (its canonical
|
||
testable shape needs both SQL tiers).
|
||
- **Contract-suite home**: in-crate `tests/conformance/` (operator,
|
||
2026-10-11) — engines are features of one crate, not sibling crates,
|
||
so a separate suite crate buys nothing; the harness is
|
||
factory-parameterized over engine constructors, rows ADR-stamped
|
||
(the alkstore ADR-017 convention adapted), mem the reference column.
|
||
- **Fuzzing**: adopted family-wide as wave 8, after ops and before
|
||
release (operator standing rule: fuzz anything worth fuzzing; it
|
||
almost always pays for itself). Target inventory is fixed at that
|
||
wave's decomposition against the matured surface.
|
||
- **Single crate, engine features** (ADR-013 §4): no engine-crate
|
||
workspace split; the feature graph is declared in the manifest from
|
||
wave 1; feature-gated engines' gates run under `--all-features`.
|
||
- **CI**: none, deliberately (family posture): CI and publishing run
|
||
manually; the merge gates (`cargo test`, `cargo clippy --all-targets
|
||
-- -D warnings`, `cargo fmt --check`, plus `--all-features` variants
|
||
once features carry code) are run coordinator-side.
|
||
|
||
## Review gates
|
||
|
||
Each wave ends in a review task (`review-wave-N`) before the next wave
|
||
decomposes. Specific gates:
|
||
|
||
- **Wave 1 review** — the one-way-door gate: the key encoding read
|
||
against ADR-002 line by line (a shape error here is a task revision;
|
||
after engines ship data it is a migration ADR); the trait + seam
|
||
shapes against ADR-003/010; the error model's altitude split against
|
||
store-api.md/ADR-012 §6.5; the feature graph against ADR-013 §4;
|
||
the named design points (single-enum error reading, Backend
|
||
visibility, trait put-form shapes, oid hex policy, async-trait
|
||
shape) audited as resolved.
|
||
- **Wave 2 review** — facade + dispatch + GC vs ADR-003/005/012 (the
|
||
three numbered unknown-length rules, the delete-window ordering
|
||
invariants, abort-as-data); stage-hygiene exact-count tests; the
|
||
suite's first columns green on mem + local; the dispatch-coverage
|
||
shape exercised.
|
||
- **Engine wave reviews (3/4/5)** — engine-vs-contract conformance:
|
||
the suite column green (server-less for sqlite; vs dockerized
|
||
postgres for the pg engines), the engine-state clause families
|
||
contract-tested per applicability, virgin-store semantics, `size`
|
||
metadata-only discipline.
|
||
- **Wave 6 review** — the fleet gate: exact-count sweep outcomes under
|
||
concurrency vs live postgres, the joint entry+pin tx atomicity, the
|
||
sweep lease semantics vs ADR-013 §2, the composite fleet-validity
|
||
predicate enforced at the constructor, kv-only put-time rejection.
|
||
- **Wave 7 review** — ops-vs-alkcall conformance (the machinery ridden,
|
||
not re-invented); ACL mapping vs OQ-09's closed-by-default; the pin
|
||
token contract vs ADR-010 §4/ADR-012 §2; no backend side-door.
|
||
- **Wave 8 review** — campaigns run clean (or findings fixed + promoted
|
||
to seeds), the corpus-replay gate green on stable, seeds committed
|
||
per the corpus policy.
|
||
- **Wave 9 review** — the pre-release gate: docs honesty audit, the
|
||
feature-graph verification re-run, all gates green (including the
|
||
fuzz replay), release-facing specs flipped `stable`. |