alkstore/docs/research/fuzzing.md

# Fuzzing alkstore: Research and Plan

**Status:** landed — adopted (2026-10-10), inserted as wave 7 in
`docs/plans/implementation.md` (release readiness renumbered to 8);
wave 1's three targets implemented, seeds committed, detached campaigns
run (§9 log), the corpus-replay gate on AGENTS.md's verification list.
Wave 2 (`mem_opseq`, §3.4) stays deferred post-release. Campaigns run
via the detached runner (§5) — **never as a foreground child of an
agent session** (the family's §5 rule, inherited verbatim).
**Date:** 2026-10-10
**Upstream precedent (inherited wholesale, not re-litigated):**
- `/workspace/@alkdev/alkcall/docs/research/fuzzing.md` — the origin doc;
  one real finding (duplicate adopt/open destroying the live channel,
  fixed 0.8.1).
- `/workspace/@alkdev/alktunnels/docs/research/fuzzing.md` — six
  harness-model findings, no crate bugs.
- `/workspace/@alkdev/alksocks/docs/research/fuzzing.md` — the most
  recent full edition; this doc borrows its structure and §7's
  successful-negative-result disposition.
- `/workspace/@alkdev/alktty/docs/plans/fuzzing.md` — the
  op-sequence-target pattern (`session_opseq`) for the deferred wave 2.

Tooling, corpus policy, runner, and the no-hosted-CI posture come from
those docs unchanged; this doc does not re-decide them, it scopes them
to this crate.

---

## 1. Should this crate be fuzzed?

**Yes — adopted 2026-10-10** (recorded in `implementation.md`'s decided
points; AGENTS.md's prior no-fuzzing posture flips with this decision).
Four lines:

1. **The family adopted it everywhere it had parse or arithmetic
   surfaces, and the pattern is mature.** Four sibling crates (alkcall,
   alktunnels, alksocks, alktty) run the identical layout: a `fuzz/`
   sidecar workspace (nightly pinned *inside* `fuzz/` only), thin
   `fuzz_target!` binaries, invariant logic in a stable-toolchain
   `fuzz/shared` crate so **corpus replay is plain `cargo test` — the
   fuzz gate** (which fits this repo's no-CI, coordinator-run-merge-
   gates posture exactly). The tooling and runner are copied wholesale;
   no new decisions are needed.

2. **This crate has already produced exactly the bug class fuzzing
   enumerates.** The wave-6 review gate caught three edge-domain
   defects by hand: the stream cursor read's `i64::MAX` overflow
   (`Bound::Excluded`), `trim_to`'s `i64::MAX` horizon endpoint, and
   the scheduler register path's first-boundary overflow rejection.
   All three live in hand-rolled arithmetic and parsing — and the
   crate deliberately owns *three* copies of that arithmetic (the
   `@every` parsers and boundary math per engine, ADR-012 §2's
   one-owner rule), plus the equal-jitter backoff curve and
   opts-stamping resolution per engine. Hand-rolled overflow-prone
   numeric code × three owners is precisely the shape the upstream
   docs adopted fuzzing for. Review found these spotwise; fuzzing
   demonstrates the domain guards across the whole input space.

3. **The contract suite's blind spot is fuzzing's strength.** The
   suite (`alkstore-contract-suite`) is example-based — curated stamps
   and inputs a human authored (no proptest). What it cannot provide:
   random *malformed* input exploration (payload bytes that are not
   what any engine would write, spec strings outside the grammar),
   stateful interleavings, and resource-bound assertions under
   adversarial sequences. The alksocks §1.3 argument applies as-is.

4. **Sequencing: cheapest now.** First release is the wave-8 deliverable;
   after that the public surface is frozen and targets become regression
   protection. All four upstream docs adopt at/just-before first publish
   for exactly this reason — and the plan's wave-7 deferral carrier is
   being discharged on schedule, before wave 8 decomposes.

**The honest caveat** (mirroring all upstream docs): the crate has no
`unsafe`, typed errors end-to-end (thiserror taxonomy, ADR-008 §5), a
no-`unwrap()`/no-panic policy enforced by review, and payload decode
delegates to serde_json — itself battle-fuzzed and recursion-limited
(oversized nesting is a typed `Error::Codec`, not a stack overflow).
Expected yield is therefore not memory-safety bugs. It is (a) the
demonstration that the typed-error / no-panic contracts hold on
adversarial input, (b) overflow-arm discovery in the hand-rolled
arithmetic (the wave-6 finding class — the crate's own track record
says these were *found by humans, spotwise*, so more arms plausibly
exist), (c) regression protection for the published surface. If the
campaigns find nothing crate-side, that is the family's
successful-negative-result posture (alksocks' first campaigns: zero
crate bugs, three harness-model findings, all valuable — the
fuzzer-as-harness-checker pattern).

## 2. Toolchain (inherited, not re-decided)

Identical to all four upstream docs: **cargo-fuzz + `arbitrary`**,
nightly confined to the `fuzz/` subtree via `fuzz/rust-toolchain.toml`
(the siblings pin `channel = "nightly"` + `llvm-tools-preview` there);
the crates themselves stay on stable. AFL/bolero/libafl/honggfuzz/
OSS-Fuzz not adopted; sanitizer: `--sanitizer=address` with debug
assertions (cargo-fuzz defaults) — no MSAN needed (no `unsafe`).

**Layout (the upstream pattern, load-bearing):**

```
fuzz/
├── Cargo.toml            alkstore-fuzz (nightly-only bins; own [workspace])
├── rust-toolchain.toml   pins nightly for this subtree only
├── fuzz_targets/         thin fuzz_target! wrappers (3-line each)
├── shared/               alkstore-fuzz-shared — STABLE-toolchain library
│   └── src/…             invariant logic + corpus replay (§3)
├── corpus/<target>/      committed hand-made seeds (§4)
├── gen_fuzz_seeds.py     deterministic seed generator (quiche pattern)
├── run-detached.sh       §5 detached runner (copy from alksocks; CWD-independent)
└── README.md
```

Upstream layout rules that carry over, in alkstore terms:

- **Invariant logic lives in `fuzz/shared/`**, not inline in the target
  binaries — the stable-toolchain corpus replay
  (`cargo test --manifest-path fuzz/shared/Cargo.toml`) runs the SAME
  invariant functions the fuzzer runs. This is the fuzz gate (§6);
  it is on AGENTS.md's verification commands (landed with wave 7, §9).
- **Workspace/publish exclusion:** this repo's root `Cargo.toml` is a
  workspace-only manifest (no root package) with an explicit members
  list, so cargo will not auto-discover `fuzz/shared/` — but the
  defensive `exclude = ["fuzz"]` on the workspace table is added anyway
  (the alkcall §2 auto-discovery failure's lesson, cheap insurance).
  Wave 8's publish-prep task verifies the fuzz sidecar cannot leak
  into any published package (`exclude` lists on the crates if needed).
- **Corpus policy:** commit hand-made seeds, gitignore grown corpora
  (quinn/h2 model). Crash inputs found by campaigns are promoted to
  committed seeds (the alksocks seed-900/901 posture).
- **wasm note:** the wave-6 constraint on `alkstore-mem` (compiles
  clean for `wasm32-unknown-unknown`, tokio sync+time only) is
  unaffected — the fuzz sidecar builds host-side only, and `shared`
  must not introduce a dependency that breaks the mem crate's wasm
  posture (it consumes the same public API).

## 3. Target inventory

Two feasibility constraints shape the list: every target should drive
already-`pub` API (the family's posture — no `#[cfg(fuzzing)]`
exposure; see §3.2 for the one snag), and campaign targets must be
in-process (the pg engine needs a live server — its paths are not
campaign targets; §3.5).

### 3.1 `payload_codec` (wave 1 — core, `alkstore/src/payload.rs`)

Input: raw `&[u8]`, plus `arbitrary`-derived `serde_json::Value`s.
Drives `encode_payload` and `payload_as::<T>` (over `Value` and a
representative typed-payload set). Invariants:

- **no-panic on any byte sequence** (the baseline).
- **typed-failure partition**: every decode failure is `Error::Codec`
  — no other error variant, no panic (the ADR-023 §1 fallibility pin,
  demonstrated across the input space).
- **deep-input loudness**: serde_json's recursion limit surfaces as
  typed `Codec`, never a stack overflow (the deep-nesting probes).
- **Value round-trip**: `encode_payload(v)` → `payload_as::<Value>`
  yields `v` for every arbitrary `Value` (canonicalization is
  serde_json's, pinned-not-regressed, not re-derived).
- **typed round-trip**: for the representative payload structs,
  `encode(Value::from(t))` decodes back to `t` — the ADR-020 §4
  "engines store exactly these bytes" contract's decode side.

Seeds: valid JSON of every Value variant, truncations, deep nesting at
and past the recursion limit, non-UTF-8 bytes, trailing garbage,
oversized numbers, duplicate keys, the `Null`-decode shape (ADR-023
§1's empty-byte-store corruption note).

Dictionary: `json.dict` (copy from a sibling).

### 3.2 `every_parse` (wave 1 — the three parsers, differential)

Input: arbitrary `&[u8]` → UTF-8-lossy `String`s (lossy is deliberate:
the engines receive `&str`, which is always valid UTF-8 — but the
generators should still stress the grammar's edge shapes). Drives all
three `parse_every_interval` implementations:

- SQLite: `alkstore-sqlite/src/substrate/queue_ops.rs`
  (`parse_every_interval`, substrate-owned, `Result<i64, String>`),
- Postgres: `alkstore-postgres/src/scheduler.rs` (`Result<i64>` →
  `InvalidSpec`),
- Mem: `alkstore-mem/src/scheduler.rs` (`Result<i64>` → `InvalidSpec`),

plus each engine's next-boundary arithmetic (the checked-add /
skip-forward paths — the wave-6 finding class). Invariants:

- **no-panic on any string** (each parser, each boundary path).
- **typed rejection**: every rejection is the parser's own typed error
  (`InvalidSpec` on pg/mem; the substrate's typed error under the
  engine's mapping — never a panic or an `Ok`).
- **three-way agreement**: the accept-set and the interval value are
  identical across the three parsers for every input. The contract
  suite (suite-pinned rows) pins behavioral equivalence on *valid*
  specs; fuzzing extends that to *arbitrary* strings — cheap, and it
  pins the ADR-009 §2 grammar three ways continuously.
- **overflow posture**: the boundary paths reject
  (checked-add → `InvalidSpec`/equivalent), never wrap or panic — the
  wave-6 scheduler-register fix's invariant, over the whole interval
  domain (`u64::MAX`-shaped digits, the 9223372036854775807 neighborhood,
  `d`-unit multipliers).
- **the `n <= 0` and giant-`n` arms**: leading-zero forms, digit
  overflow at exactly `i64::MAX/86400`, positive-zero rejection.

**The exposure design point (wave-7 decomposition resolves):** the
three parsers are `pub(crate)` — the family's "drive already-`pub` APIs"
posture has no precedent for this. Options named for decomposition:

- **(a)** drive through the public `schedule()` path against the mem
  (and SQLite — in-process, temp files) factories with arbitrary spec
  strings: pure public surface, but slower per-exec and no byte-level
  differential (behavioral only).
- **(b)** a `#[doc(hidden)]` narrow exposure mirroring the existing
  `Job::from_row` / `Schedule::new` constructor precedent
  (ADR-017 §3 consumer-read policy does not govern engine-internal
  test surfaces; the parsers are engine-owned, not contract surface) —
  keeps the fast byte-level differential target the upstream docs favor.
- **(c)** a dev-only feature flag (`fuzzing`), rejected by the family's
  no-`#[cfg(fuzzing)]` experience unless the harness shape provably
  needs it.

Lean: **(b)** for the fast differential, **(a)** as its in-suite
companion (the suite already pins behavior; (a) adds no new rows, only
optional corpus seeds). Decomposition decides; this doc does not
pre-commit.

### 3.3 `name_validation` (wave 1 — core, `alkstore/src/validation.rs`)

Input: arbitrary `&[u8]` → strings, all `NameKind`s, lengths spanning
the caps. Drives `validate_name` / `validate_shared_name` /
`validate_local_name`. Invariants: total (no panic); deterministic
accept/reject; rejection only via typed `Error::InvalidName`; the
reserved-prefix rule (`__alkstore_`, ADR-014 §1's shared namespace)
rejects in every casing/boundary form the fuzzers find; length
boundary exactness at the cap (the ±1 neighborhood).

Cheapest target; low expected yield but keeps the entry-point domain
guard demonstrated (the gate reviews audit guard-order discipline; this
demonstrates it over the whole string space).

### 3.4 `mem_opseq` (wave 2 — deferred, post-release)

The stateful op-sequence target, the alktty `session_opseq` /
alkcall `manager_routing` analog: `MemStore::new()` + an
`arbitrary`-derived op sequence (enqueue/claim/ack/reclaim/dead-letter/
sweep/publish/trim/save_offset/try_lock/lock-extend/notify/notify_tx/
schedule-tick/outbox enqueue, interleaved with tx begin/commit/drop and
close), driven over a current-thread tokio runtime (the suite's
mem-column flavor). Invariant sketch (decomposition firms it):

- no-panic across any op sequence;
- the ADR-010 queue state machine's legality (no transition the state
  machine forbids; reclaim-eats-attempt; dead-letter only from
  exhausted attempts);
- per-consumer stream offsets monotone; receiver arms correct after
  trim and close;
- tx overlay: commit-merge atomicity, drop-=-discard (no ghost rows),
  wake-at-commit — the suite's tx rows as invariants over random
  interleavings instead of authored ones;
- lock TTL expiry/re-acquisition; the loser's clean `None`.

Deferred for the same reasons alksocks deferred its wave 2: wave 1
covers the byte surfaces; this is a large harness; and the crate is
pre-release — regression protection matters most once the surface
freezes. Note the mem engine is the *only* engine fully fast enough
for this (SQLite is feasible but file-bound per-op; pg needs a server).

### 3.5 What is deliberately NOT a target

- **The pg engine's server-reachable paths** (forwarder, listener,
  pool open, DSN handling) — not in-process; campaigns cannot drive
  them without a live server. Their failure arms are owned by the
  wave-4/5/6 test suite (the forwarder reconnect tests etc.).
- **The SQLite substrate internals beyond the public surface** —
  substrate edits are governed by the lineage-fidelity diff discipline
  (ADR-012 §3 / ADR-018), not by fuzzing; its inherited test floor
  remains the owner of that surface. The `every_parse` target touches
  it only through the public/hidden exposure decision (§3.2).
- **Arithmetic beyond §3.2** (backoff curve, opts resolution) for wave
  1 — the suite pins their outputs three-way; fuzzing them is a
  wave-2 candidate if the opseq target's invariants want them. The
  equal-jitter curve's determinism makes fuzz yield unlikely.
- **wasm targets** — libFuzzer is host-side; the mem wasm acceptance
  stays compile-clean (§2's wasm note).

## 4. Scope: one implemented wave + a deferred second

**Wave 1 (wave 7 in the plan, this session's scope):** the three
targets of §3.1–3.3 — `payload_codec`, `every_parse`, `name_validation`.
All in-process, all fast (byte/grammar-level), all invariants sketched
above. Sequencing mirrors upstream: scaffold (layout of §2, runner,
dict, seed generator) → targets in parallel → seeds committed →
10–30 min detached campaigns per target → fix anything found (crash
inputs promoted to seeds) → corpus replay green.

**Wave 2 (deferred — post-release):** `mem_opseq` (§3.4). Recorded in
§3.4 so the deferral is a named scope decision, not a pending-client
hedge.

The wave's decomposition resolves the named open points: §3.2's
exposure decision, the AGENTS.md verification-list flip (corpus replay
joins the merge gates when the wave lands), and the publish-exclusion
verification (§2).

## 5. Operational isolation: campaigns never share fate with the agent session

**Inherited verbatim from the upstream docs (alksocks §5)** — an
environment constraint, not a nice-to-have. Three layers, all
mandatory: (1) libFuzzer soft limits (`-rss_limit_mb=2048`
`-malloc_limit_mb=2048`) — never `ulimit -v` under ASAN; (2) fork mode
(`-fork=1`) as blast-radius containment; (3) process detachment —
`run-detached.sh` wraps `cargo fuzz run` in `setsid` + `nohup` + full
redirection; poll, never `wait`. The alksocks runner is copied
as-is (it is CWD-independent and repo-layout-agnostic). Also
inherited: `cargo fuzz run` returns before its last fork-mode children
fully exit — `pgrep -af` to LIST, kill by explicit pid from the
runner's echoed `pid=`, never pattern-kill-and-recheck in one call.

## 6. No hosted CI

The standing policy applies identically (per all four upstream docs):
no hosted CI; no Actions/Gitea Actions workflow files. **Corpus replay
is the fuzz gate as plain `cargo test`**:
`cargo test --manifest-path fuzz/shared/Cargo.toml` replays every
committed seed through the invariant functions. It is on
AGENTS.md's verification commands (landed with wave 7, §9). Campaigns run
manually via the detached runner — e.g. before the release, or after
touching `payload.rs`, the three parsers, or `validation.rs`.

## 7. Verification commands

```bash
cargo test                                          # workspace gates (stable)
cargo test --manifest-path fuzz/shared/Cargo.toml   # corpus replay (fuzz gate, from wave 7)
cargo fuzz build                                    # in fuzz/ (nightly, ASAN)
fuzz/run-detached.sh <target>                       # campaigns (never foreground)
cargo clippy --all-targets -- -D warnings
cargo fmt --check
cargo clippy --manifest-path fuzz/shared/Cargo.toml --all-targets -- -D warnings
cargo fmt --manifest-path fuzz/shared/Cargo.toml --check
cargo check -p alkstore-mem --target wasm32-unknown-unknown   # mem's wasm acceptance
```

Campaign flag starting points (per target, tuned at wave 7):
`payload_codec -max_len=65536 -dict=json.dict`;
`every_parse -max_len=4096 -use_value_profile=1`;
`name_validation -max_len=4096 -use_value_profile=1`.

## 8. References

- Upstream docs (authoritative for tooling, runner, corpus policy,
  no-CI posture): alkcall / alktunnels / alksocks / alktty, paths in
  the header.
- RUSTSEC-2026-0037 (quinn-proto — the maintainers' fuzzing-coverage
  statement; the upstream §1 datapoint).
- This crate: ADR-008 §5 (error taxonomy), ADR-009 §2 (the `@every`
  grammar), ADR-010 (queue state machine), ADR-012 §2 (one-owner
  parsers/arithmetic), ADR-017 §3–§4 (exposure/versioning policy),
  ADR-020 §4 (payload encoding posture), ADR-023 §1 (encode
  fallibility), ADR-024 §5 (mem wasm acceptance); the wave-6 review
  gate's overflow findings (`tasks/review-wave-6.md`);
  `docs/plans/implementation.md` §Decided points (the adoption record).

---

## 9. Implementation log — wave 7 (2026-10-10)

- **2026-10-10 — wave 1's three targets implemented** (the
  decomposition's target tasks, all via the §2 layout: stable
  `fuzz/shared` invariant library, thin `fuzz_target!` binaries,
  committed seed corpora from `gen_fuzz_seeds.py` — 97 + 66 + 44).
  §3.2 resolved to the option-*(b)* `#[doc(hidden)]` exposure
  (doc-hidden re-exports of the three parsers + `skip_forward`; the
  every_parse campaign's two earlier findings — the boundary-arithmetic
  division-by-zero/negative-interval wrap and the i64 gap-subtraction
  overflow, fixed crate-side and registered as PROVENANCE D-38). The
  §3.2 suite-companion option-*(a)* added no suite rows — suite-pinned
  rows already carried equivalence; noted here per the decomposition's
  resolution scope.
- **2026-10-10 — first campaign finding: the `Value` round-trip
  invariant over-tightened (harness-model, not a crate bug).** The
  `payload_codec` campaign crashed ~38 s in on raw input
  `46741844073709.51615` (`crash-0ce050f1…`, exit 77):
  `decode(encode(v)) ≠ v` — `46741844073709.52` vs `46741844073709.516`.
  Reproduced and characterized probe-side (serde_json 1.0.152): its
  **default fast-path f64 parser is not exactly correctly-rounded** —
  parsing serde_json's own ryu serialization can drift 1–2 ULPs
  (measured over 600k random-bits finite f64s: ~30% fail one
  parse∘serialize∘parse pass, `max drift = 2.00` ULP, overwhelmingly
  exactly 1 ULP; the one-pass re-parse is **not a fixpoint** at extreme
  significand/exponent shapes), so the byte-level closure holds only
  outside float leaves. This is serde_json's *documented shipping
  behavior*, and [ADR-020](../architecture/decisions/020-enqueue-opt-semantics-and-bridges.md)
  §4 pins exactly that — "not pinned to a serde_json *feature flavor*
  … beyond default shipping behavior" — so enabling its
  `float_roundtrip` feature would be an ADR deviation, not a fix.
  The harness was the defect: it pinned the suite's example-based
  round-trip equality (authored floats are ryu-stable) over the whole
  raw input space. Fixed in `fuzz/shared`: the closure is now exact
  outside `Number` float leaves, each float leaf drift-bounded at
  2 ULPs (pinned-not-regressed), plus a decode-determinism pin (two
  parses of the same stored bytes are identical). The crash input is
  promoted to `corpus/payload_codec/seed-900`; the campaign re-ran
  clean.
- **2026-10-10 — wave-1 15-minute detached campaigns run** (§6 runner,
  `-fork=1 -rss_limit_mb=2048 -malloc_limit_mb=2048 -timeout=25`;
  flags per §7): all three exited 0, `oom/timeout/crash: 0/0/0`
  throughout, empty per-target artifact dirs at completion (the one
  first-run crash artifact above is the promoted finding's evidence).
  - `payload_codec` (post-fix re-run; the pre-fix run died at the
    finding) — 927 s, ~586k execs (~650 exec/s), 41 fork jobs,
    coverage 2395 edges (ft 7608, in-memory corp 1544 from the 98
    committed seeds).
  - `every_parse` — 927 s, ~3.7M execs (~4.1k exec/s), 41 fork jobs,
    coverage saturated at 285 edges (ft 3898, corp 805 from the 66
    committed seeds) — the three parsers' grammar space is fully
    enumerated post-fix; the earlier boundary-arithmetic findings were
    the campaign-relevant depth.
  - `name_validation` — 939 s, ~59k execs (~66 exec/s), 41 fork jobs,
    coverage 327 edges (ft 3248, corp 378 from the 44 committed
    seeds); the corpus-replay seeds already covered the domain guards,
    and the slow exec/s profile is the long-string shapes, not a
    harness defect.
  - Grown corpora were **not** merged into the committed seeds (the
    §2 policy; `fuzz/.gitignore` now tracks only `seed-*` files —
    the scaffold's directory-level negation leaked SHA-named grown
    units into `git status` during the campaigns, fixed here).
- **NO crate bugs found by the campaigns.** The sole campaign finding
  was the harness-model defect above (fixed harness-side); every
  contract invariant held across the adversarial input space: the
  typed-failure partition (`Codec` only), deep-input loudness at
  serde_json's recursion limit, the empty-byte-store loudness
  (ADR-023 §1), the typed round-trip batteries, the three-way parser
  agreement and overflow posture, and name-validation
  determinism/typedness. That is the family's
  successful-negative-result posture (alksocks §7's first campaigns:
  zero crate bugs after the two earlier wave-7 crate-side
  boundary-arithmetic fixes, which the every_parse task itself landed).