alkstore/docs/research/fuzzing.md

Fuzzing alkstore: Research and Plan

Status: landed — adopted (2026-10-10), inserted as wave 7 in docs/plans/implementation.md (release readiness renumbered to 8); wave 1's three targets implemented, seeds committed, detached campaigns run (§9 log), the corpus-replay gate on AGENTS.md's verification list. Wave 2 (mem_opseq, §3.4) stays deferred post-release. Campaigns run via the detached runner (§5) — never as a foreground child of an agent session (the family's §5 rule, inherited verbatim). Date: 2026-10-10 Upstream precedent (inherited wholesale, not re-litigated):

• /workspace/@alkdev/alkcall/docs/research/fuzzing.md — the origin doc; one real finding (duplicate adopt/open destroying the live channel, fixed 0.8.1).
• /workspace/@alkdev/alktunnels/docs/research/fuzzing.md — six harness-model findings, no crate bugs.
• /workspace/@alkdev/alksocks/docs/research/fuzzing.md — the most recent full edition; this doc borrows its structure and §7's successful-negative-result disposition.
• /workspace/@alkdev/alktty/docs/plans/fuzzing.md — the op-sequence-target pattern (session_opseq) for the deferred wave 2.

Tooling, corpus policy, runner, and the no-hosted-CI posture come from those docs unchanged; this doc does not re-decide them, it scopes them to this crate.

---

1. Should this crate be fuzzed?

Yes — adopted 2026-10-10 (recorded in implementation.md's decided points; AGENTS.md's prior no-fuzzing posture flips with this decision). Four lines:

1. The family adopted it everywhere it had parse or arithmetic surfaces, and the pattern is mature. Four sibling crates (alkcall, alktunnels, alksocks, alktty) run the identical layout: a fuzz/ sidecar workspace (nightly pinned inside fuzz/ only), thin fuzz_target! binaries, invariant logic in a stable-toolchain fuzz/shared crate so corpus replay is plain cargo test — the fuzz gate (which fits this repo's no-CI, coordinator-run-merge- gates posture exactly). The tooling and runner are copied wholesale; no new decisions are needed.

2. This crate has already produced exactly the bug class fuzzing enumerates. The wave-6 review gate caught three edge-domain defects by hand: the stream cursor read's i64::MAX overflow (Bound::Excluded), trim_to's i64::MAX horizon endpoint, and the scheduler register path's first-boundary overflow rejection. All three live in hand-rolled arithmetic and parsing — and the crate deliberately owns three copies of that arithmetic (the @every parsers and boundary math per engine, ADR-012 §2's one-owner rule), plus the equal-jitter backoff curve and opts-stamping resolution per engine. Hand-rolled overflow-prone numeric code × three owners is precisely the shape the upstream docs adopted fuzzing for. Review found these spotwise; fuzzing demonstrates the domain guards across the whole input space.

3. The contract suite's blind spot is fuzzing's strength. The suite (alkstore-contract-suite) is example-based — curated stamps and inputs a human authored (no proptest). What it cannot provide: random malformed input exploration (payload bytes that are not what any engine would write, spec strings outside the grammar), stateful interleavings, and resource-bound assertions under adversarial sequences. The alksocks §1.3 argument applies as-is.

4. Sequencing: cheapest now. First release is the wave-8 deliverable; after that the public surface is frozen and targets become regression protection. All four upstream docs adopt at/just-before first publish for exactly this reason — and the plan's wave-7 deferral carrier is being discharged on schedule, before wave 8 decomposes.

The honest caveat (mirroring all upstream docs): the crate has no unsafe, typed errors end-to-end (thiserror taxonomy, ADR-008 §5), a no-unwrap()/no-panic policy enforced by review, and payload decode delegates to serde_json — itself battle-fuzzed and recursion-limited (oversized nesting is a typed Error::Codec, not a stack overflow). Expected yield is therefore not memory-safety bugs. It is (a) the demonstration that the typed-error / no-panic contracts hold on adversarial input, (b) overflow-arm discovery in the hand-rolled arithmetic (the wave-6 finding class — the crate's own track record says these were found by humans, spotwise, so more arms plausibly exist), (c) regression protection for the published surface. If the campaigns find nothing crate-side, that is the family's successful-negative-result posture (alksocks' first campaigns: zero crate bugs, three harness-model findings, all valuable — the fuzzer-as-harness-checker pattern).

2. Toolchain (inherited, not re-decided)

Identical to all four upstream docs: cargo-fuzz + arbitrary, nightly confined to the fuzz/ subtree via fuzz/rust-toolchain.toml (the siblings pin channel = "nightly" + llvm-tools-preview there); the crates themselves stay on stable. AFL/bolero/libafl/honggfuzz/ OSS-Fuzz not adopted; sanitizer: --sanitizer=address with debug assertions (cargo-fuzz defaults) — no MSAN needed (no unsafe).

Layout (the upstream pattern, load-bearing):

fuzz/
├── Cargo.toml            alkstore-fuzz (nightly-only bins; own [workspace])
├── rust-toolchain.toml   pins nightly for this subtree only
├── fuzz_targets/         thin fuzz_target! wrappers (3-line each)
├── shared/               alkstore-fuzz-shared — STABLE-toolchain library
│   └── src/…             invariant logic + corpus replay (§3)
├── corpus/<target>/      committed hand-made seeds (§4)
├── gen_fuzz_seeds.py     deterministic seed generator (quiche pattern)
├── run-detached.sh       §5 detached runner (copy from alksocks; CWD-independent)
└── README.md

Upstream layout rules that carry over, in alkstore terms:

• Invariant logic lives in fuzz/shared/, not inline in the target binaries — the stable-toolchain corpus replay (cargo test --manifest-path fuzz/shared/Cargo.toml) runs the SAME invariant functions the fuzzer runs. This is the fuzz gate (§6); it is on AGENTS.md's verification commands (landed with wave 7, §9).
• Workspace/publish exclusion: this repo's root Cargo.toml is a workspace-only manifest (no root package) with an explicit members list, so cargo will not auto-discover fuzz/shared/ — but the defensive exclude = ["fuzz"] on the workspace table is added anyway (the alkcall §2 auto-discovery failure's lesson, cheap insurance). Wave 8's publish-prep task verifies the fuzz sidecar cannot leak into any published package (exclude lists on the crates if needed).
• Corpus policy: commit hand-made seeds, gitignore grown corpora (quinn/h2 model). Crash inputs found by campaigns are promoted to committed seeds (the alksocks seed-900/901 posture).
• wasm note: the wave-6 constraint on alkstore-mem (compiles clean for wasm32-unknown-unknown, tokio sync+time only) is unaffected — the fuzz sidecar builds host-side only, and shared must not introduce a dependency that breaks the mem crate's wasm posture (it consumes the same public API).

3. Target inventory

Two feasibility constraints shape the list: every target should drive already-pub API (the family's posture — no #[cfg(fuzzing)] exposure; see §3.2 for the one snag), and campaign targets must be in-process (the pg engine needs a live server — its paths are not campaign targets; §3.5).

3.1 payload_codec (wave 1 — core, alkstore/src/payload.rs)

Input: raw &[u8], plus arbitrary-derived serde_json::Values. Drives encode_payload and payload_as::<T> (over Value and a representative typed-payload set). Invariants:

• no-panic on any byte sequence (the baseline).
• typed-failure partition: every decode failure is Error::Codec — no other error variant, no panic (the ADR-023 §1 fallibility pin, demonstrated across the input space).
• deep-input loudness: serde_json's recursion limit surfaces as typed Codec, never a stack overflow (the deep-nesting probes).
• Value round-trip: encode_payload(v) → payload_as::<Value> yields v for every arbitrary Value (canonicalization is serde_json's, pinned-not-regressed, not re-derived).
• typed round-trip: for the representative payload structs, encode(Value::from(t)) decodes back to t — the ADR-020 §4 "engines store exactly these bytes" contract's decode side.

Seeds: valid JSON of every Value variant, truncations, deep nesting at and past the recursion limit, non-UTF-8 bytes, trailing garbage, oversized numbers, duplicate keys, the Null-decode shape (ADR-023 §1's empty-byte-store corruption note).

Dictionary: json.dict (copy from a sibling).

3.2 every_parse (wave 1 — the three parsers, differential)

Input: arbitrary &[u8] → UTF-8-lossy Strings (lossy is deliberate: the engines receive &str, which is always valid UTF-8 — but the generators should still stress the grammar's edge shapes). Drives all three parse_every_interval implementations:

• SQLite: alkstore-sqlite/src/substrate/queue_ops.rs (parse_every_interval, substrate-owned, Result<i64, String>),
• Postgres: alkstore-postgres/src/scheduler.rs (Result<i64> → InvalidSpec),
• Mem: alkstore-mem/src/scheduler.rs (Result<i64> → InvalidSpec),

plus each engine's next-boundary arithmetic (the checked-add / skip-forward paths — the wave-6 finding class). Invariants:

• no-panic on any string (each parser, each boundary path).
• typed rejection: every rejection is the parser's own typed error (InvalidSpec on pg/mem; the substrate's typed error under the engine's mapping — never a panic or an Ok).
• three-way agreement: the accept-set and the interval value are identical across the three parsers for every input. The contract suite (suite-pinned rows) pins behavioral equivalence on valid specs; fuzzing extends that to arbitrary strings — cheap, and it pins the ADR-009 §2 grammar three ways continuously.
• overflow posture: the boundary paths reject (checked-add → InvalidSpec/equivalent), never wrap or panic — the wave-6 scheduler-register fix's invariant, over the whole interval domain (u64::MAX-shaped digits, the 9223372036854775807 neighborhood, d-unit multipliers).
• the n <= 0 and giant-n arms: leading-zero forms, digit overflow at exactly i64::MAX/86400, positive-zero rejection.

The exposure design point (wave-7 decomposition resolves): the three parsers are pub(crate) — the family's "drive already-pub APIs" posture has no precedent for this. Options named for decomposition:

• (a) drive through the public schedule() path against the mem (and SQLite — in-process, temp files) factories with arbitrary spec strings: pure public surface, but slower per-exec and no byte-level differential (behavioral only).
• (b) a #[doc(hidden)] narrow exposure mirroring the existing Job::from_row / Schedule::new constructor precedent (ADR-017 §3 consumer-read policy does not govern engine-internal test surfaces; the parsers are engine-owned, not contract surface) — keeps the fast byte-level differential target the upstream docs favor.
• (c) a dev-only feature flag (fuzzing), rejected by the family's no-#[cfg(fuzzing)] experience unless the harness shape provably needs it.

Lean: (b) for the fast differential, (a) as its in-suite companion (the suite already pins behavior; (a) adds no new rows, only optional corpus seeds). Decomposition decides; this doc does not pre-commit.

3.3 name_validation (wave 1 — core, alkstore/src/validation.rs)

Input: arbitrary &[u8] → strings, all NameKinds, lengths spanning the caps. Drives validate_name / validate_shared_name / validate_local_name. Invariants: total (no panic); deterministic accept/reject; rejection only via typed Error::InvalidName; the reserved-prefix rule (__alkstore_, ADR-014 §1's shared namespace) rejects in every casing/boundary form the fuzzers find; length boundary exactness at the cap (the ±1 neighborhood).

Cheapest target; low expected yield but keeps the entry-point domain guard demonstrated (the gate reviews audit guard-order discipline; this demonstrates it over the whole string space).

3.4 mem_opseq (wave 2 — deferred, post-release)

The stateful op-sequence target, the alktty session_opseq / alkcall manager_routing analog: MemStore::new() + an arbitrary-derived op sequence (enqueue/claim/ack/reclaim/dead-letter/ sweep/publish/trim/save_offset/try_lock/lock-extend/notify/notify_tx/ schedule-tick/outbox enqueue, interleaved with tx begin/commit/drop and close), driven over a current-thread tokio runtime (the suite's mem-column flavor). Invariant sketch (decomposition firms it):

• no-panic across any op sequence;
• the ADR-010 queue state machine's legality (no transition the state machine forbids; reclaim-eats-attempt; dead-letter only from exhausted attempts);
• per-consumer stream offsets monotone; receiver arms correct after trim and close;
• tx overlay: commit-merge atomicity, drop-=-discard (no ghost rows), wake-at-commit — the suite's tx rows as invariants over random interleavings instead of authored ones;
• lock TTL expiry/re-acquisition; the loser's clean None.

Deferred for the same reasons alksocks deferred its wave 2: wave 1 covers the byte surfaces; this is a large harness; and the crate is pre-release — regression protection matters most once the surface freezes. Note the mem engine is the only engine fully fast enough for this (SQLite is feasible but file-bound per-op; pg needs a server).

3.5 What is deliberately NOT a target

• The pg engine's server-reachable paths (forwarder, listener, pool open, DSN handling) — not in-process; campaigns cannot drive them without a live server. Their failure arms are owned by the wave-4/5/6 test suite (the forwarder reconnect tests etc.).
• The SQLite substrate internals beyond the public surface — substrate edits are governed by the lineage-fidelity diff discipline (ADR-012 §3 / ADR-018), not by fuzzing; its inherited test floor remains the owner of that surface. The every_parse target touches it only through the public/hidden exposure decision (§3.2).
• Arithmetic beyond §3.2 (backoff curve, opts resolution) for wave 1 — the suite pins their outputs three-way; fuzzing them is a wave-2 candidate if the opseq target's invariants want them. The equal-jitter curve's determinism makes fuzz yield unlikely.
• wasm targets — libFuzzer is host-side; the mem wasm acceptance stays compile-clean (§2's wasm note).

4. Scope: one implemented wave + a deferred second

Wave 1 (wave 7 in the plan, this session's scope): the three targets of §3.1–3.3 — payload_codec, every_parse, name_validation. All in-process, all fast (byte/grammar-level), all invariants sketched above. Sequencing mirrors upstream: scaffold (layout of §2, runner, dict, seed generator) → targets in parallel → seeds committed → 10–30 min detached campaigns per target → fix anything found (crash inputs promoted to seeds) → corpus replay green.

Wave 2 (deferred — post-release): mem_opseq (§3.4). Recorded in §3.4 so the deferral is a named scope decision, not a pending-client hedge.

The wave's decomposition resolves the named open points: §3.2's exposure decision, the AGENTS.md verification-list flip (corpus replay joins the merge gates when the wave lands), and the publish-exclusion verification (§2).

5. Operational isolation: campaigns never share fate with the agent session

Inherited verbatim from the upstream docs (alksocks §5) — an environment constraint, not a nice-to-have. Three layers, all mandatory: (1) libFuzzer soft limits (-rss_limit_mb=2048 -malloc_limit_mb=2048) — never ulimit -v under ASAN; (2) fork mode (-fork=1) as blast-radius containment; (3) process detachment — run-detached.sh wraps cargo fuzz run in setsid + nohup + full redirection; poll, never wait. The alksocks runner is copied as-is (it is CWD-independent and repo-layout-agnostic). Also inherited: cargo fuzz run returns before its last fork-mode children fully exit — pgrep -af to LIST, kill by explicit pid from the runner's echoed pid=, never pattern-kill-and-recheck in one call.

6. No hosted CI

The standing policy applies identically (per all four upstream docs): no hosted CI; no Actions/Gitea Actions workflow files. Corpus replay is the fuzz gate as plain cargo test: cargo test --manifest-path fuzz/shared/Cargo.toml replays every committed seed through the invariant functions. It is on AGENTS.md's verification commands (landed with wave 7, §9). Campaigns run manually via the detached runner — e.g. before the release, or after touching payload.rs, the three parsers, or validation.rs.

7. Verification commands

cargo test                                          # workspace gates (stable)
cargo test --manifest-path fuzz/shared/Cargo.toml   # corpus replay (fuzz gate, from wave 7)
cargo fuzz build                                    # in fuzz/ (nightly, ASAN)
fuzz/run-detached.sh <target>                       # campaigns (never foreground)
cargo clippy --all-targets -- -D warnings
cargo fmt --check
cargo clippy --manifest-path fuzz/shared/Cargo.toml --all-targets -- -D warnings
cargo fmt --manifest-path fuzz/shared/Cargo.toml --check
cargo check -p alkstore-mem --target wasm32-unknown-unknown   # mem's wasm acceptance

Campaign flag starting points (per target, tuned at wave 7): payload_codec -max_len=65536 -dict=json.dict; every_parse -max_len=4096 -use_value_profile=1; name_validation -max_len=4096 -use_value_profile=1.

8. References

• Upstream docs (authoritative for tooling, runner, corpus policy, no-CI posture): alkcall / alktunnels / alksocks / alktty, paths in the header.
• RUSTSEC-2026-0037 (quinn-proto — the maintainers' fuzzing-coverage statement; the upstream §1 datapoint).
• This crate: ADR-008 §5 (error taxonomy), ADR-009 §2 (the @every grammar), ADR-010 (queue state machine), ADR-012 §2 (one-owner parsers/arithmetic), ADR-017 §3–§4 (exposure/versioning policy), ADR-020 §4 (payload encoding posture), ADR-023 §1 (encode fallibility), ADR-024 §5 (mem wasm acceptance); the wave-6 review gate's overflow findings (tasks/review-wave-6.md); docs/plans/implementation.md §Decided points (the adoption record).

---

9. Implementation log — wave 7 (2026-10-10)

• 2026-10-10 — wave 1's three targets implemented (the decomposition's target tasks, all via the §2 layout: stable fuzz/shared invariant library, thin fuzz_target! binaries, committed seed corpora from gen_fuzz_seeds.py — 97 + 66 + 44). §3.2 resolved to the option-(b) #[doc(hidden)] exposure (doc-hidden re-exports of the three parsers + skip_forward; the every_parse campaign's two earlier findings — the boundary-arithmetic division-by-zero/negative-interval wrap and the i64 gap-subtraction overflow, fixed crate-side and registered as PROVENANCE D-38). The §3.2 suite-companion option-(a) added no suite rows — suite-pinned rows already carried equivalence; noted here per the decomposition's resolution scope.
• 2026-10-10 — first campaign finding: the Value round-trip invariant over-tightened (harness-model, not a crate bug). The payload_codec campaign crashed ~38 s in on raw input 46741844073709.51615 (crash-0ce050f1…, exit 77): decode(encode(v)) ≠ v — 46741844073709.52 vs 46741844073709.516. Reproduced and characterized probe-side (serde_json 1.0.152): its default fast-path f64 parser is not exactly correctly-rounded — parsing serde_json's own ryu serialization can drift 1–2 ULPs (measured over 600k random-bits finite f64s: ~30% fail one parse∘serialize∘parse pass, max drift = 2.00 ULP, overwhelmingly exactly 1 ULP; the one-pass re-parse is not a fixpoint at extreme significand/exponent shapes), so the byte-level closure holds only outside float leaves. This is serde_json's documented shipping behavior, and ADR-020 §4 pins exactly that — "not pinned to a serde_json feature flavor … beyond default shipping behavior" — so enabling its float_roundtrip feature would be an ADR deviation, not a fix. The harness was the defect: it pinned the suite's example-based round-trip equality (authored floats are ryu-stable) over the whole raw input space. Fixed in fuzz/shared: the closure is now exact outside Number float leaves, each float leaf drift-bounded at 2 ULPs (pinned-not-regressed), plus a decode-determinism pin (two parses of the same stored bytes are identical). The crash input is promoted to corpus/payload_codec/seed-900; the campaign re-ran clean.
• 2026-10-10 — wave-1 15-minute detached campaigns run (§6 runner, -fork=1 -rss_limit_mb=2048 -malloc_limit_mb=2048 -timeout=25; flags per §7): all three exited 0, oom/timeout/crash: 0/0/0 throughout, empty per-target artifact dirs at completion (the one first-run crash artifact above is the promoted finding's evidence).
  • payload_codec (post-fix re-run; the pre-fix run died at the finding) — 927 s, ~586k execs (~650 exec/s), 41 fork jobs, coverage 2395 edges (ft 7608, in-memory corp 1544 from the 98 committed seeds).
  • every_parse — 927 s, ~3.7M execs (~4.1k exec/s), 41 fork jobs, coverage saturated at 285 edges (ft 3898, corp 805 from the 66 committed seeds) — the three parsers' grammar space is fully enumerated post-fix; the earlier boundary-arithmetic findings were the campaign-relevant depth.
  • name_validation — 939 s, ~59k execs (~66 exec/s), 41 fork jobs, coverage 327 edges (ft 3248, corp 378 from the 44 committed seeds); the corpus-replay seeds already covered the domain guards, and the slow exec/s profile is the long-string shapes, not a harness defect.
  • Grown corpora were not merged into the committed seeds (the §2 policy; fuzz/.gitignore now tracks only seed-* files — the scaffold's directory-level negation leaked SHA-named grown units into git status during the campaigns, fixed here).
• NO crate bugs found by the campaigns. The sole campaign finding was the harness-model defect above (fixed harness-side); every contract invariant held across the adversarial input space: the typed-failure partition (Codec only), deep-input loudness at serde_json's recursion limit, the empty-byte-store loudness (ADR-023 §1), the typed round-trip batteries, the three-way parser agreement and overflow posture, and name-validation determinism/typedness. That is the family's successful-negative-result posture (alksocks §7's first campaigns: zero crate bugs after the two earlier wave-7 crate-side boundary-arithmetic fixes, which the every_parse task itself landed).