alkblobs/tasks/engine-local.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
engine-local	local large engine — sharded files, stage-then-commit-rename, pread cursors	completed		moderate	medium	component	implementation	wave-2 engines local

Description

Implement the local large-tier engine (ungated — dependency-free, always present; ADR-003/008) in src/local.rs: the ADR-003 backend as specced.

• Constructor: takes a root directory (tests use fresh temp dirs). Layout: flat {hex-prefix}/{hex-prefix}/{hash} sharding (iroh's surviving conclusion — limits directory size on huge pools). Named design point: the filename encoding — lowercase hex of the full key bytes (Key::as_bytes, alg-tagged and self-describing per ADR-002's persistence-boundary ruling), prefixes taken from the hex string; record the decision (and the list()-decode round-trip) for the review gate.
• Staged put (the staged-put contract realized, ADR-010 §1): stage-then-commit-rename. put_staged hands out a staging cursor backed by a stage file in a staging area (e.g. {root}/staging/, unique per stage); append writes through; commit renames the stage file into its sharded committed path (creating parent dirs) — publish is the rename; every failure path converges on discard with zero residue (commit failure self-discards; there is no post-failure discard to drive). Drop is best-effort residue cleanup — and per the wave-1 review's recorded observation, this engine's hygiene tests must cover its own cancellation story (a staging cursor dropped without commit/discard leaves no residue).
• Read cursor: the streaming family — a pread loop over the committed file (immutable once committed), read_at clamping at EOF pread-style, the provided read_range composing the slice-digest convention. Blocking file work lives in spawn_blocking inside the impl (store-api.md §Concurrency posture; the store never blocks the executor).
• Whole-value put: the large tier's settled form for direct whole-value writes — write-then-rename into the sharded path, CAS first-writer-wins (an existing file for the key is never overwritten).
• size: file stat — metadata-only, never a content load.
• list: walk the sharded tree, decode filenames to key bytes — complete and well-formed by contract (a malformed list is as deadly as an incomplete one, ADR-003).
• delete: remove the committed file; Err(Error::Missing) when absent. Virgin store (fresh root): reads are no-ops, list empty.
• Exact-count stage-hygiene tests (POC #3 finding A5's shape, store-api.md invariant 3): N failed puts leave zero stage files and zero committed entries; N successful puts leave exactly N committed entries and zero stage files — over both failure modes (explicit discard, failed commit) plus the cancellation/Drop form.

The conformance column itself is suite-column-local (separate task); this task ships the engine and its in-module unit tests only.

Acceptance Criteria

• src/local.rs implements Backend completely (ungated), with the sharded layout, the streaming pread cursor, and the stage-then-commit-rename staging cursor
• Filename encoding decided and recorded (self-describing key bytes; list()-decode round-trips)
• All file I/O off the async path (spawn_blocking); no blocking on the executor
• Exact-count stage-hygiene tests green over discard, failed commit, and Drop-cancellation forms — zero residue on every failure path, exactly one committed entry per success
• Virgin-store reads are no-ops on a fresh root; size is a stat, never a content load; list is complete and well-formed
• cargo test green (local is ungated — in the default build); cargo clippy --all-targets -- -D warnings, cargo fmt --check green

References

• docs/architecture/backends-and-dispatch.md (§large tier — local engine)
• docs/architecture/decisions/003-backend-contract-and-dispatch.md (stage-then-commit-rename, sharding)
• docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract)
• docs/architecture/store-api.md (§Concurrency posture; §Invariants 3)
• docs/research/poc-largeblob-findings.md (findings A2/A5 — pread soundness, stage hygiene)
• tasks/review-wave-1.md (Notes — the cancellation-residue observation this task owns)

Notes

Filename encoding (the named design point, recorded for the review gate): committed filenames are the lowercase hex of the full key bytes (Key::as_bytes — the alg-tagged, self-describing encoding ADR-002's persistence-boundary ruling requires); shard prefixes are taken from the hex string itself ({hex[..2]}/{hex[2..4]}/{hex}), so the sharded layout is fully decodable from the filesystem tree alone. list decodes each filename back through Key::from_bytes; the round-trip is asserted over both algorithms (list_is_complete_and_well_formed, sharded_layout_and_filename_encoding). A walk entry that does not decode — foreign junk or corruption — is a fatal list error (Error::Io naming the offending path), never a silent skip: a malformed list is as deadly as an incomplete one (ADR-003). Only the engine's staging area ({root}/staging/) is a known non-shard entry and is excluded from the walk.

• Constructor — LocalBackend::new(root) (async) creates the root and the staging area in spawn_blocking. Fresh temp dirs in tests (TempRoot helper, Drop-removed idempotently).
• Blocking discipline — every file operation (constructor, stat, open, pread, append, commit-rename, CAS put's write-then-rename, delete, list walk, discard) runs in tokio::task::spawn_blocking; nothing blocks the executor. The committed file is immutable once published, so the read cursor preads against an open handle (FileExt::read_at on unix; a seek-based fallback otherwise); the staging cursor is the sole writer of its stage file, tracking the staged length in an atomic (reads of it clamp at the staged EOF).
• CAS posture — size/has ride one metadata stat; both put forms publish check-then-rename: an existing committed file is never overwritten, and a losing concurrent re-put converges bit-identically by content-addressing (worst case an idempotent rename).
• Commit-failure injection — #[cfg(test)] fail_commit on the backend/cursor, mirroring the mem engine's knob, stands in for the media failure; real failures (rename, parent creation, dedup) take the same self-discard path (stage file removed, Err returned — no post-failure discard to drive).
• Drop cancellation — the staging cursor's Drop best-effort removes a stage file left unconsumed (synchronous remove; the commit/discard paths clear the path first, so Drop is a no-op on them). Covered by drop_cancellation_leaves_zero_residue.
• The conformance column was left to suite-column-local as scoped — no tests/conformance changes in this task.

Summary

Implemented the local large-tier engine (ungated) in src/local.rs: LocalBackend implements Backend completely (name local) over an Arc<PathBuf> root. Committed layout is the flat {hex-prefix}/{hex-prefix}/{hex} sharding with lowercase-hex-of- Key::as_bytes filenames (self-describing; the two-level prefixes are taken from the hex string itself — the decision and its decode round-trip recorded in Notes). put is write-then-rename into the sharded path, CAS first-writer-wins; put_staged hands out LocalStaging — the stage-then-commit-rename contract: a unique stage file under {root}/staging/, appends write through in spawn_blocking, commit renames the stage into its sharded path (creating parents) — publish is the rename — and every failure path (explicit discard, failed commit — including injected media failure, Drop cancellation) converges on zero residue. get hands out LocalCursor, the streaming pread body over the immutable committed file (EOF-clamped reads, the provided read_range composing the slice-digest convention); size stats the file, never a content load; list walks the sharded tree, decodes filenames back to key bytes, and is fatal (Error::Io, path named) on anything that does not decode — complete and well-formed by contract; virgin-store reads are no-ops; delete carries Missing. All file I/O runs in spawn_blocking (store-api.md §Concurrency posture).

Eleven in-module unit tests mirror the mem column's suite and cover this engine's stage-hygiene story: the Backend method contract, list completeness/well-formedness plus the sharded-layout/filename-encoding round-trip, the streaming cursor read contract (full/over/tail/EOF/ zero-length/range-digest), the empty-entry round-trip, the staged happy path with the verification pass over the staged whole and both re-put forms as dedup no-ops, and the exact-count stage-hygiene gate over all failure forms (discard, commit failure, Drop cancellation — 3N failed puts left zero committed entries and zero stage files) and the success form (N successful puts left exactly N committed entries, zero stage files).

Verification: cargo test green (37 lib + 4 conformance + 3 git-interop; local ungated so included in the default build), cargo clippy --all-targets -- -D warnings green, cargo clippy --all-features --all-targets -- -D warnings green, cargo fmt --check green.