alkblobs/tasks/engine-local.md

---
id: engine-local
name: local large engine — sharded files, stage-then-commit-rename, pread cursors
status: completed
depends_on: []
scope: moderate
risk: medium
impact: component
level: implementation
tags: [wave-2, engines, local]
---

## Description

Implement the `local` large-tier engine (ungated — dependency-free,
always present; ADR-003/008) in `src/local.rs`: the ADR-003 backend as
specced.

- **Constructor**: takes a root directory (tests use fresh temp dirs).
  Layout: flat `{hex-prefix}/{hex-prefix}/{hash}` sharding (iroh's
  surviving conclusion — limits directory size on huge pools).
  **Named design point**: the filename encoding — lowercase hex of the
  full key bytes (`Key::as_bytes`, alg-tagged and self-describing per
  ADR-002's persistence-boundary ruling), prefixes taken from the hex
  string; record the decision (and the list()-decode round-trip) for
  the review gate.
- **Staged put** (the staged-put contract realized, ADR-010 §1):
  stage-then-commit-rename. `put_staged` hands out a staging cursor
  backed by a stage file in a staging area (e.g. `{root}/staging/`,
  unique per stage); `append` writes through; `commit` renames the
  stage file into its sharded committed path (creating parent dirs) —
  publish is the rename; **every failure path converges on discard with
  zero residue** (commit failure self-discards; there is no
  post-failure discard to drive). `Drop` is best-effort residue cleanup
  — and per the wave-1 review's recorded observation, this engine's
  hygiene tests must cover its own cancellation story (a staging cursor
  dropped without commit/discard leaves no residue).
- **Read cursor**: the **streaming family** — a pread loop over the
  committed file (immutable once committed), `read_at` clamping at EOF
  pread-style, the provided `read_range` composing the slice-digest
  convention. **Blocking file work lives in `spawn_blocking`** inside
  the impl (store-api.md §Concurrency posture; the store never blocks
  the executor).
- **Whole-value `put`**: the large tier's settled form for direct
  whole-value writes — write-then-rename into the sharded path, CAS
  first-writer-wins (an existing file for the key is never overwritten).
- **`size`**: file stat — metadata-only, never a content load.
- **`list`**: walk the sharded tree, decode filenames to key bytes —
  complete and well-formed by contract (a malformed list is as deadly
  as an incomplete one, ADR-003).
- **`delete`**: remove the committed file; `Err(Error::Missing)` when
  absent. Virgin store (fresh root): reads are no-ops, `list` empty.
- **Exact-count stage-hygiene tests** (POC #3 finding A5's shape,
  store-api.md invariant 3): N failed puts leave zero stage files and
  zero committed entries; N successful puts leave exactly N committed
  entries and zero stage files — over both failure modes (explicit
  discard, failed commit) plus the cancellation/Drop form.

The conformance column itself is `suite-column-local` (separate task);
this task ships the engine and its in-module unit tests only.

## Acceptance Criteria

- [x] `src/local.rs` implements `Backend` completely (ungated), with
      the sharded layout, the streaming pread cursor, and the
      stage-then-commit-rename staging cursor
- [x] Filename encoding decided and recorded (self-describing key
      bytes; list()-decode round-trips)
- [x] All file I/O off the async path (`spawn_blocking`); no blocking
      on the executor
- [x] Exact-count stage-hygiene tests green over discard, failed
      commit, and Drop-cancellation forms — zero residue on every
      failure path, exactly one committed entry per success
- [x] Virgin-store reads are no-ops on a fresh root; `size` is a stat,
      never a content load; `list` is complete and well-formed
- [x] `cargo test` green (local is ungated — in the default build);
      `cargo clippy --all-targets -- -D warnings`, `cargo fmt --check`
      green

## References

- docs/architecture/backends-and-dispatch.md (§large tier — local engine)
- docs/architecture/decisions/003-backend-contract-and-dispatch.md (stage-then-commit-rename, sharding)
- docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §1 (the staged-put contract)
- docs/architecture/store-api.md (§Concurrency posture; §Invariants 3)
- docs/research/poc-largeblob-findings.md (findings A2/A5 — pread soundness, stage hygiene)
- tasks/review-wave-1.md (Notes — the cancellation-residue observation this task owns)

## Notes

> **Filename encoding (the named design point, recorded for the review
> gate):** committed filenames are the lowercase hex of the full key
> bytes (`Key::as_bytes` — the alg-tagged, self-describing encoding
> ADR-002's persistence-boundary ruling requires); shard prefixes are
> taken from the hex string itself (`{hex[..2]}/{hex[2..4]}/{hex}`), so
> the sharded layout is fully decodable from the filesystem tree alone.
> `list` decodes each filename back through `Key::from_bytes`; the
> round-trip is asserted over both algorithms
> (`list_is_complete_and_well_formed`,
> `sharded_layout_and_filename_encoding`). A walk entry that does not
> decode — foreign junk or corruption — is a **fatal list error**
> (`Error::Io` naming the offending path), never a silent skip: a
> malformed list is as deadly as an incomplete one (ADR-003). Only the
> engine's staging area (`{root}/staging/`) is a known non-shard entry
> and is excluded from the walk.

- **Constructor** — `LocalBackend::new(root)` (async) creates the root
  and the staging area in `spawn_blocking`. Fresh temp dirs in tests
  (`TempRoot` helper, Drop-removed idempotently).
- **Blocking discipline** — every file operation (constructor, stat,
  open, pread, append, commit-rename, CAS put's write-then-rename,
  delete, list walk, discard) runs in `tokio::task::spawn_blocking`;
  nothing blocks the executor. The committed file is immutable once
  published, so the read cursor preads against an open handle
  (`FileExt::read_at` on unix; a seek-based fallback otherwise); the
  staging cursor is the sole writer of its stage file, tracking the
  staged length in an atomic (reads of it clamp at the staged EOF).
- **CAS posture** — `size`/`has` ride one metadata stat; both put forms
  publish check-then-rename: an existing committed file is never
  overwritten, and a losing concurrent re-put converges bit-identically
  by content-addressing (worst case an idempotent rename).
- **Commit-failure injection** — `#[cfg(test)] fail_commit` on the
  backend/cursor, mirroring the mem engine's knob, stands in for the
  media failure; real failures (rename, parent creation, dedup) take
  the same self-discard path (stage file removed, `Err` returned — no
  post-failure discard to drive).
- **Drop cancellation** — the staging cursor's `Drop` best-effort
  removes a stage file left unconsumed (synchronous remove; the
  commit/discard paths clear the path first, so Drop is a no-op on
  them). Covered by `drop_cancellation_leaves_zero_residue`.
- The conformance column was left to `suite-column-local` as scoped —
  no `tests/conformance` changes in this task.

## Summary

Implemented the `local` large-tier engine (ungated) in `src/local.rs`:
`LocalBackend` implements `Backend` completely (name `local`) over an
`Arc<PathBuf>` root. Committed layout is the flat
`{hex-prefix}/{hex-prefix}/{hex}` sharding with lowercase-hex-of-
`Key::as_bytes` filenames (self-describing; the two-level prefixes are
taken from the hex string itself — the decision and its decode
round-trip recorded in Notes). `put` is write-then-rename into the
sharded path, CAS first-writer-wins; `put_staged` hands out
`LocalStaging` — the stage-then-commit-rename contract: a unique stage
file under `{root}/staging/`, appends write through in `spawn_blocking`,
`commit` renames the stage into its sharded path (creating parents) —
publish is the rename — and every failure path (explicit discard,
failed commit — including injected media failure, Drop cancellation)
converges on zero residue. `get` hands out `LocalCursor`, the streaming
pread body over the immutable committed file (EOF-clamped reads, the
provided `read_range` composing the slice-digest convention); `size`
stats the file, never a content load; `list` walks the sharded tree,
decodes filenames back to key bytes, and is fatal (`Error::Io`, path
named) on anything that does not decode — complete and well-formed by
contract; virgin-store reads are no-ops; `delete` carries `Missing`.
All file I/O runs in `spawn_blocking` (store-api.md §Concurrency
posture).

Eleven in-module unit tests mirror the mem column's suite and cover
this engine's stage-hygiene story: the Backend method contract, list
completeness/well-formedness plus the sharded-layout/filename-encoding
round-trip, the streaming cursor read contract (full/over/tail/EOF/
zero-length/range-digest), the empty-entry round-trip, the staged happy
path with the verification pass over the staged whole and both re-put
forms as dedup no-ops, and the exact-count stage-hygiene gate over all
failure forms (discard, commit failure, Drop cancellation — 3N failed
puts left zero committed entries and zero stage files) and the success
form (N successful puts left exactly N committed entries, zero stage
files).

Verification: `cargo test` green (37 lib + 4 conformance + 3
git-interop; local ungated so included in the default build), `cargo
clippy --all-targets -- -D warnings` green, `cargo clippy
--all-features --all-targets -- -D warnings` green, `cargo fmt --check`
green.