feat(engines): the local large-tier engine — sharded files, stage-then-commit-rename, pread cursors
- LocalBackend: flat {hex-prefix}/{hex-prefix}/{hex} sharding over
lowercase-hex-of-Key::as_bytes filenames (self-describing; the
list-decode round-trip recorded for the review gate); virgin-store
reads are no-ops
- staged put: unique stage files under {root}/staging/, commit-rename
is the publish, CAS first-writer-wins, every failure path (discard,
failed commit, Drop cancellation) converges on zero residue
- read cursor: pread loop over the immutable committed file, EOF
clamping, provided read_range composing the slice-digest convention
- all file I/O in spawn_blocking; size is a stat, never a content load
- list walks the committed tree and is fatal (Error::Io, path named)
on anything that does not decode — complete and well-formed
- 11 in-module unit tests incl. exact-count stage hygiene over all
failure forms and the success form
Verification: cargo test (44 green), clippy --all-targets/--all-features
-D warnings, fmt --check — all clean
Task: tasks/engine-local.md
This commit is contained in:
1 parent
0b43ea1449
commit
856d436f81
2 files changed
+1100
-10
No files matched your search
+1011
-1
File diff suppressed because it is too large.
Load diff
+89
-9
@@ -1,7 +1,7 @@
|
|||||||
---
|
---
|
||||||
id: engine-local
|
id: engine-local
|
||||||
name: local large engine — sharded files, stage-then-commit-rename, pread cursors
|
name: local large engine — sharded files, stage-then-commit-rename, pread cursors
|
||||||
status: pending
|
status: completed
|
||||||
depends_on: []
|
depends_on: []
|
||||||
scope: moderate
|
scope: moderate
|
||||||
risk: medium
|
risk: medium
|
||||||
@@ -61,19 +61,19 @@ this task ships the engine and its in-module unit tests only.
|
|||||||
|
|
||||||
## Acceptance Criteria
|
## Acceptance Criteria
|
||||||
|
|
||||||
- [ ] `src/local.rs` implements `Backend` completely (ungated), with
|
- [x] `src/local.rs` implements `Backend` completely (ungated), with
|
||||||
the sharded layout, the streaming pread cursor, and the
|
the sharded layout, the streaming pread cursor, and the
|
||||||
stage-then-commit-rename staging cursor
|
stage-then-commit-rename staging cursor
|
||||||
- [ ] Filename encoding decided and recorded (self-describing key
|
- [x] Filename encoding decided and recorded (self-describing key
|
||||||
bytes; list()-decode round-trips)
|
bytes; list()-decode round-trips)
|
||||||
- [ ] All file I/O off the async path (`spawn_blocking`); no blocking
|
- [x] All file I/O off the async path (`spawn_blocking`); no blocking
|
||||||
on the executor
|
on the executor
|
||||||
- [ ] Exact-count stage-hygiene tests green over discard, failed
|
- [x] Exact-count stage-hygiene tests green over discard, failed
|
||||||
commit, and Drop-cancellation forms — zero residue on every
|
commit, and Drop-cancellation forms — zero residue on every
|
||||||
failure path, exactly one committed entry per success
|
failure path, exactly one committed entry per success
|
||||||
- [ ] Virgin-store reads are no-ops on a fresh root; `size` is a stat,
|
- [x] Virgin-store reads are no-ops on a fresh root; `size` is a stat,
|
||||||
never a content load; `list` is complete and well-formed
|
never a content load; `list` is complete and well-formed
|
||||||
- [ ] `cargo test` green (local is ungated — in the default build);
|
- [x] `cargo test` green (local is ungated — in the default build);
|
||||||
`cargo clippy --all-targets -- -D warnings`, `cargo fmt --check`
|
`cargo clippy --all-targets -- -D warnings`, `cargo fmt --check`
|
||||||
green
|
green
|
||||||
|
|
||||||
@@ -88,8 +88,88 @@ this task ships the engine and its in-module unit tests only.
|
|||||||
|
|
||||||
## Notes
|
## Notes
|
||||||
|
|
||||||
> To be filled by implementation agent
|
> **Filename encoding (the named design point, recorded for the review
|
||||||
|
> gate):** committed filenames are the lowercase hex of the full key
|
||||||
|
> bytes (`Key::as_bytes` — the alg-tagged, self-describing encoding
|
||||||
|
> ADR-002's persistence-boundary ruling requires); shard prefixes are
|
||||||
|
> taken from the hex string itself (`{hex[..2]}/{hex[2..4]}/{hex}`), so
|
||||||
|
> the sharded layout is fully decodable from the filesystem tree alone.
|
||||||
|
> `list` decodes each filename back through `Key::from_bytes`; the
|
||||||
|
> round-trip is asserted over both algorithms
|
||||||
|
> (`list_is_complete_and_well_formed`,
|
||||||
|
> `sharded_layout_and_filename_encoding`). A walk entry that does not
|
||||||
|
> decode — foreign junk or corruption — is a **fatal list error**
|
||||||
|
> (`Error::Io` naming the offending path), never a silent skip: a
|
||||||
|
> malformed list is as deadly as an incomplete one (ADR-003). Only the
|
||||||
|
> engine's staging area (`{root}/staging/`) is a known non-shard entry
|
||||||
|
> and is excluded from the walk.
|
||||||
|
|
||||||
|
- **Constructor** — `LocalBackend::new(root)` (async) creates the root
|
||||||
|
and the staging area in `spawn_blocking`. Fresh temp dirs in tests
|
||||||
|
(`TempRoot` helper, Drop-removed idempotently).
|
||||||
|
- **Blocking discipline** — every file operation (constructor, stat,
|
||||||
|
open, pread, append, commit-rename, CAS put's write-then-rename,
|
||||||
|
delete, list walk, discard) runs in `tokio::task::spawn_blocking`;
|
||||||
|
nothing blocks the executor. The committed file is immutable once
|
||||||
|
published, so the read cursor preads against an open handle
|
||||||
|
(`FileExt::read_at` on unix; a seek-based fallback otherwise); the
|
||||||
|
staging cursor is the sole writer of its stage file, tracking the
|
||||||
|
staged length in an atomic (reads of it clamp at the staged EOF).
|
||||||
|
- **CAS posture** — `size`/`has` ride one metadata stat; both put forms
|
||||||
|
publish check-then-rename: an existing committed file is never
|
||||||
|
overwritten, and a losing concurrent re-put converges bit-identically
|
||||||
|
by content-addressing (worst case an idempotent rename).
|
||||||
|
- **Commit-failure injection** — `#[cfg(test)] fail_commit` on the
|
||||||
|
backend/cursor, mirroring the mem engine's knob, stands in for the
|
||||||
|
media failure; real failures (rename, parent creation, dedup) take
|
||||||
|
the same self-discard path (stage file removed, `Err` returned — no
|
||||||
|
post-failure discard to drive).
|
||||||
|
- **Drop cancellation** — the staging cursor's `Drop` best-effort
|
||||||
|
removes a stage file left unconsumed (synchronous remove; the
|
||||||
|
commit/discard paths clear the path first, so Drop is a no-op on
|
||||||
|
them). Covered by `drop_cancellation_leaves_zero_residue`.
|
||||||
|
- The conformance column was left to `suite-column-local` as scoped —
|
||||||
|
no `tests/conformance` changes in this task.
|
||||||
|
|
||||||
## Summary
|
## Summary
|
||||||
|
|
||||||
> To be filled on completion
|
Implemented the `local` large-tier engine (ungated) in `src/local.rs`:
|
||||||
|
`LocalBackend` implements `Backend` completely (name `local`) over an
|
||||||
|
`Arc<PathBuf>` root. Committed layout is the flat
|
||||||
|
`{hex-prefix}/{hex-prefix}/{hex}` sharding with lowercase-hex-of-
|
||||||
|
`Key::as_bytes` filenames (self-describing; the two-level prefixes are
|
||||||
|
taken from the hex string itself — the decision and its decode
|
||||||
|
round-trip recorded in Notes). `put` is write-then-rename into the
|
||||||
|
sharded path, CAS first-writer-wins; `put_staged` hands out
|
||||||
|
`LocalStaging` — the stage-then-commit-rename contract: a unique stage
|
||||||
|
file under `{root}/staging/`, appends write through in `spawn_blocking`,
|
||||||
|
`commit` renames the stage into its sharded path (creating parents) —
|
||||||
|
publish is the rename — and every failure path (explicit discard,
|
||||||
|
failed commit — including injected media failure, Drop cancellation)
|
||||||
|
converges on zero residue. `get` hands out `LocalCursor`, the streaming
|
||||||
|
pread body over the immutable committed file (EOF-clamped reads, the
|
||||||
|
provided `read_range` composing the slice-digest convention); `size`
|
||||||
|
stats the file, never a content load; `list` walks the sharded tree,
|
||||||
|
decodes filenames back to key bytes, and is fatal (`Error::Io`, path
|
||||||
|
named) on anything that does not decode — complete and well-formed by
|
||||||
|
contract; virgin-store reads are no-ops; `delete` carries `Missing`.
|
||||||
|
All file I/O runs in `spawn_blocking` (store-api.md §Concurrency
|
||||||
|
posture).
|
||||||
|
|
||||||
|
Eleven in-module unit tests mirror the mem column's suite and cover
|
||||||
|
this engine's stage-hygiene story: the Backend method contract, list
|
||||||
|
completeness/well-formedness plus the sharded-layout/filename-encoding
|
||||||
|
round-trip, the streaming cursor read contract (full/over/tail/EOF/
|
||||||
|
zero-length/range-digest), the empty-entry round-trip, the staged happy
|
||||||
|
path with the verification pass over the staged whole and both re-put
|
||||||
|
forms as dedup no-ops, and the exact-count stage-hygiene gate over all
|
||||||
|
failure forms (discard, commit failure, Drop cancellation — 3N failed
|
||||||
|
puts left zero committed entries and zero stage files) and the success
|
||||||
|
form (N successful puts left exactly N committed entries, zero stage
|
||||||
|
files).
|
||||||
|
|
||||||
|
Verification: `cargo test` green (37 lib + 4 conformance + 3
|
||||||
|
git-interop; local ungated so included in the default build), `cargo
|
||||||
|
clippy --all-targets -- -D warnings` green, `cargo clippy
|
||||||
|
--all-features --all-targets -- -D warnings` green, `cargo fmt --check`
|
||||||
|
green.
|
||||||
Reference in new issue
Block a user