alkblobs/tasks/f3-local-durability.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
f3-local-durability	F-3 — the local engine's fsync-durable publish posture (ADR-015)	completed		moderate	medium	component	implementation	wave-9 review-001 correctness local durability

Description

Review-001 F-3, resolved by OQ-12 → ADR-015 (docs/architecture/decisions/015-local-engine-durability-posture.md — the task implements the ADR's unconditional terms; read it line by line before starting). The local engine (src/local.rs) performs no fsync anywhere; on power loss the rename can be durable while the data blocks are not — a committed file of zeros/garbage under a valid key, silent in the tier whose whole job is durability-of-large-content (and silent because ADR-006's no-reverify-on-read posture plus the unix pread's zero-fill never catch it).

Landing, per the ADR's Decision section:

1. fsync-durable publish — file fsync (sync_all) before the directory-entry mutation that publishes the entry; directory fsync after it — on both the put's known-length publish and the staged put's commit. Scope is the publish paths only (the correctness boundary): staging intermediates, delete paths, and dedup discard-moves are not fsynced (the ADR's asymmetry-aware scope — do not over-apply).
2. The publish is no-replace — F-3.2's check-then-act (committed.exists() → rename, where rename silently replaces) is closed by hardlink-into-place: link(stage, committed) (atomically fails when the destination exists), then remove the staging file, then the directory fsync. The loser of a publish race fails loudly and discards its stage; no committed path is ever replaced — the module's pinned invariant held by the filesystem, not by discipline. (renameat2 RENAME_NOREPLACE is the named alternative, not the shipped primitive.)
3. Short preads are errors — the unix pread_all path (src/local.rs:226) treats a short read against the known length as an error, agreeing with the non-unix read_exact path (today the platforms disagree on the corruption shape; the platform agreement is unconditional in the ADR, independent of the durability posture).
4. A partial write failure discards the stage — any write_all error mid-staging discards the staging file and fails the put (F-3.3: the staged read window can never serve hole-plus-shifted content). The stage-hygiene invariant's "every failure path discards" rule extends to mid-write failures, exact-count tested (the wave-2 hygiene test shape).

Also per the ADR: no constructor knob (rejected for v0.1.0 — do not add one), and the module doc gains the operator-facing durability contract ("commits are crash-durable and never corrupt; crashes cost recent in-flight puts, restored by re-put") — the posture row that ends the asymmetry with sqlite's documented synchronous=NORMAL.

Tests: the no-replace race (a loser's link fails, the committed file is never overwritten, the stage discards), the truncation error on the unix pread path (and the non-unix agreement), the write-failure discard exact counts, and the fsync calls' presence on the publish paths (drivable via a testable seam or instrumentation — the mechanism is the task's design point, recorded; crash-window behavior itself is simulated, not crash-tested).

Acceptance Criteria

• File fsync before publish + directory fsync after, on the known-length publish and the staged commit; staging/delete/ discard paths not fsynced (the ADR's scope held)
• The publish is no-replace hardlink-into-place; a publish race's loser fails loudly and discards; no committed path is ever replaced (race-tested)
• Short preads error on the unix path (platform agreement with the non-unix read_exact path; tested)
• A mid-write write_all failure discards the stage and fails the put — exact-count tested
• No constructor knob added; the module doc carries the operator-facing durability contract per ADR-015
• Gates green: cargo test (default), cargo test --all-features server-less, clippy --all-targets -- -D warnings across the feature variants, cargo fmt --check (no pg dependency — this task is server-less; the live-harness matrix runs at the wave gate)

References

• docs/architecture/decisions/015-local-engine-durability-posture.md (the ADR this implements — OQ-12's resolution)
• docs/reviews/001-storage-surface-general-review.md (F-3 and its compounding pieces 1-3)
• docs/plans/implementation.md (§Wave 9 — "the ADR's unconditional terms")
• src/local.rs (the whole engine: publish paths, pread_all, the staging write path, the module doc's pinned invariant)
• docs/architecture/decisions/006-verification-posture-and-transfer-encoding.md (the no-reverify-on-read posture that makes the corruption silent)
• tasks/engine-local.md, tasks/suite-column-local.md (the stage-hygiene exact-count test shapes this extends)

Notes

• The fsync-testability mechanism (the task's recorded design point): a SyncProbe instrumentation type — production-side a zero-sized default marker, cfg(test)-side an in-instance log the publish helper records its steps into (SyncEvent: FileSync/Link/StageUnlinked/DirSync). Chosen over (a) a global counter — it would cross-observe concurrent tests at default parallelism and assert impossible-to-isolate exact counts — and over (b) a fsync-trait seam — heavier surface for no extra truth. The log rides the engine instance (instrument_publish() returns the shared log's Arc), so tests assert exact event sequences — the fsyncs' presence and their order around the link (file sync before the directory-entry mutation, dir sync after) — with no cross-test pollution and no lock held across an awaited publish.
• Dedup = the losing-link arm, deterministically testable: both publish sites' exists() pre-checks were removed, so every publish is a link attempt — a publish race's loser (or a dedup re-put) is exactly the AlreadyExists arm, reachable in a sequential test. Trade recorded: a re-put now writes and fsyncs its stage copy before learning it is dedup (rare call; the uniform primitive is worth it). The engine's CAS contract stands unchanged: the loser lands Ok (dedup), stage discarded, first writer's bytes — "fails loudly" is the primitive's refusal (no silent replacement, ever); the engine-side mapping stays the dedup form the pinned tests assert.
• Non-unix note: std offers no primitive to open a directory, so the directory-fsync leg is a recorded no-op there (sync_dir cfg-split mirrors the existing pread_all split); the file-fsync leg and the no-replace link primitive are unconditional (the ADR itself names std::fs::hard_link as behaving likewise on Windows).
• The append-failure injection rides the engines' existing cfg(test) knob pattern (inject_fail_append(n) — a countdown of successful appends before the next write fails, disarming once fired per cursor), which is what exact-counts the mid-staging failure form (prefix-then-fail) and the first-write form.
• sync_dir is unix-real / non-unix-no-op; DirSync is recorded in the probe on both (the publish leg's step), so the sequence assertions are platform-stable; the truncation test is platform-true either way (write-true on unix, read_exact on non-unix).

Summary

Landed ADR-015's fsync-durable publish posture in src/local.rs (review-001 F-3, all four terms). The publish helper (fsync_publish) is shared by both forms: the publishing file syncs (sync_all) before hard_link(stage, committed) — the no-replace primitive that atomically fails with AlreadyExists when the entry exists (a losing publish race, or a dedup re-put), discards the stage copy, and lands Ok under the engine's unchanged CAS contract; a winning link unlinks the stage file (a discard-path unlink, unfsynced per the ADR's scope) and fsyncs the committed entry's parent directory. The exists() pre-checks were dropped on both publish sites so the dedup arm is the losing-link arm — the committed file is never overwritten by the filesystem, not by discipline. Staging intermediates, deletes, and discards remain unfsynced (the ADR's scope held and probed). The unix pread_all now errors on a short pread against the clamped span (UnexpectedEof, an Error::Io at the seam) — platform agreement with the non-unix read_exact fallback; a truncated entry can never serve a silent zero-filled tail. A mid-staging write_all failure discards the stage file and fails the put (F-3.3): the staged read window stays the clean written prefix, the cursor self-consumes (a later commit errors rather than publish), and the exact-count ledger is asserted over both the prefix-then-fail and first-write-fail forms. Module doc now carries the operator-facing durability contract ("commits are crash-durable and never corrupt; crashes cost recent in-flight puts, restored by re-put") and the no-replace publish primitive; no constructor knob. Four new in-module tests: the publish fsync sequences on both forms (exact order, intermediates/discard unfsynced), the no-replace race (loser's link, stage discarded, first-writer bytes, no dir fsync on dedup), the truncation error, and the exact-count write-failure ledger.

Verification: cargo test green (132 lib + 54 conformance + 4 git-interop), cargo test --all-features green against the live pg harness (287 + 125 + 4; server-less it skips cleanly — also run), cargo clippy --all-targets -- -D warnings and cargo clippy --all-features --all-targets -- -D warnings green, cargo fmt --check green, cargo doc --no-deps clean.