feat(store): the read paths — get/stat/read_range/has/list on the fall-through rule

- Public `(len, stream)` shape: `Store::get` returns
  `Option<(u64, Blob)>`; `Blob` (src/facade.rs) is a bounded, re-entrant
  windowed reader over the new crate-internal `AnyCursor` tier dispatch
  (mem materializing / local pread body families) — engine cursors
  never leak onto the surface; no chunk-iterator machinery.
- Verification-on-get pinned against ADR-006: kv-tier materialized
  reads whole-blob hash-check at get time (mismatch = Err(Verification),
  never a further fall-through); the large tier's streaming open is
  metadata + handle only (put-path verification + slice digests are its
  story); `read_range` carries slice digests only, on both tiers.
- `stat` rides the trait's `size` probe (never a content load) into
  `EntryMeta { len, algorithm }`; `has` is the pure existence probe;
  both fall through kv -> large like every key-addressed op (ADR-012
  §6.1) — consult order, not routing (never consults `tier_for`).
- Miss shapes enforced (the error module's behavioral note): both-tier
  misses are Ok(None) on get/stat, false on has, Err(Missing) on
  read_range; `src/error.rs` wave-2 deferral notes updated to enforced
  state.
- `list` decided Vec<Key>: materialized, sorted total-order union of
  both tiers (the sweep consumes the engines' enumeration directly);
  malformed pool bytes are fatal `Err(KeyInvalid)`, never a silent
  skip.
- `KeyInvalid` at the facade boundary: ops take opaque key bytes and
  convert via `Key::from_bytes` once (`boundary_key`); engines never
  see typed keys.
- Twelve read-path tests: virgin-store shapes per mode (dual-tier,
  mem-only, kv-only), the `(len, stream)` windows, fall-through,
  kv-first (decoy content in the large tier), both verification
  postures, list union/sort/pool-fatality, and malformed-key
  rejection on every op.

Task: tasks/store-read-paths.md (design points recorded in Notes for
the wave-2 review gate).

Verification: cargo test (default), cargo test --features mem, cargo
test --all-features green; cargo clippy --all-targets -- -D warnings
green in all three feature shapes; cargo fmt --check green; cargo doc
--no-deps clean beyond the pre-existing private-intra-doc-links class.
This commit is contained in:
glm-5.3-flash committed 2026-10-11 22:28:01 +00:00
1 parent 532adfa707
commit f1f505a910
7 files changed
+935 -46

No files matched your search

+5 -2
View File
@@ -21,8 +21,11 @@
//! own modes.
//!
//! The `#![allow(dead_code)]` below marks the seam's scope: `route` is
//! driven by the facade's write path (store-put-paths) and consulted by
//! the facade's read paths; the allow shrinks as those land.
//! driven by the facade's write-side tier selection (`Store::tier_for`),
//! whose own caller is the put seam (store-put-paths); the allow shrinks
//! as those land. Read-side ops do not consult `route` — the
//! fall-through consult order is the facade's (kv first, then large;
//! ADR-012 §6.1).
#![allow(dead_code)]
+11 -9
View File
@@ -26,9 +26,10 @@ pub enum Error {
/// store-api.md §Error model).
///
/// A miss on `get` and `stat` is a value, not an error — they
/// signal absence with `None` and never carry `Missing`
/// (behavioral enforcement lands with the facade in wave 2; this
/// doc comment states the contract here).
/// signal absence with `None` and never carry `Missing`. The
/// facade's read paths enforce this behaviorally: both-tier misses
/// resolve `Ok(None)` on get/stat (see `src/store.rs`'s read-path
/// tests; ADR-012 §6.5 ruling 5; store-api.md §Error model).
#[error("key missing (read_range/delete: get/stat signal absence with None)")]
Missing,
@@ -96,8 +97,9 @@ mod tests {
/// both altitudes — the trait boundary's effective members plus
/// the facade re-exposure with `GcRefuse` added, carried by doc
/// comments and construction discipline rather than two types.
/// Facade behavioral enforcement (`get`/`stat` never carry
/// `Missing`; `Missing`/`KeyInvalid` exclusions) lands in wave 2.
/// Facade behavioral enforcement is in place on the read paths
/// (`get`/`stat` never carry `Missing` — their both-tier misses
/// are `Ok(None)`, pinned by the store.rs read-path tests).
#[test]
fn one_enum_both_altitudes() {
let trait_effective: [Error; 4] = [
@@ -118,10 +120,10 @@ mod tests {
}
/// `get`/`stat` signal absence with `None` and never carry
/// `Missing` (ADR-012 §6.5 ruling 5; store-api.md §Error model).
/// Behavioral enforcement lands with the facade in wave 2; the
/// doc contract on [`Error::Missing`] is stated with this test as
/// its marker.
/// `Missing` (ADR-012 §6.5 ruling 5; store-api.md §Error model)
/// — enforced behaviorally by the facade's `get`/`stat` returning
/// `Option` (the store.rs read-path tests pin the both-tier-miss
/// shapes); this test is the doc contract's in-module marker.
#[test]
fn missing_is_exclusive_to_err_shaped_ops() {
let absence_on_get: Option<String> = None;
+70
View File
@@ -10,10 +10,80 @@
//! semantics never change shape — wave 2 wires the behavior, not the
//! surface.
use std::fmt;
use std::time::SystemTime;
use crate::backend::ReadCursor;
use crate::hashing::Algorithm;
use crate::keys::Key;
use crate::store::AnyCursor;
/// The `get` seam's public stream (`(len, stream)` — store-api.md §Read
/// path): a bounded, re-entrant **windowed reader** over a stored entry,
/// uniform across both tiers with the crate-internal read cursor
/// (ADR-010 §1) staying behind it. The kv tier's windows slice
/// materialized content (bounded by tier policy); the large tier's read
/// window-by-window over the committed file — streaming without ever
/// materializing the whole value. `read_at` is a pread-style bounded
/// window (over-reads clamp at the entry's end; a read at EOF is empty,
/// never an error); `read_range` is the same window plus the ADR-006
/// slice digest — plain SHA-256 over exactly the returned bytes, fixed
/// by convention.
///
/// Constructed by the store core only (via `get`); engine cursors never
/// leak onto this surface. A `Blob` stays valid across concurrent pool
/// activity because committed entries are immutable. No verification
/// rides the windows themselves: canonical verification fires at get
/// time on the kv tier's materialized reads (and at put time), per the
/// store core's recorded verification-on-get posture (ADR-006).
pub struct Blob {
cursor: AnyCursor,
}
impl fmt::Debug for Blob {
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
f.debug_struct("Blob")
.field("len", &self.cursor.len())
.finish()
}
}
impl Blob {
/// Store-core constructor: a `Blob` wraps a cursor-dispatched
/// engine body; consumers obtain one only through the read seam.
pub(crate) fn new(cursor: AnyCursor) -> Blob {
Blob { cursor }
}
/// The entry's length in bytes — the metadata-only length the
/// cursor body carries (the same probe `stat` rides).
pub fn len(&self) -> u64 {
self.cursor.len()
}
/// Whether the entry holds zero bytes.
pub fn is_empty(&self) -> bool {
self.cursor.len() == 0
}
/// The bytes at `[offset, offset + len)` — clamped at the entry's
/// end (pread-style short read); `offset >= len()` (EOF) is empty,
/// never an error.
pub async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, crate::Error> {
self.cursor.read_at(offset, len).await
}
/// The bytes at `[offset, offset + len)` plus the ADR-006 slice
/// digest — SHA-256 over exactly the returned bytes, for the caller
/// to carry out-of-band.
pub async fn read_range(
&self,
offset: u64,
len: u64,
) -> Result<(Vec<u8>, [u8; 32]), crate::Error> {
self.cursor.read_range(offset, len).await
}
}
/// The cheap stat probe's result (store-api.md §Read path): the entry's
/// length and its hash algorithm — the "type" at this layer is the hash
+4 -3
View File
@@ -41,9 +41,10 @@
//! a key is never overwritten — a losing race or a re-put is dedup,
//! landing as `Ok` with the stage discarded.
//!
//! The `#![allow(dead_code)]` below marks the engine's scope: it is a
//! wave-2 deliverable defined ahead of its dispatch driver. The allow
//! shrinks when dispatch names the `local` engine.
//! The `#![allow(dead_code)]` below marks the narrow remaining scope:
//! in default builds (no `mem`) nothing constructs the engine, so
//! [`LocalBackend::new`]/the staged-put bodies have no caller until
//! wave 3's facade constructors and `store-put-paths` name them.
#![allow(dead_code)]
-7
View File
@@ -16,13 +16,6 @@
//! (no media can fail here); the test-only hook noted on [`MemStaging`]
//! exercises the failure form's self-discard hygiene, standing in for the
//! media-failure injection later engines need.
//!
//! The `#![allow(dead_code)]` below marks the engine's scope: it is a
//! wave-2 deliverable defined ahead of its dispatch driver (no store-core
//! or facade wiring reaches it yet). The allow shrinks when dispatch
//! names the mem engine.
#![allow(dead_code)]
use std::collections::BTreeMap;
use std::sync::Arc;
+744 -16
View File
@@ -1,18 +1,20 @@
//! The `Store` facade type (store-api.md §What this is): the public
//! surface consumers program against — constructor modes and the
//! size-threshold dispatch seam. **Skeleton and routing only**: the ops
//! methods land against this type in `store-read-paths` (get/stat/
//! read_range/has/list with the fall-through general rule),
//! `gc-protocol` (sweep, direct delete, the pin/liveness machinery),
//! and `store-put-paths` (put, the kv-only rejection behavior). This
//! module owns the engine representation and the constructor-mode
//! machinery; the pure routing function lives in [`crate::dispatch`].
//! surface consumers program against — constructor modes, the
//! size-threshold dispatch seam, and the read-path ops (`get`/`stat`/
//! `read_range`/`has`/`list` with the fall-through general rule,
//! implemented below). The remaining ops land against this type in
//! `store-put-paths` (put, the kv-only rejection behavior) and
//! `gc-protocol` (sweep, direct delete, the pin/liveness machinery).
//! Direct `delete` is *not* a read path — it lands with the GC
//! protocol's arbitration. This module owns the engine representation
//! and the constructor-mode machinery; the pure routing function lives
//! in [`crate::dispatch`].
//!
//! **Named design point — engine representation: concrete `Store`,
//! crate-internal enum dispatch.** A `Store<Kv: Backend, Large: Backend>`
//! generic was the alternative: rejected. The engines are statically
//! known and in-crate (no `dyn` anywhere — wave-1 design point 3), and
//! they are `pub(crate)` behind the crate-internal [`Backend`][crate::backend::Backend]
//! they are `pub(crate)` behind the crate-internal [`Backend`][crate::backend]
//! trait — a generic facade parameterized over `Backend` implementers
//! would leak engine types through the public surface, force consumers
//! to name engines they must never see (the facade is **engine-blind**,
@@ -52,6 +54,68 @@
//! feature `mem` and constructs the mem engine — and every instance
//! carries its mode for introspection ([`Store::mode`]).
//!
//! **Read-path design points (recorded for the review gate):**
//!
//! - **The fall-through general rule (ADR-012 §6.1; POC #1 finding
//! 6).** Every key-addressed op consults [`KvEngine`] first, then
//! [`LargeEngine`] — there is no length routing on the read side and
//! [`Store::tier_for`] is never consulted by it (the dispatch seam's
//! write-half routing is the put path's; `route` feeds nothing on
//! reads). A miss on both tiers resolves per op's shape: `Ok(None)`
//! on get/stat, `false` on has, `Err(Missing)` on read_range.
//! - **The public stream type (named design point): `Blob`.** `get`
//! returns the pinned `(len, stream)` shape as
//! `Option<(u64, Blob)>` (store-api.md §Read path) — [`Blob`] is a
//! bounded, re-entrant windowed reader over the crate-internal
//! [`AnyCursor`], which dispatches the engine cursor bodies by tier
//! (mem's materializing body for kv, `local`'s pread body for
//! large). Windowed reads are the uniform stream shape across
//! tiers; no chunk-iterator machinery was adopted — the cursor
//! bodies *are* the streaming abstraction, and per-chunk `Stream`-
//! wrapping would re-shape them without changing the underlying
//! reads. Engine cursors never leak onto the public surface:
//! [`AnyCursor`] is `pub(crate)`, `Blob` is opaque but drivable
//! (constructed only by the store core, ADR-010 §1).
//! - **Verification-on-get (named design point, pinned against
//! ADR-006's text).** The whole-blob hash-check fires where the
//! whole content is already in hand — the kv tier's materializing
//! reads. A kv-tier `get` materializes the stored value and
//! re-derives the canonical digest over it, comparing against the
//! key (ADR-006 decision 1: "put and get paths hash-check");
//! mismatch is `Err(Error::Verification)` and **never falls
//! through further** — a verified mismatch is a pool-integrity
//! fact, and consulting another tier past it would be a guess.
//! The large tier's streaming `get` does *not* re-derive at open:
//! opening is metadata + handle only, and the tier's integrity
//! story is the put path's whole-blob verification pass (the
//! complete staged artifact was checked before commit) plus the
//! per-range slice digests `read_range` carries out-of-band
//! (ADR-006 decisions 1–2; the §Context line "git objects
//! self-verify under git's own model when read whole"). `read_range`
//! carries no canonical check on either tier — the slice digest is
//! the convention's whole answer (ADR-006 decision 2). stat/has/
//! list carry no verification (metadata and enumeration only).
//! - **`list` returns `Vec<Key>` (named design point: Vec vs
//! stream).** Whole-pool enumeration, complete by contract
//! (ADR-003 — load-bearing for GC), returned **materialized and
//! sorted in the keys' total order** (deterministic output). The
//! sweep consumes the same crate-internal enumeration directly from
//! the engines (its live-set diffs sort independently), so the
//! facade shape serves consumers, not internal needs. A lazy stream
//! was rejected: every shipped engine enumerates fully and eagerly
//! (the completeness contract *is* the GC-safety property), so a
//! pseudo-lazy wrapper adds machinery without changing the
//! underlying walk. Listed bytes failing [`Key::from_bytes`]
//! reject with `Error::KeyInvalid` — a malformed pool list is
//! fatal, never a silent skip (ADR-003's malformed-list ruling
//! applied at the facade boundary).
//! - **`KeyInvalid` at the facade boundary (ADR-002/003).** Read ops
//! take opaque key *bytes* and convert to typed `Key` at the
//! boundary via [`Key::from_bytes`]: malformed bytes reject with
//! `Error::KeyInvalid` on every key-addressed op, never a guess,
//! and the engines receive only the physical byte encoding — they
//! never see typed keys.
//!
//! **The fleet declaration defers to the engine waves.** ADR-012 §3
//! rules fleet mode a constructor declaration (`fleet: true`), never
//! inferred — but it is a constructor *error* for every non-fleet-valid
@@ -70,21 +134,24 @@
//! [`Store::large`]), and the GC state home ([`crate::gc::GcHome`])
//! the gc-protocol task fills.
//!
//! The `#![allow(dead_code)]` below marks the skeleton's scope: the
//! accessors and engine enums exist for the ops tasks' drivers, and the
//! kv engine set is empty in default builds (sqlite is wave 3). The
//! allow shrinks as the ops tasks land.
//! The `#![allow(dead_code)]` below marks the work still ahead, not the
//! read path (which is driven below): write-side `tier_for`/`threshold`
//! and the GC home wait on `store-put-paths`/`gc-protocol`, and the kv
//! engine set is empty in default builds (mem is feature-gated; sqlite
//! is wave 3). The allow shrinks as those land.
#![allow(dead_code)]
use crate::backend::{Backend, ReadCursor};
use crate::dispatch::{route, Tier};
use crate::facade::{Blob, EntryMeta};
use crate::gc::GcHome;
use crate::local::LocalBackend;
use crate::keys::Key;
use crate::local::{LocalBackend, LocalCursor};
#[cfg(feature = "mem")]
use crate::mem::MemBackend;
use crate::mem::{MemBackend, MemCursor};
#[cfg(feature = "mem")]
use crate::Error;
#[cfg(feature = "mem")]
@@ -150,6 +217,115 @@ pub(crate) enum LargeEngine {
Local(LocalBackend),
}
/// The read path's cursor-dispatch enum (ADR-010 §1's "store core
/// programs the abstraction"): tier dispatch over the engine cursor
/// bodies — the kv tier's materializing body and the large tier's
/// streaming body — behind one crate-internal value that [`Blob`] holds
/// and drives. Crate-internal: engine cursors never appear on the
/// public surface; the facade's stream shape is this enum's uniform
/// [`ReadCursor`] view (`Blob`).
pub(crate) enum AnyCursor {
/// The kv body family: the whole value materialized in memory
/// (bounded by tier policy).
#[cfg(feature = "mem")]
Kv(MemCursor),
/// The large body family: a pread cursor over the committed file.
Large(LocalCursor),
}
impl ReadCursor for AnyCursor {
fn len(&self) -> u64 {
match self {
#[cfg(feature = "mem")]
Self::Kv(cursor) => cursor.len(),
Self::Large(cursor) => cursor.len(),
}
}
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
match self {
#[cfg(feature = "mem")]
Self::Kv(cursor) => cursor.read_at(offset, len).await,
Self::Large(cursor) => cursor.read_at(offset, len).await,
}
}
}
impl KvEngine {
#[allow(unused_variables)] // kv engine set empty until wave 3 (sqlite): no-mem builds never reference `key`
async fn get(&self, key: &[u8]) -> Result<Option<AnyCursor>, Error> {
match *self {
#[cfg(feature = "mem")]
KvEngine::Mem(ref engine) => Ok(engine.get(key).await?.map(AnyCursor::Kv)),
}
}
#[allow(unused_variables)] // same empty-set scope as `get`
async fn has(&self, key: &[u8]) -> Result<bool, Error> {
match *self {
#[cfg(feature = "mem")]
KvEngine::Mem(ref engine) => engine.has(key).await,
}
}
#[allow(unused_variables)] // same empty-set scope as `get`
async fn size(&self, key: &[u8]) -> Result<Option<u64>, Error> {
match *self {
#[cfg(feature = "mem")]
KvEngine::Mem(ref engine) => engine.size(key).await,
}
}
async fn list(&self) -> Result<Vec<Vec<u8>>, Error> {
match *self {
#[cfg(feature = "mem")]
KvEngine::Mem(ref engine) => engine.list().await,
}
}
}
impl LargeEngine {
async fn get(&self, key: &[u8]) -> Result<Option<AnyCursor>, Error> {
match self {
LargeEngine::Local(engine) => Ok(engine.get(key).await?.map(AnyCursor::Large)),
}
}
async fn has(&self, key: &[u8]) -> Result<bool, Error> {
match self {
LargeEngine::Local(engine) => engine.has(key).await,
}
}
async fn size(&self, key: &[u8]) -> Result<Option<u64>, Error> {
match self {
LargeEngine::Local(engine) => engine.size(key).await,
}
}
async fn list(&self) -> Result<Vec<Vec<u8>>, Error> {
match self {
LargeEngine::Local(engine) => engine.list().await,
}
}
}
/// The facade's key-bytes decode: typed `Key` conversion at the store
/// boundary (ADR-002/003) — the one place on the read path where key
/// bytes are interpreted; anything malformed is `Error::KeyInvalid`,
/// never a guess, and never reaches an engine typed.
fn boundary_key(key: &[u8]) -> Result<Key, Error> {
Key::from_bytes(key)
}
/// Constitute the facade's typed keys from an engine's opaque list
/// rows: a failed decode is fatal (`Error::KeyInvalid`) — a malformed
/// pool list is as deadly as an incomplete one and is never silently
/// skipped (ADR-003).
fn typed_keys(raws: Vec<Vec<u8>>) -> Result<Vec<Key>, Error> {
raws.into_iter().map(|raw| boundary_key(&raw)).collect()
}
impl Store {
/// **mem-only mode** (ADR-011 §4): the mem engine alone, no
/// dispatch threshold in effect — a testing/embedder-ephemeral
@@ -268,10 +444,144 @@ impl Store {
}
}
/// The read path (store-api.md §Read path): `get`, `stat`,
/// `read_range`, `has`, `list` — every key-addressed op falling through
/// kv first, then large (ADR-012 §6.1's general rule), typed key
/// conversion at the facade boundary, and the module-doc design points
/// (the `Blob` stream shape, verification-on-get, the `Vec<Key>` list
/// shape) carried in their implementations.
impl Store {
/// `get(key) → Option<(len, stream)>`. Falls through kv → large;
/// a both-tier miss is `Ok(None)` — never `Error::Missing`
/// (ADR-012 §6.5 ruling 5). On a kv hit the materialized read is
/// whole-blob-verified against the key at get time (mismatch:
/// `Err(Error::Verification)`, never a further fall-through) and
/// the returned length is the verified materialized byte length;
/// the large tier's hit opens streaming (metadata + handle, no
/// canonical re-derivation — see the module doc), its length the
/// cursor's metadata-only probe. Both hits return the same
/// `(len, stream)` shape: the stream is [`Blob`] over the tier's
/// cursor body.
pub async fn get(&self, key: &[u8]) -> Result<Option<(u64, Blob)>, Error> {
let key = boundary_key(key)?;
let bytes = key.as_bytes();
let cursor = if let Some(cursor) = self.kv().get(&bytes).await? {
cursor
} else if let Some(cursor) = self.large_get(&bytes).await? {
cursor
} else {
return Ok(None);
};
match cursor {
#[cfg(feature = "mem")]
AnyCursor::Kv(kv) => {
let value = kv.read_at(0, kv.len()).await?;
let derived = key.algorithm().derive(&value, value.len() as u64);
if &derived != key.digest() {
return Err(Error::Verification);
}
Ok(Some((value.len() as u64, Blob::new(AnyCursor::Kv(kv)))))
}
AnyCursor::Large(large) => Ok(Some((large.len(), Blob::new(AnyCursor::Large(large))))),
}
}
/// The large tier's consult — `None` where the mode has no large
/// tier (single-tier modes cannot hold large-tier content).
async fn large_get(&self, bytes: &[u8]) -> Result<Option<AnyCursor>, Error> {
match self.large() {
Some(large) => large.get(bytes).await,
None => Ok(None),
}
}
/// `stat(key) → Option<EntryMeta>` — the cheap probe riding the
/// trait's `size` method (ADR-008): length plus the key's hash
/// algorithm, never a content load, falling through kv → large.
/// A both-tier miss is `Ok(None)`.
pub async fn stat(&self, key: &[u8]) -> Result<Option<EntryMeta>, Error> {
let key = boundary_key(key)?;
let bytes = key.as_bytes();
let len = match self.kv().size(&bytes).await? {
Some(len) => Some(len),
None => match self.large() {
Some(large) => large.size(&bytes).await?,
None => None,
},
};
Ok(len.map(|len| EntryMeta {
len,
algorithm: key.algorithm(),
}))
}
/// `read_range(key, offset, len) → (slice, slice_digest)` — the
/// slice plus the out-of-band SHA-256 slice digest (ADR-006, fixed
/// by convention), composed through the cursor's `read_range` seam.
/// Falls through kv → large exactly as [`Store::get`] does; no
/// canonical verification rides a range (the slice digest is the
/// whole answer, ADR-006 decision 2). A both-tier miss is
/// `Err(Error::Missing)` — the pinned shape difference vs `get`'s
/// `None` (ADR-012 §6.5 ruling 5).
pub async fn read_range(
&self,
key: &[u8],
offset: u64,
len: u64,
) -> Result<(Vec<u8>, [u8; 32]), Error> {
let key = boundary_key(key)?;
let bytes = key.as_bytes();
if let Some(cursor) = self.kv().get(&bytes).await? {
return cursor.read_range(offset, len).await;
}
match self.large_get(&bytes).await? {
Some(cursor) => cursor.read_range(offset, len).await,
None => Err(Error::Missing),
}
}
/// `has(key) → bool` — the pure existence probe (ADR-008's `has`
/// discipline: an existence probe per tier, `local`'s body being
/// the literal size-probe), falling through kv → large. Absent on
/// both tiers: `false`. Virgin-store reads are no-ops.
pub async fn has(&self, key: &[u8]) -> Result<bool, Error> {
let key = boundary_key(key)?;
let bytes = key.as_bytes();
if self.kv().has(&bytes).await? {
return Ok(true);
}
match self.large() {
Some(large) => large.has(&bytes).await,
None => Ok(false),
}
}
/// `list() → Vec<Key>` — whole-pool enumeration of every tier,
/// complete by contract (ADR-003), returned materialized and
/// sorted in the keys' total order (the recorded Vec-vs-stream
/// design point; see the module doc). Malformed pool bytes are
/// fatal: `Err(Error::KeyInvalid)`, never a silent skip.
pub async fn list(&self) -> Result<Vec<Key>, Error> {
let mut keys = typed_keys(self.kv().list().await?)?;
if let Some(large) = self.large() {
keys.extend(typed_keys(large.list().await?)?);
}
keys.sort();
Ok(keys)
}
}
#[cfg(all(test, feature = "mem"))]
mod tests {
use super::*;
use crate::backend::{slice_digest, Backend};
use crate::dispatch::DEFAULT_DISPATCH_THRESHOLD;
use crate::hashing::Algorithm;
use std::fs;
use std::path::{Path, PathBuf};
use std::sync::atomic::{AtomicU64, Ordering};
@@ -416,4 +726,422 @@ mod tests {
let store = Store::mem_only();
let _ = store.gc();
}
/// Entry planters straight into the tier engines (bypassing the
/// put seam, which is a later task): opaque key bytes + value,
/// exactly what the engines' CAS tables accept. Planting
/// mismatching content under a key is how the read path's
/// verification postures are observed.
async fn plant_kv(store: &Store, key: &[u8], value: &[u8]) {
match store.kv() {
KvEngine::Mem(engine) => engine.put(key, value).await.expect("kv plant"),
}
}
async fn plant_large(store: &Store, key: &[u8], value: &[u8]) {
match store.large() {
Some(LargeEngine::Local(engine)) => engine.put(key, value).await.expect("large plant"),
None => panic!("planting into a mode with no large tier"),
}
}
fn key_for(content: &[u8]) -> Key {
Key::derive(content, content.len() as u64)
}
fn sha1_key_for(content: &[u8]) -> Key {
Key::from(Algorithm::GitBlobSha1.derive(content, content.len() as u64))
}
/// The both-tier-miss shapes every read op must resolve to
/// (store-api.md §Error model; ADR-012 §6.5 ruling 5) — pinned on
/// all three constructible modes, virgin-store reads as no-ops.
async fn assert_virgin_read_shapes(store: &Store) {
let key = key_for(b"nothing ever landed").as_bytes();
assert!(
store.get(&key).await.unwrap().is_none(),
"get's both-tier miss is Ok(None)"
);
assert_eq!(store.stat(&key).await.unwrap(), None, "stat misses as None");
assert_eq!(
store.read_range(&key, 0, 16).await.unwrap_err(),
Error::Missing,
"read_range's both-tier miss is Err(Missing) — the pinned shape difference vs get"
);
assert!(
!store.has(&key).await.unwrap(),
"has's both-tier miss is false"
);
assert_eq!(
store.list().await.unwrap(),
Vec::<Key>::new(),
"virgin-store list is empty"
);
}
#[tokio::test]
async fn virgin_store_reads_are_no_ops_dual_tier() {
let root = temp_root("read-virgin-dual");
let store = Store::dual_tier_mem_local(root.path(), 1024)
.await
.expect("dual-tier constructs");
assert_virgin_read_shapes(&store).await;
}
#[tokio::test]
async fn virgin_store_reads_are_no_ops_mem_only() {
let store = Store::mem_only();
assert_virgin_read_shapes(&store).await;
}
#[tokio::test]
async fn virgin_store_reads_are_no_ops_kv_only() {
let store = Store::kv_only_mem(64);
assert_virgin_read_shapes(&store).await;
}
/// `get` returns the pinned `(len, stream)` shape from the kv tier:
/// the tuple carries the metadata length, the `Blob` serves
/// bounded windows (clamped at EOF) and the ADR-006 range seam.
#[tokio::test]
async fn get_returns_the_pinned_len_stream_shape_from_kv() {
let store = Store::mem_only();
let content = b"small-tier content";
let key = key_for(content);
plant_kv(&store, &key.as_bytes(), content).await;
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("entry present");
assert_eq!(len, content.len() as u64, "tuple len is the entry length");
assert_eq!(blob.len(), len);
assert!(!blob.is_empty());
let whole = blob.read_at(0, len).await.expect("full window");
assert_eq!(whole, content, "full window matches");
assert_eq!(
blob.read_at(0, len + 17).await.expect("over-read"),
content,
"over-read clamps at the entry's end"
);
let (slice, digest) = blob.read_range(6, 6).await.expect("ranged window");
assert_eq!(slice, &content[6..12]);
assert_eq!(
digest,
slice_digest(&content[6..12]),
"slice digest is SHA-256 over exactly the returned bytes"
);
let (full, full_digest) = blob.read_range(0, len).await.expect("full range");
assert_eq!(full, content);
assert_eq!(full_digest, slice_digest(content));
}
/// The zero-length degenerate: a kv-tier empty entry get's as
/// `Some((0, blob))`, its windows read empty-never-error, and
/// `stat` carries exact zero.
#[tokio::test]
async fn zero_length_kv_entry_reads_as_empty_blob() {
let store = Store::mem_only();
let key = key_for(b"");
plant_kv(&store, &key.as_bytes(), b"").await;
let (len, blob) = store.get(&key.as_bytes()).await.unwrap().expect("present");
assert_eq!(len, 0);
assert!(blob.is_empty());
assert_eq!(blob.read_at(0, 8).await.unwrap(), Vec::<u8>::new());
assert_eq!(
store.stat(&key.as_bytes()).await.unwrap(),
Some(EntryMeta {
len: 0,
algorithm: key.algorithm()
})
);
}
/// `get`/`stat`/`read_range`/`has` fall through to the large tier
/// on a kv miss — consult order, not length routing: a
/// sub-threshold entry planted only in the large tier is still
/// found even though `tier_for` routes its length to kv.
#[tokio::test]
async fn read_ops_fall_through_kv_miss_to_large() {
let root = temp_root("read-fall-through");
let store = Store::dual_tier_mem_local(root.path(), 1024)
.await
.expect("dual-tier constructs");
let content = b"a sub-threshold entry planted directly in the large tier";
let key = key_for(content);
assert_eq!(
store.tier_for(content.len() as u64),
Tier::Kv,
"the length *routes* kv — the read still lands via the large tier"
);
plant_large(&store, &key.as_bytes(), content).await;
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("fall-through get succeeds")
.expect("large-tier entry found via fall-through");
assert_eq!(len, content.len() as u64);
assert_eq!(blob.read_at(0, len).await.unwrap(), content);
assert_eq!(
store.stat(&key.as_bytes()).await.unwrap(),
Some(EntryMeta {
len: content.len() as u64,
algorithm: key.algorithm()
}),
"stat falls through too"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"has falls through"
);
let (slice, digest) = store
.read_range(&key.as_bytes(), 4, 8)
.await
.expect("read_range falls through");
assert_eq!(slice, &content[4..12]);
assert_eq!(digest, slice_digest(&slice));
}
/// The kv tier is the *first* consult on every key-addressed op:
/// the same key bytes planted with different (mismatching) content
/// in both tiers — get/read_range/stat/has must all answer from
/// the kv tier's entry.
#[tokio::test]
async fn kv_tier_is_the_first_consult_on_every_key_addressed_op() {
let root = temp_root("read-kv-first");
let store = Store::dual_tier_mem_local(root.path(), 1024)
.await
.expect("dual-tier constructs");
const KV_CONTENT: &[u8] = b"the kv tier's entry";
let key = key_for(KV_CONTENT);
plant_kv(&store, &key.as_bytes(), KV_CONTENT).await;
plant_large(&store, &key.as_bytes(), b"decoy content in the large tier").await;
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("kv hit wins");
assert_eq!(
blob.read_at(0, len).await.unwrap(),
KV_CONTENT,
"kv tier first — and its content verifies against the key"
);
let (slice, _) = store
.read_range(&key.as_bytes(), 0, 4)
.await
.expect("ranged");
assert_eq!(
slice,
&KV_CONTENT[0..4],
"read_range also consults kv first"
);
assert!(store.has(&key.as_bytes()).await.unwrap());
assert_eq!(
store.stat(&key.as_bytes()).await.unwrap(),
Some(EntryMeta {
len: KV_CONTENT.len() as u64,
algorithm: key.algorithm()
})
);
}
/// Verification-on-get, kv tier (ADR-006 decision 1): a
/// materialized get hash-checks against the key; mismatch is
/// `Err(Verification)` and never falls through further. `read_range`
/// by contrast carries no canonical check — the slice digest is its
/// whole verification story (ADR-006 decision 2).
#[tokio::test]
async fn kv_get_whole_blob_verifies_and_never_falls_through_past_mismatch() {
let store = Store::mem_only();
let key = key_for(b"the derived address");
let bytes = key.as_bytes();
plant_kv(&store, &bytes, b"content that is not the key").await;
assert_eq!(
store.get(&bytes).await.unwrap_err(),
Error::Verification,
"kv materialized reads hash-check at get time"
);
assert_eq!(
store.read_range(&bytes, 0, 8).await.unwrap(),
(b"content ".to_vec(), slice_digest(b"content ")),
"read_range carries no canonical check — the slice-digest posture"
);
assert!(
store.stat(&bytes).await.unwrap().is_some(),
"stat is a metadata probe, not a verifier"
);
}
/// Verification-on-get, large tier (the recorded posture's other
/// half): a streaming get opens metadata + handle only — it does
/// not re-derive the canonical digest at open. The tier's
/// integrity story is the put path's whole-blob verification pass
/// plus the per-range slice digests (ADR-006 decisions 1–2).
#[tokio::test]
async fn large_streaming_get_does_not_re_derive_at_open() {
let root = temp_root("read-large-no-reverify");
let store = Store::dual_tier_mem_local(root.path(), 8)
.await
.expect("dual-tier constructs");
let key = key_for(b"the derived address");
plant_large(&store, &key.as_bytes(), b"stored bytes").await;
assert!(
store.get(&key.as_bytes()).await.unwrap().is_some(),
"streaming open never re-derives (mismatching content and all)"
);
}
/// `stat` carries len + algorithm for both algorithms (the
/// "type" at this layer is the hash algorithm, ADR-002) and is a
/// `size` ride — the fall-through consult, `None` when absent.
#[tokio::test]
async fn stat_carries_len_and_algorithm_from_size_probes() {
let root = temp_root("read-stat");
let store = Store::dual_tier_mem_local(root.path(), 1024)
.await
.expect("dual-tier constructs");
let sha256_key = key_for(b"canonical");
plant_kv(&store, &sha256_key.as_bytes(), b"canonical").await;
let legacy_key = sha1_key_for(b"legacy");
plant_large(&store, &legacy_key.as_bytes(), b"legacy").await;
assert_eq!(
store.stat(&sha256_key.as_bytes()).await.unwrap(),
Some(EntryMeta {
len: 9,
algorithm: Algorithm::GitBlobSha256
})
);
assert_eq!(
store
.stat(&legacy_key.as_bytes())
.await
.unwrap()
.expect("legacy entry present")
.algorithm,
Algorithm::GitBlobSha1,
"the algorithm comes from the key"
);
assert_eq!(
store.stat(&sha256_key.as_bytes()).await.unwrap(),
store.stat(&sha256_key.as_bytes()).await.unwrap(),
"stat is a pure probe"
);
}
/// `list` returns the sorted, typed union of both tiers'
/// enumerations — both algorithms, dedup-free by key identity.
#[tokio::test]
async fn list_returns_the_sorted_typed_union_of_both_tiers() {
let root = temp_root("read-list");
let store = Store::dual_tier_mem_local(root.path(), 1024)
.await
.expect("dual-tier constructs");
let kv_keys = [
key_for(b"kv entry one"),
sha1_key_for(b"kv legacy"),
key_for(b"kv entry two"),
];
for key in &kv_keys {
plant_kv(&store, &key.as_bytes(), b"kv bytes").await;
}
let large_keys = [key_for(b"large entry one"), key_for(b"large entry two")];
for key in &large_keys {
plant_large(&store, &key.as_bytes(), b"large bytes").await;
}
let mut expected: Vec<Key> = kv_keys.into_iter().chain(large_keys).collect();
expected.sort();
let listed = store.list().await.expect("list succeeds");
assert_eq!(listed, expected, "complete, typed, both tiers");
for window in listed.windows(2) {
assert!(
window[0] < window[1],
"output is sorted in the keys' total order"
);
}
for key in &listed {
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"every listed key addresses a real entry"
);
}
}
/// A malformed pool list is fatal (`Error::KeyInvalid`), never a
/// silent skip — ADR-003's malformed-list ruling applied at the
/// facade boundary; the engines own their own body-level checks.
#[tokio::test]
async fn list_of_malformed_pool_keys_is_fatal() {
let store = Store::mem_only();
plant_kv(&store, &[0x07; 33], b"stray in the table").await;
assert_eq!(
store.list().await.unwrap_err(),
Error::KeyInvalid,
"malformed pool bytes are fatal on list"
);
}
/// `KeyInvalid` at the facade boundary on every key-addressed
/// op (typed conversion happens there and only there, ADR-002/003)
/// — malformed bytes reject on hit paths (a planted entry) and
/// miss paths alike, before any tier is consulted and never as a
/// `None`/`false` guess.
#[tokio::test]
async fn malformed_key_bytes_reject_keyinvalid_on_every_op() {
let store = Store::mem_only();
let anchor = key_for(b"an anchor entry");
plant_kv(&store, &anchor.as_bytes(), b"anchor").await;
let key_bytes = anchor.as_bytes();
let malformed = [
Vec::new(), // empty
key_bytes[..key_bytes.len() - 1].to_vec(), // truncated digest
[0x00u8; 33].to_vec(), // reserved tag, full width
[0x03u8; 33].to_vec(), // unknown tag
[0x01u8; 21].to_vec(), // right tag, wrong digest length
];
for raw in &malformed {
assert_eq!(
store.get(raw).await.unwrap_err(),
Error::KeyInvalid,
"get rejects malformed key bytes: {raw:?}"
);
assert_eq!(
store.stat(raw).await.unwrap_err(),
Error::KeyInvalid,
"stat rejects malformed key bytes: {raw:?}"
);
assert_eq!(
store.read_range(raw, 0, 1).await.unwrap_err(),
Error::KeyInvalid,
"read_range rejects malformed key bytes: {raw:?}"
);
assert_eq!(
store.has(raw).await.unwrap_err(),
Error::KeyInvalid,
"has rejects malformed key bytes: {raw:?}"
);
}
}
}
+101 -9
View File
@@ -1,7 +1,7 @@
---
id: store-read-paths
name: Store read paths — get/stat/read_range/has/list with dispatch fall-through
status: pending
status: completed
depends_on: [store-construction]
scope: moderate
risk: medium
@@ -55,19 +55,19 @@ Implement the facade's read-side ops on the `Store` type (store-api.md
## Acceptance Criteria
- [ ] `get`/`stat`/`read_range`/`has`/`list` implemented with the
- [x] `get`/`stat`/`read_range`/`has`/`list` implemented with the
fall-through general rule (kv first, then large), unit-tested
including the both-tier-miss shapes (`None` vs `Err(Missing)` per
op)
- [ ] The public stream type decided and recorded; engine cursors never
- [x] The public stream type decided and recorded; engine cursors never
leak into the public surface
- [ ] `stat` never fetches content (rides `size`); `EntryMeta` carries
- [x] `stat` never fetches content (rides `size`); `EntryMeta` carries
len + algorithm
- [ ] `read_range` returns slice + SHA-256 slice digest per the ADR-006
- [x] `read_range` returns slice + SHA-256 slice digest per the ADR-006
convention; verification-on-get posture decided and recorded
- [ ] Malformed key bytes reject with `KeyInvalid` at the facade
- [x] Malformed key bytes reject with `KeyInvalid` at the facade
boundary on every key-addressed op
- [ ] `cargo test` green; `cargo test --features mem` and `cargo test
- [x] `cargo test` green; `cargo test --features mem` and `cargo test
--all-features` green; clippy/fmt gates green
## References
@@ -81,8 +81,100 @@ Implement the facade's read-side ops on the `Store` type (store-api.md
## Notes
> To be filled by implementation agent
- **The public stream type decided: `Blob` (in `src/facade.rs`) — a
bounded, re-entrant windowed reader; `get` returns the pinned
`(len, stream)` shape as `Option<(u64, Blob)>`.** The engine cursor
bodies (mem's materializing body, `local`'s pread body) stay behind
a crate-internal tier-dispatch enum `AnyCursor` (`src/store.rs`)
implementing `ReadCursor`; `Blob` exposes `len()`/`is_empty()`,
`read_at(offset, len)` (pread-style clamped windows), and
`read_range(offset, len)` (window + ADR-006 slice digest), constructed
only by the store core. No chunk-iterator machinery was adopted: the
cursor bodies *are* the streaming abstraction (large tier reads
window-by-window over the committed file without ever materializing
the whole; kv is bounded by materialization policy) — a per-chunk
`Stream` wrapper would re-shape them without changing the underlying
reads. Recorded in `src/store.rs` module docs + `Blob`'s doc.
- **Verification-on-get posture (pinned against ADR-006's text):** the
whole-blob hash-check fires where the whole content is already in
hand — the kv tier's materializing reads (ADR-006 decision 1: "put
and get paths hash-check"): kv get materializes, re-derives the
canonical digest over the stored bytes, compares against the key;
mismatch is `Err(Verification)` and never falls through further (a
verified mismatch is a pool-integrity fact; consulting another tier
past it would be a guess). The large tier's streaming get does *not*
re-derive at open (metadata + handle only) — its integrity story is
the put path's whole-blob verification pass plus the per-range slice
digests (ADR-006 decisions 1–2; §Context's "git objects self-verify
under git's own model when read whole"). `read_range` carries no
canonical check on either tier (decision 2: no per-range verification
is derivable). Pinned by tests on both tiers
(`kv_get_whole_blob_verifies_and_never_falls_through_past_mismatch`,
`large_streaming_get_does_not_re_derive_at_open`).
- **`list` decided: `Vec<Key>`** — whole-pool enumeration returned
materialized and sorted in the keys' total order (deterministic,
typed, both algorithms). The sweep consumes the same crate-internal
enumeration directly from the engines (its live-set diffs sort
independently), so the facade shape serves consumers, not internal
needs; a lazy stream was rejected because every shipped engine
enumerates fully and eagerly (the completeness contract *is* the
GC-safety property) — a pseudo-lazy wrapper adds machinery without
changing the underlying walk. Listed bytes failing `Key::from_bytes`
reject `Err(KeyInvalid)`: a malformed pool list is fatal, never a
silent skip (ADR-003's malformed-list ruling applied at the facade
boundary).
- **`KeyInvalid` at the boundary:** the read ops take opaque key
*bytes* (`&[u8]`) and convert via `Key::from_bytes` at the store
boundary (`boundary_key`, one seam in `src/store.rs`) — malformed
bytes reject `Error::KeyInvalid` on get/stat/read_range/has before
any tier is consulted and never as a `None`/`false` guess; engines
receive only the physical encoding. `list()`'s pool decode is the
same one-way door's read face (above).
- **Fall-through is consult order, not routing:** the read path never
consults `tier_for`/`route` — a sub-threshold entry planted only in
the large tier is found (test
`read_ops_fall_through_kv_miss_to_large` asserts the *routing*
disagrees while the read succeeds to make the distinction
observable). kv-first is proven behaviorally by planting different
content under the same key bytes in both tiers
(`kv_tier_is_the_first_consult_on_every_key_addressed_op`).
- **Miss shapes enforced (the error module's behavioral note):**
both-tier misses are `Ok(None)` on get/stat, `false` on has,
`Err(Missing)` on read_range — pinned per mode (dual-tier/mem-only/
kv-only) via `assert_virgin_read_shapes`; `src/error.rs`'s
wave-2 deferral notes updated to the in-place state.
- **Tier dispatch plumbing:** `KvEngine`/`LargeEngine` grew
crate-internal `get`/`has`/`size`/`list` dispatch methods returning
engine-raw shapes (cursor enum / `Option<u64>` / `bool` /
`Vec<Vec<u8>>`); adding a variant per enum (waves 3–5) is slotting
in. `get`'s kv arm (and its verification) is
`#[cfg(feature = "mem")]` — in default builds (no kv engine until
wave 3's sqlite) the large/None arms fully cover the unconstructible
facade.
## Summary
> To be filled on completion
Implemented the facade's read-side ops on `Store` (`src/store.rs`):
`get(key) → Result<Option<(u64, Blob)>>` with the pinned `(len, stream)`
shape, `stat(key) → Result<Option<EntryMeta>>` riding the trait's
`size` probe, `read_range(key, offset, len) → Result<(Vec<u8>, [u8;32])>`
composed through the cursor's `read_range` slice-digest seam, `has(key)`
as the pure existence probe, and `list() → Result<Vec<Key>>` — all on
the kv-first-then-large fall-through rule with typed key conversion at
the facade boundary. Added the public stream type `Blob`
(`src/facade.rs`) over the new crate-internal `AnyCursor` cursor
dispatch and recorded the four named design points (stream shape,
verification-on-get, list return shape, `KeyInvalid` boundary) in
`src/store.rs` module docs; updated the wave-2 deferral notes in
`src/error.rs` to their enforced state and the scope notes in
`src/mem.rs`/`src/local.rs`/`src/dispatch.rs` now that the read path
drives them. Twelve read-path tests pin the miss shapes, the
fall-through order, both verification postures, the list union
(including malformed-pool fatality), and `KeyInvalid` on every
key-addressed op.
Verification: `cargo test` (default), `cargo test --features mem`,
`cargo test --all-features` all green (66 lib tests green under mem);
`cargo clippy --all-targets -- -D warnings` green in all three feature
shapes; `cargo fmt --check` green; `cargo doc --no-deps` clean of new
warning classes beyond the pre-existing private-intra-doc-links class.