- Three `_fleet` twin constructors slot the explicit `fleet: true` declaration in without reshaping the fleet-capable constructors; every pre-existing constructor stays implicitly `fleet: false` with byte-unchanged behavior (the recorded non-event landed as recorded): `kv_only_postgres_fleet` (the re-routing client posture, large = none), `dual_tier_postgres_local_fleet` (the shared-media posture), `dual_tier_postgres_pglo_fleet` (the consolidation posture). - The composite fleet-validity predicate (`fleet_validity_error`) is one function, one sentence, both tiers, run at the constructor keyed off the engine representation: kv = postgres AND (pg-lo on the same pg instance OR local on declared shared media OR none). The structural clauses (mem never, sqlite never, local needs the declaration) are enforced by the constructor set's shape and tested per clause at the predicate level; the pg-lo same-instance clause is the runtime arm: `pg_control_system().system_identifier` compared across one pooled connection per engine — mismatch or failed probe is a constructor error, never a silent pass. - The named design points decided and recorded: the shared-media assertion is a sealed dedicated type (`FleetSharedMedia::asserted()`, never a bare flag); the pin TTL is a fleet constructor parameter (default `DEFAULT_PIN_TTL` 60 s, `Duration::ZERO` rejected at entry) wired through `PostgresBackend::new_with_pin_ttl` to the renewal body (proven against the live server); the owner id mints 32 raw bytes at construction (SHA-256 fold over pid/nanos/instance counter/ASLR anchor), stable per instance, crate-internal. - The store carries its fleet state (`FleetState`: the declaration flag, the TTL, the owner id) — crate-internal (fleet parameters, not API); the one-window-host posture verified intact under fleet (kv host's both-tables GC schema, pg-lo window clauses dormant, the form branch keyed off the engine representation), and the declaration turns nothing else on yet (put/sweep run the non-fleet machinery until the driver tasks). - Constructor docs updated to the landed state (module doc's fleet-declaration paragraph, the three non-fleet recordings, the scope notes, gc.rs/postgres.rs/pg_lo.rs wave-6 notes); `cargo doc --no-deps` clean in the default build. Verification: cargo test (default) green; --features postgres and --features "postgres,pg-lo" green server-less (skipping cleanly) and against the harness server (pglo-poc, :15432 — verified live 2026- 10-12); --all-features green; clippy -D warnings green (default + postgres + postgres,pg-lo + all-features); cargo fmt --check green; cargo doc --no-deps clean.
7627 lines
303 KiB
Rust
7627 lines
303 KiB
Rust
//! The `Store` facade type (store-api.md §What this is): the public
|
||
//! surface consumers program against — constructor modes, the
|
||
//! size-threshold dispatch seam, the write path (`put`, the two internal
|
||
//! put paths behind one signature, the `Batch` batching seam, kv-only
|
||
//! rejection — implemented below), the read-path ops (`get`/`stat`/
|
||
//! `read_range`/`has`/`list` with the fall-through general rule),
|
||
//! and the GC protocol's ops (`sweep`, direct `delete`, the liveness
|
||
//! registration — arbitration-then-delete per ADR-005/012, implemented
|
||
//! below). This module owns the engine representation and the
|
||
//! constructor-mode machinery; the pure routing function lives in
|
||
//! [`crate::dispatch`].
|
||
//!
|
||
//! **Named design point — engine representation: concrete `Store`,
|
||
//! crate-internal enum dispatch.** A `Store<Kv: Backend, Large: Backend>`
|
||
//! generic was the alternative: rejected. The engines are statically
|
||
//! known and in-crate (no `dyn` anywhere — wave-1 design point 3), and
|
||
//! they are `pub(crate)` behind the crate-internal [`Backend`][crate::backend]
|
||
//! trait — a generic facade parameterized over `Backend` implementers
|
||
//! would leak engine types through the public surface, force consumers
|
||
//! to name engines they must never see (the facade is **engine-blind**,
|
||
//! ADR-007/008/010), or push `dyn` onto the surface. The concrete type
|
||
//! holds two crate-internal engine-holding enums (`KvEngine`,
|
||
//! `LargeEngine`) dispatching over the closed five-engine set; the
|
||
//! conformance factory's binding note anticipated exactly this collapse
|
||
//! (tests/conformance/factory.rs — "most plausibly the associated type
|
||
//! collapsing to the concrete facade type across all factories"). The
|
||
//! public surface is engine-blind: consumers name modes and engines
|
||
//! only through named constructors, feature-gated with their engine
|
||
//! (`mem_only` needs feature `mem`; the sqlite constructors are the
|
||
//! default build's — wave 3; the postgres constructors arrived with
|
||
//! wave 4 behind feature `postgres`; the pg-lo constructors arrived
|
||
//! with wave 5 behind feature `pg-lo` — and each slotting is exactly
|
||
//! adding an enum variant and a named constructor, without reshaping).
|
||
//!
|
||
//! **Constructor modes** (ADR-011 §4; ADR-012 §6.3):
|
||
//!
|
||
//! - **dual-tier** (the default): kv + large, size-threshold dispatch
|
||
//! via `crate::dispatch::route`. The wave-2 constructible shape is
|
||
//! `kv = mem` + `large = local` — the dispatch-coverage test shape
|
||
//! ADR-012 §6.3 names (both tiers load-bearing); the wave-3
|
||
//! constructible shape is `kv = sqlite` + `large = local` — the
|
||
//! **default-build production shape** (feature `sqlite` is
|
||
//! default-on, ADR-013 §4); the wave-4 shape is `kv = postgres` +
|
||
//! `large = local` — the **fleet-capable-kv production shape** (the
|
||
//! engine is the fleet host, ADR-012 §4; the constructor stays the
|
||
//! non-fleet declaration — the fleet declaration is the `_fleet`
|
||
//! twin constructors, the module doc's fleet-declaration
|
||
//! paragraph). The wave-5 shapes complete
|
||
//! the dual-tier constructor set: `kv = postgres` + `large = pg-lo`
|
||
//! — the **consolidation posture** (ADR-009's "one engine to
|
||
//! operate"; that constructor completes the fleet-valid engine
|
||
//! set, the module doc's recorded forward reference) — and `kv =
|
||
//! mem` + `large = pg-lo` — the **one-window-host rule's named
|
||
//! test topology** (ADR-012 §5: the durable window hosts on the
|
||
//! large engine when the kv engine is not SQL-backed).
|
||
//! Over-threshold content routes to the large tier; at-or-below to
|
||
//! kv.
|
||
//! - **kv-only**: no large tier; over-threshold puts reject **at put
|
||
//! time** (ADR-012 §6.2 — known-length immediately; unknown-length at
|
||
//! mid-stream threshold overflow). The rejection behavior itself
|
||
//! lands with the put seam; the threshold this mode carries is the
|
||
//! rejection point's value. The wave-2 engine is mem (testing);
|
||
//! the wave-3 engine is sqlite (the durable single-node medium); the
|
||
//! wave-4 engine is postgres (the fleet host's kv-only shape).
|
||
//! pg-lo is not a kv engine — kv-only modes take kv engines only,
|
||
//! and no `kv_only(pg-lo)` constructor exists.
|
||
//! - **mem-only**: the mem engine alone, **no dispatch threshold in
|
||
//! effect** (everything is local and small) — a testing/embedder-
|
||
//! ephemeral posture, never production. Structurally distinct from
|
||
//! kv-only precisely by that threshold absence.
|
||
//!
|
||
//! pg-lo constructors are dual-tier only (pg-lo is a large engine;
|
||
//! the pairing is a valid non-fleet construction per ADR-012 §5),
|
||
//! and no sqlite + pg-lo constructor is named this wave — added on
|
||
//! demand.
|
||
//!
|
||
//! Modes are validated structurally — unrepresentable states are
|
||
//! unrepresentable by construction: dual-tier's signature constructs
|
||
//! the large engine itself, mem-only's constructor exists only with
|
||
//! feature `mem` and constructs the mem engine — and every instance
|
||
//! carries its mode for introspection ([`Store::mode`]).
|
||
//!
|
||
//! **Read-path design points (recorded for the review gate):**
|
||
//!
|
||
//! - **The fall-through general rule (ADR-012 §6.1; POC #1 finding
|
||
//! 6).** Every key-addressed op consults `KvEngine` first, then
|
||
//! `LargeEngine` — there is no length routing on the read side and
|
||
//! `Store::tier_for` is never consulted by it (the dispatch seam's
|
||
//! write-half routing is the put path's; `route` feeds nothing on
|
||
//! reads). A miss on both tiers resolves per op's shape: `Ok(None)`
|
||
//! on get/stat, `false` on has, `Err(Missing)` on read_range.
|
||
//! - **The public stream type (named design point): `Blob`.** `get`
|
||
//! returns the pinned `(len, stream)` shape as
|
||
//! `Option<(u64, Blob)>` (store-api.md §Read path) — [`Blob`] is a
|
||
//! bounded, re-entrant windowed reader over the crate-internal
|
||
//! `AnyCursor`, which dispatches the engine cursor bodies by tier
|
||
//! — the kv materializing bodies behind `KvCursor` on the one
|
||
//! arm, the large streaming bodies behind `LargeCursor` on the
|
||
//! other (both sub-enums the recorded design points). Windowed
|
||
//! reads are the uniform stream shape across
|
||
//! tiers; no chunk-iterator machinery was adopted — the cursor
|
||
//! bodies *are* the streaming abstraction, and per-chunk `Stream`-
|
||
//! wrapping would re-shape them without changing the underlying
|
||
//! reads. Engine cursors never leak onto the public surface:
|
||
//! `AnyCursor` is `pub(crate)`, `Blob` is opaque but drivable
|
||
//! (constructed only by the store core, ADR-010 §1).
|
||
//! - **Verification-on-get (named design point, pinned against
|
||
//! ADR-006's text).** The whole-blob hash-check fires where the
|
||
//! whole content is already in hand — the kv tier's materializing
|
||
//! reads. A kv-tier `get` materializes the stored value and
|
||
//! re-derives the canonical digest over it, comparing against the
|
||
//! key (ADR-006 decision 1: "put and get paths hash-check");
|
||
//! mismatch is `Err(Error::Verification)` and **never falls
|
||
//! through further** — a verified mismatch is a pool-integrity
|
||
//! fact, and consulting another tier past it would be a guess.
|
||
//! The large tier's streaming `get` does *not* re-derive at open:
|
||
//! opening is metadata + handle only, and the tier's integrity
|
||
//! story is the put path's whole-blob verification pass (the
|
||
//! complete staged artifact was checked before commit) plus the
|
||
//! per-range slice digests `read_range` carries out-of-band
|
||
//! (ADR-006 decisions 1–2; the §Context line "git objects
|
||
//! self-verify under git's own model when read whole"). `read_range`
|
||
//! carries no canonical check on either tier — the slice digest is
|
||
//! the convention's whole answer (ADR-006 decision 2). stat/has/
|
||
//! list carry no verification (metadata and enumeration only).
|
||
//! - **`list` returns `Vec<Key>` (named design point: Vec vs
|
||
//! stream).** Whole-pool enumeration, complete by contract
|
||
//! (ADR-003 — load-bearing for GC), returned **materialized and
|
||
//! sorted in the keys' total order** (deterministic output). The
|
||
//! sweep consumes the same crate-internal enumeration directly from
|
||
//! the engines (its live-set diffs sort independently), so the
|
||
//! facade shape serves consumers, not internal needs. A lazy stream
|
||
//! was rejected: every shipped engine enumerates fully and eagerly
|
||
//! (the completeness contract *is* the GC-safety property), so a
|
||
//! pseudo-lazy wrapper adds machinery without changing the
|
||
//! underlying walk. Listed bytes failing [`Key::from_bytes`]
|
||
//! reject with `Error::KeyInvalid` — a malformed pool list is
|
||
//! fatal, never a silent skip (ADR-003's malformed-list ruling
|
||
//! applied at the facade boundary).
|
||
//! - **`KeyInvalid` at the facade boundary (ADR-002/003).** Read ops
|
||
//! take opaque key *bytes* and convert to typed `Key` at the
|
||
//! boundary via [`Key::from_bytes`]: malformed bytes reject with
|
||
//! `Error::KeyInvalid` on every key-addressed op, never a guess,
|
||
//! and the engines receive only the physical byte encoding — they
|
||
//! never see typed keys.
|
||
//!
|
||
//! **Write-path design points (recorded for the review gate):**
|
||
//!
|
||
//! - **The stream parameter shape: `put(len, stream: impl AsyncRead +
|
||
//! Unpin + Send)`** — generic over the concrete reader, no boxing.
|
||
//! The family's no-`dyn` posture (wave-1 design point 3: no `dyn`
|
||
//! anywhere) carries to the write seam: the reader is monomorphized,
|
||
//! and a caller holding a boxed reader passes the box (a
|
||
//! `Box<dyn AsyncRead + Unpin + Send>` is itself `AsyncRead + Unpin +
|
||
//! Send` under tokio's blanket impl). `Send` (and no `'static`:
|
||
//! nothing is spawned) keeps spawn-site bounds resolvable for the
|
||
//! ops layer (wave 7), matching the ADR-005/ops posture. A boxed
|
||
//! trait-object parameter was rejected as the *default* shape — it
|
||
//! would force an allocation on every call (including the known-
|
||
//! length path's) to carry a posture the family has already ruled
|
||
//! against. Callers who need erasure box at their call site.
|
||
//! - **The declared-len mismatch case (`Verification`).** A known-
|
||
//! length stream that yields fewer or more bytes than `len` rejects
|
||
//! with `Error::Verification` on either side: the derivation's
|
||
//! preamble carries the *declared* length, so a mismatched artifact's
|
||
//! canonical address can never equal the stored key (a later get
|
||
//! re-derives over the actual length and hash-checks — storing under
|
||
//! the declared-length derivation would break pool integrity by
|
||
//! construction). Zero residue on either side: the kv path buffers
|
||
//! and publishes nothing; the staged path drives
|
||
//! `crate::backend::backend`'s `StagingCursor::discard`. The error family is
|
||
//! ADR-pinned to five members (ADR-012 §6.5); `Verification` is the
|
||
//! put-path hash-check family member and this is a content-≠-key
|
||
//! fact (the ADR-006 whole-blob posture applied at put time).
|
||
//! - **The kv-only over-threshold rejection's error carrier (named
|
||
//! design point).** ADR-012 §6.2's put-time rejection rides
|
||
//! `Error::Io(String)` with a descriptive message: the pinned family
|
||
//! (ADR-012 §6.5) holds no dedicated variant, and `Missing`/
|
||
//! `Verification`/`KeyInvalid`/`GcRefuse` are strictly narrower
|
||
//! families — the rejection is an operational/capability fact about
|
||
//! the store's medium posture, the bucket `Io(String)` covers
|
||
//! (stringly, so the message names mode, threshold, and the
|
||
//! rejected length). Recorded for the review gate: a dedicated
|
||
//! variant for embedders that need typed matchability would be a
|
||
//! new failure family — a new ADR.
|
||
//! - **The staged-put seam resolves the key at commit (the wave-2
|
||
//! amendment of the wave-1 handout shape).** The canonical key is
|
||
//! the derivation over preamble + content and completes at stream
|
||
//! end — after the staging cursor exists (both large-tier put paths
|
||
//! stream content once into the cursor while hashing; ADR-003). The
|
||
//! `crate::backend::backend`'s `put_staged` handout is therefore
|
||
//! keyless and the staging cursor's `commit` carries the
|
||
//! store-core-resolved key (CAS first-writer-wins as the whole-value
|
||
//! put). The wave-1 notes' `put_staged(key)` handout predates this
|
||
//! task's flow realization; no ADR text pins the key's arrival
|
||
//! point, and the amendment is recorded for review-wave-2.
|
||
//! - **`Batch` surface: `batch()` + `put_in_batch(&mut Batch, ...)`.**
|
||
//! The batch-scoped pin group is minted by the store core
|
||
//! ([`crate::gc`]); a member put runs the lone-put flow and
|
||
//! contributes its pin to the group (`Batch::add` — the minted guard
|
||
//! is defused, the group's one-step release owns the protection).
|
||
//! Members publish pin-before-publish exactly as lone puts do; the
|
||
//! group's pins drop together on [`Batch::commit`] or drop (the
|
||
//! `gc-protocol` machinery drives).
|
||
//!
|
||
//! **The fleet declaration (ADR-012 §3; landed — wave 6's
|
||
//! `store-fleet-construction`).** Fleet mode is a constructor
|
||
//! declaration, never inferred: every pre-existing constructor is
|
||
//! implicitly `fleet: false` (the ADR-005 in-process protocol applies
|
||
//! verbatim regardless of engine — pins are in-process guards, the
|
||
//! sweep lease is the in-process lock, none of the `EngineFleetState`
|
||
//! clauses run — behavior byte-unchanged), and the **`_fleet`
|
||
//! twin constructors** declare `fleet: true`:
|
||
//! `Store::kv_only_postgres_fleet` (the re-routing client posture —
|
||
//! large = none, pool large content rides the ops surface, wave 7's
|
||
//! wiring), `Store::dual_tier_postgres_local_fleet` (the shared-media
|
||
//! posture — the declaration requires the sealed
|
||
//! `FleetSharedMedia` assertion, ADR-008 §3), and
|
||
//! `Store::dual_tier_postgres_pglo_fleet` (the consolidation posture —
|
||
//! ADR-012 §4's same-instance clause runtime-probed). **The composite
|
||
//! fleet-validity predicate** (`fleet_validity_error` — the one-
|
||
//! sentence rule of backends-and-dispatch.md, both tiers) is enforced
|
||
//! at the constructor, keyed off the engine representation like every
|
||
//! other dispatch in this store: a configuration is fleet-valid iff
|
||
//! `kv = postgres` AND (`large = pg-lo` on the same pg instance as the
|
||
//! kv engine OR `large = local` on the declared shared media OR
|
||
//! `large = none`) — plus the explicit declaration. `fleet: true` is a
|
||
//! constructor error for every non-fleet-valid configuration; the
|
||
//! predicate's mem/sqlite clauses are enforced structurally (no
|
||
//! `_fleet` constructor is named for a mem or sqlite kv engine — the
|
||
//! recorded "not fleet-anything" postures) with the predicate's
|
||
//! rejection clauses unit-tested per clause at the predicate level,
|
||
//! and the pg-lo same-instance clause is the one runtime-checkable
|
||
//! clause: the instance's `system_identifier` (via
|
||
//! `pg_control_system()`, PG14+) compared across one connection per
|
||
//! engine — a mismatch **or a failed probe** is a constructor error,
|
||
//! never a silent pass. **The fleet constructor parameters, not API**
|
||
//! (ADR-008 §2): the pin TTL (default `DEFAULT_PIN_TTL`, 60 s — the
|
||
//! store owns the TTL and the TTL/3 renewal cadence; the engine's
|
||
//! `pin_renew` successor-time body reads the TTL it was constructed
|
||
//! with) and the large-tier declaration (the mode's name, the sealed
|
||
//! shared-media assertion, or the pg-lo shape). The store carries the
|
||
//! fleet state once declared — the flag (the put/sweep paths branch on
|
||
//! it, never on re-deriving from the engine set), the pin TTL, and the
|
||
//! store-instance id (the pin row's `owner` half, minted at
|
||
//! construction, 32 raw bytes, crate-internal): `FleetState`, never
|
||
//! public API (ADR-008 §3: fleet constructor parameters, not API). The
|
||
//! fleet declaration slots in without reshaping the fleet-capable
|
||
//! constructors (the wave-4/5 recorded non-event landed as recorded):
|
||
//! the pre-existing constructors' signatures and behavior are
|
||
//! unchanged and remain implicitly `fleet: false`.
|
||
//!
|
||
//! **The dispatch seam the ops tasks program through:** tier selection
|
||
//! for a given length (`Store::tier_for` composing
|
||
//! `crate::dispatch::route` per mode), the fall-through consult
|
||
//! order — **kv first, then large** — that every key-addressed op
|
||
//! consumes (ADR-012 §6.1's general rule; `Store::kv` then
|
||
//! `Store::large`), and the GC state home (`GcHome`,
|
||
//! crate-internal)
|
||
//! the gc-protocol task fills.
|
||
//!
|
||
//! **The durable delete window hosts on exactly one SQL-backed engine
|
||
//! per store instance (ADR-012 §5's one-window-host rule) — the kv
|
||
//! engine when it is SQL-backed, otherwise the large engine.** Landed
|
||
//! in three constructor stages. Sqlite and postgres — the SQL-backed
|
||
//! kv engines — host: their constructors run the GC schema's
|
||
//! engine-neutral call (`ensure_gc_schema` on the `EngineWindowState`
|
||
//! seam) at construction, dispatched off the engine representation
|
||
//! (`ensure_gc_schema_at_construction`). Pg-lo's dormancy landed
|
||
//! with the wave-5 wiring: on a dual-SQL instance (kv =
|
||
//! sqlite/postgres + large = pg-lo) the kv host's call runs and
|
||
//! pg-lo's window clauses are **dormant** (the pg-lo schema carries
|
||
//! no `gc_window`), while on the kv = mem + large = pg-lo shape the
|
||
//! **pg-lo host's** call
|
||
//! runs (`gc_window` in the pg-lo schema) — the rule's large-host
|
||
//! arm, the window's completion. The sweep's form branch keys off
|
||
//! the same representation (`sweep_window`'s kv-engine-enum match,
|
||
//! whose mem arm consults the large engine precisely when it hosts
|
||
//! the window — never a flag): the **durable form** (ADR-008 §2
|
||
//! staged re-arbitrated delete; ADR-012 §6.7 the executor is the
|
||
//! next sweep) on every window host, the inline form verbatim where
|
||
//! the window is in-process (mem/local, ADR-005 protocol inline).
|
||
//! Every SQL-backed-host constructor carries the delete-window delay
|
||
//! (ADR-008 §2: default 5 min, constructor-tunable per deployment —
|
||
//! [`DEFAULT_DELETE_WINDOW_DELAY`]; `0` is a legal degenerate tuning,
|
||
//! execute-on-next-sweep) as the cadence knob the durable form's
|
||
//! staging consumes (`visible_after = mark_end + window`) — on
|
||
//! either host; the mem/local constructors' inline form has no delay
|
||
//! in effect.
|
||
//!
|
||
//! The `#![allow(dead_code)]` below marks the work still ahead, not
|
||
//! the GC protocol (the window host's construction call and the
|
||
//! durable form are driven below on either host). The fleet
|
||
//! declaration has landed (the constructors' predicate, parameters,
|
||
//! and fleet state above); the fleet mechanism's store-core consumers
|
||
//! hold the remaining scope — the joint-tx put driver and the fleet
|
||
//! sweep (`store-fleet-put`, `store-fleet-sweep`).
|
||
|
||
#![allow(dead_code)]
|
||
|
||
use std::collections::BTreeSet;
|
||
#[cfg(feature = "postgres")]
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use std::sync::Arc;
|
||
#[cfg(feature = "postgres")]
|
||
use std::time::UNIX_EPOCH;
|
||
use std::time::{Duration, SystemTime};
|
||
|
||
use tokio::io::{AsyncRead, AsyncReadExt};
|
||
|
||
use crate::backend::{Backend, ReadCursor, StagingCursor};
|
||
use crate::dispatch::{route, Tier};
|
||
use crate::facade::{
|
||
Batch, Blob, EntryMeta, GcAbortCause, LivenessSource, Pin, ProtectCallback, Registration,
|
||
SweepReport,
|
||
};
|
||
use crate::gc::{Arbitration, EngineWindowState, GcHome, WindowCandidate};
|
||
use crate::hashing::{Algorithm, BlobHasher};
|
||
use crate::keys::Key;
|
||
use crate::local::{LocalBackend, LocalCursor, LocalStaging};
|
||
|
||
#[cfg(feature = "mem")]
|
||
use crate::mem::{MemBackend, MemCursor};
|
||
#[cfg(any(feature = "postgres", all(feature = "pg-lo", feature = "mem")))]
|
||
use crate::pg_common::PgOpts;
|
||
#[cfg(feature = "pg-lo")]
|
||
use crate::pg_lo::{PgLoBackend, PgLoCursor, PgLoStaging};
|
||
#[cfg(feature = "postgres")]
|
||
use crate::postgres::{PostgresBackend, PostgresCursor};
|
||
#[cfg(feature = "sqlite")]
|
||
use crate::sqlite::{SqliteBackend, SqliteCursor};
|
||
|
||
#[cfg(feature = "postgres")]
|
||
use sha2::{Digest as _, Sha256};
|
||
|
||
use crate::Error;
|
||
|
||
#[cfg(any(feature = "mem", feature = "postgres", feature = "sqlite"))]
|
||
use std::path::PathBuf;
|
||
|
||
use self::StoreMode::{DualTier, KvOnly, MemOnly};
|
||
|
||
/// The shipped default **delete-window delay** (ADR-008 §2): the
|
||
/// staged candidates in the durable window become visible for
|
||
/// execution this long after the sweep staged them — 5 minutes,
|
||
/// constructor-tunable per deployment. `Duration::ZERO` is a legal
|
||
/// degenerate tuning (execute-on-next-sweep; the test tuning). The
|
||
/// sqlite constructors take the delay explicitly and their docs point
|
||
/// here as the shipped default; the mem/local inline form has no
|
||
/// delay in effect and the constant is irrelevant to it.
|
||
pub const DEFAULT_DELETE_WINDOW_DELAY: Duration = Duration::from_secs(5 * 60);
|
||
|
||
/// The shipped default **pin TTL** (ADR-008 §2): a fleet mode's pin
|
||
/// rows renew to `now + pin_ttl` on the guard's TTL/3 cadence — 60
|
||
/// seconds, the fleet constructors' default tuning (`FleetState`'s
|
||
/// carried value; the engine's `pin_renew` successor-time body reads
|
||
/// the TTL its backend was constructed with). Constructor-tunable
|
||
/// per deployment, never API (ADR-008 §2: fleet constructor
|
||
/// parameters, not API). `Duration::ZERO` is a constructor error —
|
||
/// the TTL/3 renewal cadence degenerates at zero and a zero TTL
|
||
/// expires every pin at once.
|
||
#[cfg(feature = "postgres")]
|
||
pub const DEFAULT_PIN_TTL: Duration = Duration::from_secs(crate::postgres::DEFAULT_PIN_TTL_SECS);
|
||
|
||
/// The operator's explicit **shared-media assertion** (ADR-008 §3's
|
||
/// enforceable-seam bullet): the declaration a `local` large tier
|
||
/// needs under `fleet: true` — the operator's assertion that the
|
||
/// constructor's `local` root is fleet-shared media. **Named design
|
||
/// point — the declaration's type shape:** a dedicated sealed type
|
||
/// with a private seal field, not a bare flag — the caller cannot
|
||
/// construct it accidentally and must name [`FleetSharedMedia::asserted`]
|
||
/// at the call site (the ADR's "explicit declaration" wording's
|
||
/// reading: a named type the call site cannot misread), while the
|
||
/// predicate-level alternative (`LocalUndeclared`) has no constructor
|
||
/// to reach it from. The assertion asserts — cross-node truth ("is
|
||
/// the declared media actually shared?") remains the deployment's
|
||
/// verified invariant, the detection symptom documented (ADR-008 §3's
|
||
/// seam; ADR-012 §3's misconfiguration reading): the constructor
|
||
/// cannot prove cross-node fact, only admit the declared posture.
|
||
#[cfg(feature = "postgres")]
|
||
pub struct FleetSharedMedia {
|
||
_seal: (),
|
||
}
|
||
|
||
#[cfg(feature = "postgres")]
|
||
impl FleetSharedMedia {
|
||
/// Mint the shared-media assertion — the operator's explicit
|
||
/// declaration at the fleet constructor's call site (the type's
|
||
/// sealed shape makes this the only minting path).
|
||
pub fn asserted() -> Self {
|
||
Self { _seal: () }
|
||
}
|
||
}
|
||
|
||
/// The store's **fleet state** — what the constructor's `fleet: true`
|
||
/// declaration leaves the store carrying (crate-internal, never
|
||
/// public API — ADR-008 §3: fleet constructor parameters, not API):
|
||
/// the declaration's flag (the put/sweep paths branch on the state's
|
||
/// presence, never on re-deriving from the engine set), the pin TTL
|
||
/// the constructor took, and the **store-instance id** (`owner`,
|
||
/// ADR-011's vocabulary — the pin row's identity half).
|
||
pub(crate) struct FleetState {
|
||
/// The pin TTL the fleet constructor took (default
|
||
/// [`DEFAULT_PIN_TTL`]) — the store owns the TTL and the TTL/3
|
||
/// renewal cadence; the engine's `pin_renew` successor-time body
|
||
/// reads the TTL its backend was constructed with.
|
||
pin_ttl: Duration,
|
||
/// The store-instance id ([`mint_owner_id`]'s minting) — carried
|
||
/// as the `owner` bytes of every [`crate::gc::PinRow`] the store
|
||
/// mints.
|
||
owner: Vec<u8>,
|
||
}
|
||
|
||
impl FleetState {
|
||
/// The pin TTL the fleet constructor took.
|
||
pub(crate) fn pin_ttl(&self) -> Duration {
|
||
self.pin_ttl
|
||
}
|
||
|
||
/// The store-instance id — the `owner` bytes of the pin rows this
|
||
/// instance mints.
|
||
pub(crate) fn owner(&self) -> &[u8] {
|
||
&self.owner
|
||
}
|
||
}
|
||
|
||
/// The composite fleet-validity predicate's **kv descriptor** — the
|
||
/// engine-set shape the predicate reasons over, derived off the
|
||
/// engine representation ([`FleetKvShape::of`]) like every other
|
||
/// dispatch in this store, never a flag.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
enum FleetKvShape {
|
||
/// The fleet kv engine (ADR-011 §4's constructor table's only
|
||
/// fleet-valid kv row).
|
||
Postgres,
|
||
/// The durable single-node kv engine — never fleet-valid (a
|
||
/// sqlite fleet is a recorded non-option, ADR-010 §3).
|
||
Sqlite,
|
||
/// The contract-reference ephemeral engine — never fleet-valid
|
||
/// (ADR-011 §4).
|
||
Mem,
|
||
}
|
||
|
||
/// The composite fleet-validity predicate's **large descriptor**.
|
||
#[derive(Debug, Clone, Copy)]
|
||
enum FleetLargeShape {
|
||
/// No large tier — the kv-only re-routing posture (ADR-008 §3):
|
||
/// pool large content rides the ops surface.
|
||
None,
|
||
/// The `local` engine with the operator's shared-media
|
||
/// declaration ([`FleetSharedMedia`]).
|
||
LocalSharedMedia,
|
||
/// The `local` engine undeclared — the predicate's rejected arm:
|
||
/// a fleet-mode constructor requires the explicit shared-media
|
||
/// assertion (ADR-008 §3).
|
||
LocalUndeclared,
|
||
/// The pg-lo engine — fleet-valid under ADR-012 §4's
|
||
/// same-instance clause (the constructor-level probe).
|
||
PgLo,
|
||
}
|
||
|
||
/// **The composite fleet-validity predicate** — backends-and-dispatch.md's
|
||
/// one-sentence rule, both tiers, as code: a configuration is
|
||
/// fleet-valid iff **kv = postgres AND (large = pg-lo on the same pg
|
||
/// instance as the kv engine OR large = `local` on declared shared
|
||
/// media OR large = none)**. `None` = fleet-valid; `Some` = the
|
||
/// constructor error `fleet: true` carries (ADR-012 §3). The
|
||
/// enforcement map: the `_fleet` constructors are named only for the
|
||
/// shapes the predicate accepts — the structural clauses (kv ≠
|
||
/// postgres, mem anywhere, `local` without the shared-media
|
||
/// declaration) have no fleet constructor to reach them through (the
|
||
/// recorded "not fleet-anything" postures carry no fleet parameter)
|
||
/// and are enforced by that constructor-set shape plus this
|
||
/// predicate's rejection arms (unit-tested per clause, the
|
||
/// predicate's error being the constructor error a would-be
|
||
/// construction carries); the pg-lo shape's runtime-checkable clause
|
||
/// — the same-instance probe (ADR-012 §4) — runs at its constructor
|
||
/// alongside this call. Derived off the engine representation;
|
||
/// crate-internal (fleet parameters, not API).
|
||
fn fleet_validity_error(kv: FleetKvShape, large: FleetLargeShape) -> Option<Error> {
|
||
match (kv, large) {
|
||
(FleetKvShape::Postgres, FleetLargeShape::None) => None,
|
||
(FleetKvShape::Postgres, FleetLargeShape::LocalSharedMedia) => None,
|
||
(FleetKvShape::Postgres, FleetLargeShape::PgLo) => None,
|
||
(FleetKvShape::Postgres, FleetLargeShape::LocalUndeclared) => Some(Error::Io(
|
||
"fleet: the local large tier rejects without the shared-media declaration \
|
||
(ADR-008 \u{a7}3): a fleet-mode constructor requires the explicit \
|
||
FleetSharedMedia assertion"
|
||
.to_owned(),
|
||
)),
|
||
(kv, _) => Some(Error::Io(format!(
|
||
"fleet: the kv engine {kv:?} rejects the fleet declaration (ADR-012 \u{a7}3): \
|
||
kv = postgres is the composite fleet-validity rule's kv clause — \
|
||
{kv:?} is never fleet-valid"
|
||
))),
|
||
}
|
||
}
|
||
|
||
impl FleetKvShape {
|
||
/// The predicate kv descriptor derived off the engine
|
||
/// representation.
|
||
#[cfg(any(feature = "mem", feature = "postgres", feature = "sqlite"))]
|
||
fn of(engine: &KvEngine) -> FleetKvShape {
|
||
match engine {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => FleetKvShape::Mem,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(_) => FleetKvShape::Postgres,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(_) => FleetKvShape::Sqlite,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// **The store-instance id's minting** (named design point): 32 raw
|
||
/// bytes — a SHA-256 fold over the process id, the minting instant's
|
||
/// wall-clock nanos, a process-global instance counter, and an
|
||
/// address-space-randomized anchor — minted once at construction,
|
||
/// stable for the instance's life, carried as the `owner` bytes of
|
||
/// every [`crate::gc::PinRow`] the store mints. Uniqueness is the
|
||
/// contract, not secrecy: the counter distinguishes instances in one
|
||
/// process by construction; time + pid + the ASLR anchor distinguish
|
||
/// processes. The id is crate-internal (never public API — ADR-008:
|
||
/// fleet constructor parameters, not API).
|
||
#[cfg(feature = "postgres")]
|
||
fn mint_owner_id() -> Vec<u8> {
|
||
static INSTANCE_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = INSTANCE_SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let nanos = SystemTime::now()
|
||
.duration_since(UNIX_EPOCH)
|
||
.map(|d| d.as_nanos())
|
||
.unwrap_or(0);
|
||
let anchor = &seq as *const u64 as usize;
|
||
let mut hasher = Sha256::new();
|
||
hasher.update(u64::from(std::process::id()).to_le_bytes());
|
||
hasher.update(nanos.to_le_bytes());
|
||
hasher.update(seq.to_le_bytes());
|
||
hasher.update((anchor as u64).to_le_bytes());
|
||
hasher.finalize().to_vec()
|
||
}
|
||
|
||
/// The store instance — the public facade above the tier engines
|
||
/// (ADR-011 §1's vocabulary: *one running instantiation of the store
|
||
/// facade — one engine set per the constructor mode, plus dispatch and
|
||
/// GC state*; **the GC arbitration domain** for single-instance
|
||
/// topologies).
|
||
///
|
||
/// Engine-blind by construction: the engines are held in crate-internal
|
||
/// enums dispatched statically (see the module's engine-representation
|
||
/// design point); consumers name modes only. The GC state home
|
||
/// (`GcHome`, crate-internal) rides as a field — the store instance owns
|
||
/// all GC state (ADR-010 §2), which the gc-protocol task fills.
|
||
pub struct Store {
|
||
/// The kv-tier engine — the first consult in every fall-through.
|
||
kv: KvEngine,
|
||
/// The large-tier engine — `None` in single-tier modes.
|
||
large: Option<LargeEngine>,
|
||
/// The dispatch threshold in effect where the mode has one; mem-only
|
||
/// carries the shipped default but never consults it — "no dispatch
|
||
/// threshold in effect" is the mode's ruling (ADR-011 §4), not a
|
||
/// value absence.
|
||
threshold: u64,
|
||
/// The delete-window delay the constructor took (ADR-008 §2) —
|
||
/// the durable window's cadence knob the sweep's durable form
|
||
/// consumes (`window_stage`'s `visible_after = mark_end + window`
|
||
/// on the `EngineWindowState` seam, on either host — the kv
|
||
/// engine's or the large engine's). The in-process inline form
|
||
/// has no delay in effect: those constructors (mem-only,
|
||
/// kv-only(mem), dual-tier mem+local) carry
|
||
/// [`DEFAULT_DELETE_WINDOW_DELAY`], never consulted.
|
||
window_delay: Duration,
|
||
/// The constructor mode; the dispatch/put/read/gc paths branch on it.
|
||
mode: StoreMode,
|
||
/// The GC state home (ADR-010 §2: the store core owns GC state)
|
||
/// — the gc-protocol task's machinery lands here.
|
||
gc: GcHome,
|
||
/// The constructor's fleet declaration — `Some` exactly when a
|
||
/// `_fleet` constructor declared `fleet: true` (never inferred,
|
||
/// ADR-012 §3): the flag the put/sweep paths branch on (crate-
|
||
/// internally, never re-derived from the engine set), the pin
|
||
/// TTL, and the store-instance id (`FleetState`). Under
|
||
/// `fleet: false` — every pre-existing constructor, behavior
|
||
/// byte-unchanged — the ADR-005 in-process protocol applies
|
||
/// verbatim regardless of engine.
|
||
fleet: Option<FleetState>,
|
||
}
|
||
|
||
/// The constructor modes (ADR-011 §4's pinned set; ADR-012 §6.3's
|
||
/// dual-tier ruling). Three variants, ADR-pinned — a new mode is a new
|
||
/// ADR.
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||
pub enum StoreMode {
|
||
/// kv + large, size-threshold dispatch between them (the default).
|
||
DualTier,
|
||
/// kv alone; over-threshold puts reject at put time (ADR-012 §6.2).
|
||
KvOnly,
|
||
/// The mem engine alone, no dispatch threshold in effect — never
|
||
/// production (ADR-011 §4).
|
||
MemOnly,
|
||
}
|
||
|
||
/// The kv-tier engine set — the closed per-tier engine enum of the
|
||
/// engine-representation design point. Feature-gated variants slot in
|
||
/// as their engine waves land: sqlite (wave 3 — landed, the default),
|
||
/// postgres (wave 4 — landed, feature-gated default-off).
|
||
///
|
||
/// The variant payloads are structurally dissimilar by design — the
|
||
/// in-process engine holds one `Arc`, the SQL-backed engines hold
|
||
/// their connection postures inline — and every dispatch rides the
|
||
/// enum by reference (`&self` matches only), so the size difference
|
||
/// costs nothing.
|
||
#[allow(clippy::large_enum_variant)]
|
||
pub(crate) enum KvEngine {
|
||
/// The contract-reference ephemeral engine (ADR-011 §2).
|
||
#[cfg(feature = "mem")]
|
||
Mem(MemBackend),
|
||
/// The default build's kv engine — the first SQL-backed engine,
|
||
/// the durable delete window's host (ADR-012 §5's one-window-host
|
||
/// rule: the window is the kv engine's when the kv engine is
|
||
/// SQL-backed).
|
||
#[cfg(feature = "sqlite")]
|
||
Sqlite(SqliteBackend),
|
||
/// The second SQL-backed engine — the fleet host (ADR-012 §4) and
|
||
/// a one-window-host alongside sqlite: its constructors run the
|
||
/// GC schema's engine-neutral call (the postgres row's body
|
||
/// creates `gc_pins` + `gc_window`) and its sweeps run the durable
|
||
/// window form on it. Feature `postgres`, default-off.
|
||
#[cfg(feature = "postgres")]
|
||
Postgres(PostgresBackend),
|
||
}
|
||
|
||
/// The large-tier engine set. `local` is ungated (dependency-free,
|
||
/// always present, ADR-013 §4); pg-lo landed behind feature `pg-lo`
|
||
/// (wave 5) — its slotting is exactly the enum-variant-plus-named-
|
||
/// constructor pattern the module doc pins.
|
||
pub(crate) enum LargeEngine {
|
||
/// The sharded-filesystem engine (ADR-003/008).
|
||
Local(LocalBackend),
|
||
/// The pg-lo large-object engine (ADR-009): postgres Large
|
||
/// Objects hold the bytes, the companion table is the addressing
|
||
/// authority. Feature `pg-lo`, default-off (it does not imply
|
||
/// feature `postgres`, ADR-013 §4).
|
||
#[cfg(feature = "pg-lo")]
|
||
PgLo(PgLoBackend),
|
||
}
|
||
|
||
/// **The large-arm shape over multiple large engines (named design
|
||
/// point): a `LargeCursor` sub-enum behind [`AnyCursor`]'s one large
|
||
/// arm — the recorded [`KvCursor`] sub-enum pattern's large-tier
|
||
/// twin, on the same bounded-memory framing as the kv side.** The
|
||
/// alternative — one `AnyCursor` variant per large engine — would
|
||
/// copy the large streaming arm into the tier dispatch per engine,
|
||
/// and the tier dispatch's variant count must stay bounded so
|
||
/// `Blob`-tier consumers (the verification posture's arm shapes in
|
||
/// `Store::get`) stay a two-arm tier dispatch while large engines
|
||
/// multiply inside this one. Recorded for the review gate and landed
|
||
/// with wave 5: pg-lo slotted one arm in here (exactly the shape the
|
||
/// design point named, no reshaping).
|
||
pub(crate) enum LargeCursor {
|
||
/// The pread body over the committed file (ADR-003/008).
|
||
Local(LocalCursor),
|
||
/// The descriptorless window body over the companion row —
|
||
/// `lo_get` windows (ADR-009).
|
||
#[cfg(feature = "pg-lo")]
|
||
PgLo(PgLoCursor),
|
||
}
|
||
|
||
impl ReadCursor for LargeCursor {
|
||
fn len(&self) -> u64 {
|
||
match self {
|
||
Self::Local(cursor) => cursor.len(),
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.len(),
|
||
}
|
||
}
|
||
|
||
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
|
||
match self {
|
||
Self::Local(cursor) => cursor.read_at(offset, len).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.read_at(offset, len).await,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// **The kv-arm shape over multiple kv engines (named design point):
|
||
/// a `KvCursor` sub-enum behind [`AnyCursor`]'s one kv arm.** The
|
||
/// alternative — one `AnyCursor` variant per kv engine — would copy
|
||
/// the kv materializing arm (and its consumers: the verification-on-
|
||
/// get body, `Blob`'s drive) per engine without adding a behavior;
|
||
/// the sub-enum keeps [`AnyCursor`] a two-arm tier dispatch (kv /
|
||
/// large) while kv engines multiply inside this one. Recorded for the
|
||
/// review gate and landed with wave 4: postgres slotted one arm in
|
||
/// here (exactly the predicted shape, no reshaping).
|
||
pub(crate) enum KvCursor {
|
||
/// The contract-reference ephemeral body (ADR-011 §2).
|
||
#[cfg(feature = "mem")]
|
||
Mem(MemCursor),
|
||
/// The default build's materializing body: the row's value loaded
|
||
/// once, reads in-memory slices (the sqlite engine's record).
|
||
#[cfg(feature = "sqlite")]
|
||
Sqlite(SqliteCursor),
|
||
/// The postgres body — the same materializing shape, the value
|
||
/// loaded once from the engine's pooled row (the postgres
|
||
/// engine's record).
|
||
#[cfg(feature = "postgres")]
|
||
Postgres(PostgresCursor),
|
||
}
|
||
|
||
impl ReadCursor for KvCursor {
|
||
fn len(&self) -> u64 {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
Self::Mem(cursor) => cursor.len(),
|
||
#[cfg(feature = "postgres")]
|
||
Self::Postgres(cursor) => cursor.len(),
|
||
#[cfg(feature = "sqlite")]
|
||
Self::Sqlite(cursor) => cursor.len(),
|
||
}
|
||
}
|
||
|
||
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
Self::Mem(cursor) => cursor.read_at(offset, len).await,
|
||
#[cfg(feature = "postgres")]
|
||
Self::Postgres(cursor) => cursor.read_at(offset, len).await,
|
||
#[cfg(feature = "sqlite")]
|
||
Self::Sqlite(cursor) => cursor.read_at(offset, len).await,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The read path's cursor-dispatch enum (ADR-010 §1's "store core
|
||
/// programs the abstraction"): tier dispatch over the engine cursor
|
||
/// bodies — the kv tier's materializing body and the large tier's
|
||
/// streaming body — behind one crate-internal value that [`Blob`] holds
|
||
/// and drives. Crate-internal: engine cursors never appear on the
|
||
/// public surface; the facade's stream shape is this enum's uniform
|
||
/// [`ReadCursor`] view (`Blob`). The per-tier body families live
|
||
/// behind the sub-enums — [`KvCursor`] for the kv arm, [`LargeCursor`]
|
||
/// for the large arm (both the recorded design points).
|
||
pub(crate) enum AnyCursor {
|
||
/// The kv body family: the whole value materialized in memory
|
||
/// (bounded by tier policy).
|
||
Kv(KvCursor),
|
||
/// The large body family: streaming readers over the committed
|
||
/// home, behind the large-tier sub-enum.
|
||
Large(LargeCursor),
|
||
}
|
||
|
||
impl ReadCursor for AnyCursor {
|
||
fn len(&self) -> u64 {
|
||
match self {
|
||
Self::Kv(cursor) => cursor.len(),
|
||
Self::Large(cursor) => cursor.len(),
|
||
}
|
||
}
|
||
|
||
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
|
||
match self {
|
||
Self::Kv(cursor) => cursor.read_at(offset, len).await,
|
||
Self::Large(cursor) => cursor.read_at(offset, len).await,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The write path's staging-cursor dispatch (ADR-010 §1: the store core
|
||
/// programs the abstraction): the large tier's stage bodies behind one
|
||
/// crate-internal value, the put seam's twin of [`AnyCursor`]. The kv
|
||
/// tier never stages on the facade path (the whole-value put is its
|
||
/// settled form); a kv-degenerate staged body variant arrives only if
|
||
/// the conformance suite demands the staged form through the facade.
|
||
/// Crate-internal — engine cursors never leak onto the public surface.
|
||
/// The stage bodies live behind [`LargeStaging`] (the sub-enum design
|
||
/// point's staging-side twin).
|
||
pub(crate) enum AnyStaging {
|
||
/// The large body family: stage bodies behind the large-tier
|
||
/// sub-enum.
|
||
Large(LargeStaging),
|
||
}
|
||
|
||
impl ReadCursor for AnyStaging {
|
||
fn len(&self) -> u64 {
|
||
match self {
|
||
Self::Large(cursor) => cursor.len(),
|
||
}
|
||
}
|
||
|
||
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
|
||
match self {
|
||
Self::Large(cursor) => cursor.read_at(offset, len).await,
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The staged arm's sub-enum — the [`LargeCursor`]/staging-side twin
|
||
/// of the `KvCursor` pattern: the local stage-file body and pg-lo's
|
||
/// held-transaction body behind [`AnyStaging`]'s one large arm rather
|
||
/// than one `AnyStaging` variant per large engine (the same
|
||
/// bounded-variant argument; landed with wave 5). The variant payloads
|
||
/// are structurally dissimilar by design (the file-descriptor posture
|
||
/// vs the held pooled checkout + tx) and every dispatch rides the enum
|
||
/// by value-consumption or `&self`, so the size difference costs
|
||
/// nothing — the recorded `KvEngine` enum's allow posture.
|
||
#[allow(clippy::large_enum_variant)]
|
||
pub(crate) enum LargeStaging {
|
||
/// The stage-file body under the stage-hygiene protocol,
|
||
/// published by commit-rename (ADR-003/008).
|
||
Local(LocalStaging),
|
||
/// The held-transaction body: the stage lives in uncommitted LO
|
||
/// pages, published by the companion-row insert + COMMIT
|
||
/// (ADR-009).
|
||
#[cfg(feature = "pg-lo")]
|
||
PgLo(PgLoStaging),
|
||
}
|
||
|
||
impl ReadCursor for LargeStaging {
|
||
fn len(&self) -> u64 {
|
||
match self {
|
||
Self::Local(cursor) => cursor.len(),
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.len(),
|
||
}
|
||
}
|
||
|
||
async fn read_at(&self, offset: u64, len: u64) -> Result<Vec<u8>, Error> {
|
||
match self {
|
||
Self::Local(cursor) => cursor.read_at(offset, len).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.read_at(offset, len).await,
|
||
}
|
||
}
|
||
}
|
||
|
||
impl StagingCursor for LargeStaging {
|
||
async fn append(&mut self, chunk: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
Self::Local(cursor) => cursor.append(chunk).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.append(chunk).await,
|
||
}
|
||
}
|
||
|
||
async fn commit(self, key: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
Self::Local(cursor) => cursor.commit(key).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.commit(key).await,
|
||
}
|
||
}
|
||
|
||
async fn discard(self) {
|
||
match self {
|
||
Self::Local(cursor) => cursor.discard().await,
|
||
#[cfg(feature = "pg-lo")]
|
||
Self::PgLo(cursor) => cursor.discard().await,
|
||
}
|
||
}
|
||
}
|
||
|
||
impl StagingCursor for AnyStaging {
|
||
async fn append(&mut self, chunk: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
Self::Large(cursor) => cursor.append(chunk).await,
|
||
}
|
||
}
|
||
|
||
async fn commit(self, key: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
Self::Large(cursor) => cursor.commit(key).await,
|
||
}
|
||
}
|
||
|
||
async fn discard(self) {
|
||
match self {
|
||
Self::Large(cursor) => cursor.discard().await,
|
||
}
|
||
}
|
||
}
|
||
|
||
impl KvEngine {
|
||
async fn put(&self, key: &[u8], value: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.put(key, value).await,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.put(key, value).await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.put(key, value).await,
|
||
}
|
||
}
|
||
|
||
async fn get(&self, key: &[u8]) -> Result<Option<AnyCursor>, Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => Ok(engine
|
||
.get(key)
|
||
.await?
|
||
.map(|c| AnyCursor::Kv(KvCursor::Mem(c)))),
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => Ok(engine
|
||
.get(key)
|
||
.await?
|
||
.map(|c| AnyCursor::Kv(KvCursor::Postgres(c)))),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => Ok(engine
|
||
.get(key)
|
||
.await?
|
||
.map(|c| AnyCursor::Kv(KvCursor::Sqlite(c)))),
|
||
}
|
||
}
|
||
|
||
/// The GC schema's engine-neutral kv-host arm (ADR-010 §2's
|
||
/// companion seam; ADR-012 §5's applicability table): a SQL-backed
|
||
/// kv engine creates the tables its applicability row names —
|
||
/// sqlite's body creates `gc_window` only, postgres's creates
|
||
/// `gc_pins` + `gc_window` (the fleet host's both-tables row) — and
|
||
/// an in-process kv engine is structurally nothing (the
|
||
/// local/mem row of the table: those engines execute the window
|
||
/// protocol inline, ADR-005, and carry no GC schema). The
|
||
/// store-core construction call is
|
||
/// [`ensure_gc_schema_at_construction`] — the one-window-host
|
||
/// rule's dispatch, which consults this method on the SQL-backed
|
||
/// kv arms and the large tier only when the kv engine is not
|
||
/// SQL-backed.
|
||
async fn ensure_gc_schema(&self) -> Result<(), Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => Ok(()),
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.ensure_gc_schema().await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.ensure_gc_schema().await,
|
||
}
|
||
}
|
||
|
||
async fn has(&self, key: &[u8]) -> Result<bool, Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.has(key).await,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.has(key).await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.has(key).await,
|
||
}
|
||
}
|
||
|
||
async fn size(&self, key: &[u8]) -> Result<Option<u64>, Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.size(key).await,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.size(key).await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.size(key).await,
|
||
}
|
||
}
|
||
|
||
async fn delete(&self, key: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.delete(key).await,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.delete(key).await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.delete(key).await,
|
||
}
|
||
}
|
||
|
||
async fn list(&self) -> Result<Vec<Vec<u8>>, Error> {
|
||
match self {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.list().await,
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.list().await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.list().await,
|
||
}
|
||
}
|
||
}
|
||
|
||
impl LargeEngine {
|
||
/// The staged form's handout (ADR-010 §1): the put seam's large path
|
||
/// streams into this cursor while hashing; the key resolves at the
|
||
/// staging cursor's commit.
|
||
async fn put_staged(&self) -> Result<AnyStaging, Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => Ok(engine
|
||
.put_staged()
|
||
.await
|
||
.map(|c| AnyStaging::Large(LargeStaging::Local(c)))?),
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => Ok(engine
|
||
.put_staged()
|
||
.await
|
||
.map(|c| AnyStaging::Large(LargeStaging::PgLo(c)))?),
|
||
}
|
||
}
|
||
|
||
async fn get(&self, key: &[u8]) -> Result<Option<AnyCursor>, Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => Ok(engine
|
||
.get(key)
|
||
.await?
|
||
.map(|c| AnyCursor::Large(LargeCursor::Local(c)))),
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => Ok(engine
|
||
.get(key)
|
||
.await?
|
||
.map(|c| AnyCursor::Large(LargeCursor::PgLo(c)))),
|
||
}
|
||
}
|
||
|
||
async fn has(&self, key: &[u8]) -> Result<bool, Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => engine.has(key).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => engine.has(key).await,
|
||
}
|
||
}
|
||
|
||
async fn size(&self, key: &[u8]) -> Result<Option<u64>, Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => engine.size(key).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => engine.size(key).await,
|
||
}
|
||
}
|
||
|
||
async fn delete(&self, key: &[u8]) -> Result<(), Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => engine.delete(key).await,
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => engine.delete(key).await,
|
||
}
|
||
}
|
||
|
||
async fn list(&self) -> Result<Vec<Vec<u8>>, Error> {
|
||
match self {
|
||
LargeEngine::Local(engine) => engine.list().await,
|
||
#[cfg(feature = "pg-lo")]
|
||
LargeEngine::PgLo(engine) => engine.list().await,
|
||
}
|
||
}
|
||
|
||
/// The large tier's arm of the one-window-host rule's construction
|
||
/// call: the pg-lo host's `ensure_gc_schema` (creating `gc_window`
|
||
/// in the pg-lo schema), consulted by
|
||
/// [`ensure_gc_schema_at_construction`] only when the kv engine is
|
||
/// not SQL-backed. No arm exists for `local` — it is
|
||
/// seam-excluded by construction (ADR-010/012 §5).
|
||
#[cfg(feature = "pg-lo")]
|
||
async fn ensure_gc_schema(&self) -> Result<(), Error> {
|
||
match self {
|
||
LargeEngine::PgLo(engine) => engine.ensure_gc_schema().await,
|
||
LargeEngine::Local(_) => Ok(()),
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The facade's key-bytes decode: typed `Key` conversion at the store
|
||
/// boundary (ADR-002/003) — the one place on the read path where key
|
||
/// bytes are interpreted; anything malformed is `Error::KeyInvalid`,
|
||
/// never a guess, and never reaches an engine typed.
|
||
fn boundary_key(key: &[u8]) -> Result<Key, Error> {
|
||
Key::from_bytes(key)
|
||
}
|
||
|
||
/// Constitute the facade's typed keys from an engine's opaque list
|
||
/// rows: a failed decode is fatal (`Error::KeyInvalid`) — a malformed
|
||
/// pool list is as deadly as an incomplete one and is never silently
|
||
/// skipped (ADR-003).
|
||
fn typed_keys(raws: Vec<Vec<u8>>) -> Result<Vec<Key>, Error> {
|
||
raws.into_iter().map(|raw| boundary_key(&raw)).collect()
|
||
}
|
||
|
||
/// **The GC schema's engine-neutral construction call under the
|
||
/// one-window-host rule (ADR-012 §5)** — the rule's store-core
|
||
/// dispatch, run at construction exactly once per instance: the kv
|
||
/// engine's arm when the kv engine is SQL-backed (its
|
||
/// `ensure_gc_schema` creates its applicability row's tables; the
|
||
/// dual-SQL instance's pg-lo window clauses stay dormant — never
|
||
/// consulted here), the large tier's arm only when the kv engine is
|
||
/// not SQL-backed (kv = mem + large = pg-lo — the pg-lo host's call
|
||
/// creating `gc_window` in the pg-lo schema). The in-process
|
||
/// combinations (mem alone, local large) are structurally nothing: no
|
||
/// GC schema exists on their side of the rule. The dispatch keys off
|
||
/// the engine representation, never a flag.
|
||
async fn ensure_gc_schema_at_construction(
|
||
kv: &KvEngine,
|
||
large: Option<&LargeEngine>,
|
||
) -> Result<(), Error> {
|
||
// `large` is consulted only by the mem arm (the one-window-host
|
||
// rule's large-host arm); absent feature `mem` no arm reads it.
|
||
#[cfg(not(feature = "mem"))]
|
||
let _ = large;
|
||
match kv {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => match large {
|
||
#[cfg(feature = "pg-lo")]
|
||
Some(LargeEngine::PgLo(engine)) => engine.ensure_gc_schema().await,
|
||
_ => Ok(()),
|
||
},
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.ensure_gc_schema().await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.ensure_gc_schema().await,
|
||
}
|
||
}
|
||
|
||
impl Store {
|
||
/// **mem-only mode** (ADR-011 §4): the mem engine alone, no
|
||
/// dispatch threshold in effect — a testing/embedder-ephemeral
|
||
/// posture, never production. Every length routes to the (sole)
|
||
/// mem engine; there is no large tier and nothing to reject.
|
||
#[cfg(feature = "mem")]
|
||
pub fn mem_only() -> Store {
|
||
Store {
|
||
kv: KvEngine::Mem(MemBackend::default()),
|
||
large: None,
|
||
threshold: crate::dispatch::DEFAULT_DISPATCH_THRESHOLD,
|
||
window_delay: DEFAULT_DELETE_WINDOW_DELAY,
|
||
mode: MemOnly,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
}
|
||
}
|
||
|
||
/// **kv-only mode** (ADR-011 §4; ADR-012 §6.2) with the mem kv
|
||
/// engine: no large tier; `threshold` is the put-time rejection
|
||
/// point — known-length puts over it reject immediately,
|
||
/// unknown-length puts at mid-stream threshold overflow (the put
|
||
/// seam's behavior; zero residue on either rejection). Routing in
|
||
/// this mode is kv for every length.
|
||
#[cfg(feature = "mem")]
|
||
pub fn kv_only_mem(threshold: u64) -> Store {
|
||
Store {
|
||
kv: KvEngine::Mem(MemBackend::default()),
|
||
large: None,
|
||
threshold,
|
||
window_delay: DEFAULT_DELETE_WINDOW_DELAY,
|
||
mode: KvOnly,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
}
|
||
}
|
||
|
||
/// **dual-tier mode** (the default, ADR-011 §4) in the wave-2
|
||
/// constructible shape — `kv = mem` + `large = local`, the
|
||
/// dispatch-coverage test shape ADR-012 §6.3 names: both tiers
|
||
/// load-bearing, routed by size-threshold dispatch at `threshold`
|
||
/// (constructor-tunable; [`crate::dispatch::DEFAULT_DISPATCH_THRESHOLD`]
|
||
/// is the shipped default).
|
||
///
|
||
/// Constructing the large engine creates the root directory —
|
||
/// fallible I/O, hence a fallible constructor. The mem kv engine
|
||
/// cannot fail.
|
||
///
|
||
/// Not fleet-anything: mem is never fleet-valid (ADR-011 §4).
|
||
#[cfg(feature = "mem")]
|
||
pub async fn dual_tier_mem_local(
|
||
root: impl Into<PathBuf>,
|
||
threshold: u64,
|
||
) -> Result<Store, Error> {
|
||
Ok(Store {
|
||
kv: KvEngine::Mem(MemBackend::default()),
|
||
large: Some(LargeEngine::Local(LocalBackend::new(root).await?)),
|
||
threshold,
|
||
window_delay: DEFAULT_DELETE_WINDOW_DELAY,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **kv-only mode** (ADR-011 §4; ADR-012 §6.2) with the sqlite kv
|
||
/// engine — the durable single-node kv-only shape: no large tier;
|
||
/// `threshold` is the put-time rejection point (known-length puts
|
||
/// over it reject immediately, unknown-length puts at mid-stream
|
||
/// threshold overflow — the put seam's behavior, identical to the
|
||
/// mem kv-only wiring; zero residue on either rejection). Routing
|
||
/// in this mode is kv for every length.
|
||
///
|
||
/// Opening (or creating) the database file is fallible I/O; the
|
||
/// constructor also runs the GC schema's engine-neutral call
|
||
/// (`ensure_gc_schema` — sqlite's body creates `gc_window` only,
|
||
/// ADR-012 §5), so a construction failure is `Err(Error::Io)` per
|
||
/// the fallible-constructor pattern `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob — [`DEFAULT_DELETE_WINDOW_DELAY`] is the shipped default,
|
||
/// `Duration::ZERO` the legal degenerate tuning. Not fleet-
|
||
/// anything: a sqlite fleet is a recorded non-option (ADR-010 §3).
|
||
#[cfg(feature = "sqlite")]
|
||
pub async fn kv_only_sqlite(
|
||
database: impl Into<PathBuf>,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Sqlite(SqliteBackend::new(database).await?);
|
||
ensure_gc_schema_at_construction(&kv, None).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: None,
|
||
threshold,
|
||
window_delay,
|
||
mode: KvOnly,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode** (the default, ADR-011 §4) in the default-build
|
||
/// production shape — `kv = sqlite` + `large = local` (feature
|
||
/// `sqlite` is default-on, ADR-013 §4): size-threshold dispatch at
|
||
/// `threshold` (constructor-tunable;
|
||
/// [`crate::dispatch::DEFAULT_DISPATCH_THRESHOLD`] is the shipped
|
||
/// default), at-or-below on the kv tier, over-threshold staged into
|
||
/// the large tier.
|
||
///
|
||
/// Fallible on two accounts (file I/O): opening/creating the pool
|
||
/// database — which also runs the GC schema's engine-neutral call
|
||
/// (`ensure_gc_schema`; sqlite's body creates `gc_window` only,
|
||
/// ADR-012 §5's one-window-host rule: the durable window is the kv
|
||
/// engine's when the kv engine is SQL-backed) — and creating the
|
||
/// local root. A construction failure is `Err(Error::Io)` per the
|
||
/// fallible-constructor pattern `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob — [`DEFAULT_DELETE_WINDOW_DELAY`] is the shipped default,
|
||
/// `Duration::ZERO` the legal degenerate tuning. Not fleet-
|
||
/// anything: a sqlite fleet is a recorded non-option (ADR-010 §3).
|
||
#[cfg(feature = "sqlite")]
|
||
pub async fn dual_tier_sqlite_local(
|
||
database: impl Into<PathBuf>,
|
||
root: impl Into<PathBuf>,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Sqlite(SqliteBackend::new(database).await?);
|
||
let large = LargeEngine::Local(LocalBackend::new(root).await?);
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **kv-only mode** (ADR-011 §4; ADR-012 §6.2) with the postgres kv
|
||
/// engine — the fleet host's kv-only shape: no large tier;
|
||
/// `threshold` is the put-time rejection point (known-length puts
|
||
/// over it reject immediately, unknown-length puts at mid-stream
|
||
/// threshold overflow — the put seam's behavior, identical to the
|
||
/// mem and sqlite kv-only wiring; zero residue on either
|
||
/// rejection). Routing in this mode is kv for every length.
|
||
///
|
||
/// `config` is the connection settings — the tokio-postgres
|
||
/// [`tokio_postgres::Config`]-parseable DSN form; `opts` carries
|
||
/// the engine-configuration knobs (schema name, pool size,
|
||
/// `synchronous_commit`), riding the constructor parameters, never
|
||
/// invented defaults. Opening the engine (config parse, pool
|
||
/// build, connect, the bootstrap DDL) is fallible I/O; the
|
||
/// constructor also runs the GC schema's engine-neutral call
|
||
/// (`ensure_gc_schema` — the postgres row's body creates
|
||
/// `gc_pins` + `gc_window`, the fleet host's both-tables row,
|
||
/// ADR-012 §5), so a construction failure is `Err(Error::Io)` per
|
||
/// the fallible-constructor pattern `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob — [`DEFAULT_DELETE_WINDOW_DELAY`] is the shipped default,
|
||
/// `Duration::ZERO` the legal degenerate tuning.
|
||
///
|
||
/// **Non-fleet (ADR-012 §3):** this constructor is implicitly
|
||
/// `fleet: false` — the ADR-005 in-process protocol applies
|
||
/// verbatim regardless of engine (pins are in-process guards, the
|
||
/// sweep lease is the in-process lock, none of the
|
||
/// `EngineFleetState` clauses run), behavior byte-unchanged. The
|
||
/// `fleet: true` declaration is this constructor's [`Store::kv_only_postgres_fleet`]
|
||
/// twin (wave 6's landed declaration — the composite
|
||
/// fleet-validity predicate plus the fleet parameters), slotted in
|
||
/// without reshaping; this signature carries no fleet parameter.
|
||
#[cfg(feature = "postgres")]
|
||
pub async fn kv_only_postgres(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Postgres(PostgresBackend::new(config, opts).await?);
|
||
ensure_gc_schema_at_construction(&kv, None).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: None,
|
||
threshold,
|
||
window_delay,
|
||
mode: KvOnly,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode** (the default, ADR-011 §4) in the
|
||
/// fleet-capable-kv production shape — `kv = postgres` +
|
||
/// `large = local`: size-threshold dispatch at `threshold`
|
||
/// (constructor-tunable;
|
||
/// [`crate::dispatch::DEFAULT_DISPATCH_THRESHOLD`] is the shipped
|
||
/// default), at-or-below on the kv tier, over-threshold staged
|
||
/// into the large tier.
|
||
///
|
||
/// `config` is the connection settings — the tokio-postgres
|
||
/// [`tokio_postgres::Config`]-parseable DSN form; `opts` carries
|
||
/// the engine-configuration knobs (schema name, pool size,
|
||
/// `synchronous_commit`), riding the constructor parameters, never
|
||
/// invented defaults.
|
||
///
|
||
/// Fallible on two accounts (I/O): opening the engine (config
|
||
/// parse, pool build, connect, the bootstrap DDL) — which also
|
||
/// runs the GC schema's engine-neutral call (`ensure_gc_schema`;
|
||
/// the postgres row's body creates `gc_pins` + `gc_window`,
|
||
/// ADR-012 §5's one-window-host rule: the durable window is the
|
||
/// kv engine's when the kv engine is SQL-backed, so the postgres
|
||
/// host creates both GC tables per its applicability row) — and
|
||
/// creating the local root. A construction failure is
|
||
/// `Err(Error::Io)` per the fallible-constructor pattern
|
||
/// `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob — [`DEFAULT_DELETE_WINDOW_DELAY`] is the shipped default,
|
||
/// `Duration::ZERO` the legal degenerate tuning.
|
||
///
|
||
/// **Non-fleet (ADR-012 §3):** this constructor is implicitly
|
||
/// `fleet: false` — the ADR-005 in-process protocol applies
|
||
/// verbatim regardless of engine (pins are in-process guards, the
|
||
/// sweep lease is the in-process lock, none of the
|
||
/// `EngineFleetState` clauses run), behavior byte-unchanged. The
|
||
/// `fleet: true` declaration is this constructor's [`Store::dual_tier_postgres_local_fleet`]
|
||
/// twin (wave 6's landed declaration — the composite
|
||
/// fleet-validity predicate plus the fleet parameters, the
|
||
/// shared-media assertion included), slotted in without
|
||
/// reshaping; this signature carries no fleet parameter.
|
||
#[cfg(feature = "postgres")]
|
||
pub async fn dual_tier_postgres_local(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
root: impl Into<PathBuf>,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Postgres(PostgresBackend::new(config, opts).await?);
|
||
let large = LargeEngine::Local(LocalBackend::new(root).await?);
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode** (the default, ADR-011 §4) in the
|
||
/// **consolidation posture** (ADR-009: "one engine to operate") —
|
||
/// `kv = postgres` + `large = pg-lo`: size-threshold dispatch at
|
||
/// `threshold` (constructor-tunable;
|
||
/// [`crate::dispatch::DEFAULT_DISPATCH_THRESHOLD`] is the shipped
|
||
/// default), at-or-below on the kv tier, over-threshold staged
|
||
/// into the large tier's held-transaction staging cursor.
|
||
///
|
||
/// **Connection-settings design point (two pairs):** this
|
||
/// constructor takes a DSN + [`PgOpts`] pair per engine (kv first,
|
||
/// large second). Cross-instance kv=postgres + large=pg-lo is a
|
||
/// valid non-fleet construction (ADR-012 §4); the consolidation
|
||
/// shape passes **the same DSN and opts to both** — the pool
|
||
/// content lives in one pg instance the kv tier rides (the
|
||
/// recorded corollary for a future conformance-suite factory:
|
||
/// minting one engine-owned schema and passing it to both engines
|
||
/// is equally valid — both engines' table names are distinct, so
|
||
/// the one-schema form holds `blobs`, `lo_entries`, and the GC
|
||
/// tables together; the two-schema form splits them — either is a
|
||
/// legitimate mint for this constructor's shape).
|
||
///
|
||
/// Fallible on I/O — engine opens (config parse, pool build,
|
||
/// connect, the bootstrap DDL) and the GC schema call
|
||
/// ([`ensure_gc_schema_at_construction`]: the kv host runs per the
|
||
/// one-window-host rule, creating `gc_pins` + `gc_window` in the
|
||
/// kv schema — the pg-lo window clauses are **dormant** on this
|
||
/// dual-SQL instance, so the pg-lo schema carries no `gc_window`).
|
||
/// A construction failure is `Err(Error::Io)` per the
|
||
/// fallible-constructor pattern `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob — [`DEFAULT_DELETE_WINDOW_DELAY`] is the shipped default,
|
||
/// `Duration::ZERO` the legal degenerate tuning.
|
||
///
|
||
/// **Non-fleet (ADR-012 §3):** this constructor is implicitly
|
||
/// `fleet: false` — the ADR-005 in-process protocol applies
|
||
/// verbatim regardless of engine (pins are in-process guards, the
|
||
/// sweep lease is the in-process lock, none of the
|
||
/// `EngineFleetState` clauses run), behavior byte-unchanged. The
|
||
/// `fleet: true` declaration is this constructor's [`Store::dual_tier_postgres_pglo_fleet`]
|
||
/// twin (wave 6's landed declaration — the composite fleet-validity
|
||
/// predicate plus the fleet parameters, the same-instance probe
|
||
/// included), slotted in without reshaping; this signature carries
|
||
/// no fleet parameter. This constructor completed
|
||
/// the fleet-valid engine set — the module doc's recorded forward
|
||
/// reference, resolved: with pg-lo's landing, every fleet-valid
|
||
/// engine had its named constructor before the declaration's
|
||
/// predicate and parameters followed.
|
||
#[cfg(all(feature = "postgres", feature = "pg-lo"))]
|
||
pub async fn dual_tier_postgres_pglo(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
large_config: &str,
|
||
large_opts: PgOpts,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Postgres(PostgresBackend::new(config, opts).await?);
|
||
let large = LargeEngine::PgLo(PgLoBackend::new(large_config, large_opts).await?);
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// **kv-only mode, `fleet: true`** (ADR-012 §3's declaration;
|
||
/// ADR-011 §4; ADR-012 §6.2) — the fleet re-routing client posture
|
||
/// (ADR-008 §3): the postgres kv engine alone, no large tier — the
|
||
/// composite fleet-validity predicate's `large = none` arm, the
|
||
/// client node's shape whose over-threshold pool content rides the
|
||
/// ops surface (the re-routing composition, wave 7's wiring; the
|
||
/// over-threshold put-time rejection, ADR-012 §6.2, unchanged
|
||
/// under fleet).
|
||
///
|
||
/// `config`/`opts`/`threshold`/`window_delay` carry exactly the
|
||
/// [`Store::kv_only_postgres`] twin's meaning (the non-fleet
|
||
/// posture stays byte-unchanged there). `pin_ttl` (ADR-008 §2) is
|
||
/// the fleet constructor parameter — the pin rows' renewal window
|
||
/// the store owns and renews on the TTL/3 cadence, the engine's
|
||
/// `pin_renew` successor-time body reading the TTL this
|
||
/// constructor passed to its backend; [`DEFAULT_PIN_TTL`] is the
|
||
/// shipped default (60 s). `Duration::ZERO` is a constructor
|
||
/// error at entry (the TTL/3 cadence degenerates at zero; a zero
|
||
/// TTL expires every pin at once) — validated before any I/O.
|
||
///
|
||
/// **The composite fleet-validity predicate runs at the
|
||
/// constructor** (`fleet_validity_error`, keyed off the engine
|
||
/// representation like every dispatch in this store): kv = postgres
|
||
/// (structural — the only kv engine a fleet constructor names;
|
||
/// the predicate's mem/sqlite rejection arms are the clause's
|
||
/// testing home) and `large = none` — the re-routing arm. The
|
||
/// store carries the fleet state on success (`Store::fleet`):
|
||
/// the declaration's flag, the pin TTL, and the store-instance id
|
||
/// (`mint_owner_id`).
|
||
#[cfg(feature = "postgres")]
|
||
pub async fn kv_only_postgres_fleet(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
pin_ttl: Duration,
|
||
) -> Result<Store, Error> {
|
||
if pin_ttl.is_zero() {
|
||
return Err(Error::Io(
|
||
"fleet: pin_ttl must be positive — the TTL/3 renewal cadence degenerates \
|
||
at zero, and a zero-TTL pin expires at once"
|
||
.to_owned(),
|
||
));
|
||
}
|
||
let kv =
|
||
KvEngine::Postgres(PostgresBackend::new_with_pin_ttl(config, opts, pin_ttl).await?);
|
||
if let Some(err) = fleet_validity_error(FleetKvShape::of(&kv), FleetLargeShape::None) {
|
||
return Err(err);
|
||
}
|
||
ensure_gc_schema_at_construction(&kv, None).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: None,
|
||
threshold,
|
||
window_delay,
|
||
mode: KvOnly,
|
||
gc: GcHome::default(),
|
||
fleet: Some(FleetState {
|
||
pin_ttl,
|
||
owner: mint_owner_id(),
|
||
}),
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode, `fleet: true`** (ADR-012 §3's declaration;
|
||
/// ADR-011 §4) — the fleet shared-media posture: `kv = postgres` +
|
||
/// `large = local`, the composite fleet-validity predicate's
|
||
/// declared-shared-media arm (ADR-008 §3's enforceable seam). The
|
||
/// joint entry+pin transaction is unachievable across media, so
|
||
/// the put path's ordering under fleet is pin-row-first,
|
||
/// publish-second (ADR-012 §4; the put path's wiring is
|
||
/// `store-fleet-put`'s).
|
||
///
|
||
/// `config`/`opts`/`root`/`threshold`/`window_delay` carry exactly
|
||
/// the [`Store::dual_tier_postgres_local`] twin's meaning (the
|
||
/// non-fleet posture stays byte-unchanged there). `pin_ttl` is the
|
||
/// fleet constructor parameter ([`DEFAULT_PIN_TTL`] the shipped
|
||
/// default; `Duration::ZERO` a constructor error at entry). **The
|
||
/// large-tier declaration** rides [`FleetSharedMedia`] — the
|
||
/// operator's explicit shared-media assertion, a named type the
|
||
/// call site cannot construct accidentally; `local` without it
|
||
/// never reaches a fleet constructor (the predicate's
|
||
/// `LocalUndeclared` arm is the rule's textual form). The
|
||
/// composite fleet-validity predicate runs at the constructor
|
||
/// (kv = postgres and the declared shared media); the cross-node
|
||
/// truth behind the assertion stays the deployment's verified
|
||
/// invariant (ADR-008 §3's seam). The store carries the fleet
|
||
/// state on success (`Store::fleet`).
|
||
#[cfg(feature = "postgres")]
|
||
pub async fn dual_tier_postgres_local_fleet(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
root: impl Into<PathBuf>,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
pin_ttl: Duration,
|
||
shared_media: FleetSharedMedia,
|
||
) -> Result<Store, Error> {
|
||
if pin_ttl.is_zero() {
|
||
return Err(Error::Io(
|
||
"fleet: pin_ttl must be positive — the TTL/3 renewal cadence degenerates \
|
||
at zero, and a zero-TTL pin expires at once"
|
||
.to_owned(),
|
||
));
|
||
}
|
||
let kv =
|
||
KvEngine::Postgres(PostgresBackend::new_with_pin_ttl(config, opts, pin_ttl).await?);
|
||
let large = LargeEngine::Local(LocalBackend::new(root).await?);
|
||
let _ = shared_media;
|
||
if let Some(err) =
|
||
fleet_validity_error(FleetKvShape::of(&kv), FleetLargeShape::LocalSharedMedia)
|
||
{
|
||
return Err(err);
|
||
}
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: Some(FleetState {
|
||
pin_ttl,
|
||
owner: mint_owner_id(),
|
||
}),
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode, `fleet: true`** (ADR-012 §3's declaration;
|
||
/// ADR-009's consolidation posture) — `kv = postgres` + `large =
|
||
/// pg-lo`, **the same pg instance** (ADR-012 §4's checkable
|
||
/// clause, the predicate's runtime arm): the joint entry+pin tx
|
||
/// the fleet put path drives (`store-fleet-put`) is expressible
|
||
/// over the shared instance.
|
||
///
|
||
/// `config`/`opts`/`large_config`/`large_opts`/`threshold`/
|
||
/// `window_delay` carry exactly the [`Store::dual_tier_postgres_pglo`]
|
||
/// twin's meaning (cross-instance kv=postgres + large=pg-lo stays
|
||
/// valid only as the non-fleet twin). `pin_ttl` is the fleet
|
||
/// constructor parameter ([`DEFAULT_PIN_TTL`] the shipped
|
||
/// default; `Duration::ZERO` a constructor error at entry) — it
|
||
/// configures the **kv engine's** backend (`pin_renew` is the kv
|
||
/// host's clause; the pg-lo engine hosts no fleet state, ADR-012
|
||
/// §5).
|
||
///
|
||
/// **The pg-lo same-instance probe (named design point):** the
|
||
/// instance's `system_identifier` (via `pg_control_system()`,
|
||
/// PG14+) read across one pooled connection per engine and
|
||
/// compared — server-side, immune to DSN aliases (textual DSN
|
||
/// comparison was rejected: hostname vs IP vs alias cannot decide
|
||
/// instance identity). A mismatch, **or any probe failure** (a
|
||
/// pre-PG14 server, a denied function call, an unreachable
|
||
/// instance), is a constructor error — the probe errors closed,
|
||
/// never a silent pass. The composite fleet-validity predicate
|
||
/// runs at the constructor alongside the probe; the store carries
|
||
/// the fleet state on success (`Store::fleet`).
|
||
#[cfg(all(feature = "postgres", feature = "pg-lo"))]
|
||
pub async fn dual_tier_postgres_pglo_fleet(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
large_config: &str,
|
||
large_opts: PgOpts,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
pin_ttl: Duration,
|
||
) -> Result<Store, Error> {
|
||
if pin_ttl.is_zero() {
|
||
return Err(Error::Io(
|
||
"fleet: pin_ttl must be positive — the TTL/3 renewal cadence degenerates \
|
||
at zero, and a zero-TTL pin expires at once"
|
||
.to_owned(),
|
||
));
|
||
}
|
||
let kv_backend = PostgresBackend::new_with_pin_ttl(config, opts, pin_ttl).await?;
|
||
let large_backend = PgLoBackend::new(large_config, large_opts).await?;
|
||
let kv_id = kv_backend.system_identifier().await?;
|
||
let large_id = large_backend.system_identifier().await?;
|
||
let kv = KvEngine::Postgres(kv_backend);
|
||
let large = LargeEngine::PgLo(large_backend);
|
||
if let Some(err) = fleet_validity_error(FleetKvShape::of(&kv), FleetLargeShape::PgLo) {
|
||
return Err(err);
|
||
}
|
||
if kv_id != large_id {
|
||
return Err(Error::Io(format!(
|
||
"fleet: the kv engine and the pg-lo large engine resolve to different pg \
|
||
instances (system_identifier {kv_id} vs {large_id}) — ADR-012 \u{a7}4's \
|
||
same-instance clause; cross-instance pg-lo is valid only as fleet: false"
|
||
)));
|
||
}
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: Some(FleetState {
|
||
pin_ttl,
|
||
owner: mint_owner_id(),
|
||
}),
|
||
})
|
||
}
|
||
|
||
/// **dual-tier mode** (the default, ADR-011 §4) in the
|
||
/// one-window-host rule's **named test topology** (ADR-012 §5) —
|
||
/// `kv = mem` + `large = pg-lo`: the durable delete window hosts
|
||
/// on the **large engine** when the kv engine is not SQL-backed;
|
||
/// the constructor runs the pg-lo host's `ensure_gc_schema` call
|
||
/// ([`ensure_gc_schema_at_construction`]'s large-host arm —
|
||
/// `gc_window` in the pg-lo schema) and its sweeps run the
|
||
/// durable form on the pg-lo host, with the inline form nowhere in
|
||
/// this shape's story (the mem kv arm of the form branch consults
|
||
/// the large engine precisely when it hosts the window;
|
||
/// ADR-012 §5's non-SQL inline form exists for mem/local *without*
|
||
/// a large SQL host).
|
||
///
|
||
/// `config` is the connection settings — the tokio-postgres
|
||
/// [`tokio_postgres::Config`]-parseable DSN form; `opts` carries
|
||
/// the engine-configuration knobs (schema name, pool size,
|
||
/// `synchronous_commit`), riding the constructor parameters, never
|
||
/// invented defaults. The kv tier is the mem engine — fallible
|
||
/// only on the pg-lo engine's I/O (config parse, pool build,
|
||
/// connect, the bootstrap DDL, the GC schema call): a construction
|
||
/// failure is `Err(Error::Io)` per the fallible-constructor
|
||
/// pattern `dual_tier_mem_local` set.
|
||
///
|
||
/// `window_delay` (ADR-008 §2) is the durable window's cadence
|
||
/// knob on the pg-lo host — [`DEFAULT_DELETE_WINDOW_DELAY`] is the
|
||
/// shipped default, `Duration::ZERO` the legal degenerate tuning.
|
||
///
|
||
/// Not fleet-anything: mem is never fleet-valid (ADR-011 §4).
|
||
#[cfg(all(feature = "mem", feature = "pg-lo"))]
|
||
pub async fn dual_tier_mem_pglo(
|
||
config: &str,
|
||
opts: PgOpts,
|
||
threshold: u64,
|
||
window_delay: Duration,
|
||
) -> Result<Store, Error> {
|
||
let kv = KvEngine::Mem(MemBackend::default());
|
||
let large = LargeEngine::PgLo(PgLoBackend::new(config, opts).await?);
|
||
ensure_gc_schema_at_construction(&kv, Some(&large)).await?;
|
||
Ok(Store {
|
||
kv,
|
||
large: Some(large),
|
||
threshold,
|
||
window_delay,
|
||
mode: DualTier,
|
||
gc: GcHome::default(),
|
||
fleet: None,
|
||
})
|
||
}
|
||
|
||
/// The constructor mode (ADR-011 §4's vocabulary) — introspection
|
||
/// for embedders and tests; the dispatch/put/read paths branch on
|
||
/// it crate-internally.
|
||
pub fn mode(&self) -> StoreMode {
|
||
self.mode
|
||
}
|
||
|
||
/// The active dispatch threshold — `Some` in dual-tier (the routing
|
||
/// boundary) and kv-only (the put-time rejection point); `None` in
|
||
/// mem-only mode ("no dispatch threshold in effect", ADR-011 §4).
|
||
pub(crate) fn threshold(&self) -> Option<u64> {
|
||
match self.mode {
|
||
DualTier | KvOnly => Some(self.threshold),
|
||
MemOnly => None,
|
||
}
|
||
}
|
||
|
||
/// The delete-window delay the constructor took (ADR-008 §2) —
|
||
/// crate-internal: the sweep's SQL-backed form consumes it when
|
||
/// staging candidates (`visible_after = mark_end + window`);
|
||
/// introspection for the construction tests. `Duration::ZERO`
|
||
/// (a constructor-passed value) is the legal degenerate tuning —
|
||
/// execute-on-next-sweep.
|
||
pub(crate) fn window_delay(&self) -> Duration {
|
||
self.window_delay
|
||
}
|
||
|
||
/// The store's fleet state — `Some` exactly when the constructor
|
||
/// declared `fleet: true` (the declaration is explicit, never
|
||
/// inferred, ADR-012 §3): the put/sweep paths branch on the
|
||
/// state's presence, never on re-deriving from the engine set.
|
||
/// Crate-internal introspection (`FleetState`): the fleet
|
||
/// parameters are constructor parameters, not API (ADR-008 §3).
|
||
pub(crate) fn fleet(&self) -> Option<&FleetState> {
|
||
self.fleet.as_ref()
|
||
}
|
||
|
||
/// **Write-side tier selection for a given content length** — the
|
||
/// dispatch seam's routing half (the put seam programs through
|
||
/// this; read paths do not consult it). Dual-tier:
|
||
/// [`crate::dispatch::route`]'s pure function at the instance's
|
||
/// threshold (exactly-at-threshold stays kv — the pinned boundary
|
||
/// reading). kv-only: kv for every length — the over-threshold
|
||
/// *rejection* is the put seam's behavior keyed off
|
||
/// [`Store::threshold`] (ADR-012 §6.2), not a routing answer.
|
||
/// Mem-only: kv — there is no other tier and no threshold in
|
||
/// effect.
|
||
pub(crate) fn tier_for(&self, len: u64) -> Tier {
|
||
match self.mode {
|
||
DualTier => route(len, self.threshold),
|
||
KvOnly | MemOnly => Tier::Kv,
|
||
}
|
||
}
|
||
|
||
/// **The fall-through consult order — kv first** (ADR-012 §6.1's
|
||
/// general rule): every key-addressed op consults the kv tier
|
||
/// first; large comes after ([`Store::large`]). A miss on both
|
||
/// tiers resolves per op's shape (`None` on get/stat,
|
||
/// `Err(Missing)` on read_range/delete).
|
||
pub(crate) fn kv(&self) -> &KvEngine {
|
||
&self.kv
|
||
}
|
||
|
||
/// **…then large** — the second consult; `None` in single-tier
|
||
/// modes (mem-only, kv-only), where large-tier content cannot
|
||
/// exist.
|
||
pub(crate) fn large(&self) -> Option<&LargeEngine> {
|
||
self.large.as_ref()
|
||
}
|
||
|
||
/// The GC state home (ADR-010 §2: the store core owns all GC
|
||
/// state) — the gc-protocol task's pin registry, liveness sources,
|
||
/// protect callback, and in-process delete window land here and
|
||
/// drive the sweep and direct delete from this facade.
|
||
pub(crate) fn gc(&self) -> &GcHome {
|
||
&self.gc
|
||
}
|
||
}
|
||
|
||
/// The read path (store-api.md §Read path): `get`, `stat`,
|
||
/// `read_range`, `has`, `list` — every key-addressed op falling through
|
||
/// kv first, then large (ADR-012 §6.1's general rule), typed key
|
||
/// conversion at the facade boundary, and the module-doc design points
|
||
/// (the `Blob` stream shape, verification-on-get, the `Vec<Key>` list
|
||
/// shape) carried in their implementations.
|
||
impl Store {
|
||
/// `get(key) → Option<(len, stream)>`. Falls through kv → large;
|
||
/// a both-tier miss is `Ok(None)` — never `Error::Missing`
|
||
/// (ADR-012 §6.5 ruling 5). On a kv hit the materialized read is
|
||
/// whole-blob-verified against the key at get time (mismatch:
|
||
/// `Err(Error::Verification)`, never a further fall-through) and
|
||
/// the returned length is the verified materialized byte length;
|
||
/// the large tier's hit opens streaming (metadata + handle, no
|
||
/// canonical re-derivation — see the module doc), its length the
|
||
/// cursor's metadata-only probe. Both hits return the same
|
||
/// `(len, stream)` shape: the stream is [`Blob`] over the tier's
|
||
/// cursor body.
|
||
pub async fn get(&self, key: &[u8]) -> Result<Option<(u64, Blob)>, Error> {
|
||
let key = boundary_key(key)?;
|
||
let bytes = key.as_bytes();
|
||
|
||
let cursor = if let Some(cursor) = self.kv().get(&bytes).await? {
|
||
cursor
|
||
} else if let Some(cursor) = self.large_get(&bytes).await? {
|
||
cursor
|
||
} else {
|
||
return Ok(None);
|
||
};
|
||
|
||
match cursor {
|
||
AnyCursor::Kv(kv) => {
|
||
let value = kv.read_at(0, kv.len()).await?;
|
||
let derived = key.algorithm().derive(&value, value.len() as u64);
|
||
if &derived != key.digest() {
|
||
return Err(Error::Verification);
|
||
}
|
||
Ok(Some((value.len() as u64, Blob::new(AnyCursor::Kv(kv)))))
|
||
}
|
||
AnyCursor::Large(large) => Ok(Some((large.len(), Blob::new(AnyCursor::Large(large))))),
|
||
}
|
||
}
|
||
|
||
/// The large tier's consult — `None` where the mode has no large
|
||
/// tier (single-tier modes cannot hold large-tier content).
|
||
async fn large_get(&self, bytes: &[u8]) -> Result<Option<AnyCursor>, Error> {
|
||
match self.large() {
|
||
Some(large) => large.get(bytes).await,
|
||
None => Ok(None),
|
||
}
|
||
}
|
||
|
||
/// `stat(key) → Option<EntryMeta>` — the cheap probe riding the
|
||
/// trait's `size` method (ADR-008): length plus the key's hash
|
||
/// algorithm, never a content load, falling through kv → large.
|
||
/// A both-tier miss is `Ok(None)`.
|
||
pub async fn stat(&self, key: &[u8]) -> Result<Option<EntryMeta>, Error> {
|
||
let key = boundary_key(key)?;
|
||
let bytes = key.as_bytes();
|
||
|
||
let len = match self.kv().size(&bytes).await? {
|
||
Some(len) => Some(len),
|
||
None => match self.large() {
|
||
Some(large) => large.size(&bytes).await?,
|
||
None => None,
|
||
},
|
||
};
|
||
Ok(len.map(|len| EntryMeta {
|
||
len,
|
||
algorithm: key.algorithm(),
|
||
}))
|
||
}
|
||
|
||
/// `read_range(key, offset, len) → (slice, slice_digest)` — the
|
||
/// slice plus the out-of-band SHA-256 slice digest (ADR-006, fixed
|
||
/// by convention), composed through the cursor's `read_range` seam.
|
||
/// Falls through kv → large exactly as [`Store::get`] does; no
|
||
/// canonical verification rides a range (the slice digest is the
|
||
/// whole answer, ADR-006 decision 2). A both-tier miss is
|
||
/// `Err(Error::Missing)` — the pinned shape difference vs `get`'s
|
||
/// `None` (ADR-012 §6.5 ruling 5).
|
||
pub async fn read_range(
|
||
&self,
|
||
key: &[u8],
|
||
offset: u64,
|
||
len: u64,
|
||
) -> Result<(Vec<u8>, [u8; 32]), Error> {
|
||
let key = boundary_key(key)?;
|
||
let bytes = key.as_bytes();
|
||
|
||
if let Some(cursor) = self.kv().get(&bytes).await? {
|
||
return cursor.read_range(offset, len).await;
|
||
}
|
||
match self.large_get(&bytes).await? {
|
||
Some(cursor) => cursor.read_range(offset, len).await,
|
||
None => Err(Error::Missing),
|
||
}
|
||
}
|
||
|
||
/// `has(key) → bool` — the pure existence probe (ADR-008's `has`
|
||
/// discipline: an existence probe per tier, `local`'s body being
|
||
/// the literal size-probe), falling through kv → large. Absent on
|
||
/// both tiers: `false`. Virgin-store reads are no-ops.
|
||
pub async fn has(&self, key: &[u8]) -> Result<bool, Error> {
|
||
let key = boundary_key(key)?;
|
||
let bytes = key.as_bytes();
|
||
|
||
if self.kv().has(&bytes).await? {
|
||
return Ok(true);
|
||
}
|
||
match self.large() {
|
||
Some(large) => large.has(&bytes).await,
|
||
None => Ok(false),
|
||
}
|
||
}
|
||
|
||
/// `list() → Vec<Key>` — whole-pool enumeration of every tier,
|
||
/// complete by contract (ADR-003), returned materialized and
|
||
/// sorted in the keys' total order (the recorded Vec-vs-stream
|
||
/// design point; see the module doc). Malformed pool bytes are
|
||
/// fatal: `Err(Error::KeyInvalid)`, never a silent skip.
|
||
pub async fn list(&self) -> Result<Vec<Key>, Error> {
|
||
let mut keys = typed_keys(self.kv().list().await?)?;
|
||
if let Some(large) = self.large() {
|
||
keys.extend(typed_keys(large.list().await?)?);
|
||
}
|
||
keys.sort();
|
||
Ok(keys)
|
||
}
|
||
}
|
||
|
||
/// The write path's streaming chunk granularity — every put loop reads
|
||
/// the stream in windows of this size. Sized inside the kv tier's
|
||
/// economics regime (bounded memory per iteration; engine bodies
|
||
/// aggregate upstream — the pg-lo engine's measured 512 KiB
|
||
/// `lowrite` knee, [`crate::pg_lo::LOWRITE_CHUNK`], aggregates these
|
||
/// windows into its C2 statement shape; that engine's constant is the
|
||
/// authority there, landed with its wave).
|
||
const PUT_CHUNK: usize = 64 * 1024;
|
||
|
||
/// The write path (store-api.md §Write path): `put` — the two internal
|
||
/// paths behind one signature, the `Batch` batching seam, and the
|
||
/// kv-only rejection point (ADR-012 §6.2), on the pin-before-publish
|
||
/// ordering (ADR-005). The named design points (stream shape, len
|
||
/// mismatch, kv-only carrier, key-at-commit amendment) are recorded in
|
||
/// the module doc; their behaviors are pinned by the tests below.
|
||
impl Store {
|
||
/// **`put(len, stream) → (Key, Pin)`** — the streaming put seam
|
||
/// (store-api.md §Write path; POC #3 finding A1): two internal
|
||
/// paths behind one signature, routing a pure function of content
|
||
/// length (ADR-003) — no content ever migrates tiers.
|
||
///
|
||
/// **Known-length (`Some(len)`) — the encouraged path** (covers
|
||
/// git objects, `stat`-derived files, `Content-Length`-bearing
|
||
/// uploads, network offers): the preamble (`"blob <len>\0"`)
|
||
/// initializes the derivation before content flows, so one pass
|
||
/// suffices — no staging needed for hashing. ≤ the threshold → the
|
||
/// kv tier's whole-value put; > the threshold → the large tier's
|
||
/// staged put (content streams into the staging cursor while
|
||
/// hashing; the derivation completes at stream end; commit-rename
|
||
/// publishes). A stream that yields fewer or more bytes than the
|
||
/// declared `len` rejects with `Error::Verification` — zero residue
|
||
/// either side (the module doc's named design point).
|
||
///
|
||
/// **Unknown-length (`None`):** content buffers in memory up to the
|
||
/// dispatch threshold, ADR-003's option (d) semantics pinned in the
|
||
/// three numbered rules:
|
||
///
|
||
/// 1. **Mid-stream threshold overflow** (non-restartable sources
|
||
/// are normal): the buffered prefix flushes *into a fresh stage
|
||
/// file* and the stream continues appending into it — there is
|
||
/// no "restart the stream" requirement anywhere.
|
||
/// 2. **Verification runs over the eventual stored artifact as a
|
||
/// whole**: the memory buffer for the sub-threshold kv commit;
|
||
/// the complete staged bytes for the large commit — the
|
||
/// derivation pass restarts over preamble + staged bytes (the
|
||
/// preamble is inside the hashed input, ADR-002), reading the
|
||
/// staged artifact back through the staging cursor's `ReadCursor`
|
||
/// seam.
|
||
/// 3. **Stream error before commit**: no entry lands and no stage
|
||
/// file remains — buffer discard on the failure path, exactly
|
||
/// like stage discard (the stage-hygiene invariant extended to
|
||
/// the buffered case).
|
||
///
|
||
/// **Pin-before-publish (ADR-005):** the pin refcount is minted
|
||
/// under the arbitration lock *before* the entry becomes visible —
|
||
/// an entry cannot appear in a sweep's `list()` without already
|
||
/// carrying its pin. A dedup re-put acquires no new pool entry; the
|
||
/// returned `Pin` refcounts the existing one under the same lock.
|
||
///
|
||
/// **kv-only mode (ADR-012 §6.2):** a known-length put over the
|
||
/// threshold rejects immediately; an unknown-length put rejects at
|
||
/// mid-stream threshold overflow — lengths are not known at
|
||
/// construction, so put time is the only possible rejection point.
|
||
/// Zero residue on either rejection.
|
||
pub async fn put(
|
||
&self,
|
||
len: Option<u64>,
|
||
stream: impl AsyncRead + Unpin + Send,
|
||
) -> Result<(Key, Pin), Error> {
|
||
match len {
|
||
Some(len) => {
|
||
if self.mode == KvOnly && len > self.threshold {
|
||
return Err(Error::Io(format!(
|
||
"kv-only mode rejects an over-threshold put: declared len {len} > threshold {}",
|
||
self.threshold
|
||
)));
|
||
}
|
||
self.put_known(len, stream).await
|
||
}
|
||
None => self.put_unknown(stream).await,
|
||
}
|
||
}
|
||
|
||
/// **The batch-scoped pin group** (store-api.md §Write path; ADR-005
|
||
/// §Marks 2). Multi-put writes — manifest writes are exactly this —
|
||
/// group under one batch-scoped pin: [`Store::put_in_batch`] runs
|
||
/// each member's lone-put flow and contributes its pin to the
|
||
/// group; the group's pins drop together on [`Batch::commit`] or
|
||
/// drop (no early single-pin release; member liveness never
|
||
/// outlives the batch — batch drop is the failure path).
|
||
pub fn batch(&self) -> Batch {
|
||
self.gc().new_batch()
|
||
}
|
||
|
||
/// A member put inside a [`Batch`] — the batching put seam
|
||
/// (store-api.md §Write path). Member puts publish pin-before-
|
||
/// publish exactly as lone puts do: the two paths behind
|
||
/// [`Store::put`]'s one signature run, and the minted guard is
|
||
/// contributed to the group instead of returned. The batch's pins
|
||
/// drop together on [`Batch::commit`] or batch drop.
|
||
pub async fn put_in_batch(
|
||
&self,
|
||
batch: &mut Batch,
|
||
len: Option<u64>,
|
||
stream: impl AsyncRead + Unpin + Send,
|
||
) -> Result<Key, Error> {
|
||
let (key, pin) = self.put(len, stream).await?;
|
||
batch.add(pin);
|
||
Ok(key)
|
||
}
|
||
|
||
/// The known-length internal path: one streaming pass, the preamble
|
||
/// initialized into the derivation up front. Routing by the declared
|
||
/// length (the pure function); a stream that yields fewer or more
|
||
/// bytes than `len` rejects `Verification` with zero residue on
|
||
/// either tier.
|
||
async fn put_known<R: AsyncRead + Unpin + Send>(
|
||
&self,
|
||
len: u64,
|
||
mut stream: R,
|
||
) -> Result<(Key, Pin), Error> {
|
||
let mut hasher = BlobHasher::new(Algorithm::GitBlobSha256, len);
|
||
let mut chunk = vec![0u8; PUT_CHUNK];
|
||
let mut received = 0u64;
|
||
|
||
match self.tier_for(len) {
|
||
Tier::Kv => {
|
||
let mut value: Vec<u8> = Vec::new();
|
||
loop {
|
||
let n = stream
|
||
.read(&mut chunk)
|
||
.await
|
||
.map_err(|e| Error::Io(e.to_string()))?;
|
||
if n == 0 {
|
||
break;
|
||
}
|
||
received = received.saturating_add(n as u64);
|
||
if received > len {
|
||
return Err(Error::Verification);
|
||
}
|
||
hasher.update(&chunk[..n]);
|
||
value.extend_from_slice(&chunk[..n]);
|
||
}
|
||
if received != len {
|
||
return Err(Error::Verification);
|
||
}
|
||
let key = Key::from(hasher.finish());
|
||
let pin = self.gc().acquire(&key).await;
|
||
self.kv().put(&key.as_bytes(), &value).await?;
|
||
Ok((key, pin))
|
||
}
|
||
Tier::Large => {
|
||
let engine = match self.large() {
|
||
Some(engine) => engine,
|
||
None => return Err(Error::Io("no large tier in this mode".to_owned())),
|
||
};
|
||
let mut stage = engine.put_staged().await?;
|
||
let fed: Result<(), Error> = 'stream: loop {
|
||
let n = match stream.read(&mut chunk).await {
|
||
Ok(n) => n,
|
||
Err(e) => break 'stream Err(Error::Io(e.to_string())),
|
||
};
|
||
if n == 0 {
|
||
break 'stream Ok(());
|
||
}
|
||
received = received.saturating_add(n as u64);
|
||
if received > len {
|
||
break 'stream Err(Error::Verification);
|
||
}
|
||
hasher.update(&chunk[..n]);
|
||
if let Err(e) = stage.append(&chunk[..n]).await {
|
||
break 'stream Err(e);
|
||
}
|
||
};
|
||
match fed {
|
||
Err(err) => {
|
||
stage.discard().await;
|
||
Err(err)
|
||
}
|
||
Ok(()) if received != len => {
|
||
stage.discard().await;
|
||
Err(Error::Verification)
|
||
}
|
||
Ok(()) => {
|
||
let key = Key::from(hasher.finish());
|
||
let pin = self.gc().acquire(&key).await;
|
||
stage.commit(&key.as_bytes()).await?;
|
||
Ok((key, pin))
|
||
}
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The unknown-length internal path: buffer to the dispatch
|
||
/// threshold; on mid-stream overflow the buffered prefix flushes
|
||
/// into a fresh stage file and the stream continues appending
|
||
/// (rule 1) — kv-only mode rejects at that same overflow
|
||
/// (ADR-012 §6.2). At stream end, the derivation pass runs over the
|
||
/// eventual stored artifact as a whole: the buffer (kv form) or the
|
||
/// staged bytes read back through the cursor's `ReadCursor` seam
|
||
/// (staged form, rule 2). Every failure path discards (rule 3).
|
||
async fn put_unknown<R: AsyncRead + Unpin + Send>(
|
||
&self,
|
||
mut stream: R,
|
||
) -> Result<(Key, Pin), Error> {
|
||
let mut buffered = Vec::with_capacity(PUT_CHUNK);
|
||
let mut stage: Option<AnyStaging> = None;
|
||
let mut streamed_total = 0u64;
|
||
let mut chunk = vec![0u8; PUT_CHUNK];
|
||
|
||
let fed: Result<(), Error> = 'feed: loop {
|
||
let n = match stream.read(&mut chunk).await {
|
||
Ok(n) => n,
|
||
Err(e) => break 'feed Err(Error::Io(e.to_string())),
|
||
};
|
||
if n == 0 {
|
||
break 'feed Ok(());
|
||
}
|
||
streamed_total = streamed_total.saturating_add(n as u64);
|
||
|
||
if let Some(staged) = stage.as_mut() {
|
||
let append = staged.append(&chunk[..n]).await;
|
||
if let Err(e) = append {
|
||
if let Some(cursor) = stage.take() {
|
||
cursor.discard().await;
|
||
}
|
||
break 'feed Err(e);
|
||
}
|
||
continue;
|
||
}
|
||
|
||
let overflow = match self.threshold() {
|
||
Some(t) => (buffered.len() as u64).saturating_add(n as u64) > t,
|
||
None => false,
|
||
};
|
||
if !overflow {
|
||
buffered.extend_from_slice(&chunk[..n]);
|
||
continue;
|
||
}
|
||
match self.mode {
|
||
KvOnly => {
|
||
let t = self.threshold;
|
||
break 'feed Err(Error::Io(format!(
|
||
"kv-only mode rejects an over-threshold put: streamed {streamed_total} bytes past the threshold {t}"
|
||
)));
|
||
}
|
||
_ => match self.large() {
|
||
Some(engine) => {
|
||
let mut staged = match engine.put_staged().await {
|
||
Ok(staged) => staged,
|
||
Err(e) => break 'feed Err(e),
|
||
};
|
||
if let Err(e) = staged.append(&buffered).await {
|
||
staged.discard().await;
|
||
break 'feed Err(e);
|
||
}
|
||
if let Err(e) = staged.append(&chunk[..n]).await {
|
||
staged.discard().await;
|
||
break 'feed Err(e);
|
||
}
|
||
stage = Some(staged);
|
||
}
|
||
None => {
|
||
break 'feed Err(Error::Io("no large tier in this mode".to_owned()));
|
||
}
|
||
},
|
||
}
|
||
};
|
||
|
||
match fed {
|
||
Err(err) => {
|
||
if let Some(cursor) = stage.take() {
|
||
cursor.discard().await;
|
||
}
|
||
Err(err)
|
||
}
|
||
Ok(()) => match stage.take() {
|
||
Some(staged) => {
|
||
// Rule 2, staged form: the derivation pass restarts
|
||
// over preamble + staged bytes, the artifact read
|
||
// back through the cursor's ReadCursor seam.
|
||
let mut hasher = BlobHasher::new(Algorithm::GitBlobSha256, streamed_total);
|
||
let mut offset = 0u64;
|
||
let derived: Result<(), Error> = 'pass: loop {
|
||
if offset >= streamed_total {
|
||
break 'pass Ok(());
|
||
}
|
||
let want = (streamed_total - offset).min(PUT_CHUNK as u64);
|
||
let bytes = match staged.read_at(offset, want).await {
|
||
Ok(bytes) => bytes,
|
||
Err(e) => break 'pass Err(e),
|
||
};
|
||
if bytes.is_empty() {
|
||
break 'pass Err(Error::Io(
|
||
"staged artifact under-read during the derivation pass".to_owned(),
|
||
));
|
||
}
|
||
hasher.update(&bytes);
|
||
offset += bytes.len() as u64;
|
||
};
|
||
match derived {
|
||
Err(err) => {
|
||
staged.discard().await;
|
||
Err(err)
|
||
}
|
||
Ok(()) => {
|
||
let key = Key::from(hasher.finish());
|
||
let pin = self.gc().acquire(&key).await;
|
||
staged.commit(&key.as_bytes()).await?;
|
||
Ok((key, pin))
|
||
}
|
||
}
|
||
}
|
||
None => {
|
||
// Rule 2, buffer form: one derivation pass over the
|
||
// complete buffered artifact — the kv commit's whole
|
||
// stored value.
|
||
if buffered.len() as u64 != streamed_total {
|
||
return Err(Error::Io(
|
||
"buffered artifact length disagrees with the streamed total".to_owned(),
|
||
));
|
||
}
|
||
let mut hasher = BlobHasher::new(Algorithm::GitBlobSha256, streamed_total);
|
||
hasher.update(&buffered);
|
||
let key = Key::from(hasher.finish());
|
||
let pin = self.gc().acquire(&key).await;
|
||
self.kv().put(&key.as_bytes(), &buffered).await?;
|
||
Ok((key, pin))
|
||
}
|
||
},
|
||
}
|
||
}
|
||
}
|
||
|
||
/// **The delete-window commit batch size** — ~100/batch, iroh's proven
|
||
/// shape (ADR-005 §Sweep; POC #1 finding 3's named shape adopted).
|
||
const SWEEP_DELETE_BATCH: usize = 100;
|
||
|
||
/// The GC protocol's facade surface (ADR-005; gc-and-namespaces.md;
|
||
/// store-api.md §Lifecycle, §Invariants 4/5): the liveness
|
||
/// registration verbs, the direct `delete` with its arbitration-then-
|
||
/// delete refusal, and the `sweep()` delete-window protocol. This is
|
||
/// the architecture's most intricate state machine in its single-node
|
||
/// form — the fleet extension (DB-backed pins, sweep lease, SQL
|
||
/// arbitration) extends it without changing its invariants.
|
||
///
|
||
/// **The state machine, one instance at a time.** All shared GC state
|
||
/// rides the crate-internal `GcHome` behind the crate-internal
|
||
/// `Store::gc` accessor: one arbitration lock
|
||
/// over per-key pin refcounts, in-flight delete markers, and the
|
||
/// registered source lists ([`crate::gc`]'s module doc carries the
|
||
/// named design points — the marker mechanism that closes the
|
||
/// sweep-vs-put race without engine I/O under the lock, and the
|
||
/// single-node sweep lease).
|
||
///
|
||
/// **Abort-as-data** (ADR-012 §1): every sweep returns
|
||
/// `Ok(SweepReport)`; aborts are `GcAbortCause` values in the report,
|
||
/// never errors. `Error::GcRefuse` is the direct-delete refusal alone,
|
||
/// decided here at the facade — it never fires below the facade (the
|
||
/// error module's altitude discipline). The full protection set
|
||
/// (ADR-012 §6.4): pinned, re-observed live via a registered source,
|
||
/// or named by a protect callback.
|
||
impl Store {
|
||
/// Register a [`LivenessSource`] — the live-shared GC seam (ADR-005
|
||
/// §Marks 1; `register_liveness_source` verbatim; POC #1 finding
|
||
/// 7's finding is the verb's pinning reason). The source is
|
||
/// consulted **live** at mark time and at delete-time arbitration —
|
||
/// never a snapshot at registration. The returned
|
||
/// [`Registration`] token deregisters via
|
||
/// [`Store::unregister_liveness_source`] (ADR-012 §6.4's correction
|
||
/// flow). Zero registered sources make every sweep abort with
|
||
/// [`GcAbortCause::NoLivenessSources`] — the safe default: a
|
||
/// consumer without registered liveness sources cannot have its
|
||
/// content deleted.
|
||
pub fn register_liveness_source(&self, source: Arc<dyn LivenessSource>) -> Registration {
|
||
self.gc.register_source(source)
|
||
}
|
||
|
||
/// Deregister a liveness source previously registered on this
|
||
/// store. `false` when the token was already consumed or never
|
||
/// came from [`Store::register_liveness_source`].
|
||
pub fn unregister_liveness_source(&self, registration: Registration) -> bool {
|
||
self.gc.unregister_source(registration)
|
||
}
|
||
|
||
/// Register a [`ProtectCallback`] — ADR-005 §Marks 3's protection
|
||
/// source. Consulted pre-sweep (it may add externally-known hashes
|
||
/// or fail the sweep into [`GcAbortCause::ProtectFailed`] report
|
||
/// data) and at direct-delete arbitration, where a
|
||
/// protect-callback-named key is refused (`Error::GcRefuse`, ADR-012
|
||
/// §6.4's full protection set). Registration replaces nothing and
|
||
/// composes: each callback's set unions into the protection.
|
||
pub fn register_protect_callback(&self, callback: Arc<dyn ProtectCallback>) -> Registration {
|
||
self.gc.register_protect(callback)
|
||
}
|
||
|
||
/// Deregister a protect callback previously registered on this
|
||
/// store. `false` when the token was already consumed or never
|
||
/// came from [`Store::register_protect_callback`].
|
||
pub fn unregister_protect_callback(&self, registration: Registration) -> bool {
|
||
self.gc.unregister_protect(registration)
|
||
}
|
||
|
||
/// **Direct `delete(key)`** — fall-through consult (kv first, then
|
||
/// large; ADR-012 §6.4's "arbitration-then-delete where the entry
|
||
/// lives"), then the full protection set arbitration at delete
|
||
/// time: a key that is pinned, re-observed live via a registered
|
||
/// source, or named by a protect callback is refused with
|
||
/// [`Error::GcRefuse`] — nothing deleted. Unprotected keys delete
|
||
/// (across both homes when a degenerate pool holds the key twice);
|
||
/// absent keys carry [`Error::Missing`]; malformed key bytes
|
||
/// [`Error::KeyInvalid`]. A failed protect callback is refused
|
||
/// conservatively (`GcRefuse`, nothing deleted) — the direct-path
|
||
/// reading of abort-as-data's safe side, recorded in the facade
|
||
/// module doc.
|
||
///
|
||
/// Outside-of-sweep deletion is unusual by posture — most deletion
|
||
/// flows through sweeps (ADR-005); an embedder's correction flow
|
||
/// deregisters/re-registers around the delete (ADR-012 §6.4).
|
||
pub async fn delete(&self, key: &[u8]) -> Result<(), Error> {
|
||
let key = boundary_key(key)?;
|
||
let bytes = key.as_bytes();
|
||
|
||
let present = if self.kv().has(&bytes).await? {
|
||
true
|
||
} else {
|
||
match self.large() {
|
||
Some(large) => large.has(&bytes).await?,
|
||
None => false,
|
||
}
|
||
};
|
||
if !present {
|
||
return Err(Error::Missing);
|
||
}
|
||
|
||
match self.arbitrated_delete(&key, true).await? {
|
||
crate::gc::DeleteOutcome::Cancelled => Err(Error::GcRefuse),
|
||
crate::gc::DeleteOutcome::Deleted(true) => Ok(()),
|
||
// The entry vanished between the existence probe and the
|
||
// window: some other path resolved it after the probe.
|
||
crate::gc::DeleteOutcome::Deleted(false) => Err(Error::Missing),
|
||
}
|
||
}
|
||
|
||
/// **`sweep()`** — the explicit mark-and-sweep with the delete
|
||
/// window (ADR-005 §Sweep), in two forms under one signature: the
|
||
/// **durable form** runs on the delete window's SQL-backed host
|
||
/// (ADR-012 §5's one-window-host rule — the SQL-backed kv engine
|
||
/// when it is SQL-backed: sqlite and postgres; the pg-lo large
|
||
/// host otherwise: kv = mem + large = pg-lo; the branch keys off
|
||
/// the engine representation, never a flag), and the
|
||
/// **inline form** (ADR-012 §5: `local`/`mem` stage candidates
|
||
/// in-process and execute the window protocol inline in the same
|
||
/// sweep) runs verbatim elsewhere. Sweeps are never errors:
|
||
/// every abort is report data in `SweepReport.aborted`
|
||
/// (`Ok(SweepReport)`), and a completed sweep's counts are exact
|
||
/// relative to its form's schedule — the inline form reports
|
||
/// `candidates_staged == deleted + cancelled_by_arbitration` over its
|
||
/// same-sweep staged set; the durable form reports
|
||
/// `candidates_staged` as the candidates staged *this* sweep and
|
||
/// `deleted`/`cancelled` as the *overdue* set executed this sweep
|
||
/// (staged and executed are different sets there — the delay
|
||
/// separates them, ADR-012 §6.7; `deleted`/`cancelled` are both 0 on
|
||
/// any abort).
|
||
///
|
||
/// The protocol, as run here:
|
||
///
|
||
/// 1. **Single-flight**: the in-process sweep lease (ADR-013 §2's
|
||
/// single-node analog); a contended second sweeper aborts with
|
||
/// [`GcAbortCause::SweeperLock`], nothing deleted, nothing
|
||
/// staged or executed.
|
||
/// 2. **Zero registered liveness sources** aborts with
|
||
/// [`GcAbortCause::NoLivenessSources`] before anything is marked —
|
||
/// the safe default (nothing deleted).
|
||
/// 3. **Protect**: each registered [`ProtectCallback`] is consulted
|
||
/// pre-sweep; its keys union into the live set; a callback that
|
||
/// fails to answer aborts with [`GcAbortCause::ProtectFailed`] —
|
||
/// a flaky protection source skips the sweep rather than risk
|
||
/// deletion (iroh's `ProtectOutcome::Abort` conclusion). On the
|
||
/// durable form this consult's coverage extends to the executor
|
||
/// phase, so an abort here precedes any overdue execution too.
|
||
/// 4. **Durable form (SQL-backed kv engine)**, per sweep under the
|
||
/// lease: **execute overdue** first (window_take re-arbitrates
|
||
/// each overdue candidate at execution time — pin-before-publish
|
||
/// during the window delay cancels the candidate — and deletes in
|
||
/// batches), then **mark** (unchanged), then **stage** the fresh
|
||
/// candidate-dead set at `visible_after = mark_end + window` (the
|
||
/// constructor-tunable delay, ADR-008 §2). The executor of the
|
||
/// staged rows is the next sweep (ADR-008 §2; ADR-012 §6.7).
|
||
/// **Inline form**: the mark phase enumerates the whole pool
|
||
/// (both tiers' complete `list()`, ADR-003 — a malformed pool
|
||
/// list is fatal `Err(KeyInvalid)`); the live set computed from
|
||
/// all three sources (registered sources, pins, protect
|
||
/// callbacks); the candidate-dead set staged (sorted, in-process).
|
||
/// 5. **Delete window**: per candidate, **arbitration at delete
|
||
/// time** (not once upfront) under the arbitration lock — pinned
|
||
/// or re-observed-live cancels the candidate (counted in
|
||
/// `cancelled_by_arbitration`); otherwise the candidate deletes
|
||
/// (counted in `deleted`), in ~100-key batches.
|
||
///
|
||
/// **The test-asserted property** (ADR-005): a visible pool entry is
|
||
/// never deleted while liveness protects it, and an in-flight put
|
||
/// is never deleted by a sweep started before it committed — the
|
||
/// in-flight delete markers plus the blocked pin mint are the
|
||
/// mechanism ([`crate::gc`]'s module doc), and the concurrency
|
||
/// tests below assert it (the durable form's race holds *across the
|
||
/// window delay* — a put landing during the delay pins in-process
|
||
/// and the execution-time re-arbitration cancels the candidate).
|
||
pub async fn sweep(&self) -> Result<SweepReport, Error> {
|
||
let started = SystemTime::now();
|
||
|
||
if self.gc.registered_source_count() == 0 {
|
||
return Ok(Self::sweep_report(
|
||
0,
|
||
0,
|
||
0,
|
||
Some(GcAbortCause::NoLivenessSources),
|
||
started,
|
||
));
|
||
}
|
||
|
||
if !self.gc.try_begin_sweep() {
|
||
return Ok(Self::sweep_report(
|
||
0,
|
||
0,
|
||
0,
|
||
Some(GcAbortCause::SweeperLock),
|
||
started,
|
||
));
|
||
}
|
||
|
||
let result = self.sweep_window(started).await;
|
||
self.gc.end_sweep();
|
||
result
|
||
}
|
||
|
||
/// The sweep body, under the sweep lease: the protect consult
|
||
/// (whose abort shape is [`GcAbortCause::ProtectFailed`] — and whose
|
||
/// coverage extends to the durable form's execution phase), then
|
||
/// the form branch on the delete window's host — the engine
|
||
/// representation itself (ADR-012 §5's one-window-host rule),
|
||
/// never a flag: the **durable form** on a SQL-backed kv engine
|
||
/// (its generic host arm shared by every `EngineWindowState`
|
||
/// engine) and on the pg-lo large host when the kv engine is mem
|
||
/// (the mem arm's large-host completion), the **inline form**
|
||
/// verbatim on mem/local otherwise. Operational failures (engine
|
||
/// I/O, a malformed pool or window list) are the `Err` half of the
|
||
/// return shape (ADR-012 §1); every deliberate no-op is report data
|
||
/// instead.
|
||
async fn sweep_window(&self, started: SystemTime) -> Result<SweepReport, Error> {
|
||
let mut live: BTreeSet<Key> = BTreeSet::new();
|
||
|
||
for callback in self.gc.registered_protects() {
|
||
match callback.protect() {
|
||
Some(keys) => {
|
||
live.extend(keys);
|
||
}
|
||
None => {
|
||
return Ok(Self::sweep_report(
|
||
0,
|
||
0,
|
||
0,
|
||
Some(GcAbortCause::ProtectFailed),
|
||
started,
|
||
));
|
||
}
|
||
}
|
||
}
|
||
|
||
match &self.kv {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => match self.large() {
|
||
#[cfg(feature = "pg-lo")]
|
||
Some(LargeEngine::PgLo(host)) => {
|
||
self.sweep_window_durable(host, live, started).await
|
||
}
|
||
_ => self.sweep_window_inline(live, started).await,
|
||
},
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => self.sweep_window_durable(engine, live, started).await,
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => self.sweep_window_durable(engine, live, started).await,
|
||
}
|
||
}
|
||
|
||
/// The durable form's body, shared by every window-host arm of
|
||
/// the form branch (the branch keys off the engine
|
||
/// representation, ADR-012 §5's one-window-host rule — the host
|
||
/// this runs on is the SQL-backed engine the window lives on: the
|
||
/// kv engine's SQL-backed arms and pg-lo's large-host arm). The
|
||
/// overdue staged rows execute first — each re-arbitrated at
|
||
/// execution time — then this sweep's own mark and stage at
|
||
/// `visible_after = mark_end + window`.
|
||
async fn sweep_window_durable<W: EngineWindowState>(
|
||
&self,
|
||
host: &W,
|
||
live: BTreeSet<Key>,
|
||
started: SystemTime,
|
||
) -> Result<SweepReport, Error> {
|
||
let (deleted, cancelled_by_arbitration) = self.execute_overdue_window(host).await?;
|
||
let candidates_staged = self.mark_and_stage_window(host, live).await?;
|
||
Ok(Self::sweep_report(
|
||
candidates_staged,
|
||
deleted,
|
||
cancelled_by_arbitration,
|
||
None,
|
||
started,
|
||
))
|
||
}
|
||
|
||
/// The inline form (ADR-012 §5): the mark → arbitrate-at-delete-
|
||
/// time → batched commit body, unchanged — the non-SQL engines
|
||
/// stage candidates in-process and execute the window protocol
|
||
/// inline in the same sweep. Preserved verbatim across the durable
|
||
/// form's arrival.
|
||
async fn sweep_window_inline(
|
||
&self,
|
||
mut live: BTreeSet<Key>,
|
||
started: SystemTime,
|
||
) -> Result<SweepReport, Error> {
|
||
let mut pool = typed_keys(self.kv().list().await?)?;
|
||
if let Some(large) = self.large() {
|
||
pool.extend(typed_keys(large.list().await?)?);
|
||
}
|
||
live.extend(self.gc.pinned_keys());
|
||
for source in self.gc.registered_sources() {
|
||
live.extend(source.live_keys());
|
||
}
|
||
|
||
let candidates: Vec<Key> = {
|
||
let enumerated: BTreeSet<Key> = pool.into_iter().collect();
|
||
let mut candidates: Vec<Key> = enumerated.difference(&live).cloned().collect();
|
||
candidates.sort();
|
||
candidates
|
||
};
|
||
let candidates_staged = candidates.len() as u64;
|
||
|
||
let mut deleted = 0u64;
|
||
let mut cancelled_by_arbitration = 0u64;
|
||
for chunk in candidates.chunks(SWEEP_DELETE_BATCH) {
|
||
for key in chunk {
|
||
match self.arbitrated_delete(key, false).await? {
|
||
// A candidate already resolved elsewhere (deleted
|
||
// between mark and its window turn) settles here
|
||
// too: the completed sweep's counts stay exact over
|
||
// the staged set.
|
||
crate::gc::DeleteOutcome::Deleted(true) => deleted += 1,
|
||
_ => cancelled_by_arbitration += 1,
|
||
}
|
||
}
|
||
}
|
||
|
||
Ok(Self::sweep_report(
|
||
candidates_staged,
|
||
deleted,
|
||
cancelled_by_arbitration,
|
||
None,
|
||
started,
|
||
))
|
||
}
|
||
|
||
/// **The durable form's executor step** (ADR-012 §6.7): take the
|
||
/// overdue staged candidates (`visible_after <= now`) off the
|
||
/// window and execute each through the same
|
||
/// [`Store::arbitrated_delete`] path — the re-arbitration happens
|
||
/// at execution time, so a put landing (pin-before-publish) during
|
||
/// the window delay cancels its candidate here — in ~100-key
|
||
/// batches. The consult posture on this path is the sweep's protect
|
||
/// consult (recorded named design point): it already ran for this
|
||
/// sweep, above, and its coverage extends to the executor phase —
|
||
/// so these arbitrations carry `consult_protect = false`, the same
|
||
/// posture as the inline form's delete-time arbitration. A vanished
|
||
/// candidate (deleted during the delay by a direct delete) settles
|
||
/// into the cancelled count.
|
||
async fn execute_overdue_window<W: EngineWindowState>(
|
||
&self,
|
||
host: &W,
|
||
) -> Result<(u64, u64), Error> {
|
||
let overdue: Vec<Key> = typed_keys(host.window_take(SystemTime::now()).await?)?;
|
||
let mut deleted = 0u64;
|
||
let mut cancelled_by_arbitration = 0u64;
|
||
for chunk in overdue.chunks(SWEEP_DELETE_BATCH) {
|
||
for key in chunk {
|
||
match self.arbitrated_delete(key, false).await? {
|
||
crate::gc::DeleteOutcome::Deleted(true) => deleted += 1,
|
||
_ => cancelled_by_arbitration += 1,
|
||
}
|
||
}
|
||
}
|
||
Ok((deleted, cancelled_by_arbitration))
|
||
}
|
||
|
||
/// **The durable form's mark + stage step**: the mark phase exactly
|
||
/// the inline form's (pool enumerated, live set from pins/registered
|
||
/// sources plus the protect consult's keys, candidate-dead set
|
||
/// computed), then the candidate-dead set staged into the window at
|
||
/// `visible_after = mark_end + window` — the constructor-tunable
|
||
/// delay (ADR-008 §2; the store carries the value its constructor
|
||
/// took), so execution waits at earliest for the next sweep past
|
||
/// the deadline. Returns the staged-candidate count.
|
||
async fn mark_and_stage_window<W: EngineWindowState>(
|
||
&self,
|
||
host: &W,
|
||
mut live: BTreeSet<Key>,
|
||
) -> Result<u64, Error> {
|
||
let mut pool = typed_keys(self.kv().list().await?)?;
|
||
if let Some(large) = self.large() {
|
||
pool.extend(typed_keys(large.list().await?)?);
|
||
}
|
||
live.extend(self.gc.pinned_keys());
|
||
for source in self.gc.registered_sources() {
|
||
live.extend(source.live_keys());
|
||
}
|
||
|
||
let enumerated: BTreeSet<Key> = pool.into_iter().collect();
|
||
let mut candidates: Vec<Key> = enumerated.difference(&live).cloned().collect();
|
||
candidates.sort();
|
||
|
||
let mark_end = SystemTime::now();
|
||
let visible_after = mark_end.checked_add(self.window_delay).ok_or_else(|| {
|
||
Error::Io("the delete-window delay overflows the system clock".to_owned())
|
||
})?;
|
||
let staged: Vec<WindowCandidate> = candidates
|
||
.iter()
|
||
.map(|key| WindowCandidate {
|
||
key: key.as_bytes().to_vec(),
|
||
visible_after,
|
||
})
|
||
.collect();
|
||
host.window_stage(&staged).await?;
|
||
Ok(candidates.len() as u64)
|
||
}
|
||
|
||
/// A finished report constructor — every sweep outcome is a
|
||
/// completed record (ADR-012 §1: the report is data, not an error
|
||
/// carrier).
|
||
#[allow(clippy::too_many_arguments)]
|
||
fn sweep_report(
|
||
candidates_staged: u64,
|
||
deleted: u64,
|
||
cancelled_by_arbitration: u64,
|
||
aborted: Option<GcAbortCause>,
|
||
started: SystemTime,
|
||
) -> SweepReport {
|
||
SweepReport {
|
||
candidates_staged,
|
||
deleted,
|
||
cancelled_by_arbitration,
|
||
aborted,
|
||
started_at: started,
|
||
finished_at: SystemTime::now(),
|
||
}
|
||
}
|
||
|
||
/// Arbitrate, then delete where the entry lives, with the
|
||
/// in-flight delete marker held across the delete and released on
|
||
/// every path (the store-side composition of [`GcHome::arbitrate`]
|
||
/// + [`GcHome::conclude_delete`]).
|
||
async fn arbitrated_delete(
|
||
&self,
|
||
key: &Key,
|
||
consult_protect: bool,
|
||
) -> Result<crate::gc::DeleteOutcome, Error> {
|
||
match self.gc.arbitrate(key, consult_protect).await? {
|
||
Arbitration::Cancelled => Ok(crate::gc::DeleteOutcome::Cancelled),
|
||
Arbitration::Proceed => {
|
||
let result = self.delete_everywhere(key).await;
|
||
self.gc.conclude_delete(key);
|
||
Ok(crate::gc::DeleteOutcome::Deleted(result?))
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Delete the key's pool entry wherever it physically lives — the
|
||
/// kv home and the large home (normal pools hold exactly one;
|
||
/// immutability means both homes of a degenerate duplicate are the
|
||
/// same dead content). `false` when the entry was already absent:
|
||
/// per-home `Missing` is not an operational error here, it is the
|
||
/// presence answer.
|
||
async fn delete_everywhere(&self, key: &Key) -> Result<bool, Error> {
|
||
let bytes = key.as_bytes();
|
||
let mut deleted = match self.kv().delete(&bytes).await {
|
||
Ok(()) => true,
|
||
Err(Error::Missing) => false,
|
||
Err(err) => return Err(err),
|
||
};
|
||
if let Some(large) = self.large() {
|
||
match large.delete(&bytes).await {
|
||
Ok(()) => deleted = true,
|
||
Err(Error::Missing) => {}
|
||
Err(err) => return Err(err),
|
||
}
|
||
}
|
||
Ok(deleted)
|
||
}
|
||
}
|
||
|
||
#[cfg(all(test, feature = "mem"))]
|
||
mod tests {
|
||
use super::*;
|
||
use crate::backend::{slice_digest, Backend};
|
||
use crate::dispatch::DEFAULT_DISPATCH_THRESHOLD;
|
||
use crate::hashing::Algorithm;
|
||
use std::fs;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use std::time::{SystemTime, UNIX_EPOCH};
|
||
|
||
/// A fresh temp root per test (mirrors `local.rs`'s helper); Drop
|
||
/// removes the whole tree idempotently.
|
||
struct TempRoot(PathBuf);
|
||
|
||
fn temp_root(tag: &'static str) -> TempRoot {
|
||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let nanos = SystemTime::now()
|
||
.duration_since(UNIX_EPOCH)
|
||
.map(|d| d.as_nanos())
|
||
.unwrap_or(0);
|
||
let dir = std::env::temp_dir().join(format!(
|
||
"alkblobs-store-{tag}-{}-{:x}-{seq}",
|
||
std::process::id(),
|
||
nanos
|
||
));
|
||
fs::create_dir_all(&dir).expect("temp root created");
|
||
TempRoot(dir)
|
||
}
|
||
|
||
impl TempRoot {
|
||
fn path(&self) -> &Path {
|
||
&self.0
|
||
}
|
||
}
|
||
|
||
impl Drop for TempRoot {
|
||
fn drop(&mut self) {
|
||
let _ = fs::remove_dir_all(&self.0);
|
||
}
|
||
}
|
||
|
||
/// mem-only constructs and validates: the ephemeral mode's shape —
|
||
/// no large tier, no threshold in effect, kv routing for every
|
||
/// length (ADR-011 §4).
|
||
#[cfg(feature = "mem")]
|
||
#[test]
|
||
fn mem_only_constructs_and_validates() {
|
||
let store = Store::mem_only();
|
||
assert_eq!(store.mode(), MemOnly);
|
||
assert_eq!(store.threshold(), None, "no dispatch threshold in effect");
|
||
assert!(store.large().is_none(), "mem-only has no large tier");
|
||
|
||
for len in [0u64, 1, DEFAULT_DISPATCH_THRESHOLD + 1, u64::MAX] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
Tier::Kv,
|
||
"mem-only routes every length to its sole tier"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// kv-only constructs and validates: no large tier; the threshold
|
||
/// is the *rejection point's* carrier (ADR-012 §6.2), not a routing
|
||
/// answer — every length routes kv there, and the put seam rejects
|
||
/// known-length over-threshold puts immediately (its behavior).
|
||
#[cfg(feature = "mem")]
|
||
#[test]
|
||
fn kv_only_mem_constructs_and_validates() {
|
||
let store = Store::kv_only_mem(1024);
|
||
assert_eq!(store.mode(), KvOnly);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert!(store.large().is_none(), "kv-only has no large tier");
|
||
|
||
for len in [0u64, 1023, 1024, 1025, u64::MAX] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
Tier::Kv,
|
||
"kv-only routes every length to its sole tier"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The ADR-012 §6.3 dispatch-coverage shape: `kv = mem` +
|
||
/// `large = local` dual-tier construction works, both tiers
|
||
/// load-bearing, dispatch threshold active — boundary values route
|
||
/// deterministically (at/at-below threshold kv, above large), and
|
||
/// tunability is proven against a non-default threshold.
|
||
#[cfg(feature = "mem")]
|
||
#[tokio::test]
|
||
async fn dual_tier_mem_local_constructs_and_routes() {
|
||
let root = temp_root("dual-tier");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier mem+local constructs");
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"length {len} routes deterministically"
|
||
);
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"same length ⇒ same tier (the purity invariant, re-checked)"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The shipped default threshold is the constructor-tunable
|
||
/// baseline: dual-tier at `DEFAULT_DISPATCH_THRESHOLD` routes at
|
||
/// the 128 KiB boundary exactly as the pure function does.
|
||
#[cfg(feature = "mem")]
|
||
#[tokio::test]
|
||
async fn dual_tier_routes_at_the_default_threshold() {
|
||
let root = temp_root("dual-tier-default");
|
||
let store = Store::dual_tier_mem_local(root.path(), DEFAULT_DISPATCH_THRESHOLD)
|
||
.await
|
||
.expect("dual-tier constructs at the default threshold");
|
||
|
||
for (len, tier) in [
|
||
(131_071, Tier::Kv),
|
||
(131_072, Tier::Kv),
|
||
(131_073, Tier::Large),
|
||
] {
|
||
assert_eq!(store.tier_for(len), tier, "length {len} at 128 KiB");
|
||
}
|
||
}
|
||
|
||
/// The GC state home rides as a field of the store instance
|
||
/// (ADR-010 §2 — the store instance owns GC state) and is
|
||
/// reachable crate-internally: the gc-protocol task's machinery
|
||
/// lands against this handle.
|
||
#[cfg(feature = "mem")]
|
||
#[test]
|
||
fn gc_state_home_rides_on_the_foundation() {
|
||
let store = Store::mem_only();
|
||
let _ = store.gc();
|
||
}
|
||
|
||
/// Entry planters straight into the tier engines (bypassing the
|
||
/// put seam, which is a later task): opaque key bytes + value,
|
||
/// exactly what the engines' CAS tables accept. Planting
|
||
/// mismatching content under a key is how the read path's
|
||
/// verification postures are observed.
|
||
async fn plant_kv(store: &Store, key: &[u8], value: &[u8]) {
|
||
match store.kv() {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(engine) => engine.put(key, value).await.expect("kv plant"),
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine.put(key, value).await.expect("kv plant"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine.put(key, value).await.expect("kv plant"),
|
||
}
|
||
}
|
||
|
||
async fn plant_large(store: &Store, key: &[u8], value: &[u8]) {
|
||
match store.large() {
|
||
Some(LargeEngine::Local(engine)) => engine.put(key, value).await.expect("large plant"),
|
||
#[cfg(feature = "pg-lo")]
|
||
Some(LargeEngine::PgLo(_)) => panic!("these tests construct local-tier stores"),
|
||
None => panic!("planting into a mode with no large tier"),
|
||
}
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
fn sha1_key_for(content: &[u8]) -> Key {
|
||
Key::from(Algorithm::GitBlobSha1.derive(content, content.len() as u64))
|
||
}
|
||
|
||
/// The both-tier-miss shapes every read op must resolve to
|
||
/// (store-api.md §Error model; ADR-012 §6.5 ruling 5) — pinned on
|
||
/// all three constructible modes, virgin-store reads as no-ops.
|
||
async fn assert_virgin_read_shapes(store: &Store) {
|
||
let key = key_for(b"nothing ever landed").as_bytes();
|
||
|
||
assert!(
|
||
store.get(&key).await.unwrap().is_none(),
|
||
"get's both-tier miss is Ok(None)"
|
||
);
|
||
assert_eq!(store.stat(&key).await.unwrap(), None, "stat misses as None");
|
||
assert_eq!(
|
||
store.read_range(&key, 0, 16).await.unwrap_err(),
|
||
Error::Missing,
|
||
"read_range's both-tier miss is Err(Missing) — the pinned shape difference vs get"
|
||
);
|
||
assert!(
|
||
!store.has(&key).await.unwrap(),
|
||
"has's both-tier miss is false"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"virgin-store list is empty"
|
||
);
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn virgin_store_reads_are_no_ops_dual_tier() {
|
||
let root = temp_root("read-virgin-dual");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
assert_virgin_read_shapes(&store).await;
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn virgin_store_reads_are_no_ops_mem_only() {
|
||
let store = Store::mem_only();
|
||
assert_virgin_read_shapes(&store).await;
|
||
}
|
||
|
||
#[tokio::test]
|
||
async fn virgin_store_reads_are_no_ops_kv_only() {
|
||
let store = Store::kv_only_mem(64);
|
||
assert_virgin_read_shapes(&store).await;
|
||
}
|
||
|
||
/// `get` returns the pinned `(len, stream)` shape from the kv tier:
|
||
/// the tuple carries the metadata length, the `Blob` serves
|
||
/// bounded windows (clamped at EOF) and the ADR-006 range seam.
|
||
#[tokio::test]
|
||
async fn get_returns_the_pinned_len_stream_shape_from_kv() {
|
||
let store = Store::mem_only();
|
||
let content = b"small-tier content";
|
||
let key = key_for(content);
|
||
plant_kv(&store, &key.as_bytes(), content).await;
|
||
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(len, content.len() as u64, "tuple len is the entry length");
|
||
assert_eq!(blob.len(), len);
|
||
assert!(!blob.is_empty());
|
||
|
||
let whole = blob.read_at(0, len).await.expect("full window");
|
||
assert_eq!(whole, content, "full window matches");
|
||
assert_eq!(
|
||
blob.read_at(0, len + 17).await.expect("over-read"),
|
||
content,
|
||
"over-read clamps at the entry's end"
|
||
);
|
||
|
||
let (slice, digest) = blob.read_range(6, 6).await.expect("ranged window");
|
||
assert_eq!(slice, &content[6..12]);
|
||
assert_eq!(
|
||
digest,
|
||
slice_digest(&content[6..12]),
|
||
"slice digest is SHA-256 over exactly the returned bytes"
|
||
);
|
||
|
||
let (full, full_digest) = blob.read_range(0, len).await.expect("full range");
|
||
assert_eq!(full, content);
|
||
assert_eq!(full_digest, slice_digest(content));
|
||
}
|
||
|
||
/// The zero-length degenerate: a kv-tier empty entry get's as
|
||
/// `Some((0, blob))`, its windows read empty-never-error, and
|
||
/// `stat` carries exact zero.
|
||
#[tokio::test]
|
||
async fn zero_length_kv_entry_reads_as_empty_blob() {
|
||
let store = Store::mem_only();
|
||
let key = key_for(b"");
|
||
plant_kv(&store, &key.as_bytes(), b"").await;
|
||
|
||
let (len, blob) = store.get(&key.as_bytes()).await.unwrap().expect("present");
|
||
assert_eq!(len, 0);
|
||
assert!(blob.is_empty());
|
||
assert_eq!(blob.read_at(0, 8).await.unwrap(), Vec::<u8>::new());
|
||
|
||
assert_eq!(
|
||
store.stat(&key.as_bytes()).await.unwrap(),
|
||
Some(EntryMeta {
|
||
len: 0,
|
||
algorithm: key.algorithm()
|
||
})
|
||
);
|
||
}
|
||
|
||
/// `get`/`stat`/`read_range`/`has` fall through to the large tier
|
||
/// on a kv miss — consult order, not length routing: a
|
||
/// sub-threshold entry planted only in the large tier is still
|
||
/// found even though `tier_for` routes its length to kv.
|
||
#[tokio::test]
|
||
async fn read_ops_fall_through_kv_miss_to_large() {
|
||
let root = temp_root("read-fall-through");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content = b"a sub-threshold entry planted directly in the large tier";
|
||
let key = key_for(content);
|
||
|
||
assert_eq!(
|
||
store.tier_for(content.len() as u64),
|
||
Tier::Kv,
|
||
"the length *routes* kv — the read still lands via the large tier"
|
||
);
|
||
plant_large(&store, &key.as_bytes(), content).await;
|
||
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("fall-through get succeeds")
|
||
.expect("large-tier entry found via fall-through");
|
||
assert_eq!(len, content.len() as u64);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), content);
|
||
|
||
assert_eq!(
|
||
store.stat(&key.as_bytes()).await.unwrap(),
|
||
Some(EntryMeta {
|
||
len: content.len() as u64,
|
||
algorithm: key.algorithm()
|
||
}),
|
||
"stat falls through too"
|
||
);
|
||
assert!(
|
||
store.has(&key.as_bytes()).await.unwrap(),
|
||
"has falls through"
|
||
);
|
||
|
||
let (slice, digest) = store
|
||
.read_range(&key.as_bytes(), 4, 8)
|
||
.await
|
||
.expect("read_range falls through");
|
||
assert_eq!(slice, &content[4..12]);
|
||
assert_eq!(digest, slice_digest(&slice));
|
||
}
|
||
|
||
/// The kv tier is the *first* consult on every key-addressed op:
|
||
/// the same key bytes planted with different (mismatching) content
|
||
/// in both tiers — get/read_range/stat/has must all answer from
|
||
/// the kv tier's entry.
|
||
#[tokio::test]
|
||
async fn kv_tier_is_the_first_consult_on_every_key_addressed_op() {
|
||
let root = temp_root("read-kv-first");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
const KV_CONTENT: &[u8] = b"the kv tier's entry";
|
||
let key = key_for(KV_CONTENT);
|
||
plant_kv(&store, &key.as_bytes(), KV_CONTENT).await;
|
||
plant_large(&store, &key.as_bytes(), b"decoy content in the large tier").await;
|
||
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("kv hit wins");
|
||
assert_eq!(
|
||
blob.read_at(0, len).await.unwrap(),
|
||
KV_CONTENT,
|
||
"kv tier first — and its content verifies against the key"
|
||
);
|
||
|
||
let (slice, _) = store
|
||
.read_range(&key.as_bytes(), 0, 4)
|
||
.await
|
||
.expect("ranged");
|
||
assert_eq!(
|
||
slice,
|
||
&KV_CONTENT[0..4],
|
||
"read_range also consults kv first"
|
||
);
|
||
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
assert_eq!(
|
||
store.stat(&key.as_bytes()).await.unwrap(),
|
||
Some(EntryMeta {
|
||
len: KV_CONTENT.len() as u64,
|
||
algorithm: key.algorithm()
|
||
})
|
||
);
|
||
}
|
||
|
||
/// Verification-on-get, kv tier (ADR-006 decision 1): a
|
||
/// materialized get hash-checks against the key; mismatch is
|
||
/// `Err(Verification)` and never falls through further. `read_range`
|
||
/// by contrast carries no canonical check — the slice digest is its
|
||
/// whole verification story (ADR-006 decision 2).
|
||
#[tokio::test]
|
||
async fn kv_get_whole_blob_verifies_and_never_falls_through_past_mismatch() {
|
||
let store = Store::mem_only();
|
||
let key = key_for(b"the derived address");
|
||
let bytes = key.as_bytes();
|
||
plant_kv(&store, &bytes, b"content that is not the key").await;
|
||
|
||
assert_eq!(
|
||
store.get(&bytes).await.unwrap_err(),
|
||
Error::Verification,
|
||
"kv materialized reads hash-check at get time"
|
||
);
|
||
assert_eq!(
|
||
store.read_range(&bytes, 0, 8).await.unwrap(),
|
||
(b"content ".to_vec(), slice_digest(b"content ")),
|
||
"read_range carries no canonical check — the slice-digest posture"
|
||
);
|
||
assert!(
|
||
store.stat(&bytes).await.unwrap().is_some(),
|
||
"stat is a metadata probe, not a verifier"
|
||
);
|
||
}
|
||
|
||
/// Verification-on-get, large tier (the recorded posture's other
|
||
/// half): a streaming get opens metadata + handle only — it does
|
||
/// not re-derive the canonical digest at open. The tier's
|
||
/// integrity story is the put path's whole-blob verification pass
|
||
/// plus the per-range slice digests (ADR-006 decisions 1–2).
|
||
#[tokio::test]
|
||
async fn large_streaming_get_does_not_re_derive_at_open() {
|
||
let root = temp_root("read-large-no-reverify");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let key = key_for(b"the derived address");
|
||
plant_large(&store, &key.as_bytes(), b"stored bytes").await;
|
||
|
||
assert!(
|
||
store.get(&key.as_bytes()).await.unwrap().is_some(),
|
||
"streaming open never re-derives (mismatching content and all)"
|
||
);
|
||
}
|
||
|
||
/// `stat` carries len + algorithm for both algorithms (the
|
||
/// "type" at this layer is the hash algorithm, ADR-002) and is a
|
||
/// `size` ride — the fall-through consult, `None` when absent.
|
||
#[tokio::test]
|
||
async fn stat_carries_len_and_algorithm_from_size_probes() {
|
||
let root = temp_root("read-stat");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
|
||
let sha256_key = key_for(b"canonical");
|
||
plant_kv(&store, &sha256_key.as_bytes(), b"canonical").await;
|
||
let legacy_key = sha1_key_for(b"legacy");
|
||
plant_large(&store, &legacy_key.as_bytes(), b"legacy").await;
|
||
|
||
assert_eq!(
|
||
store.stat(&sha256_key.as_bytes()).await.unwrap(),
|
||
Some(EntryMeta {
|
||
len: 9,
|
||
algorithm: Algorithm::GitBlobSha256
|
||
})
|
||
);
|
||
assert_eq!(
|
||
store
|
||
.stat(&legacy_key.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("legacy entry present")
|
||
.algorithm,
|
||
Algorithm::GitBlobSha1,
|
||
"the algorithm comes from the key"
|
||
);
|
||
assert_eq!(
|
||
store.stat(&sha256_key.as_bytes()).await.unwrap(),
|
||
store.stat(&sha256_key.as_bytes()).await.unwrap(),
|
||
"stat is a pure probe"
|
||
);
|
||
}
|
||
|
||
/// `list` returns the sorted, typed union of both tiers'
|
||
/// enumerations — both algorithms, dedup-free by key identity.
|
||
#[tokio::test]
|
||
async fn list_returns_the_sorted_typed_union_of_both_tiers() {
|
||
let root = temp_root("read-list");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
|
||
let kv_keys = [
|
||
key_for(b"kv entry one"),
|
||
sha1_key_for(b"kv legacy"),
|
||
key_for(b"kv entry two"),
|
||
];
|
||
for key in &kv_keys {
|
||
plant_kv(&store, &key.as_bytes(), b"kv bytes").await;
|
||
}
|
||
let large_keys = [key_for(b"large entry one"), key_for(b"large entry two")];
|
||
for key in &large_keys {
|
||
plant_large(&store, &key.as_bytes(), b"large bytes").await;
|
||
}
|
||
|
||
let mut expected: Vec<Key> = kv_keys.into_iter().chain(large_keys).collect();
|
||
expected.sort();
|
||
|
||
let listed = store.list().await.expect("list succeeds");
|
||
assert_eq!(listed, expected, "complete, typed, both tiers");
|
||
for window in listed.windows(2) {
|
||
assert!(
|
||
window[0] < window[1],
|
||
"output is sorted in the keys' total order"
|
||
);
|
||
}
|
||
for key in &listed {
|
||
assert!(
|
||
store.has(&key.as_bytes()).await.unwrap(),
|
||
"every listed key addresses a real entry"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// A malformed pool list is fatal (`Error::KeyInvalid`), never a
|
||
/// silent skip — ADR-003's malformed-list ruling applied at the
|
||
/// facade boundary; the engines own their own body-level checks.
|
||
#[tokio::test]
|
||
async fn list_of_malformed_pool_keys_is_fatal() {
|
||
let store = Store::mem_only();
|
||
plant_kv(&store, &[0x07; 33], b"stray in the table").await;
|
||
|
||
assert_eq!(
|
||
store.list().await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"malformed pool bytes are fatal on list"
|
||
);
|
||
}
|
||
|
||
/// `KeyInvalid` at the facade boundary on every key-addressed
|
||
/// op (typed conversion happens there and only there, ADR-002/003)
|
||
/// — malformed bytes reject on hit paths (a planted entry) and
|
||
/// miss paths alike, before any tier is consulted and never as a
|
||
/// `None`/`false` guess.
|
||
#[tokio::test]
|
||
async fn malformed_key_bytes_reject_keyinvalid_on_every_op() {
|
||
let store = Store::mem_only();
|
||
let anchor = key_for(b"an anchor entry");
|
||
plant_kv(&store, &anchor.as_bytes(), b"anchor").await;
|
||
|
||
let key_bytes = anchor.as_bytes();
|
||
let malformed = [
|
||
Vec::new(), // empty
|
||
key_bytes[..key_bytes.len() - 1].to_vec(), // truncated digest
|
||
[0x00u8; 33].to_vec(), // reserved tag, full width
|
||
[0x03u8; 33].to_vec(), // unknown tag
|
||
[0x01u8; 21].to_vec(), // right tag, wrong digest length
|
||
];
|
||
for raw in &malformed {
|
||
assert_eq!(
|
||
store.get(raw).await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"get rejects malformed key bytes: {raw:?}"
|
||
);
|
||
assert_eq!(
|
||
store.stat(raw).await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"stat rejects malformed key bytes: {raw:?}"
|
||
);
|
||
assert_eq!(
|
||
store.read_range(raw, 0, 1).await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"read_range rejects malformed key bytes: {raw:?}"
|
||
);
|
||
assert_eq!(
|
||
store.has(raw).await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"has rejects malformed key bytes: {raw:?}"
|
||
);
|
||
}
|
||
}
|
||
|
||
// ------------------------------------------------------------------
|
||
// The GC protocol (the gc-protocol work): pin registry, registered
|
||
// liveness sources, protect callback, delete windows, direct-delete
|
||
// refusal — ADR-005/012/013 on the facade.
|
||
// ------------------------------------------------------------------
|
||
|
||
/// A set-backed liveness source — shared-handle semantics, the POC
|
||
/// finding-7 shape: the store consults the same set this test
|
||
/// mutates, so its answers are live-shared, never snapshots.
|
||
#[derive(Default)]
|
||
struct SetSource {
|
||
live: parking_lot::Mutex<BTreeSet<Key>>,
|
||
}
|
||
|
||
impl SetSource {
|
||
fn add(&self, key: Key) {
|
||
self.live.lock().insert(key);
|
||
}
|
||
|
||
fn remove(&self, key: &Key) {
|
||
self.live.lock().remove(key);
|
||
}
|
||
}
|
||
|
||
impl LivenessSource for SetSource {
|
||
fn live_keys(&self) -> Vec<Key> {
|
||
self.live.lock().iter().cloned().collect()
|
||
}
|
||
|
||
fn contains(&self, key: &Key) -> bool {
|
||
self.live.lock().contains(key)
|
||
}
|
||
}
|
||
|
||
/// A set-backed protect callback; `None` models the
|
||
/// failed-to-answer case (abort/refuse safe side).
|
||
#[derive(Default)]
|
||
struct SetProtect {
|
||
set: parking_lot::Mutex<Option<Vec<Key>>>,
|
||
}
|
||
|
||
impl SetProtect {
|
||
fn set_keys(&self, keys: Vec<Key>) {
|
||
*self.set.lock() = Some(keys);
|
||
}
|
||
|
||
fn fail(&self) {
|
||
*self.set.lock() = None;
|
||
}
|
||
}
|
||
|
||
impl ProtectCallback for SetProtect {
|
||
fn protect(&self) -> Option<Vec<Key>> {
|
||
self.set.lock().clone()
|
||
}
|
||
}
|
||
|
||
/// A protect callback that blocks mid-consult (the sweep lease's
|
||
/// contended-window fixture): it announces its entry on a channel,
|
||
/// waits on a second, then answers with an empty protection set.
|
||
struct LatchProtect {
|
||
entered: std::sync::mpsc::Sender<()>,
|
||
release: parking_lot::Mutex<Option<std::sync::mpsc::Receiver<()>>>,
|
||
}
|
||
|
||
impl ProtectCallback for LatchProtect {
|
||
fn protect(&self) -> Option<Vec<Key>> {
|
||
self.entered.send(()).ok();
|
||
let receiver = self.release.lock().take();
|
||
if let Some(receiver) = receiver {
|
||
let _ = receiver.recv();
|
||
}
|
||
Some(Vec::new())
|
||
}
|
||
}
|
||
|
||
/// `sweep()` with zero registered liveness sources is the safe
|
||
/// default: `Ok(SweepReport)` carrying `NoLivenessSources`, nothing
|
||
/// deleted, the pool intact (store-api.md §Lifecycle).
|
||
#[tokio::test]
|
||
async fn sweep_with_zero_sources_aborts_as_data_and_deletes_nothing() {
|
||
let store = Store::mem_only();
|
||
let key = key_for(b"unprotected content");
|
||
plant_kv(&store, &key.as_bytes(), b"unprotected content").await;
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, Some(GcAbortCause::NoLivenessSources));
|
||
assert_eq!(report.candidates_staged, 0);
|
||
assert_eq!(report.deleted, 0);
|
||
assert_eq!(report.cancelled_by_arbitration, 0);
|
||
assert!(
|
||
store.has(&key.as_bytes()).await.unwrap(),
|
||
"an aborted sweep deletes nothing"
|
||
);
|
||
}
|
||
|
||
/// With sources registered, an empty pool sweeps to a zero report
|
||
/// — no abort, exact zero counts.
|
||
#[tokio::test]
|
||
async fn sweep_with_sources_and_an_empty_pool_is_a_zero_report() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, None);
|
||
assert_eq!(report.candidates_staged, 0);
|
||
assert_eq!(report.deleted, 0);
|
||
assert_eq!(report.cancelled_by_arbitration, 0);
|
||
}
|
||
|
||
/// Exact counts on a mixed pool (store-api.md invariant 4): the
|
||
/// registered source's key and a pinned key mark live, both tiers'
|
||
/// dead entries stage and delete — `candidates_staged == deleted +
|
||
/// cancelled_by_arbitration`.
|
||
#[tokio::test]
|
||
async fn sweep_counts_are_exact_across_both_tiers() {
|
||
let root = temp_root("gc-exact-counts");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
|
||
let live = key_for(b"kept by the registered source");
|
||
plant_kv(&store, &live.as_bytes(), b"kept by the registered source").await;
|
||
source.add(live.clone());
|
||
|
||
let pinned = key_for(b"kept by the pin");
|
||
plant_kv(&store, &pinned.as_bytes(), b"kept by the pin").await;
|
||
let pin = store.gc().acquire(&pinned).await;
|
||
|
||
let kv_dead = key_for(b"dead kv entry");
|
||
plant_kv(&store, &kv_dead.as_bytes(), b"dead kv entry").await;
|
||
let large_dead = key_for(b"dead large entry with bulk");
|
||
plant_large(
|
||
&store,
|
||
&large_dead.as_bytes(),
|
||
b"dead large entry with bulk",
|
||
)
|
||
.await;
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, None);
|
||
assert_eq!(report.candidates_staged, 2);
|
||
assert_eq!(report.deleted, 2, "both tiers' dead entries deleted");
|
||
assert_eq!(report.cancelled_by_arbitration, 0);
|
||
assert_eq!(
|
||
report.candidates_staged,
|
||
report.deleted + report.cancelled_by_arbitration,
|
||
"the completed sweep's counts are exact over the staged set"
|
||
);
|
||
assert!(store.has(&live.as_bytes()).await.unwrap());
|
||
assert!(store.has(&pinned.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&kv_dead.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&large_dead.as_bytes()).await.unwrap());
|
||
|
||
drop(pin);
|
||
}
|
||
|
||
/// A dedup re-put's guard refcounts the existing entry: two pins
|
||
/// compose, each drop decrements, and only the last drop exposes
|
||
/// the entry to a sweep (ADR-005's dedup-refcount ruling).
|
||
#[tokio::test]
|
||
async fn dedup_reput_guards_refcount_under_the_arbitration_lock() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
let content = b"refcounted content";
|
||
let key = key_for(content);
|
||
plant_kv(&store, &key.as_bytes(), content).await;
|
||
|
||
let first = store.gc().acquire(&key).await;
|
||
let second = store.gc().acquire(&key).await;
|
||
assert_eq!(store.gc().pin_count(&key), 2, "dedup re-put refcounts");
|
||
drop(first);
|
||
assert_eq!(store.gc().pin_count(&key), 1);
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.candidates_staged, 0, "one refcount still protects");
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
|
||
drop(second);
|
||
assert_eq!(store.gc().pin_count(&key), 0, "zero removes the entry row");
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.candidates_staged, 1);
|
||
assert_eq!(report.deleted, 1);
|
||
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// Batch pins protect the whole group; batch drop releases every
|
||
/// member in one ordered step — no early single-pin release
|
||
/// (ADR-005 §Marks 2).
|
||
#[tokio::test]
|
||
async fn batch_pins_drop_together_in_one_ordered_step() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
let a = key_for(b"batch member a");
|
||
let b = key_for(b"batch member b");
|
||
plant_kv(&store, &a.as_bytes(), b"batch member a").await;
|
||
plant_kv(&store, &b.as_bytes(), b"batch member b").await;
|
||
|
||
let mut batch = store.gc().new_batch();
|
||
batch.add(store.gc().acquire(&a).await);
|
||
batch.add(store.gc().acquire(&b).await);
|
||
assert_eq!(batch.len(), 2);
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.candidates_staged, 0, "the batch protects both");
|
||
assert!(store.has(&a.as_bytes()).await.unwrap());
|
||
assert!(store.has(&b.as_bytes()).await.unwrap());
|
||
|
||
drop(batch);
|
||
assert_eq!(
|
||
store.gc().pin_count(&a) + store.gc().pin_count(&b),
|
||
0,
|
||
"batch drop released every member refcount"
|
||
);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.candidates_staged, 2);
|
||
assert_eq!(report.deleted, 2);
|
||
}
|
||
|
||
/// The protect callback adds externally-known hashes (they stay
|
||
/// alive) and aborts as report data on failure — nothing deleted,
|
||
/// `ProtectFailed` in `Ok(SweepReport)` (ADR-012 §1).
|
||
#[tokio::test]
|
||
async fn protect_callback_adds_keys_and_aborts_as_data() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
let protect = Arc::new(SetProtect::default());
|
||
store.register_protect_callback(protect.clone());
|
||
|
||
let named = key_for(b"protected by the callback");
|
||
plant_kv(&store, &named.as_bytes(), b"protected by the callback").await;
|
||
let dead = key_for(b"dead while the callback answers");
|
||
plant_kv(&store, &dead.as_bytes(), b"dead while the callback answers").await;
|
||
protect.set_keys(vec![named.clone()]);
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, None);
|
||
assert_eq!(report.candidates_staged, 1, "the named key never staged");
|
||
assert_eq!(report.deleted, 1);
|
||
assert!(store.has(&named.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&dead.as_bytes()).await.unwrap());
|
||
|
||
protect.fail();
|
||
let another_dead = key_for(b"dead while the callback fails");
|
||
plant_kv(
|
||
&store,
|
||
&another_dead.as_bytes(),
|
||
b"dead while the callback fails",
|
||
)
|
||
.await;
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, Some(GcAbortCause::ProtectFailed));
|
||
assert_eq!(
|
||
report.deleted, 0,
|
||
"a failed protection source deletes nothing"
|
||
);
|
||
assert!(store.has(&another_dead.as_bytes()).await.unwrap());
|
||
assert!(store.has(&named.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// The live-shared seam: registration never snapshots — content
|
||
/// added to the set after registering is seen by the next sweep;
|
||
/// content removed exposes the entry again (POC #1 finding 7).
|
||
#[tokio::test]
|
||
async fn liveness_sources_are_consulted_live_not_snapshotted() {
|
||
let store = Store::mem_only();
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
|
||
let key = key_for(b"live-shared content");
|
||
plant_kv(&store, &key.as_bytes(), b"live-shared content").await;
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1, "unprotected at mark, deleted");
|
||
|
||
plant_kv(&store, &key.as_bytes(), b"live-shared content").await;
|
||
source.add(key.clone());
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
report.candidates_staged, 0,
|
||
"the registration after registration is consulted live"
|
||
);
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
|
||
source.remove(&key);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1, "deregistration re-exposes the entry");
|
||
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// Unregistration consumes the token (returning `false` the second
|
||
/// time) and the corrected zero-source posture aborts sweeps again
|
||
/// — ADR-012 §6.4's correction-flow prerequisite.
|
||
#[tokio::test]
|
||
async fn unregistration_consumes_the_token_and_restores_the_safe_default() {
|
||
let store = Store::mem_only();
|
||
let registration = store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
assert!(
|
||
store.unregister_liveness_source(registration),
|
||
"the token deregisters once"
|
||
);
|
||
assert!(
|
||
!store.unregister_liveness_source(registration),
|
||
"the token does not deregister twice"
|
||
);
|
||
|
||
let protect_token = store.register_protect_callback(Arc::new(SetProtect::default()));
|
||
assert!(
|
||
!store.unregister_liveness_source(protect_token),
|
||
"a protect token is not a source token"
|
||
);
|
||
assert!(store.unregister_protect_callback(protect_token));
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, Some(GcAbortCause::NoLivenessSources));
|
||
}
|
||
|
||
/// Direct `delete` refuses the full protection set with the typed
|
||
/// `Error::GcRefuse` — pinned, registered-source-live,
|
||
/// protect-callback-named (ADR-012 §6.4) — and deletes unprotected
|
||
/// keys through the fall-through dispatch; absent keys carry
|
||
/// `Err(Missing)`; malformed bytes `Err(KeyInvalid)`.
|
||
#[tokio::test]
|
||
async fn direct_delete_refuses_the_full_protection_set() {
|
||
let root = temp_root("gc-direct-delete");
|
||
let store = Store::dual_tier_mem_local(root.path(), 1024)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
let protect = Arc::new(SetProtect::default());
|
||
store.register_protect_callback(protect.clone());
|
||
|
||
let pinned = key_for(b"pinned refusal");
|
||
plant_kv(&store, &pinned.as_bytes(), b"pinned refusal").await;
|
||
let _pin = store.gc().acquire(&pinned).await;
|
||
|
||
let source_live = key_for(b"source-live refusal");
|
||
plant_large(&store, &source_live.as_bytes(), b"source-live refusal").await;
|
||
source.add(source_live.clone());
|
||
|
||
let protect_named = key_for(b"protect-named refusal");
|
||
plant_kv(&store, &protect_named.as_bytes(), b"protect-named refusal").await;
|
||
protect.set_keys(vec![protect_named.clone()]);
|
||
|
||
let unprotected_kv = key_for(b"unprotected kv deletes");
|
||
plant_kv(
|
||
&store,
|
||
&unprotected_kv.as_bytes(),
|
||
b"unprotected kv deletes",
|
||
)
|
||
.await;
|
||
let unprotected_large = key_for(b"unprotected large deletes");
|
||
plant_large(
|
||
&store,
|
||
&unprotected_large.as_bytes(),
|
||
b"unprotected large deletes",
|
||
)
|
||
.await;
|
||
|
||
assert_eq!(
|
||
store.delete(&pinned.as_bytes()).await.unwrap_err(),
|
||
Error::GcRefuse,
|
||
"pinned keys refuse"
|
||
);
|
||
assert!(
|
||
store.has(&pinned.as_bytes()).await.unwrap(),
|
||
"nothing deleted on refusal"
|
||
);
|
||
assert_eq!(
|
||
store.delete(&source_live.as_bytes()).await.unwrap_err(),
|
||
Error::GcRefuse,
|
||
"registered-source-live keys refuse"
|
||
);
|
||
assert!(store.has(&source_live.as_bytes()).await.unwrap());
|
||
assert_eq!(
|
||
store.delete(&protect_named.as_bytes()).await.unwrap_err(),
|
||
Error::GcRefuse,
|
||
"protect-callback-named keys refuse"
|
||
);
|
||
assert!(store.has(&protect_named.as_bytes()).await.unwrap());
|
||
|
||
store
|
||
.delete(&unprotected_kv.as_bytes())
|
||
.await
|
||
.expect("unprotected kv keys delete");
|
||
assert!(!store.has(&unprotected_kv.as_bytes()).await.unwrap());
|
||
store
|
||
.delete(&unprotected_large.as_bytes())
|
||
.await
|
||
.expect("unprotected large keys delete through the fall-through");
|
||
assert!(!store.has(&unprotected_large.as_bytes()).await.unwrap());
|
||
|
||
let absent = key_for(b"never planted");
|
||
assert_eq!(
|
||
store.delete(&absent.as_bytes()).await.unwrap_err(),
|
||
Error::Missing,
|
||
"absent keys carry Missing"
|
||
);
|
||
assert_eq!(
|
||
store.delete(&[0x03; 33]).await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"malformed key bytes reject at the boundary"
|
||
);
|
||
}
|
||
|
||
/// Sequential evidence of the conversion ordering primitive: after
|
||
/// the ordered step, the pin refcount is released **and** the
|
||
/// registration is visible — under one arbitration-lock section
|
||
/// (ADR-005's pinned conversion ordering).
|
||
#[tokio::test]
|
||
async fn pin_conversion_releases_and_registers_in_one_ordered_step() {
|
||
let store = Store::mem_only();
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
|
||
let key = key_for(b"converted protection");
|
||
let pin = store.gc().acquire(&key).await;
|
||
assert_eq!(store.gc().pin_count(&key), 1);
|
||
|
||
store.gc().convert(pin, || source.add(key.clone()));
|
||
|
||
assert_eq!(
|
||
store.gc().pin_count(&key),
|
||
0,
|
||
"the pin's release is one half of the ordered step"
|
||
);
|
||
assert!(
|
||
source.contains(&key),
|
||
"the registration is the other half — the same section"
|
||
);
|
||
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
report.candidates_staged, 0,
|
||
"the converted registration protects through the source"
|
||
);
|
||
}
|
||
|
||
/// The ordered step under race: a pin converted into a source is
|
||
/// never an unprotected instant — a concurrent direct delete must
|
||
/// observe the pin or the registration at every schedule and
|
||
/// refuse, never delete (ADR-005's conversion-ordering invariant,
|
||
/// raced).
|
||
#[tokio::test]
|
||
async fn conversion_raced_against_direct_delete_never_deletes() {
|
||
let store = Store::mem_only();
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
|
||
for round in 0..64 {
|
||
let content = format!("converted protection round {round}");
|
||
let key = key_for(content.as_bytes());
|
||
let bytes = key.as_bytes();
|
||
plant_kv(&store, &bytes, content.as_bytes()).await;
|
||
|
||
let pin = store.gc().acquire(&key).await;
|
||
let (result, ()) = tokio::join!(store.delete(&bytes), async {
|
||
store.gc().convert(pin, {
|
||
let source = Arc::clone(&source);
|
||
let key = key.clone();
|
||
move || source.add(key)
|
||
})
|
||
},);
|
||
assert_eq!(
|
||
result,
|
||
Err(Error::GcRefuse),
|
||
"round {round}: no schedule leaves the key unprotected "
|
||
);
|
||
assert!(
|
||
store.has(&bytes).await.unwrap(),
|
||
"nothing deleted while the protection converted"
|
||
);
|
||
|
||
source.remove(&key);
|
||
store
|
||
.delete(&bytes)
|
||
.await
|
||
.expect("unprotected now, deletes");
|
||
}
|
||
}
|
||
|
||
/// The test-asserted property (ADR-005): a sweep racing an
|
||
/// in-flight pinned put never deletes the protected entry — the
|
||
/// pin-before-publish ordering holds from the sweep side, and the
|
||
/// delete-then-recover arm lands byte-identical. Counts stay exact
|
||
/// in every schedule, and the surviving pin keeps protecting.
|
||
#[tokio::test]
|
||
async fn sweep_racing_a_pinned_put_never_deletes_the_protected_entry() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
for round in 0..64 {
|
||
let content = format!("race round {round}");
|
||
let key = key_for(content.as_bytes());
|
||
let bytes = key.as_bytes();
|
||
|
||
plant_kv(&store, &bytes, content.as_bytes()).await;
|
||
|
||
let (report, pin) = tokio::join!(store.sweep(), async {
|
||
let pin = store.gc().acquire(&key).await;
|
||
plant_kv(&store, &bytes, content.as_bytes()).await;
|
||
pin
|
||
},);
|
||
|
||
let report = report.expect("sweeps are never errors");
|
||
assert_eq!(report.aborted, None, "round {round}");
|
||
assert_eq!(
|
||
report.candidates_staged,
|
||
report.deleted + report.cancelled_by_arbitration,
|
||
"round {round}: exact counts in every schedule"
|
||
);
|
||
|
||
assert!(
|
||
store.has(&bytes).await.unwrap(),
|
||
"round {round}: the entry survives (canceled, or delete-then-recovered)"
|
||
);
|
||
let (_, blob) = store
|
||
.get(&bytes)
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(
|
||
blob.read_at(0, content.len() as u64).await.unwrap(),
|
||
content.as_bytes(),
|
||
"round {round}: the recovered re-put is byte-identical"
|
||
);
|
||
|
||
drop(pin);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
report.deleted, 1,
|
||
"round {round}: with the pin dropped the entry is dead — exactly one deletion"
|
||
);
|
||
assert!(!store.has(&bytes).await.unwrap());
|
||
}
|
||
}
|
||
|
||
/// An in-flight put whose publish lands after the sweep's mark is
|
||
/// never enumerated by it — the sweep started before the commit
|
||
/// and cannot delete what its mark never saw (ADR-005's
|
||
/// pin-before-publish invariant, the fresh-put arm).
|
||
#[tokio::test]
|
||
async fn put_publishing_after_a_sweep_started_is_never_deleted() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let content = b"published after the mark";
|
||
let key = key_for(content);
|
||
let bytes = key.as_bytes();
|
||
let dead = key_for(b"a dead witness for the concurrent sweep");
|
||
plant_kv(
|
||
&store,
|
||
&dead.as_bytes(),
|
||
b"a dead witness for the concurrent sweep",
|
||
)
|
||
.await;
|
||
|
||
let (report, pin) = tokio::join!(store.sweep(), async {
|
||
let pin = store.gc().acquire(&key).await;
|
||
tokio::task::yield_now().await;
|
||
tokio::task::yield_now().await;
|
||
plant_kv(&store, &bytes, content).await;
|
||
pin
|
||
},);
|
||
|
||
let report = report.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1, "the dead witness deleted");
|
||
assert_eq!(
|
||
report.candidates_staged, 1,
|
||
"the in-flight put's entry never listed — nothing staged for it"
|
||
);
|
||
assert!(
|
||
store.has(&bytes).await.unwrap(),
|
||
"the published entry survived"
|
||
);
|
||
assert_eq!(store.gc().pin_count(&key), 1, "the pin still protects");
|
||
|
||
drop(pin);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1, "now unprotected, deleted");
|
||
}
|
||
|
||
/// `has`/`get` during a delete window observe no torn state: each
|
||
/// observation is either the untouched full content or clean
|
||
/// absence — existence flips atomically per key, content never
|
||
/// partial (the pool's immutable-entry posture).
|
||
#[tokio::test]
|
||
async fn reads_during_a_delete_window_observe_no_torn_state() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let content = b"whole-content visibility under the window";
|
||
let key = key_for(content);
|
||
let bytes = key.as_bytes();
|
||
plant_kv(&store, &bytes, content).await;
|
||
|
||
let observer = {
|
||
let store = &store;
|
||
let bytes = bytes.clone();
|
||
let content_len = content.len() as u64;
|
||
async move {
|
||
let mut seen_true = false;
|
||
let mut seen_absence = false;
|
||
for _ in 0..200 {
|
||
if store.has(&bytes).await.unwrap() {
|
||
seen_true = true;
|
||
assert!(
|
||
!seen_absence,
|
||
"no re-appearance: existence flips once per key"
|
||
);
|
||
let (_, blob) = store
|
||
.get(&bytes)
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("has said true");
|
||
let whole = blob.read_at(0, content_len).await.unwrap();
|
||
assert_eq!(
|
||
whole, content,
|
||
"an observed entry presents its exact, whole content"
|
||
);
|
||
} else {
|
||
seen_absence = true;
|
||
}
|
||
tokio::task::yield_now().await;
|
||
}
|
||
(seen_true, seen_absence)
|
||
}
|
||
};
|
||
|
||
let ((seen_true, seen_absence), report) = tokio::join!(observer, store.sweep());
|
||
let report = report.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1);
|
||
assert!(seen_true || seen_absence, "the observer ran alongside");
|
||
assert!(
|
||
!store.has(&bytes).await.unwrap(),
|
||
"after the window the entry is gone"
|
||
);
|
||
}
|
||
|
||
/// The single-node sweep is single-flight — the in-process analog
|
||
/// of the named exclusive sweep lease (ADR-013 §2): a second
|
||
/// sweeper contending behind the first aborts as report data
|
||
/// (`SweeperLock`), nothing deleted.
|
||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||
async fn concurrent_sweeps_single_flight_and_the_second_aborts_as_data() {
|
||
let store = Arc::new(Store::mem_only());
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
|
||
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
|
||
store.register_protect_callback(Arc::new(LatchProtect {
|
||
entered: entered_tx,
|
||
release: parking_lot::Mutex::new(Some(release_rx)),
|
||
}));
|
||
|
||
let dead = key_for(b"dead entry under the contended sweeps");
|
||
plant_kv(
|
||
&store,
|
||
&dead.as_bytes(),
|
||
b"dead entry under the contended sweeps",
|
||
)
|
||
.await;
|
||
|
||
let first = {
|
||
let store = Arc::clone(&store);
|
||
tokio::spawn(async move { store.sweep().await })
|
||
};
|
||
|
||
while entered_rx
|
||
.recv_timeout(std::time::Duration::from_millis(500))
|
||
.is_err()
|
||
{
|
||
tokio::task::yield_now().await;
|
||
}
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
second.aborted,
|
||
Some(GcAbortCause::SweeperLock),
|
||
"the contended sweeper aborts as report data"
|
||
);
|
||
assert_eq!(second.deleted, 0);
|
||
assert!(
|
||
store.has(&dead.as_bytes()).await.unwrap(),
|
||
"the contended sweep deleted nothing"
|
||
);
|
||
|
||
release_tx.send(()).ok();
|
||
let first = first
|
||
.await
|
||
.expect("first sweep task joined")
|
||
.expect("first sweep returns a report");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(first.deleted, 1, "the lease holder sweeps");
|
||
assert!(!store.has(&dead.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// Sweeps surface operational failures as errors (ADR-012 §1's
|
||
/// error half): a malformed pool list is fatal `KeyInvalid`, never
|
||
/// a silent skip — through the sweep's own mark enumeration too.
|
||
#[tokio::test]
|
||
async fn sweep_of_a_malformed_pool_list_is_an_operational_error() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
plant_kv(&store, &[0x07; 33], b"stray in the table").await;
|
||
|
||
assert_eq!(
|
||
store.sweep().await.unwrap_err(),
|
||
Error::KeyInvalid,
|
||
"a malformed pool list is fatal through the sweep"
|
||
);
|
||
}
|
||
|
||
// ------------------------------------------------------------------
|
||
// The put seam (store-put-paths): the two internal paths behind one
|
||
// signature, the three numbered unknown-length rules, pin-before-
|
||
// publish with dedup refcounting, the Batch seam, kv-only rejection
|
||
// (ADR-003/005/010/012 §6.2) — on the facade's write half.
|
||
// ------------------------------------------------------------------
|
||
|
||
use std::collections::VecDeque;
|
||
use std::io::Cursor;
|
||
use std::pin::Pin as TestPin;
|
||
use std::task::{Context, Poll};
|
||
use tokio::io::ReadBuf;
|
||
|
||
/// A stream cut at controlled chunk boundaries (`sizes`, in order;
|
||
/// tail data falls out whole) — the fixture that makes mid-stream
|
||
/// overflow, flush ordering, and staged continuation observable
|
||
/// past the put loop's own chunking.
|
||
struct Chunked {
|
||
data: Vec<u8>,
|
||
cursor: usize,
|
||
sizes: VecDeque<usize>,
|
||
}
|
||
|
||
impl Chunked {
|
||
fn new(data: &[u8], sizes: &[usize]) -> Chunked {
|
||
Chunked {
|
||
data: data.to_vec(),
|
||
cursor: 0,
|
||
sizes: sizes.iter().copied().collect(),
|
||
}
|
||
}
|
||
}
|
||
|
||
impl AsyncRead for Chunked {
|
||
fn poll_read(
|
||
mut self: TestPin<&mut Self>,
|
||
_cx: &mut Context<'_>,
|
||
buf: &mut ReadBuf<'_>,
|
||
) -> Poll<std::io::Result<()>> {
|
||
if self.cursor >= self.data.len() || buf.remaining() == 0 {
|
||
return Poll::Ready(Ok(()));
|
||
}
|
||
let remaining = self.data.len() - self.cursor;
|
||
let wanted = self
|
||
.sizes
|
||
.front()
|
||
.copied()
|
||
.unwrap_or(remaining)
|
||
.min(remaining)
|
||
.min(buf.remaining());
|
||
buf.put_slice(&self.data[self.cursor..self.cursor + wanted]);
|
||
self.cursor += wanted;
|
||
if let Some(front) = self.sizes.front_mut() {
|
||
*front -= wanted;
|
||
if *front == 0 {
|
||
self.sizes.pop_front();
|
||
}
|
||
}
|
||
Poll::Ready(Ok(()))
|
||
}
|
||
}
|
||
|
||
/// A stream that delivers `fail_after` bytes then errors mid-stream —
|
||
/// the non-restartable source's failure (rule 3's shape).
|
||
struct Failing {
|
||
data: Vec<u8>,
|
||
cursor: usize,
|
||
fail_after: usize,
|
||
}
|
||
|
||
impl Failing {
|
||
fn new(data: &[u8], fail_after: usize) -> Failing {
|
||
Failing {
|
||
data: data.to_vec(),
|
||
cursor: 0,
|
||
fail_after,
|
||
}
|
||
}
|
||
}
|
||
|
||
impl AsyncRead for Failing {
|
||
fn poll_read(
|
||
mut self: TestPin<&mut Self>,
|
||
_cx: &mut Context<'_>,
|
||
buf: &mut ReadBuf<'_>,
|
||
) -> Poll<std::io::Result<()>> {
|
||
let message = "stream failure (test fixture)";
|
||
if self.cursor >= self.fail_after {
|
||
return Poll::Ready(Err(std::io::Error::other(message)));
|
||
}
|
||
let wanted = (self.data.len() - self.cursor)
|
||
.min(self.fail_after - self.cursor)
|
||
.min(buf.remaining());
|
||
if wanted == 0 {
|
||
return Poll::Ready(Err(std::io::Error::other(message)));
|
||
}
|
||
buf.put_slice(&self.data[self.cursor..self.cursor + wanted]);
|
||
self.cursor += wanted;
|
||
Poll::Ready(Ok(()))
|
||
}
|
||
}
|
||
|
||
/// A stream nobody may ever read — the fixture proving kv-only's
|
||
/// known-length rejection happens *immediately*, before content
|
||
/// flows (ADR-012 §6.2).
|
||
struct Unreadable;
|
||
|
||
impl AsyncRead for Unreadable {
|
||
fn poll_read(
|
||
self: TestPin<&mut Self>,
|
||
_cx: &mut Context<'_>,
|
||
_buf: &mut ReadBuf<'_>,
|
||
) -> Poll<std::io::Result<()>> {
|
||
Poll::Ready(Err(std::io::Error::other(
|
||
"the stream must not be read (test fixture)",
|
||
)))
|
||
}
|
||
}
|
||
|
||
/// Files currently in the local tier's staging area — the residue
|
||
/// counter of the store-level stage-hygiene assertions.
|
||
fn store_stage_file_count(root: &Path) -> usize {
|
||
match fs::read_dir(root.join("staging")) {
|
||
Ok(entries) => entries.count(),
|
||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => 0,
|
||
Err(e) => panic!("staging dir unreadable: {e}"),
|
||
}
|
||
}
|
||
|
||
/// INV-1, kv tier, known-length path: `put(Some(len), stream)`
|
||
/// round-trips byte-identical, lands in the routed tier (kv), and
|
||
/// the pin rides out with the put.
|
||
#[tokio::test]
|
||
async fn known_length_put_round_trips_on_the_kv_tier() {
|
||
let root = temp_root("put-known-kv");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content = b"a known-length small blob";
|
||
let canonical = key_for(content);
|
||
|
||
let (key, pin) = store
|
||
.put(Some(content.len() as u64), Cursor::new(&content[..]))
|
||
.await
|
||
.expect("kv put publishes");
|
||
assert_eq!(key, canonical, "the canonical derivation addresses the put");
|
||
assert_eq!(store.gc().pin_count(&canonical), 1, "the pin rode out");
|
||
assert!(
|
||
store.kv().has(&canonical.as_bytes()).await.unwrap(),
|
||
"landed in the routed tier (kv)"
|
||
);
|
||
assert!(
|
||
!store
|
||
.large()
|
||
.unwrap()
|
||
.has(&canonical.as_bytes())
|
||
.await
|
||
.unwrap(),
|
||
"no large-tier copy on the kv route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&canonical.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(len, content.len() as u64);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), &content[..]);
|
||
|
||
drop(pin);
|
||
}
|
||
|
||
/// INV-1, large tier, known-length path: over-threshold known
|
||
/// content streams into the staged put with hashing in the same
|
||
/// pass, the derivation completing at stream end; commit-rename
|
||
/// publishes; zero stage files remain.
|
||
#[tokio::test]
|
||
async fn known_length_put_round_trips_on_the_large_tier_through_the_staged_put() {
|
||
let root = temp_root("put-known-large");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content = vec![42u8; 300];
|
||
let canonical = key_for(&content);
|
||
|
||
let (key, pin) = store
|
||
.put(Some(300), Cursor::new(&content[..]))
|
||
.await
|
||
.expect("large staged put publishes");
|
||
assert_eq!(key, canonical);
|
||
assert_eq!(store.gc().pin_count(&canonical), 1);
|
||
assert!(
|
||
store
|
||
.large()
|
||
.unwrap()
|
||
.has(&canonical.as_bytes())
|
||
.await
|
||
.unwrap(),
|
||
"landed in the routed tier (large, commit-rename)"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&canonical.as_bytes()).await.unwrap(),
|
||
"no kv copy on the large route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&canonical.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(len, 300);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), content);
|
||
assert_eq!(
|
||
store_stage_file_count(root.path()),
|
||
0,
|
||
"commit-rename left zero stage files"
|
||
);
|
||
|
||
drop(pin);
|
||
}
|
||
|
||
/// INV-1, both tiers, unknown-length path: sub-threshold content
|
||
/// commits through the buffered kv form; over-threshold content
|
||
/// flushed-through staging — both round-trip byte-identical and
|
||
/// both arrive under the canonical derivation over the eventual
|
||
/// stored artifact as a whole (rule 2 asserted by key equality:
|
||
/// anything but preamble + the wholeArtifact derivation would
|
||
/// derive a different key).
|
||
#[tokio::test]
|
||
async fn unknown_length_put_round_trips_on_both_tiers() {
|
||
let root = temp_root("put-unknown");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
|
||
let small = b"small unknown-length content";
|
||
let (key, pin) = store
|
||
.put(None, Cursor::new(&small[..]))
|
||
.await
|
||
.expect("buffered put publishes");
|
||
assert_eq!(key, key_for(small), "rule 2, buffer form");
|
||
assert!(store.kv().has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
|
||
let big: Vec<u8> = (0..200u32).map(|i| (i % 251) as u8).collect();
|
||
let (key, pin) = store
|
||
.put(None, Chunked::new(&big, &[7; 30]))
|
||
.await
|
||
.expect("staged put publishes");
|
||
assert_eq!(key, key_for(&big), "rule 2, staged-bytes form");
|
||
assert!(
|
||
store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"the over-threshold unknown put landed large"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(len, big.len() as u64);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), big);
|
||
assert_eq!(
|
||
store_stage_file_count(root.path()),
|
||
0,
|
||
"zero stage files on success"
|
||
);
|
||
drop(pin);
|
||
}
|
||
|
||
/// Rule 1 (ADR-003 option (d)): threshold exceeded mid-stream — the
|
||
/// buffered prefix flushes into a fresh stage file and the stream
|
||
/// *continues appending into it*; no restart anywhere, nothing
|
||
/// migrates tiers, and the flushed artifact derives canonically.
|
||
#[tokio::test]
|
||
async fn mid_stream_overflow_flushes_into_a_fresh_stage_and_the_stream_continues() {
|
||
let root = temp_root("put-rule1-flush");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content: Vec<u8> = (0..40u32).map(|i| (i % 199) as u8).collect();
|
||
// Chunks of 3: two fill the buffer to 6; the third (would-be 9)
|
||
// crosses the threshold mid-stream and triggers the flush; the
|
||
// rest stream on into the stage.
|
||
let chunked = Chunked::new(&content, &[3; 14]);
|
||
|
||
let (key, pin) = store
|
||
.put(None, chunked)
|
||
.await
|
||
.expect("mid-stream overflow flushes; the stream continues");
|
||
assert_eq!(
|
||
key,
|
||
key_for(&content),
|
||
"the flushed artifact derives canonically"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"nothing migrated to the kv tier on a large-route put"
|
||
);
|
||
assert!(
|
||
store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"the stage holds the whole streamed artifact"
|
||
);
|
||
let (_, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(
|
||
blob.read_at(0, content.len() as u64).await.unwrap(),
|
||
content,
|
||
"the flushed prefix and the continued stream are one artifact"
|
||
);
|
||
assert_eq!(
|
||
store_stage_file_count(root.path()),
|
||
0,
|
||
"the stage published by rename; zero residue"
|
||
);
|
||
drop(pin);
|
||
}
|
||
|
||
/// The dispatch seam's boundary through the put path: an
|
||
/// unknown-length stream ending exactly at the threshold stays kv
|
||
/// (≤ threshold routes kv; the overflow is strictly past it).
|
||
#[tokio::test]
|
||
async fn unknown_length_exactly_at_threshold_stays_kv() {
|
||
let root = temp_root("put-boundary");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content = b"01234567"; // exactly 8 bytes
|
||
let chunked = Chunked::new(content, &[4, 4]);
|
||
|
||
let (key, pin) = store
|
||
.put(None, chunked)
|
||
.await
|
||
.expect("at-threshold stream commits through the buffer");
|
||
assert_eq!(key, key_for(content));
|
||
assert!(
|
||
store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"exactly-at-threshold stays kv"
|
||
);
|
||
assert!(
|
||
!store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"no large-tier copy on the boundary"
|
||
);
|
||
drop(pin);
|
||
}
|
||
|
||
/// Rule 3 (ADR-003): a stream error before commit leaves no entry
|
||
/// and no stage file — on the buffered kv form, on the flushed
|
||
/// staged form, and on both known-length tiers; the stage-hygiene
|
||
/// invariant extended to the buffered case.
|
||
#[tokio::test]
|
||
async fn stream_errors_before_commit_leave_no_entry_and_no_stage_file() {
|
||
let root = temp_root("put-rule3-error");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content: Vec<u8> = (0..40u32).map(|i| (i % 173) as u8).collect();
|
||
|
||
// Flushed staged form: the overflow flush ran, then the stream
|
||
// errored — the stage discards.
|
||
let err = store
|
||
.put(None, Failing::new(&content, 12))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
matches!(err, Error::Io(_)),
|
||
"stream failures surface as Io: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"no entry landed from the errored staging put"
|
||
);
|
||
assert_eq!(
|
||
store_stage_file_count(root.path()),
|
||
0,
|
||
"no stage file remains"
|
||
);
|
||
|
||
// Buffered kv form: error before the threshold — the buffer
|
||
// discards exactly like a stage.
|
||
let err = store
|
||
.put(None, Failing::new(&content, 5))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(matches!(err, Error::Io(_)));
|
||
assert_eq!(store.list().await.unwrap(), Vec::<Key>::new());
|
||
assert_eq!(store_stage_file_count(root.path()), 0);
|
||
|
||
// Known-length large path: mid-stage failure discards.
|
||
let err = store
|
||
.put(Some(40), Failing::new(&content, 12))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(matches!(err, Error::Io(_)));
|
||
assert_eq!(store.list().await.unwrap(), Vec::<Key>::new());
|
||
assert_eq!(store_stage_file_count(root.path()), 0);
|
||
|
||
// Known-length kv path: never published.
|
||
let err = store
|
||
.put(Some(40), Failing::new(&content, 5))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(matches!(err, Error::Io(_)));
|
||
assert_eq!(store.list().await.unwrap(), Vec::<Key>::new());
|
||
assert_eq!(store_stage_file_count(root.path()), 0);
|
||
}
|
||
|
||
/// Pin-before-publish (ADR-005) on both put paths and both tiers:
|
||
/// a sweep running alongside an in-flight put never deletes the
|
||
/// protected entry — the pin is minted before the entry can appear
|
||
/// in any `list()` — and every completed schedule keeps exact
|
||
/// counts. The surviving pin protects; dropping it re-exposes.
|
||
#[tokio::test]
|
||
async fn pin_before_publish_holds_on_both_put_paths_and_both_tiers() {
|
||
let root = temp_root("put-pin");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let big: Vec<u8> = (0..200u32).map(|i| (i % 241) as u8).collect();
|
||
let big_other: Vec<u8> = (0..288u32).map(|i| (i % 223) as u8).collect();
|
||
let rounds: [(Vec<u8>, Option<u64>); 4] = [
|
||
(b"pinned known small".to_vec(), Some(18)),
|
||
(b"still-pinned unknown small content".to_vec(), None),
|
||
(big, Some(200)),
|
||
(big_other, None),
|
||
];
|
||
|
||
for (round, (content, len)) in rounds.iter().enumerate() {
|
||
let expected = key_for(content);
|
||
let (report, put) = tokio::join!(store.sweep(), async {
|
||
store.put(*len, Cursor::new(&content[..])).await
|
||
});
|
||
let report = report.expect("sweeps are never errors");
|
||
let (key, pin) = put.expect("put published");
|
||
assert_eq!(key, expected, "round {round}");
|
||
assert_eq!(report.aborted, None, "round {round}");
|
||
assert_eq!(
|
||
report.candidates_staged,
|
||
report.deleted + report.cancelled_by_arbitration,
|
||
"round {round}: exact counts in every schedule"
|
||
);
|
||
assert!(
|
||
store.has(&key.as_bytes()).await.unwrap(),
|
||
"round {round}: the entry survived the race"
|
||
);
|
||
assert_eq!(
|
||
store.gc().pin_count(&expected),
|
||
1,
|
||
"round {round}: the minted pin still protects"
|
||
);
|
||
let (_, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(
|
||
blob.read_at(0, content.len() as u64).await.unwrap(),
|
||
&content[..],
|
||
"round {round}: the published entry is byte-identical"
|
||
);
|
||
|
||
drop(pin);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1, "round {round}: unpinned, deleted");
|
||
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
||
}
|
||
}
|
||
|
||
/// Dedup (store-api.md invariant 2; ADR-005): a re-put of identical
|
||
/// content acquires no new pool entry and refcounts the existing
|
||
/// one — across put paths (known vs unknown) and across both tiers;
|
||
/// the refcounts compose under the same lock and drop in sequence.
|
||
#[tokio::test]
|
||
async fn dedup_reputs_refcount_the_existing_entry_from_any_path() {
|
||
let root = temp_root("put-dedup");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let content = b"deduplicated content";
|
||
let (known_key, first) = store
|
||
.put(Some(content.len() as u64), Cursor::new(&content[..]))
|
||
.await
|
||
.expect("known-path put");
|
||
let (unknown_key, second) = store
|
||
.put(None, Cursor::new(&content[..]))
|
||
.await
|
||
.expect("unknown-path re-put");
|
||
assert_eq!(known_key, unknown_key, "one derivation, one address");
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
vec![known_key.clone()],
|
||
"one pool entry"
|
||
);
|
||
assert_eq!(
|
||
store.gc().pin_count(&known_key),
|
||
2,
|
||
"the re-put refcounted the existing entry"
|
||
);
|
||
|
||
drop(first);
|
||
assert_eq!(store.gc().pin_count(&known_key), 1);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.candidates_staged, 0, "one refcount still protects");
|
||
|
||
drop(second);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1);
|
||
assert!(!store.has(&known_key.as_bytes()).await.unwrap());
|
||
|
||
// The same composition across the large tier: two staged puts
|
||
// of identical content — one entry, two refcounts.
|
||
let big = [91u8; 200];
|
||
let (first_key, first) = store
|
||
.put(Some(200), Cursor::new(&big[..]))
|
||
.await
|
||
.expect("known staged put");
|
||
let (_, second) = store
|
||
.put(None, Cursor::new(&big[..]))
|
||
.await
|
||
.expect("unknown staged re-put");
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
vec![first_key.clone()],
|
||
"one pool entry across both staged paths"
|
||
);
|
||
assert_eq!(store.gc().pin_count(&first_key), 2);
|
||
drop(first);
|
||
drop(second);
|
||
}
|
||
|
||
/// `Batch` (store-api.md §Write path; ADR-005 §Marks 2): multi-put
|
||
/// writes group under one batch-scoped pin — members publish pin-
|
||
/// before-publish exactly as lone puts do, both tiers, and the
|
||
/// group's pins drop together on batch commit.
|
||
#[tokio::test]
|
||
async fn batch_groups_multi_put_writes_and_members_drop_together() {
|
||
let root = temp_root("put-batch");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let small_a = b"manifest member a";
|
||
let small_b = b"manifest member b";
|
||
let big = vec![77u8; 200];
|
||
|
||
let mut batch = store.batch();
|
||
assert!(batch.is_empty(), "a fresh batch protects nothing");
|
||
let key_a = store
|
||
.put_in_batch(&mut batch, Some(17), Cursor::new(&small_a[..]))
|
||
.await
|
||
.expect("batch member a");
|
||
let key_b = store
|
||
.put_in_batch(&mut batch, None, Cursor::new(&small_b[..]))
|
||
.await
|
||
.expect("batch member b");
|
||
let key_big = store
|
||
.put_in_batch(&mut batch, Some(200), Cursor::new(&big[..]))
|
||
.await
|
||
.expect("batch member large");
|
||
assert_eq!(batch.len(), 3);
|
||
assert_eq!(key_a, key_for(small_a));
|
||
assert_eq!(key_b, key_for(small_b));
|
||
assert_eq!(key_big, key_for(&big));
|
||
|
||
for key in [&key_a, &key_b, &key_big] {
|
||
assert_eq!(
|
||
store.gc().pin_count(key),
|
||
1,
|
||
"each member's pin rides in the group"
|
||
);
|
||
}
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
report.candidates_staged, 0,
|
||
"the batch-scoped pin protects the whole group"
|
||
);
|
||
assert!(store.has(&key_a.as_bytes()).await.unwrap());
|
||
assert!(store.has(&key_big.as_bytes()).await.unwrap());
|
||
|
||
batch.commit();
|
||
for key in [&key_a, &key_b, &key_big] {
|
||
assert_eq!(
|
||
store.gc().pin_count(key),
|
||
0,
|
||
"no early single-pin release and no late release either"
|
||
);
|
||
}
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 3, "unprotected after the batch ends");
|
||
assert!(!store.has(&key_a.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&key_b.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&key_big.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// Batch drop is the failure path: a member put error leaves the
|
||
/// landed members batch-protected and contributes nothing to the
|
||
/// group; member liveness never outlives the batch.
|
||
#[tokio::test]
|
||
async fn batch_member_put_failure_does_not_endanger_landed_members() {
|
||
let root = temp_root("put-batch-failure");
|
||
let store = Store::dual_tier_mem_local(root.path(), 64)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let landed = b"the member that landed";
|
||
let mut batch = store.batch();
|
||
let key = store
|
||
.put_in_batch(&mut batch, Some(22), Cursor::new(&landed[..]))
|
||
.await
|
||
.expect("landed member published");
|
||
assert_eq!(store.gc().pin_count(&key), 1);
|
||
|
||
let content: Vec<u8> = (0..40u32).map(|i| (i % 157) as u8).collect();
|
||
let err = store
|
||
.put_in_batch(&mut batch, None, Failing::new(&content, 12))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
matches!(err, Error::Io(_)),
|
||
"the failed member surfaces its error"
|
||
);
|
||
assert_eq!(batch.len(), 1, "the failed member joined nothing");
|
||
assert_eq!(
|
||
store.gc().pin_count(&key),
|
||
1,
|
||
"landed members stay protected"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
vec![key.clone()],
|
||
"the failed member published no entry"
|
||
);
|
||
|
||
drop(batch);
|
||
assert_eq!(
|
||
store.gc().pin_count(&key),
|
||
0,
|
||
"batch drop released every member together"
|
||
);
|
||
let report = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(report.deleted, 1);
|
||
}
|
||
|
||
/// kv-only mode's put-time rejection (ADR-012 §6.2): known-length
|
||
/// over-threshold rejects *immediately* (the stream is never read);
|
||
/// unknown-length rejects at mid-stream threshold overflow — zero
|
||
/// residue on either rejection, and at/exactly-threshold puts land.
|
||
#[tokio::test]
|
||
async fn kv_only_mode_rejects_over_threshold_puts_at_the_two_points() {
|
||
let store = Store::kv_only_mem(64);
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let err = store.put(Some(65), Unreadable).await.unwrap_err();
|
||
assert!(
|
||
err.to_string().contains("kv-only mode rejects"),
|
||
"the immediate rejection carries its carrier message: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
|
||
let at_threshold = vec![9u8; 64];
|
||
let (key, pin) = store
|
||
.put(Some(64), Cursor::new(&at_threshold[..]))
|
||
.await
|
||
.expect("exactly-at-threshold known put lands");
|
||
assert_eq!(key, key_for(&at_threshold));
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
|
||
let streamed = vec![3u8; 100];
|
||
let err = store
|
||
.put(None, Chunked::new(&streamed, &[10; 10]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string()
|
||
.contains("kv-only mode rejects an over-threshold put"),
|
||
"the overflow rejection is the mode's, not the stream's: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
vec![key_for(&at_threshold)],
|
||
"zero residue"
|
||
);
|
||
|
||
let just_under = vec![5u8; 64];
|
||
let (key, pin) = store
|
||
.put(None, Chunked::new(&just_under, &[8; 8]))
|
||
.await
|
||
.expect("at-threshold unknown stream lands through the buffer");
|
||
assert_eq!(key, key_for(&just_under));
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
}
|
||
|
||
/// The declared-len mismatch (the put seam's named design point):
|
||
/// a known-length stream yielding fewer or more bytes than `len`
|
||
/// rejects with `Error::Verification` on either tier — zero residue
|
||
/// either way (nothing published, no stage file left).
|
||
#[tokio::test]
|
||
async fn declared_len_mismatch_rejects_verification_with_zero_residue() {
|
||
let root = temp_root("put-mismatch");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let content = b"0123456789"; // 10 bytes
|
||
|
||
assert_eq!(
|
||
store
|
||
.put(Some(5), Cursor::new(&content[..]))
|
||
.await
|
||
.unwrap_err(),
|
||
Error::Verification,
|
||
"more bytes than declared (kv route)"
|
||
);
|
||
assert_eq!(
|
||
store
|
||
.put(Some(20), Cursor::new(&content[..]))
|
||
.await
|
||
.unwrap_err(),
|
||
Error::Verification,
|
||
"fewer bytes than declared (kv route)"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue on the kv mismatches"
|
||
);
|
||
|
||
let big = [5u8; 32];
|
||
assert_eq!(
|
||
store
|
||
.put(Some(16), Cursor::new(&big[..]))
|
||
.await
|
||
.unwrap_err(),
|
||
Error::Verification,
|
||
"more bytes than declared (large route)"
|
||
);
|
||
assert_eq!(
|
||
store
|
||
.put(Some(64), Cursor::new(&big[..]))
|
||
.await
|
||
.unwrap_err(),
|
||
Error::Verification,
|
||
"fewer bytes than declared (large route)"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue on the staged mismatches"
|
||
);
|
||
assert_eq!(
|
||
store_stage_file_count(root.path()),
|
||
0,
|
||
"the discarded stages left zero stage files"
|
||
);
|
||
}
|
||
|
||
/// mem-only accepts every length on both put paths — no dispatch
|
||
/// threshold in effect (ADR-011 §4), nothing to reject; the sole
|
||
/// mem engine holds even over-default-threshold content.
|
||
#[tokio::test]
|
||
async fn mem_only_accepts_every_length_on_both_put_paths() {
|
||
let store = Store::mem_only();
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
assert_eq!(store.threshold(), None, "no threshold in effect");
|
||
|
||
let small = b"mem-only content";
|
||
let (key, pin) = store
|
||
.put(Some(16), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("known put");
|
||
assert_eq!(key, key_for(small));
|
||
drop(pin);
|
||
|
||
let big = vec![1u8; (DEFAULT_DISPATCH_THRESHOLD + 1) as usize];
|
||
let (key, pin) = store
|
||
.put(None, Cursor::new(&big[..]))
|
||
.await
|
||
.expect("over-default-threshold unknown put routes to the sole kv engine");
|
||
assert_eq!(key, key_for(&big));
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("entry present");
|
||
assert_eq!(len, big.len() as u64);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), big);
|
||
drop(pin);
|
||
}
|
||
|
||
/// The zero-length degenerate on both put paths: empty content
|
||
/// round-trips through its canonical key (one entry, both paths
|
||
/// dedup into it), reading empty-never-error.
|
||
#[tokio::test]
|
||
async fn zero_length_puts_round_trip_on_both_paths() {
|
||
let root = temp_root("put-empty");
|
||
let store = Store::dual_tier_mem_local(root.path(), 8)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
let empty = b"";
|
||
|
||
let (known, pin) = store
|
||
.put(Some(0), Cursor::new(&empty[..]))
|
||
.await
|
||
.expect("known empty put");
|
||
assert_eq!(known, key_for(empty));
|
||
assert!(store.kv().has(&known.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
|
||
let (unknown, pin) = store
|
||
.put(None, Cursor::new(&empty[..]))
|
||
.await
|
||
.expect("unknown empty put");
|
||
assert_eq!(unknown, known, "empty content dedups into one entry");
|
||
assert_eq!(store.list().await.unwrap(), vec![known.clone()]);
|
||
drop(pin);
|
||
|
||
let (len, blob) = store
|
||
.get(&known.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("entry present");
|
||
assert_eq!(len, 0);
|
||
assert!(blob.is_empty());
|
||
assert_eq!(blob.read_at(0, 8).await.unwrap(), Vec::<u8>::new());
|
||
}
|
||
}
|
||
|
||
/// The wave-3 constructor wiring's tests (feature `sqlite`): the
|
||
/// sqlite modes construct and introspect correctly, the GC schema ran
|
||
/// at construction, the window-delay default is the pinned constant
|
||
/// and is constructor-tunable, the kv-only rejection wiring is the
|
||
/// mode's put seam unchanged, and the default-build production shape
|
||
/// (kv = sqlite + large = local) round-trips end-to-end through both
|
||
/// tiers. All-features builds share this module (mem on alongside):
|
||
/// the engine accessors carry cfg arms for that.
|
||
#[cfg(all(test, feature = "sqlite"))]
|
||
mod sqlite_construction_tests {
|
||
use super::*;
|
||
use std::fs;
|
||
use std::io::Cursor;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use std::time::{SystemTime, UNIX_EPOCH};
|
||
|
||
/// A fresh temp database file (and sibling stage/root dir) per
|
||
/// test; Drop removes the file tree and the WAL sidecars
|
||
/// idempotently (the sqlite engine's test helper's shape).
|
||
struct TempSite {
|
||
database: PathBuf,
|
||
root: PathBuf,
|
||
}
|
||
|
||
impl TempSite {
|
||
fn path(&self) -> &Path {
|
||
&self.root
|
||
}
|
||
|
||
fn drop_sidecar(&self, suffix: &str) {
|
||
let base = self.database.to_string_lossy().into_owned();
|
||
let _ = fs::remove_file(format!("{base}{suffix}"));
|
||
}
|
||
}
|
||
|
||
impl Drop for TempSite {
|
||
fn drop(&mut self) {
|
||
self.drop_sidecar("-wal");
|
||
self.drop_sidecar("-shm");
|
||
let _ = fs::remove_file(&self.database);
|
||
let _ = fs::remove_dir_all(&self.root);
|
||
}
|
||
}
|
||
|
||
fn temp_site(tag: &str) -> TempSite {
|
||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let nanos = SystemTime::now()
|
||
.duration_since(UNIX_EPOCH)
|
||
.map(|d| d.as_nanos())
|
||
.unwrap_or(0);
|
||
let base = std::env::temp_dir().join(format!(
|
||
"alkblobs-store-sqlite-{tag}-{}-{nanos:x}-{seq}",
|
||
std::process::id()
|
||
));
|
||
fs::create_dir_all(&base).expect("temp site created");
|
||
TempSite {
|
||
database: base.join("pool.db"),
|
||
root: base.join("local-root"),
|
||
}
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
/// The sqlite kv engine behind the constructed store — the schema
|
||
/// observations' instrument.
|
||
fn sqlite_engine(store: &Store) -> &SqliteBackend {
|
||
match store.kv() {
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine,
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct sqlite stores"),
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(_) => panic!("these tests construct sqlite stores"),
|
||
}
|
||
}
|
||
|
||
/// The engine-neutral GC-schema call ran at construction: the pool
|
||
/// table exists (the engine's own open), `gc_window` exists (the
|
||
/// store-core call), `gc_pins` never (sqlite's applicability row,
|
||
/// ADR-012 §5's one-window-host rule — the kv engine hosts the
|
||
/// window and nothing else).
|
||
async fn assert_gc_schema_ran(store: &Store) {
|
||
let engine = sqlite_engine(store);
|
||
assert!(
|
||
engine.table_exists("blobs").await.unwrap(),
|
||
"the pool table exists at construction"
|
||
);
|
||
assert!(
|
||
engine.table_exists("gc_window").await.unwrap(),
|
||
"ensure_gc_schema ran at construction: the window table exists in the fresh database"
|
||
);
|
||
assert!(
|
||
!engine.table_exists("gc_pins").await.unwrap(),
|
||
"gc_pins is never created on sqlite (ADR-012 §5)"
|
||
);
|
||
}
|
||
|
||
/// The dual-tier construction: both tiers present, mode/threshold
|
||
/// introspection exact, boundary routing deterministic, and the
|
||
/// GC schema ran.
|
||
#[tokio::test]
|
||
async fn dual_tier_sqlite_local_constructs_and_routes() {
|
||
let site = temp_site("dual-tier");
|
||
let store = Store::dual_tier_sqlite_local(
|
||
&site.database,
|
||
site.path(),
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("dual-tier sqlite+local constructs");
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"length {len} routes deterministically"
|
||
);
|
||
}
|
||
|
||
assert_gc_schema_ran(&store).await;
|
||
}
|
||
|
||
/// The kv-only construction: no large tier, the threshold is the
|
||
/// rejection point's carrier, and the rejection wiring is the put
|
||
/// seam's unchanged — known-length over-threshold rejects
|
||
/// immediately (zero residue), unknown-length at mid-stream
|
||
/// overflow, and exactly-at-threshold puts land.
|
||
#[tokio::test]
|
||
async fn kv_only_sqlite_constructs_and_rejects_over_threshold_puts() {
|
||
let site = temp_site("kv-only");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, DEFAULT_DELETE_WINDOW_DELAY)
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
assert_eq!(store.mode(), KvOnly);
|
||
assert_eq!(store.threshold(), Some(64));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_none(), "kv-only has no large tier");
|
||
assert_gc_schema_ran(&store).await;
|
||
|
||
for len in [0u64, 63, 64, 65, u64::MAX] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
Tier::Kv,
|
||
"kv-only routes every length kv"
|
||
);
|
||
}
|
||
|
||
let err = store
|
||
.put(Some(65), Cursor::new(&b"never read"[..]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string().contains("kv-only mode rejects"),
|
||
"the immediate rejection is the mode's: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
|
||
let streamed = [3u8; 100];
|
||
let err = store
|
||
.put(None, Cursor::new(&streamed[..]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string()
|
||
.contains("kv-only mode rejects an over-threshold put"),
|
||
"the overflow rejection is the mode's, not the stream's: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
|
||
let at_threshold = vec![9u8; 64];
|
||
let (key, pin) = store
|
||
.put(Some(64), Cursor::new(&at_threshold[..]))
|
||
.await
|
||
.expect("exactly-at-threshold known put lands");
|
||
assert_eq!(key, key_for(&at_threshold));
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
}
|
||
|
||
/// The window-delay parameter: the shipped default is the pinned
|
||
/// 5-minute constant, the constructor's value passes through the
|
||
/// store's introspection, and `Duration::ZERO` is a legal
|
||
/// degenerate tuning (execute-on-next-sweep).
|
||
#[tokio::test]
|
||
async fn window_delay_default_is_pinned_and_constructor_tunable() {
|
||
assert_eq!(
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::from_secs(5 * 60),
|
||
"the shipped default is 5 minutes (ADR-008 §2)"
|
||
);
|
||
|
||
let site = temp_site("delay-default");
|
||
let store = Store::dual_tier_sqlite_local(
|
||
&site.database,
|
||
site.path(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("constructs at the shipped default");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(300));
|
||
|
||
let site = temp_site("delay-tuned");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::from_secs(90))
|
||
.await
|
||
.expect("constructs at a custom tuning");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(90));
|
||
|
||
let site = temp_site("delay-zero");
|
||
let store = Store::dual_tier_sqlite_local(&site.database, site.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("the degenerate zero tuning is legal");
|
||
assert_eq!(store.window_delay(), Duration::ZERO);
|
||
|
||
let site = temp_site("delay-long");
|
||
Store::kv_only_sqlite(&site.database, 64, Duration::from_secs(6000))
|
||
.await
|
||
.expect("a long delay is a legal tuning too");
|
||
}
|
||
|
||
/// The default-build production shape round-trips end-to-end: a
|
||
/// sub-threshold put lands on the kv tier (sqlite), an
|
||
/// over-threshold put stages into the large tier (local), both
|
||
/// get back byte-identical, and the pool list is the sorted union
|
||
/// of both tiers.
|
||
#[tokio::test]
|
||
async fn dual_tier_sqlite_shape_round_trips_end_to_end_through_both_tiers() {
|
||
let site = temp_site("round-trip");
|
||
let store = Store::dual_tier_sqlite_local(&site.database, site.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
|
||
let small = b"a sub-threshold blob";
|
||
let (key, pin) = store
|
||
.put(Some(small.len() as u64), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("kv put publishes");
|
||
assert_eq!(key, key_for(small));
|
||
assert!(
|
||
store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the kv tier (sqlite)"
|
||
);
|
||
assert!(
|
||
!store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"no large-tier copy on the kv route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("entry present");
|
||
assert_eq!(len, small.len() as u64);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), &small[..]);
|
||
drop(pin);
|
||
|
||
let big = vec![42u8; 200];
|
||
let (key, pin) = store
|
||
.put(Some(200), Cursor::new(&big[..]))
|
||
.await
|
||
.expect("large staged put publishes");
|
||
assert_eq!(key, key_for(&big));
|
||
assert!(
|
||
store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the large tier (commit-rename)"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"no kv copy on the large route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("entry present");
|
||
assert_eq!(len, 200);
|
||
assert_eq!(blob.read_at(0, len).await.unwrap(), big);
|
||
drop(pin);
|
||
|
||
let listed = store.list().await.expect("list succeeds");
|
||
assert_eq!(listed.len(), 2, "one entry per tier, both enumerated");
|
||
for window in listed.windows(2) {
|
||
assert!(window[0] < window[1], "list is sorted");
|
||
}
|
||
|
||
// A re-put dedups into the existing entry (CAS first-writer-
|
||
// wins, through the facade's put).
|
||
let (_, pin) = store
|
||
.put(Some(small.len() as u64), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("dedup re-put");
|
||
assert_eq!(store.list().await.unwrap().len(), 2, "still two entries");
|
||
drop(pin);
|
||
}
|
||
}
|
||
|
||
/// The sweep's durable form (feature `sqlite`): the staged
|
||
/// re-arbitrated delete (ADR-008 §2; ADR-012 §5/§6.7) — the first
|
||
/// durable delete window, single-node — beside the inline form. The
|
||
/// two-sweep exact-count shapes, the race across the delay, the
|
||
/// vanished-candidate form, the abort shapes, and the delay knob's
|
||
/// staging semantics.
|
||
#[cfg(all(test, feature = "sqlite"))]
|
||
mod sqlite_window_tests {
|
||
use super::*;
|
||
use crate::backend::Backend;
|
||
use std::fs;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use std::time::{SystemTime, UNIX_EPOCH};
|
||
|
||
/// A fresh temp site per test (the construction tests' helper
|
||
/// shape); Drop removes the file tree and the WAL sidecars.
|
||
struct TempSite {
|
||
database: PathBuf,
|
||
root: PathBuf,
|
||
}
|
||
|
||
impl TempSite {
|
||
fn path(&self) -> &Path {
|
||
&self.root
|
||
}
|
||
|
||
fn drop_sidecar(&self, suffix: &str) {
|
||
let base = self.database.to_string_lossy().into_owned();
|
||
let _ = fs::remove_file(format!("{base}{suffix}"));
|
||
}
|
||
}
|
||
|
||
impl Drop for TempSite {
|
||
fn drop(&mut self) {
|
||
self.drop_sidecar("-wal");
|
||
self.drop_sidecar("-shm");
|
||
let _ = fs::remove_file(&self.database);
|
||
let _ = fs::remove_dir_all(&self.root);
|
||
}
|
||
}
|
||
|
||
fn temp_site(tag: &str) -> TempSite {
|
||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let nanos = SystemTime::now()
|
||
.duration_since(UNIX_EPOCH)
|
||
.map(|d| d.as_nanos())
|
||
.unwrap_or(0);
|
||
let base = std::env::temp_dir().join(format!(
|
||
"alkblobs-store-window-{tag}-{}-{nanos:x}-{seq}",
|
||
std::process::id()
|
||
));
|
||
fs::create_dir_all(&base).expect("temp site created");
|
||
TempSite {
|
||
database: base.join("pool.db"),
|
||
root: base.join("local-root"),
|
||
}
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
fn sqlite_engine(store: &Store) -> &SqliteBackend {
|
||
match store.kv() {
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct sqlite stores"),
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(_) => panic!("these tests construct sqlite stores"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(engine) => engine,
|
||
}
|
||
}
|
||
|
||
async fn plant_kv(store: &Store, key: &Key, value: &[u8]) {
|
||
sqlite_engine(store)
|
||
.put(&key.as_bytes(), value)
|
||
.await
|
||
.expect("kv plant");
|
||
}
|
||
|
||
async fn plant_large(store: &Store, key: &Key, value: &[u8]) {
|
||
match store.large() {
|
||
Some(LargeEngine::Local(engine)) => engine
|
||
.put(&key.as_bytes(), value)
|
||
.await
|
||
.expect("large plant"),
|
||
#[cfg(feature = "pg-lo")]
|
||
Some(LargeEngine::PgLo(_)) => panic!("these tests construct local-tier stores"),
|
||
None => panic!("planting into a mode with no large tier"),
|
||
}
|
||
}
|
||
|
||
/// A set-backed liveness source — the live-shared seam, shared
|
||
/// with the wave-2 tests' fixture shape.
|
||
#[derive(Default)]
|
||
struct SetSource {
|
||
live: parking_lot::Mutex<BTreeSet<Key>>,
|
||
}
|
||
|
||
impl SetSource {
|
||
fn add(&self, key: Key) {
|
||
self.live.lock().insert(key);
|
||
}
|
||
}
|
||
|
||
impl LivenessSource for SetSource {
|
||
fn live_keys(&self) -> Vec<Key> {
|
||
self.live.lock().iter().cloned().collect()
|
||
}
|
||
|
||
fn contains(&self, key: &Key) -> bool {
|
||
self.live.lock().contains(key)
|
||
}
|
||
}
|
||
|
||
/// A set-backed protect callback; `None` models the
|
||
/// failed-to-answer case.
|
||
#[derive(Default)]
|
||
struct SetProtect {
|
||
set: parking_lot::Mutex<Option<Vec<Key>>>,
|
||
}
|
||
|
||
impl SetProtect {
|
||
fn set_keys(&self, keys: Vec<Key>) {
|
||
*self.set.lock() = Some(keys);
|
||
}
|
||
|
||
fn fail(&self) {
|
||
*self.set.lock() = None;
|
||
}
|
||
}
|
||
|
||
impl ProtectCallback for SetProtect {
|
||
fn protect(&self) -> Option<Vec<Key>> {
|
||
self.set.lock().clone()
|
||
}
|
||
}
|
||
|
||
/// A protect callback that blocks mid-consult — the contention
|
||
/// window for the single-flight assert on a sqlite store.
|
||
struct LatchProtect {
|
||
entered: std::sync::mpsc::Sender<()>,
|
||
release: parking_lot::Mutex<Option<std::sync::mpsc::Receiver<()>>>,
|
||
}
|
||
|
||
impl ProtectCallback for LatchProtect {
|
||
fn protect(&self) -> Option<Vec<Key>> {
|
||
self.entered.send(()).ok();
|
||
let receiver = self.release.lock().take();
|
||
if let Some(receiver) = receiver {
|
||
let _ = receiver.recv();
|
||
}
|
||
Some(Vec::new())
|
||
}
|
||
}
|
||
|
||
/// A non-destructive read of the window is not on the seam — but a
|
||
/// far-horizon `window_take` drains it entirely, which is exactly
|
||
/// the observation the assertions below need (and leaves the
|
||
/// window clean for the next step of the test).
|
||
async fn window_drain(store: &Store) -> Vec<Key> {
|
||
let far = SystemTime::now()
|
||
.checked_add(Duration::from_secs(365 * 24 * 3600))
|
||
.unwrap();
|
||
let raws = sqlite_engine(store).window_take(far).await.unwrap();
|
||
typed_keys(raws).expect("the window holds typed keys")
|
||
}
|
||
|
||
/// The two-sweep exact-count shape at the degenerate delay (the
|
||
/// test tuning, ADR-008 §2): sweep 1 stages N — nothing deleted,
|
||
/// the pool intact; sweep 2 executes exactly N —
|
||
/// `deleted + cancelled == N` against the staged set, staged-this-
|
||
/// sweep and executed-this-sweep different sets, and the window
|
||
/// left empty.
|
||
#[tokio::test]
|
||
async fn durable_two_sweep_exact_counts_at_delay_zero() {
|
||
let site = temp_site("two-sweep");
|
||
let store = Store::dual_tier_sqlite_local(&site.database, site.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("dual-tier sqlite+local constructs");
|
||
let source = Arc::new(SetSource::default());
|
||
store.register_liveness_source(source.clone());
|
||
|
||
let live = key_for(b"kept by the registered source");
|
||
plant_kv(&store, &live, b"kept by the registered source").await;
|
||
source.add(live.clone());
|
||
|
||
let dead_kv = key_for(b"dead kv entry in the window");
|
||
plant_kv(&store, &dead_kv, b"dead kv entry in the window").await;
|
||
let dead_large = key_for(b"dead large entry in the window");
|
||
plant_large(&store, &dead_large, b"dead large entry in the window").await;
|
||
|
||
let first = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(
|
||
first.candidates_staged, 2,
|
||
"both tiers' dead entries staged"
|
||
);
|
||
assert_eq!(
|
||
first.deleted, 0,
|
||
"staging is not executing — the delay separates"
|
||
);
|
||
assert_eq!(first.cancelled_by_arbitration, 0);
|
||
|
||
assert!(
|
||
store.has(&dead_kv.as_bytes()).await.unwrap(),
|
||
"sweep 1 staged, did not execute: the kv entry is intact"
|
||
);
|
||
assert!(
|
||
store.has(&dead_large.as_bytes()).await.unwrap(),
|
||
"sweep 1 staged, did not execute: the large entry is intact"
|
||
);
|
||
assert!(store.has(&live.as_bytes()).await.unwrap());
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(second.aborted, None);
|
||
assert_eq!(second.candidates_staged, 0, "nothing newly dead");
|
||
assert_eq!(
|
||
second.deleted + second.cancelled_by_arbitration,
|
||
2,
|
||
"sweep 2 executed exactly the staged set"
|
||
);
|
||
assert_eq!(second.deleted, 2);
|
||
assert_eq!(second.cancelled_by_arbitration, 0);
|
||
|
||
assert!(!store.has(&dead_kv.as_bytes()).await.unwrap());
|
||
assert!(!store.has(&dead_large.as_bytes()).await.unwrap());
|
||
assert!(store.has(&live.as_bytes()).await.unwrap());
|
||
assert!(
|
||
window_drain(&store).await.is_empty(),
|
||
"the executed rows left the window"
|
||
);
|
||
}
|
||
|
||
/// The delay knob is the staging's cadence (ADR-008 §2): at a
|
||
/// future-facing delay the staged rows are not overdue at
|
||
/// `window_take(now)` — no execution, no deletion — and a repeated
|
||
/// sweep re-stages the still-dead set without duplicating rows.
|
||
#[tokio::test]
|
||
async fn durable_stage_at_a_future_delay_waits_past_the_window() {
|
||
let site = temp_site("future-delay");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::from_secs(600))
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let dead_a = key_for(b"dead under a future-facing delay a");
|
||
plant_kv(&store, &dead_a, b"dead under a future-facing delay a").await;
|
||
let dead_b = key_for(b"dead under a future-facing delay b");
|
||
plant_kv(&store, &dead_b, b"dead under a future-facing delay b").await;
|
||
|
||
let first = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(first.candidates_staged, 2);
|
||
assert_eq!(first.deleted, 0, "nothing is overdue at staging time");
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(second.candidates_staged, 2, "still dead, re-staged");
|
||
assert_eq!(
|
||
second.deleted, 0,
|
||
"the window delay pushed execution out again"
|
||
);
|
||
|
||
let drained = window_drain(&store).await;
|
||
let mut expected = vec![dead_a.clone(), dead_b.clone()];
|
||
expected.sort();
|
||
assert_eq!(drained, expected);
|
||
assert!(store.has(&dead_a.as_bytes()).await.unwrap());
|
||
assert!(store.has(&dead_b.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// The race across the delay (the window form of ADR-005's
|
||
/// test-asserted property): a put landing during the window (the
|
||
/// pin minted between the staging sweep and the executor sweep,
|
||
/// pin-before-publish) survives — the execution-time re-arbitration
|
||
/// cancels its candidate — while the unpinned twin deletes. The
|
||
/// cancelled candidate re-stages (and executes) only once its
|
||
/// protection drops.
|
||
#[tokio::test]
|
||
async fn durable_race_across_the_delay_put_during_window_survives() {
|
||
let site = temp_site("race-across-delay");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::ZERO)
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let pinned = key_for(b"the twin a put lands under during the window");
|
||
plant_kv(
|
||
&store,
|
||
&pinned,
|
||
b"the twin a put lands under during the window",
|
||
)
|
||
.await;
|
||
let unpinned = key_for(b"the unpinned twin");
|
||
plant_kv(&store, &unpinned, b"the unpinned twin").await;
|
||
|
||
let staged = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(staged.candidates_staged, 2);
|
||
assert_eq!(staged.deleted, 0);
|
||
|
||
{
|
||
let _pin = store.gc().acquire(&pinned).await;
|
||
assert_eq!(store.gc().pin_count(&pinned), 1, "pin-before-publish");
|
||
|
||
let executed = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(executed.aborted, None);
|
||
assert_eq!(
|
||
executed.deleted + executed.cancelled_by_arbitration,
|
||
2,
|
||
"execution-time re-arbitration ran over the taken set"
|
||
);
|
||
assert_eq!(executed.deleted, 1, "the unpinned twin deleted");
|
||
assert_eq!(
|
||
executed.cancelled_by_arbitration, 1,
|
||
"the pinned candidate cancelled"
|
||
);
|
||
assert!(
|
||
store.has(&pinned.as_bytes()).await.unwrap(),
|
||
"a put landing during the window survives the next sweep's execution"
|
||
);
|
||
assert!(!store.has(&unpinned.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
// The pin dropped: the still-present candidate re-stages, and
|
||
// then (and only then) executes — its window row was consumed
|
||
// by the cancellation, not by a delete.
|
||
let restaged = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
restaged.candidates_staged, 1,
|
||
"unpinned, the candidate re-stages"
|
||
);
|
||
assert_eq!(restaged.deleted, 0);
|
||
|
||
let executed = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(executed.deleted, 1, "the re-staged candidate executes");
|
||
assert!(!store.has(&pinned.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// The vanished-candidate form: a staged candidate deleted during
|
||
/// the delay by a direct delete settles into
|
||
/// `cancelled_by_arbitration` at execution time — the vanished-
|
||
/// candidate accounting carries over (nothing to delete, counts
|
||
/// stay exact).
|
||
#[tokio::test]
|
||
async fn durable_vanished_candidate_settles_into_cancelled() {
|
||
let site = temp_site("vanished");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::ZERO)
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let victim = key_for(b"deleted during the delay");
|
||
plant_kv(&store, &victim, b"deleted during the delay").await;
|
||
|
||
let staged = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(staged.candidates_staged, 1);
|
||
|
||
store
|
||
.delete(&victim.as_bytes())
|
||
.await
|
||
.expect("the direct delete lands on the unprotected entry");
|
||
|
||
let executed = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(executed.aborted, None);
|
||
assert_eq!(executed.candidates_staged, 0);
|
||
assert_eq!(executed.deleted, 0);
|
||
assert_eq!(
|
||
executed.cancelled_by_arbitration, 1,
|
||
"the vanished candidate settles into the cancelled count"
|
||
);
|
||
assert!(window_drain(&store).await.is_empty());
|
||
}
|
||
|
||
/// The abort shapes on a sqlite store delete nothing, execute
|
||
/// nothing overdue, and stage nothing: zero sources
|
||
/// (`NoLivenessSources`), a failing protect callback
|
||
/// (`ProtectFailed`), and lease contention (`SweeperLock`).
|
||
#[tokio::test]
|
||
async fn durable_aborts_delete_nothing_and_execute_nothing() {
|
||
let site = temp_site("aborts-sources");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::ZERO)
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
let dead = key_for(b"dead while no sources are registered");
|
||
plant_kv(&store, &dead, b"dead while no sources are registered").await;
|
||
|
||
let aborted = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(aborted.aborted, Some(GcAbortCause::NoLivenessSources));
|
||
assert_eq!(aborted.candidates_staged, 0);
|
||
assert_eq!(aborted.deleted, 0);
|
||
assert!(store.has(&dead.as_bytes()).await.unwrap());
|
||
assert!(window_drain(&store).await.is_empty(), "nothing staged");
|
||
}
|
||
|
||
/// `ProtectFailed` aborts before the executor step on the durable
|
||
/// form: an overdue row staged by an earlier sweep is not taken
|
||
/// while the protection consult fails — abort precedes execution.
|
||
#[tokio::test]
|
||
async fn durable_protect_failed_aborts_before_executing_the_overdue_set() {
|
||
let site = temp_site("aborts-protect");
|
||
let store = Store::kv_only_sqlite(&site.database, 64, Duration::ZERO)
|
||
.await
|
||
.expect("kv-only sqlite constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
let protect = Arc::new(SetProtect::default());
|
||
store.register_protect_callback(protect.clone());
|
||
|
||
let staged_dead = key_for(b"staged dead while the consult answers");
|
||
plant_kv(
|
||
&store,
|
||
&staged_dead,
|
||
b"staged dead while the consult answers",
|
||
)
|
||
.await;
|
||
protect.set_keys(Vec::new());
|
||
let first = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(first.candidates_staged, 1);
|
||
|
||
protect.fail();
|
||
let later_dead = key_for(b"dead while the consult fails");
|
||
plant_kv(&store, &later_dead, b"dead while the consult fails").await;
|
||
|
||
let aborted = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(aborted.aborted, Some(GcAbortCause::ProtectFailed));
|
||
assert_eq!(aborted.candidates_staged, 0);
|
||
assert_eq!(aborted.deleted, 0);
|
||
assert_eq!(aborted.cancelled_by_arbitration, 0);
|
||
assert!(
|
||
store.has(&staged_dead.as_bytes()).await.unwrap(),
|
||
"the overdue candidate was not executed past the abort"
|
||
);
|
||
assert!(
|
||
store.has(&later_dead.as_bytes()).await.unwrap(),
|
||
"and nothing was deleted"
|
||
);
|
||
assert_eq!(
|
||
window_drain(&store).await,
|
||
vec![staged_dead],
|
||
"the overdue row still waits in the window"
|
||
);
|
||
}
|
||
|
||
/// Contention on the single-node lease aborts the second sweeper
|
||
/// as report data on the durable form too — nothing executed,
|
||
/// nothing staged, nothing deleted (`Ok(SweepReport)`, ADR-012 §1).
|
||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||
async fn durable_lease_contention_aborts_the_second_sweeper_as_data() {
|
||
let site = temp_site("aborts-lease");
|
||
let store = Arc::new(
|
||
Store::kv_only_sqlite(&site.database, 64, Duration::ZERO)
|
||
.await
|
||
.expect("kv-only sqlite constructs"),
|
||
);
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
|
||
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
|
||
store.register_protect_callback(Arc::new(LatchProtect {
|
||
entered: entered_tx,
|
||
release: parking_lot::Mutex::new(Some(release_rx)),
|
||
}));
|
||
|
||
let dead = key_for(b"dead under the contended sweeps");
|
||
plant_kv(&store, &dead, b"dead under the contended sweeps").await;
|
||
|
||
let first = {
|
||
let store = Arc::clone(&store);
|
||
tokio::spawn(async move { store.sweep().await })
|
||
};
|
||
|
||
while entered_rx
|
||
.recv_timeout(std::time::Duration::from_millis(500))
|
||
.is_err()
|
||
{
|
||
tokio::task::yield_now().await;
|
||
}
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(second.aborted, Some(GcAbortCause::SweeperLock));
|
||
assert_eq!(second.deleted, 0);
|
||
assert_eq!(second.candidates_staged, 0, "nothing staged");
|
||
assert!(store.has(&dead.as_bytes()).await.unwrap());
|
||
assert!(window_drain(&store).await.is_empty(), "nothing executed");
|
||
|
||
release_tx.send(()).ok();
|
||
let first = first
|
||
.await
|
||
.expect("first sweep task joined")
|
||
.expect("first sweep returns a report");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(first.candidates_staged, 1, "the lease holder stages");
|
||
assert_eq!(first.deleted, 0, "and executes nothing at staging time");
|
||
assert!(
|
||
store.has(&dead.as_bytes()).await.unwrap(),
|
||
"the lease holder staged, did not execute"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The wave-4 constructor wiring's tests (feature `postgres`): the
|
||
/// postgres modes construct and introspect correctly, the GC schema
|
||
/// ran at construction (both GC tables in the fresh schema — the
|
||
/// fleet host's both-tables row), the window-delay default is the
|
||
/// pinned constant and is constructor-tunable, the kv-only rejection
|
||
/// wiring is the mode's put seam unchanged, the fleet-capable-kv
|
||
/// shape (kv = postgres + large = local) round-trips end-to-end
|
||
/// through both tiers, and the sweep's form branch runs the durable
|
||
/// form on the postgres host. Env-driven — the engine tests'
|
||
/// convention — skipping cleanly server-less.
|
||
#[cfg(all(test, feature = "postgres"))]
|
||
mod postgres_construction_tests {
|
||
use super::*;
|
||
use crate::postgres::test_support::SchemaJanitor;
|
||
use std::fs;
|
||
use std::io::Cursor;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use std::time::{SystemTime, UNIX_EPOCH};
|
||
|
||
/// A fresh temp root per test (mirrors the mem construction
|
||
/// tests' helper); Drop removes the tree idempotently.
|
||
struct TempRoot(PathBuf);
|
||
|
||
fn temp_root(tag: &'static str) -> TempRoot {
|
||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let nanos = SystemTime::now()
|
||
.duration_since(UNIX_EPOCH)
|
||
.map(|d| d.as_nanos())
|
||
.unwrap_or(0);
|
||
let dir = std::env::temp_dir().join(format!(
|
||
"alkblobs-store-pg-{tag}-{}-{:x}-{seq}",
|
||
std::process::id(),
|
||
nanos
|
||
));
|
||
fs::create_dir_all(&dir).expect("temp root created");
|
||
TempRoot(dir)
|
||
}
|
||
|
||
impl TempRoot {
|
||
fn path(&self) -> &Path {
|
||
&self.0
|
||
}
|
||
}
|
||
|
||
impl Drop for TempRoot {
|
||
fn drop(&mut self) {
|
||
let _ = fs::remove_dir_all(&self.0);
|
||
}
|
||
}
|
||
|
||
/// A fresh engine-owned schema per test — the postgres suite
|
||
/// factory's fresh-schema-per-open convention — so the GC-schema
|
||
/// assertions observe genuine creation.
|
||
static SCHEMA_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
|
||
fn fresh_schema(tag: &str) -> String {
|
||
format!(
|
||
"abt_sctor_{}_{}_{}",
|
||
tag,
|
||
std::process::id(),
|
||
SCHEMA_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
)
|
||
}
|
||
|
||
fn env_nonempty(name: &str) -> Option<String> {
|
||
std::env::var(name).ok().filter(|v| !v.is_empty())
|
||
}
|
||
|
||
/// The harness connection settings ride the environment
|
||
/// (`ALKBLOBS_PG_HOST` required; port/user/password/db default to
|
||
/// the harness server's values). Absent → the caller skips.
|
||
fn harness_dsn() -> Option<String> {
|
||
let host = env_nonempty("ALKBLOBS_PG_HOST")?;
|
||
Some(format!(
|
||
"host={host} port={} user={} password={} dbname={}",
|
||
env_nonempty("ALKBLOBS_PG_PORT").unwrap_or_else(|| "15432".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_USER").unwrap_or_else(|| "postgres".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_PASSWORD").unwrap_or_else(|| "poc".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_DB").unwrap_or_else(|| "blobs".to_owned()),
|
||
))
|
||
}
|
||
|
||
/// The skip note — printed exactly once per skipped test.
|
||
fn skip_note() {
|
||
eprintln!(
|
||
"skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB"
|
||
);
|
||
}
|
||
|
||
/// A fresh-schema opts for the harness server — with the schema
|
||
/// janitor that drops it at test end (success, early return, or
|
||
/// unwind) — or `None` (skip note printed) server-less.
|
||
fn fresh_opts(tag: &str) -> Option<(PgOpts, SchemaJanitor)> {
|
||
let dsn = harness_dsn()?;
|
||
let schema = fresh_schema(tag);
|
||
Some((
|
||
PgOpts {
|
||
schema: schema.clone(),
|
||
..Default::default()
|
||
},
|
||
SchemaJanitor::new(&dsn, &schema),
|
||
))
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
/// The postgres kv engine behind the constructed store — the
|
||
/// schema observations' instrument.
|
||
fn postgres_engine(store: &Store) -> &PostgresBackend {
|
||
match store.kv() {
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine,
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct postgres stores"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(_) => panic!("these tests construct postgres stores"),
|
||
}
|
||
}
|
||
|
||
/// The engine-neutral GC-schema call ran at construction on the
|
||
/// fleet host's applicability row: the pool table exists (the
|
||
/// engine's own bootstrap) and *both* GC tables exist in the
|
||
/// fresh schema (postgres's both-tables row, ADR-012 §5).
|
||
async fn assert_gc_schema_ran(store: &Store) {
|
||
let engine = postgres_engine(store);
|
||
assert!(
|
||
engine.table_exists("blobs").await.unwrap(),
|
||
"the pool table exists at construction"
|
||
);
|
||
assert!(
|
||
engine.table_exists("gc_window").await.unwrap(),
|
||
"ensure_gc_schema ran at construction: the window table exists in the fresh schema"
|
||
);
|
||
assert!(
|
||
engine.table_exists("gc_pins").await.unwrap(),
|
||
"ensure_gc_schema ran at construction: the pins table exists (the fleet host's both-tables row)"
|
||
);
|
||
}
|
||
|
||
/// The dual-tier construction: both tiers present, mode/threshold/
|
||
/// window-delay introspection exact, boundary routing
|
||
/// deterministic, and the GC schema ran (both GC tables in the
|
||
/// fresh schema).
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_local_constructs_and_routes() {
|
||
let Some((opts, _janitor)) = fresh_opts("dual-tier") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let root = temp_root("dual-tier");
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let store = Store::dual_tier_postgres_local(
|
||
&dsn,
|
||
opts,
|
||
root.path(),
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("dual-tier postgres+local constructs");
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"length {len} routes deterministically"
|
||
);
|
||
}
|
||
|
||
assert_gc_schema_ran(&store).await;
|
||
}
|
||
|
||
/// The kv-only construction: no large tier, the threshold is the
|
||
/// rejection point's carrier, and the rejection wiring is the put
|
||
/// seam's unchanged — known-length over-threshold rejects
|
||
/// immediately (zero residue), unknown-length at mid-stream
|
||
/// overflow, and exactly-at-threshold puts land.
|
||
#[tokio::test]
|
||
async fn kv_only_postgres_constructs_and_rejects_over_threshold_puts() {
|
||
let Some((opts, _janitor)) = fresh_opts("kv-only") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let store = Store::kv_only_postgres(&dsn, opts, 64, DEFAULT_DELETE_WINDOW_DELAY)
|
||
.await
|
||
.expect("kv-only postgres constructs");
|
||
assert_eq!(store.mode(), KvOnly);
|
||
assert_eq!(store.threshold(), Some(64));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_none(), "kv-only has no large tier");
|
||
assert_gc_schema_ran(&store).await;
|
||
|
||
for len in [0u64, 63, 64, 65, u64::MAX] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
Tier::Kv,
|
||
"kv-only routes every length kv"
|
||
);
|
||
}
|
||
|
||
let err = store
|
||
.put(Some(65), Cursor::new(&b"never read"[..]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string().contains("kv-only mode rejects"),
|
||
"the immediate rejection is the mode's: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
|
||
let streamed = [3u8; 100];
|
||
let err = store
|
||
.put(None, Cursor::new(&streamed[..]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string()
|
||
.contains("kv-only mode rejects an over-threshold put"),
|
||
"the overflow rejection is the mode's, not the stream's: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
|
||
let at_threshold = vec![9u8; 64];
|
||
let (key, pin) = store
|
||
.put(Some(64), Cursor::new(&at_threshold[..]))
|
||
.await
|
||
.expect("exactly-at-threshold known put lands");
|
||
assert_eq!(key, key_for(&at_threshold));
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
}
|
||
|
||
/// The window-delay parameter: the shipped default is the pinned
|
||
/// 5-minute constant, the constructor's value passes through the
|
||
/// store's introspection, and `Duration::ZERO` is a legal
|
||
/// degenerate tuning (execute-on-next-sweep).
|
||
#[tokio::test]
|
||
async fn window_delay_default_is_pinned_and_constructor_tunable() {
|
||
assert_eq!(
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::from_secs(5 * 60),
|
||
"the shipped default is 5 minutes (ADR-008 §2)"
|
||
);
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("delay-default") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let root = temp_root("delay-default");
|
||
let store = Store::dual_tier_postgres_local(
|
||
&dsn,
|
||
opts,
|
||
root.path(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("constructs at the shipped default");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(300));
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("delay-tuned") else {
|
||
return;
|
||
};
|
||
let store = Store::kv_only_postgres(&dsn, opts, 64, Duration::from_secs(90))
|
||
.await
|
||
.expect("constructs at a custom tuning");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(90));
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("delay-zero") else {
|
||
return;
|
||
};
|
||
let root = temp_root("delay-zero");
|
||
let store = Store::dual_tier_postgres_local(&dsn, opts, root.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("the degenerate zero tuning is legal");
|
||
assert_eq!(store.window_delay(), Duration::ZERO);
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("delay-long") else {
|
||
return;
|
||
};
|
||
Store::kv_only_postgres(&dsn, opts, 64, Duration::from_secs(6000))
|
||
.await
|
||
.expect("a long delay is a legal tuning too");
|
||
}
|
||
|
||
/// The fleet-capable-kv shape round-trips end-to-end: a
|
||
/// sub-threshold put lands on the kv tier (postgres), an
|
||
/// over-threshold put stages into the large tier (local), both
|
||
/// get back byte-identical, and the pool list is the sorted union
|
||
/// of both tiers.
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_shape_round_trips_end_to_end_through_both_tiers() {
|
||
let Some((opts, _janitor)) = fresh_opts("round-trip") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let root = temp_root("round-trip");
|
||
let store = Store::dual_tier_postgres_local(&dsn, opts, root.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
|
||
let small = b"a sub-threshold blob";
|
||
let (key, pin) = store
|
||
.put(Some(small.len() as u64), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("kv put publishes");
|
||
assert_eq!(key, key_for(small));
|
||
assert!(
|
||
store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the kv tier (postgres)"
|
||
);
|
||
assert!(
|
||
!store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"no large-tier copy on the kv route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("kv entry present");
|
||
assert_eq!(len, small.len() as u64, "whole-blob length exact");
|
||
assert_eq!(
|
||
blob.read_at(0, len).await.expect("full window"),
|
||
small,
|
||
"kv-tier round-trip byte-identical"
|
||
);
|
||
drop(pin);
|
||
|
||
let large = vec![7u8; 128];
|
||
let (key, pin) = store
|
||
.put(Some(large.len() as u64), Cursor::new(&large[..]))
|
||
.await
|
||
.expect("staged put publishes");
|
||
assert_eq!(key, key_for(&large));
|
||
assert!(
|
||
store.large().unwrap().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the large tier (local)"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"no kv-tier copy on the large route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.expect("get succeeds")
|
||
.expect("large entry present");
|
||
assert_eq!(len, large.len() as u64);
|
||
assert_eq!(
|
||
blob.read_at(0, len).await.expect("full window"),
|
||
large,
|
||
"large-tier round-trip byte-identical"
|
||
);
|
||
drop(pin);
|
||
|
||
let mut expected: Vec<Key> = [key_for(small), key_for(&large)].to_vec();
|
||
expected.sort();
|
||
assert_eq!(
|
||
store.list().await.expect("list succeeds"),
|
||
expected,
|
||
"the pool list is the sorted union of both tiers"
|
||
);
|
||
}
|
||
|
||
/// A set-backed liveness source (the construction tests' shared
|
||
/// sweep instrument).
|
||
#[derive(Default)]
|
||
struct SetSource {
|
||
live: parking_lot::Mutex<BTreeSet<Key>>,
|
||
}
|
||
|
||
impl LivenessSource for SetSource {
|
||
fn live_keys(&self) -> Vec<Key> {
|
||
self.live.lock().iter().cloned().collect()
|
||
}
|
||
|
||
fn contains(&self, key: &Key) -> bool {
|
||
self.live.lock().contains(key)
|
||
}
|
||
}
|
||
|
||
/// The sweep's form branch runs the **durable form** on the
|
||
/// postgres host (the branch keyed off the kv-engine enum,
|
||
/// ADR-012 §5's one-window-host rule): a zero-delay sweep stages
|
||
/// this sweep's dead candidates and executes none of them (the
|
||
/// inline form would delete in the same sweep), and the *next*
|
||
/// sweep executes the overdue set through time-of-delete
|
||
/// re-arbitration.
|
||
#[tokio::test]
|
||
async fn sweep_runs_the_durable_form_on_the_postgres_host() {
|
||
let source = Arc::new(SetSource::default());
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("durable-sweep") else {
|
||
drop(source);
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let root = temp_root("durable-sweep");
|
||
let store = Store::dual_tier_postgres_local(&dsn, opts, root.path(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("dual-tier constructs");
|
||
store.register_liveness_source(source.clone());
|
||
|
||
let dead = key_for(b"a dead candidate under the durable form");
|
||
store
|
||
.put(
|
||
Some(dead_content().len() as u64),
|
||
Cursor::new(&dead_content()[..]),
|
||
)
|
||
.await
|
||
.expect("the candidate's put publishes");
|
||
assert_eq!(store.list().await.unwrap(), vec![dead.clone()]);
|
||
|
||
let first = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(
|
||
first.candidates_staged, 1,
|
||
"the dead candidate staged into the durable window"
|
||
);
|
||
assert_eq!(
|
||
first.deleted, 0,
|
||
"the durable form executes nothing in its own staging sweep"
|
||
);
|
||
assert!(
|
||
store.has(&dead.as_bytes()).await.unwrap(),
|
||
"the staged candidate still waits out its window"
|
||
);
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(second.aborted, None);
|
||
assert_eq!(
|
||
second.deleted, 1,
|
||
"the next sweep executed the overdue candidate"
|
||
);
|
||
assert_eq!(
|
||
second.candidates_staged, 0,
|
||
"the pool holds nothing dead after the execution"
|
||
);
|
||
assert!(
|
||
!store.has(&dead.as_bytes()).await.unwrap(),
|
||
"the expired candidate is gone"
|
||
);
|
||
}
|
||
|
||
/// The content the durable-sweep test's candidate carries — the
|
||
/// key derives over it, the entry is genuinely dead (nothing
|
||
/// liveness-protects it; the registered source set stays empty).
|
||
fn dead_content() -> Vec<u8> {
|
||
b"a dead candidate under the durable form".to_vec()
|
||
}
|
||
}
|
||
|
||
/// The wave-5 constructor wiring's tests (features `pg-lo` + `mem`):
|
||
/// the one-window-host rule's named test topology — the durable
|
||
/// window's large host. The mode constructs and introspects correctly,
|
||
/// the GC schema ran on the **pg-lo host** at construction (`gc_window`
|
||
/// in the pg-lo schema; `gc_pins` never), the window-delay default is
|
||
/// the pinned constant and is constructor-tunable (`0` legal), the
|
||
/// shape round-trips end-to-end through both tiers with LO placement
|
||
/// asserted (kv entries never in the pg database), and the sweep's
|
||
/// form branch runs the durable form on the pg-lo host. Env-driven
|
||
/// (`ALKBLOBS_PG_*`), skipping cleanly server-less.
|
||
#[cfg(all(test, feature = "pg-lo", feature = "mem"))]
|
||
mod mem_pglo_construction_tests {
|
||
use super::*;
|
||
use crate::backend::Backend;
|
||
use crate::pg_common::test_support::DatabaseJanitor;
|
||
use crate::pg_lo::PgLoBackend;
|
||
use std::io::Cursor;
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use tokio_postgres::NoTls;
|
||
|
||
static DB_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
|
||
fn env_nonempty(name: &str) -> Option<String> {
|
||
std::env::var(name).ok().filter(|v| !v.is_empty())
|
||
}
|
||
|
||
/// The harness connection settings ride the environment
|
||
/// (`ALKBLOBS_PG_HOST` required; port/user/password/db default to
|
||
/// the harness server's values). Absent → the caller skips.
|
||
fn harness_dsn() -> Option<String> {
|
||
let host = env_nonempty("ALKBLOBS_PG_HOST")?;
|
||
Some(format!(
|
||
"host={host} port={} user={} password={} dbname={}",
|
||
env_nonempty("ALKBLOBS_PG_PORT").unwrap_or_else(|| "15432".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_USER").unwrap_or_else(|| "postgres".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_PASSWORD").unwrap_or_else(|| "poc".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_DB").unwrap_or_else(|| "blobs".to_owned()),
|
||
))
|
||
}
|
||
|
||
/// The skip note — printed exactly once per skipped test.
|
||
fn skip_note() {
|
||
eprintln!(
|
||
"skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB"
|
||
);
|
||
}
|
||
|
||
/// A fresh dedicated database per test (LOs are database-level —
|
||
/// the C7 lesson): the test's constructor opens its engine inside
|
||
/// it, and a `DatabaseJanitor` drop-guard (`WITH (FORCE)`) removes
|
||
/// the whole database at test end — success, early return, or
|
||
/// unwind — zero residue on the shared harness database.
|
||
struct TestDatabase {
|
||
/// The DSN into the dedicated database (the constructor's
|
||
/// connection settings).
|
||
dsn: String,
|
||
_janitor: DatabaseJanitor,
|
||
}
|
||
|
||
fn admin_dsn(dsn: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname=postgres")
|
||
}
|
||
|
||
fn db_dsn(dsn: &str, db: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname={db}")
|
||
}
|
||
|
||
async fn fresh_db(tag: &str) -> Option<TestDatabase> {
|
||
let dsn = harness_dsn()?;
|
||
let db = format!(
|
||
"abtmplo_{}_{}_{:x}",
|
||
tag,
|
||
std::process::id(),
|
||
DB_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
);
|
||
let admin = admin_dsn(&dsn);
|
||
{
|
||
let (client, conn) = tokio_postgres::connect(&admin, NoTls)
|
||
.await
|
||
.expect("admin connect");
|
||
tokio::spawn(async move {
|
||
let _ = conn.await;
|
||
});
|
||
client
|
||
.execute(
|
||
&format!(
|
||
"CREATE DATABASE {}",
|
||
crate::pg_common::quote_identifier(&db)
|
||
),
|
||
&[],
|
||
)
|
||
.await
|
||
.expect("create test database");
|
||
}
|
||
Some(TestDatabase {
|
||
dsn: db_dsn(&dsn, &db),
|
||
_janitor: DatabaseJanitor::new(&admin, &db),
|
||
})
|
||
}
|
||
|
||
fn pglo_engine(store: &Store) -> &PgLoBackend {
|
||
match store.large() {
|
||
Some(LargeEngine::PgLo(engine)) => engine,
|
||
Some(LargeEngine::Local(_)) | None => {
|
||
panic!("these tests construct pg-lo-tier stores")
|
||
}
|
||
}
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
/// The one-window-host rule's named test topology constructs and
|
||
/// introspects: both tiers present, the dispatch threshold active,
|
||
/// and `ensure_gc_schema` ran at construction on the **pg-lo host**
|
||
/// (`gc_window` in the pg-lo schema; `gc_pins` never — the pg-lo
|
||
/// row of ADR-012 §5's applicability table).
|
||
#[tokio::test]
|
||
async fn dual_tier_mem_pglo_constructs_and_validates() {
|
||
let Some(site) = fresh_db("mem-pglo").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_mem_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("mem+pglo constructs");
|
||
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"length {len} routes deterministically"
|
||
);
|
||
}
|
||
|
||
let lo = pglo_engine(&store);
|
||
assert!(
|
||
lo.table_exists("lo_entries").await.unwrap(),
|
||
"the pg-lo engine's companion table bootstrapped at construction"
|
||
);
|
||
assert!(
|
||
lo.table_exists("gc_window").await.unwrap(),
|
||
"ensure_gc_schema ran at construction on the pg-lo host: \
|
||
gc_window in the pg-lo schema (the one-window-host rule's \
|
||
large-host arm)"
|
||
);
|
||
assert!(
|
||
!lo.table_exists("gc_pins").await.unwrap(),
|
||
"gc_pins is never created by pg-lo (fleet pins live on the kv host)"
|
||
);
|
||
}
|
||
|
||
/// The mem-pglo shape round-trips end-to-end: a sub-threshold
|
||
/// put lands on the kv tier (mem) — never in the pg database —
|
||
/// an over-threshold put stages into the pg-lo tier (the
|
||
/// companion row + LO present), both get back byte-identical,
|
||
/// and the pool list is the sorted union of both tiers.
|
||
#[tokio::test]
|
||
async fn dual_tier_mem_pglo_shape_round_trips_with_lo_placement() {
|
||
let Some(site) = fresh_db("mem-pglo-rt").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_mem_pglo(&site.dsn, PgOpts::default(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("mem+pglo constructs");
|
||
|
||
let small = b"a sub-threshold blob";
|
||
let (key, pin) = store
|
||
.put(Some(small.len() as u64), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("kv put publishes");
|
||
assert_eq!(key, key_for(small));
|
||
assert!(
|
||
store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the kv tier (mem)"
|
||
);
|
||
let lo = pglo_engine(&store);
|
||
assert!(
|
||
!lo.has(&key.as_bytes()).await.unwrap(),
|
||
"kv entries never land in the pg database"
|
||
);
|
||
drop(pin);
|
||
|
||
let big = vec![7u8; 200];
|
||
let (key, pin) = store
|
||
.put(Some(200), Cursor::new(&big[..]))
|
||
.await
|
||
.expect("staged put publishes");
|
||
assert_eq!(key, key_for(&big));
|
||
assert!(
|
||
lo.has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the pg-lo tier: the companion row is present"
|
||
);
|
||
assert_eq!(
|
||
lo.size(&key.as_bytes()).await.unwrap(),
|
||
Some(200),
|
||
"the companion row's size is exact"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"no kv copy on the large route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("entry present");
|
||
assert_eq!(len, 200);
|
||
assert_eq!(
|
||
blob.read_at(0, len).await.unwrap(),
|
||
big,
|
||
"the LO's content round-trips byte-identical (large placement)"
|
||
);
|
||
drop(pin);
|
||
|
||
let mut expected: Vec<Key> = [key_for(small), key_for(&big)].to_vec();
|
||
expected.sort();
|
||
assert_eq!(
|
||
store.list().await.expect("list succeeds"),
|
||
expected,
|
||
"the pool list is the sorted union of both tiers"
|
||
);
|
||
}
|
||
|
||
/// The window-delay parameter on the pg-lo host: the shipped
|
||
/// default is the pinned 5-minute constant, the constructor's
|
||
/// value passes through the store's introspection, and
|
||
/// `Duration::ZERO` is a legal degenerate tuning (execute-on-
|
||
/// next-sweep).
|
||
#[tokio::test]
|
||
async fn window_delay_default_is_pinned_and_constructor_tunable() {
|
||
assert_eq!(
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::from_secs(5 * 60),
|
||
"the shipped default is 5 minutes (ADR-008 §2)"
|
||
);
|
||
|
||
let Some(site) = fresh_db("mem-pglo-delay").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_mem_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("constructs at the shipped default");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(300));
|
||
|
||
let store =
|
||
Store::dual_tier_mem_pglo(&site.dsn, PgOpts::default(), 64, Duration::from_secs(90))
|
||
.await
|
||
.expect("constructs at a custom tuning");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(90));
|
||
|
||
let store = Store::dual_tier_mem_pglo(&site.dsn, PgOpts::default(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("the degenerate zero tuning is legal");
|
||
assert_eq!(store.window_delay(), Duration::ZERO);
|
||
|
||
Store::dual_tier_mem_pglo(&site.dsn, PgOpts::default(), 64, Duration::from_secs(6000))
|
||
.await
|
||
.expect("a long delay is a legal tuning too");
|
||
}
|
||
|
||
/// A set-backed liveness source — the construction tests' shared
|
||
/// sweep instrument.
|
||
#[derive(Default)]
|
||
struct SetSource {
|
||
live: parking_lot::Mutex<BTreeSet<Key>>,
|
||
}
|
||
|
||
impl LivenessSource for SetSource {
|
||
fn live_keys(&self) -> Vec<Key> {
|
||
self.live.lock().iter().cloned().collect()
|
||
}
|
||
|
||
fn contains(&self, key: &Key) -> bool {
|
||
self.live.lock().contains(key)
|
||
}
|
||
}
|
||
|
||
/// The sweep's form branch runs the **durable form on the pg-lo
|
||
/// host** (the branch keyed off the engine representation — the mem
|
||
/// arm consulting the large engine, never a flag, ADR-012 §5): a
|
||
/// zero-delay sweep stages this sweep's dead candidate into the
|
||
/// pg-lo schema's `gc_window` and executes none of them (the
|
||
/// inline form — mem kv, no large host — would delete in the same
|
||
/// sweep, so this is the form-branch-is-large-host proof), and the
|
||
/// *next* sweep executes the overdue set re-arbitrated at
|
||
/// execution time.
|
||
#[tokio::test]
|
||
async fn sweep_runs_the_durable_form_on_the_pg_lo_host() {
|
||
let Some(site) = fresh_db("mem-pglo-sweep").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_mem_pglo(&site.dsn, PgOpts::default(), 64, Duration::ZERO)
|
||
.await
|
||
.expect("mem+pglo constructs");
|
||
store.register_liveness_source(Arc::new(SetSource::default()));
|
||
|
||
let content = b"a dead candidate under the pg-lo host's window";
|
||
let (dead, _) = store
|
||
.put(Some(content.len() as u64), Cursor::new(&content[..]))
|
||
.await
|
||
.expect("the candidate's put publishes");
|
||
assert_eq!(store.list().await.unwrap(), vec![dead.clone()]);
|
||
|
||
let first = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(first.aborted, None);
|
||
assert_eq!(
|
||
first.candidates_staged, 1,
|
||
"the dead candidate staged into the pg-lo schema's gc_window"
|
||
);
|
||
assert_eq!(
|
||
first.deleted, 0,
|
||
"the durable form executes nothing in its own staging sweep \
|
||
(the form-branch-is-large-host proof)"
|
||
);
|
||
assert!(
|
||
store.has(&dead.as_bytes()).await.unwrap(),
|
||
"the staged candidate still waits out its window"
|
||
);
|
||
|
||
let second = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(second.aborted, None);
|
||
assert_eq!(
|
||
second.candidates_staged, 0,
|
||
"nothing newly dead after the execution"
|
||
);
|
||
assert_eq!(
|
||
second.deleted, 1,
|
||
"the next sweep executed the overdue set, re-arbitrated at \
|
||
execution time"
|
||
);
|
||
assert!(!store.has(&dead.as_bytes()).await.unwrap());
|
||
}
|
||
}
|
||
|
||
/// The wave-5 constructor wiring's tests (features `pg-lo` +
|
||
/// `postgres`): the **consolidation posture** (ADR-009) — the
|
||
/// constructor completes the fleet-valid engine set. The mode
|
||
/// constructs and introspects correctly, the GC schema ran on the
|
||
/// **kv host** (both GC tables in the kv schema; the pg-lo window
|
||
/// clauses dormant — no `gc_window` in the pg-lo schema), the
|
||
/// window-delay default is the pinned constant and is
|
||
/// constructor-tunable (`0` legal), and the consolidation shape
|
||
/// round-trips end-to-end through both tiers with LO placement
|
||
/// asserted. Env-driven (`ALKBLOBS_PG_*`), skipping cleanly
|
||
/// server-less.
|
||
#[cfg(all(test, feature = "pg-lo", feature = "postgres"))]
|
||
mod postgres_pglo_construction_tests {
|
||
use super::*;
|
||
use crate::pg_common::test_support::DatabaseJanitor;
|
||
use std::io::Cursor;
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use tokio_postgres::NoTls;
|
||
|
||
static DB_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
|
||
fn env_nonempty(name: &str) -> Option<String> {
|
||
std::env::var(name).ok().filter(|v| !v.is_empty())
|
||
}
|
||
|
||
/// The harness connection settings ride the environment
|
||
/// (`ALKBLOBS_PG_HOST` required; port/user/password/db default to
|
||
/// the harness server's values). Absent → the caller skips.
|
||
fn harness_dsn() -> Option<String> {
|
||
let host = env_nonempty("ALKBLOBS_PG_HOST")?;
|
||
Some(format!(
|
||
"host={host} port={} user={} password={} dbname={}",
|
||
env_nonempty("ALKBLOBS_PG_PORT").unwrap_or_else(|| "15432".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_USER").unwrap_or_else(|| "postgres".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_PASSWORD").unwrap_or_else(|| "poc".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_DB").unwrap_or_else(|| "blobs".to_owned()),
|
||
))
|
||
}
|
||
|
||
/// The skip note — printed exactly once per skipped test.
|
||
fn skip_note() {
|
||
eprintln!(
|
||
"skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB"
|
||
);
|
||
}
|
||
|
||
/// A fresh dedicated database per test (LOs are database-level —
|
||
/// the C7 lesson): the drop-guard removes the whole database —
|
||
/// both engines' schemas with it — at test end in every exit path.
|
||
struct TestDatabase {
|
||
/// The DSN into the dedicated database (the constructor's
|
||
/// connection settings for both engines).
|
||
dsn: String,
|
||
_janitor: DatabaseJanitor,
|
||
}
|
||
|
||
fn admin_dsn(dsn: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname=postgres")
|
||
}
|
||
|
||
fn db_dsn(dsn: &str, db: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname={db}")
|
||
}
|
||
|
||
async fn fresh_db(tag: &str) -> Option<TestDatabase> {
|
||
let dsn = harness_dsn()?;
|
||
let db = format!(
|
||
"abtmpglo_{}_{}_{:x}",
|
||
tag,
|
||
std::process::id(),
|
||
DB_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
);
|
||
let admin = admin_dsn(&dsn);
|
||
{
|
||
let (client, conn) = tokio_postgres::connect(&admin, NoTls)
|
||
.await
|
||
.expect("admin connect");
|
||
tokio::spawn(async move {
|
||
let _ = conn.await;
|
||
});
|
||
client
|
||
.execute(
|
||
&format!(
|
||
"CREATE DATABASE {}",
|
||
crate::pg_common::quote_identifier(&db)
|
||
),
|
||
&[],
|
||
)
|
||
.await
|
||
.expect("create test database");
|
||
}
|
||
Some(TestDatabase {
|
||
dsn: db_dsn(&dsn, &db),
|
||
_janitor: DatabaseJanitor::new(&admin, &db),
|
||
})
|
||
}
|
||
|
||
/// A fresh engine-owned schema per test (the host-split test's
|
||
/// instrument: two distinct schemas so the dormancy assertion
|
||
/// observes the pg-lo schema genuinely without `gc_window`).
|
||
fn fresh_schema(tag: &str, seq: u64) -> String {
|
||
format!("abtmpglo_{tag}_{}_{seq:x}", std::process::id())
|
||
}
|
||
|
||
fn pglo_engine(store: &Store) -> &crate::pg_lo::PgLoBackend {
|
||
match store.large() {
|
||
Some(LargeEngine::PgLo(engine)) => engine,
|
||
Some(LargeEngine::Local(_)) | None => {
|
||
panic!("these tests construct pg-lo-tier stores")
|
||
}
|
||
}
|
||
}
|
||
|
||
fn postgres_engine(store: &Store) -> &PostgresBackend {
|
||
match store.kv() {
|
||
#[cfg(feature = "postgres")]
|
||
KvEngine::Postgres(engine) => engine,
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct postgres stores"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(_) => panic!("these tests construct postgres stores"),
|
||
}
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
/// The consolidation constructor constructs and introspects; the
|
||
/// GC schema ran on the **kv host** (both GC tables in the kv
|
||
/// schema) and the pg-lo window clauses are **dormant** (no
|
||
/// `gc_window` in the pg-lo schema — two distinct schemas here so
|
||
/// the dormancy assertion observes genuine absence, ADR-012 §5's
|
||
/// dual-SQL shape).
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_pglo_constructs_with_the_kv_host_winning_the_window() {
|
||
let Some(site) = fresh_db("consolidation").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let seq = DB_SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let kv_schema = fresh_schema("kv", seq);
|
||
let lo_schema = fresh_schema("lo", seq);
|
||
let store = Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
PgOpts {
|
||
schema: kv_schema,
|
||
..Default::default()
|
||
},
|
||
&site.dsn,
|
||
PgOpts {
|
||
schema: lo_schema,
|
||
..Default::default()
|
||
},
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("consolidation constructs");
|
||
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert_eq!(store.threshold(), Some(1024));
|
||
assert_eq!(store.window_delay(), DEFAULT_DELETE_WINDOW_DELAY);
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"length {len} routes deterministically"
|
||
);
|
||
}
|
||
|
||
let kv = postgres_engine(&store);
|
||
assert!(
|
||
kv.table_exists("blobs").await.unwrap(),
|
||
"the pool table exists at construction"
|
||
);
|
||
assert!(
|
||
kv.table_exists("gc_window").await.unwrap(),
|
||
"the kv host's call ran at construction: gc_window in the kv schema"
|
||
);
|
||
assert!(
|
||
kv.table_exists("gc_pins").await.unwrap(),
|
||
"the fleet host's both-tables row: gc_pins in the kv schema"
|
||
);
|
||
|
||
let lo = pglo_engine(&store);
|
||
assert!(
|
||
lo.table_exists("lo_entries").await.unwrap(),
|
||
"the pg-lo engine's companion table bootstrapped at construction"
|
||
);
|
||
assert!(
|
||
!lo.table_exists("gc_window").await.unwrap(),
|
||
"the pg-lo window clauses are dormant on the dual-SQL instance \
|
||
(ADR-012 §5): the pg-lo schema carries no gc_window"
|
||
);
|
||
assert!(
|
||
!lo.table_exists("gc_pins").await.unwrap(),
|
||
"gc_pins is never created by pg-lo (fleet pins live on the kv host)"
|
||
);
|
||
}
|
||
|
||
/// The consolidation shape round-trips end-to-end in the recorded
|
||
/// corollary form — the same DSN and opts to both engines (one
|
||
/// minted schema holding `blobs`, `lo_entries`, and the GC tables
|
||
///; ADR-012 §4's co-tenancy): a sub-threshold put lands on the kv
|
||
/// tier and never in the pg-lo companion table; an over-threshold
|
||
/// put stages into the pg-lo tier (the companion row + LO
|
||
/// present); both get back byte-identical; the pool list is the
|
||
/// sorted union.
|
||
#[tokio::test]
|
||
async fn consolidation_shape_round_trips_end_to_end_through_both_tiers() {
|
||
let Some(site) = fresh_db("consolidation-rt").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let opts = PgOpts::default();
|
||
let store = Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
opts.clone(),
|
||
&site.dsn,
|
||
opts,
|
||
64,
|
||
Duration::ZERO,
|
||
)
|
||
.await
|
||
.expect("consolidation constructs");
|
||
|
||
let small = b"a sub-threshold blob";
|
||
let (key, pin) = store
|
||
.put(Some(small.len() as u64), Cursor::new(&small[..]))
|
||
.await
|
||
.expect("kv put publishes");
|
||
assert_eq!(key, key_for(small));
|
||
assert!(
|
||
store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the kv tier (postgres)"
|
||
);
|
||
let lo = pglo_engine(&store);
|
||
assert!(
|
||
!lo.has(&key.as_bytes()).await.unwrap(),
|
||
"the sub-threshold key never entered the pg-lo companion table"
|
||
);
|
||
drop(pin);
|
||
|
||
let big = vec![42u8; 200];
|
||
let (key, pin) = store
|
||
.put(Some(200), Cursor::new(&big[..]))
|
||
.await
|
||
.expect("staged put publishes");
|
||
assert_eq!(key, key_for(&big));
|
||
assert!(
|
||
lo.has(&key.as_bytes()).await.unwrap(),
|
||
"landed on the pg-lo tier: the companion row is present"
|
||
);
|
||
assert_eq!(
|
||
lo.size(&key.as_bytes()).await.unwrap(),
|
||
Some(200),
|
||
"the companion row's size is exact"
|
||
);
|
||
assert!(
|
||
!store.kv().has(&key.as_bytes()).await.unwrap(),
|
||
"no kv copy on the large route"
|
||
);
|
||
let (len, blob) = store
|
||
.get(&key.as_bytes())
|
||
.await
|
||
.unwrap()
|
||
.expect("entry present");
|
||
assert_eq!(len, 200);
|
||
assert_eq!(
|
||
blob.read_at(0, len).await.unwrap(),
|
||
big,
|
||
"the LO's content round-trips byte-identical (LO placement)"
|
||
);
|
||
drop(pin);
|
||
|
||
let mut expected: Vec<Key> = [key_for(small), key_for(&big)].to_vec();
|
||
expected.sort();
|
||
assert_eq!(
|
||
store.list().await.expect("list succeeds"),
|
||
expected,
|
||
"the pool list is the sorted union of both tiers"
|
||
);
|
||
}
|
||
|
||
/// The window-delay parameter on the consolidation constructor:
|
||
/// the shipped default is the pinned 5-minute constant, tunable,
|
||
/// `0` legal.
|
||
#[tokio::test]
|
||
async fn window_delay_default_is_pinned_and_constructor_tunable() {
|
||
assert_eq!(
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::from_secs(5 * 60),
|
||
"the shipped default is 5 minutes (ADR-008 §2)"
|
||
);
|
||
|
||
let Some(site) = fresh_db("consolidation-delay").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
)
|
||
.await
|
||
.expect("constructs at the shipped default");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(300));
|
||
|
||
let store = Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
Duration::from_secs(90),
|
||
)
|
||
.await
|
||
.expect("constructs at a custom tuning");
|
||
assert_eq!(store.window_delay(), Duration::from_secs(90));
|
||
|
||
let store = Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
Duration::ZERO,
|
||
)
|
||
.await
|
||
.expect("the degenerate zero tuning is legal");
|
||
assert_eq!(store.window_delay(), Duration::ZERO);
|
||
|
||
Store::dual_tier_postgres_pglo(
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
&site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
Duration::from_secs(6000),
|
||
)
|
||
.await
|
||
.expect("a long delay is a legal tuning too");
|
||
}
|
||
}
|
||
|
||
/// The composite fleet-validity predicate's per-clause tests (the
|
||
/// predicate is the fleet constructors' admission gate; the mem and
|
||
/// sqlite clauses have no fleet constructor to reach them through —
|
||
/// the recorded "not fleet-anything" postures — so the rejection arms
|
||
/// are exercised at the predicate level, the same error the would-be
|
||
/// construction carries). Runs in every build: pure vocabulary, no
|
||
/// engine I/O.
|
||
#[cfg(test)]
|
||
mod fleet_validity_predicate_tests {
|
||
use super::*;
|
||
|
||
/// Every fleet-valid shape admits: kv = postgres and (large = none
|
||
/// — the re-routing arm, OR `local` on the declared shared media,
|
||
/// OR pg-lo — the same-instance clause's runtime probe rides its
|
||
/// constructor). (backends-and-dispatch.md, "The composite
|
||
/// fleet-validity rule", one sentence, both tiers.)
|
||
#[test]
|
||
fn every_fleet_valid_shape_admits() {
|
||
assert_eq!(
|
||
fleet_validity_error(FleetKvShape::Postgres, FleetLargeShape::None),
|
||
None,
|
||
"the re-routing posture (large = none) is fleet-valid with kv = postgres"
|
||
);
|
||
assert_eq!(
|
||
fleet_validity_error(FleetKvShape::Postgres, FleetLargeShape::LocalSharedMedia),
|
||
None,
|
||
"the declared shared-media arm is fleet-valid"
|
||
);
|
||
assert_eq!(
|
||
fleet_validity_error(FleetKvShape::Postgres, FleetLargeShape::PgLo),
|
||
None,
|
||
"the pg-lo arm is fleet-valid (its same-instance clause probes at the constructor)"
|
||
);
|
||
}
|
||
|
||
/// `mem` anywhere rejects (ADR-012 §3): mem is never fleet-valid
|
||
/// (ADR-011 §4) — every large shape over a mem kv engine is the
|
||
/// constructor error.
|
||
#[test]
|
||
fn mem_anywhere_rejects() {
|
||
for large in [
|
||
FleetLargeShape::None,
|
||
FleetLargeShape::LocalSharedMedia,
|
||
FleetLargeShape::LocalUndeclared,
|
||
FleetLargeShape::PgLo,
|
||
] {
|
||
let err = fleet_validity_error(FleetKvShape::Mem, large)
|
||
.expect("mem is never fleet-valid, any large shape");
|
||
assert!(
|
||
err.to_string().contains("Mem is never fleet-valid"),
|
||
"mem is the kv engine, large = {large:?}: {err}"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The sqlite kv engine rejects (ADR-012 §3's "any kv engine other
|
||
/// than postgres"): a sqlite fleet is a recorded non-option (ADR-010
|
||
/// §3).
|
||
#[test]
|
||
fn sqlite_rejects() {
|
||
for large in [
|
||
FleetLargeShape::None,
|
||
FleetLargeShape::LocalSharedMedia,
|
||
FleetLargeShape::LocalUndeclared,
|
||
FleetLargeShape::PgLo,
|
||
] {
|
||
assert!(
|
||
fleet_validity_error(FleetKvShape::Sqlite, large).is_some(),
|
||
"the sqlite kv engine, large = {large:?}, rejects"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// `local` without the shared-media declaration rejects (ADR-008
|
||
/// §3): a fleet-mode constructor requires the explicit
|
||
/// [`FleetSharedMedia`] assertion — the sealed declaration type
|
||
/// keeps every real call site on the declared arm, and this is the
|
||
/// rule's rejected arm.
|
||
#[test]
|
||
fn local_without_the_shared_media_declaration_rejects() {
|
||
let err = fleet_validity_error(FleetKvShape::Postgres, FleetLargeShape::LocalUndeclared)
|
||
.expect("the undeclared arm rejects");
|
||
assert!(
|
||
err.to_string().contains("shared-media declaration"),
|
||
"the error names the missing declaration: {err}"
|
||
);
|
||
}
|
||
}
|
||
|
||
/// The wave-6 fleet declaration's constructor tests (feature
|
||
/// `postgres`): the `_fleet` constructors' fleet state introspection
|
||
/// (flag, TTL, owner id), the TTL's engine wiring, the admission gate
|
||
/// (a zero TTL errors closed at entry), and the declaration's behavioral
|
||
/// nothing-else posture — put/sweep on a fleet store run exactly the
|
||
/// non-fleet store's machinery (the fleet mechanism turns on with the
|
||
/// later put/sweep driver tasks, not with the declaration).
|
||
/// Env-driven (`ALKBLOBS_PG_*`), skipping cleanly server-less.
|
||
#[cfg(all(test, feature = "postgres"))]
|
||
mod postgres_fleet_construction_tests {
|
||
use super::*;
|
||
use crate::postgres::test_support::SchemaJanitor;
|
||
use std::fs;
|
||
use std::io::Cursor;
|
||
use std::path::{Path, PathBuf};
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
|
||
/// A fresh temp root per test; Drop removes the tree idempotently.
|
||
struct TempRoot(PathBuf);
|
||
|
||
fn temp_root(tag: &'static str) -> TempRoot {
|
||
static SEQ: AtomicU64 = AtomicU64::new(0);
|
||
let seq = SEQ.fetch_add(1, Ordering::Relaxed);
|
||
let dir = std::env::temp_dir().join(format!(
|
||
"alkblobs-store-fleet-{tag}-{}-{:x}-{seq}",
|
||
std::process::id(),
|
||
seq,
|
||
));
|
||
fs::create_dir_all(&dir).expect("temp root created");
|
||
TempRoot(dir)
|
||
}
|
||
|
||
impl TempRoot {
|
||
fn path(&self) -> &Path {
|
||
&self.0
|
||
}
|
||
}
|
||
|
||
impl Drop for TempRoot {
|
||
fn drop(&mut self) {
|
||
let _ = fs::remove_dir_all(&self.0);
|
||
}
|
||
}
|
||
|
||
static SCHEMA_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
|
||
fn fresh_schema(tag: &str) -> String {
|
||
format!(
|
||
"abtfleet_{}_{}_{:x}",
|
||
tag,
|
||
std::process::id(),
|
||
SCHEMA_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
)
|
||
}
|
||
|
||
fn env_nonempty(name: &str) -> Option<String> {
|
||
std::env::var(name).ok().filter(|v| !v.is_empty())
|
||
}
|
||
|
||
fn harness_dsn() -> Option<String> {
|
||
let host = env_nonempty("ALKBLOBS_PG_HOST")?;
|
||
Some(format!(
|
||
"host={host} port={} user={} password={} dbname={}",
|
||
env_nonempty("ALKBLOBS_PG_PORT").unwrap_or_else(|| "15432".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_USER").unwrap_or_else(|| "postgres".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_PASSWORD").unwrap_or_else(|| "poc".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_DB").unwrap_or_else(|| "blobs".to_owned()),
|
||
))
|
||
}
|
||
|
||
fn skip_note() {
|
||
eprintln!(
|
||
"skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB"
|
||
);
|
||
}
|
||
|
||
fn fresh_opts(tag: &str) -> Option<(PgOpts, SchemaJanitor)> {
|
||
let dsn = harness_dsn()?;
|
||
let schema = fresh_schema(tag);
|
||
Some((
|
||
PgOpts {
|
||
schema: schema.clone(),
|
||
..Default::default()
|
||
},
|
||
SchemaJanitor::new(&dsn, &schema),
|
||
))
|
||
}
|
||
|
||
fn key_for(content: &[u8]) -> Key {
|
||
Key::derive(content, content.len() as u64)
|
||
}
|
||
|
||
fn postgres_engine(store: &Store) -> &PostgresBackend {
|
||
match store.kv() {
|
||
KvEngine::Postgres(engine) => engine,
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct postgres stores"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(_) => panic!("these tests construct postgres stores"),
|
||
}
|
||
}
|
||
|
||
async fn assert_gc_schema_ran(store: &Store) {
|
||
let engine = postgres_engine(store);
|
||
assert!(
|
||
engine.table_exists("blobs").await.unwrap(),
|
||
"the pool table exists at construction"
|
||
);
|
||
assert!(
|
||
engine.table_exists("gc_window").await.unwrap(),
|
||
"ensure_gc_schema ran at construction"
|
||
);
|
||
assert!(
|
||
engine.table_exists("gc_pins").await.unwrap(),
|
||
"the fleet host's both-tables row: gc_pins exists"
|
||
);
|
||
}
|
||
|
||
/// The shipped default pin TTL is the pinned 60-second constant
|
||
/// (ADR-008 §2) — the fleet constructors' default tuning, the
|
||
/// constant the engine's `pin_renew` successor-time body renews
|
||
/// against.
|
||
#[test]
|
||
fn default_pin_ttl_is_the_pinned_sixty_seconds() {
|
||
assert_eq!(
|
||
DEFAULT_PIN_TTL,
|
||
Duration::from_secs(crate::postgres::DEFAULT_PIN_TTL_SECS),
|
||
"the store's shipped default is the engine constant's seconds"
|
||
);
|
||
assert_eq!(
|
||
DEFAULT_PIN_TTL,
|
||
Duration::from_secs(60),
|
||
"the shipped default is 60 s (ADR-008 §2)"
|
||
);
|
||
}
|
||
|
||
/// The store-instance id's minting: 32 raw bytes, unique per
|
||
/// instance (the process-global counter distinguishes instances in
|
||
/// one process; time + pid + the ASLR anchor distinguish
|
||
/// processes) — minted at construction, carried as the pin rows'
|
||
/// `owner` half, crate-internal.
|
||
#[test]
|
||
fn owner_id_mints_unique_and_fixed_width() {
|
||
let first = mint_owner_id();
|
||
let second = mint_owner_id();
|
||
assert_eq!(first.len(), 32, "the SHA-256 fold's width");
|
||
assert_eq!(second.len(), 32);
|
||
assert_ne!(
|
||
first, second,
|
||
"two instances in one process mint distinct ids"
|
||
);
|
||
}
|
||
|
||
/// A zero pin TTL errors closed at entry — before any I/O — never
|
||
/// a silent pass (the TTL/3 renewal cadence degenerates at zero
|
||
/// and a zero-TTL pin expires at once).
|
||
#[tokio::test]
|
||
async fn zero_pin_ttl_is_a_constructor_error_at_entry() {
|
||
let err = Store::kv_only_postgres_fleet(
|
||
"host=no-such-host.invalid port=1 user=x password=x dbname=x",
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::ZERO,
|
||
)
|
||
.await
|
||
.err()
|
||
.expect("the entry validation rejects before any connection attempt");
|
||
assert!(
|
||
err.to_string().contains("pin_ttl must be positive"),
|
||
"the zero TTL rejects before any connection attempt: {err}"
|
||
);
|
||
|
||
let err = Store::dual_tier_postgres_local_fleet(
|
||
"host=no-such-host.invalid port=1 user=x password=x dbname=x",
|
||
PgOpts::default(),
|
||
std::env::temp_dir().join("alkblobs-fleet-zero-ttl"),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::ZERO,
|
||
FleetSharedMedia::asserted(),
|
||
)
|
||
.await
|
||
.err()
|
||
.expect("the entry validation rejects on the shared-media shape too");
|
||
assert!(
|
||
err.to_string().contains("pin_ttl must be positive"),
|
||
"the zero TTL rejects at entry on the shared-media shape too: {err}"
|
||
);
|
||
}
|
||
|
||
/// The fleet re-routing client posture constructs and introspects:
|
||
/// the fleet state carries the declaration's flag, the pin TTL
|
||
/// (default and constructor-tuned), and the store-instance id —
|
||
/// distinct across two instances — and the GC schema ran (the
|
||
/// fleet host's both-tables row). The over-threshold put-time
|
||
/// rejection (ADR-012 §6.2) is unchanged under fleet.
|
||
#[tokio::test]
|
||
async fn kv_only_postgres_fleet_constructs_and_introspects_fleet_state() {
|
||
let Some((opts, _janitor)) = fresh_opts("fleet-kv-only") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let store = Store::kv_only_postgres_fleet(
|
||
&dsn,
|
||
opts,
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
)
|
||
.await
|
||
.expect("the fleet kv-only posture constructs");
|
||
assert_eq!(store.mode(), KvOnly);
|
||
assert!(store.large().is_none(), "no large tier (large = none)");
|
||
let fleet = store
|
||
.fleet()
|
||
.expect("fleet: true declares: the state carries");
|
||
assert_eq!(fleet.pin_ttl(), DEFAULT_PIN_TTL);
|
||
assert_eq!(fleet.owner().len(), 32);
|
||
assert_gc_schema_ran(&store).await;
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(63, Tier::Kv),
|
||
(64, Tier::Kv),
|
||
(65, Tier::Kv),
|
||
(u64::MAX, Tier::Kv),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"kv-only routes every length kv, fleet or not"
|
||
);
|
||
}
|
||
|
||
let err = store
|
||
.put(Some(65), Cursor::new(&b"never read"[..]))
|
||
.await
|
||
.unwrap_err();
|
||
assert!(
|
||
err.to_string().contains("kv-only mode rejects"),
|
||
"the put-time rejection is unchanged under the declaration: {err}"
|
||
);
|
||
assert_eq!(
|
||
store.list().await.unwrap(),
|
||
Vec::<Key>::new(),
|
||
"zero residue"
|
||
);
|
||
drop(store);
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("fleet-kv-ttl") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::kv_only_postgres_fleet(
|
||
&dsn,
|
||
opts,
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::from_secs(90),
|
||
)
|
||
.await
|
||
.expect("constructs at a custom pin TTL");
|
||
let fleet = store.fleet().expect("fleet state carries");
|
||
assert_eq!(
|
||
fleet.pin_ttl(),
|
||
Duration::from_secs(90),
|
||
"the TTL is a fleet constructor parameter"
|
||
);
|
||
let same_instance_owner = fleet.owner().to_vec();
|
||
drop(store);
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("fleet-kv-owner2") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let other = Store::kv_only_postgres_fleet(
|
||
&dsn,
|
||
opts,
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
)
|
||
.await
|
||
.expect("a second instance constructs");
|
||
assert_ne!(
|
||
other.fleet().expect("fleet state carries").owner(),
|
||
same_instance_owner,
|
||
"two instances never share the store-instance id"
|
||
);
|
||
}
|
||
|
||
/// A declared fleet store behaves exactly as the non-fleet store
|
||
/// does until the fleet machinery's drivers land: a put lands on
|
||
/// the kv tier, and the sweep's form branch — keyed off the engine
|
||
/// representation, never the declaration — runs the durable form
|
||
/// on the postgres host (stage this sweep, execute the next).
|
||
#[tokio::test]
|
||
async fn fleet_declaration_turns_nothing_else_on_yet() {
|
||
struct SetSource;
|
||
impl LivenessSource for SetSource {
|
||
fn live_keys(&self) -> Vec<Key> {
|
||
Vec::new()
|
||
}
|
||
fn contains(&self, _key: &Key) -> bool {
|
||
false
|
||
}
|
||
}
|
||
|
||
let Some((opts, _janitor)) = fresh_opts("fleet-inert") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let store = Store::kv_only_postgres_fleet(&dsn, opts, 64, Duration::ZERO, DEFAULT_PIN_TTL)
|
||
.await
|
||
.expect("constructs");
|
||
store.register_liveness_source(Arc::new(SetSource));
|
||
|
||
let content = b"a dead candidate on a declared fleet store";
|
||
let (key, pin) = store
|
||
.put(Some(content.len() as u64), Cursor::new(&content[..]))
|
||
.await
|
||
.expect("the put publishes through the in-process protocol");
|
||
assert_eq!(key, key_for(content));
|
||
assert!(store.has(&key.as_bytes()).await.unwrap());
|
||
drop(pin);
|
||
|
||
let staged = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(staged.aborted, None);
|
||
assert_eq!(
|
||
staged.candidates_staged, 1,
|
||
"the durable form staged on the kv host (the form branch keys off the \
|
||
engine representation)"
|
||
);
|
||
assert_eq!(staged.deleted, 0);
|
||
|
||
let executed = store.sweep().await.expect("sweeps are never errors");
|
||
assert_eq!(
|
||
executed.deleted, 1,
|
||
"the next sweep executed the overdue set"
|
||
);
|
||
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
||
}
|
||
|
||
/// The fleet shared-media posture constructs and introspects; the
|
||
/// GC schema ran on the kv host (the one-window-host rule intact
|
||
/// under fleet — kv = postgres hosts the window; there is no
|
||
/// second SQL engine to host anything else).
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_local_fleet_constructs_and_introspects() {
|
||
let Some((opts, _janitor)) = fresh_opts("fleet-shared-media") else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let dsn = harness_dsn().expect("harness dsn re-read");
|
||
let root = temp_root("fleet-shared-media");
|
||
let store = Store::dual_tier_postgres_local_fleet(
|
||
&dsn,
|
||
opts,
|
||
root.path(),
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
FleetSharedMedia::asserted(),
|
||
)
|
||
.await
|
||
.expect("the fleet shared-media posture constructs");
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert!(store.large().is_some(), "both tiers present");
|
||
let fleet = store
|
||
.fleet()
|
||
.expect("fleet: true declares: the state carries");
|
||
assert_eq!(fleet.pin_ttl(), DEFAULT_PIN_TTL);
|
||
assert_eq!(fleet.owner().len(), 32);
|
||
assert_gc_schema_ran(&store).await;
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"the declaration does not disturb the dispatch"
|
||
);
|
||
}
|
||
}
|
||
}
|
||
|
||
/// The wave-6 fleet declaration's pg-lo-side constructor tests
|
||
/// (features `postgres` + `pg-lo`): the consolidation posture's fleet
|
||
/// twin — the pg-lo same-instance probe (ADR-012 §4) green on the
|
||
/// same instance (including the instance-level shape: two distinct
|
||
/// *databases* on one instance are one probe-matching instance), the
|
||
/// probe's error-closed negative (a DSN that cannot resolve to the kv
|
||
/// engine's instance is a constructor error, never a silent pass), and
|
||
/// the one-window-host posture (the window hosts on the kv engine; the
|
||
/// pg-lo window clauses stay dormant). Env-driven (`ALKBLOBS_PG_*`),
|
||
/// skipping cleanly server-less.
|
||
#[cfg(all(test, feature = "pg-lo", feature = "postgres"))]
|
||
mod postgres_pglo_fleet_construction_tests {
|
||
use super::*;
|
||
use crate::pg_common::test_support::DatabaseJanitor;
|
||
use std::sync::atomic::{AtomicU64, Ordering};
|
||
use tokio_postgres::NoTls;
|
||
|
||
static DB_SEQ: AtomicU64 = AtomicU64::new(0);
|
||
|
||
fn env_nonempty(name: &str) -> Option<String> {
|
||
std::env::var(name).ok().filter(|v| !v.is_empty())
|
||
}
|
||
|
||
fn harness_dsn() -> Option<String> {
|
||
let host = env_nonempty("ALKBLOBS_PG_HOST")?;
|
||
Some(format!(
|
||
"host={host} port={} user={} password={} dbname={}",
|
||
env_nonempty("ALKBLOBS_PG_PORT").unwrap_or_else(|| "15432".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_USER").unwrap_or_else(|| "postgres".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_PASSWORD").unwrap_or_else(|| "poc".to_owned()),
|
||
env_nonempty("ALKBLOBS_PG_DB").unwrap_or_else(|| "blobs".to_owned()),
|
||
))
|
||
}
|
||
|
||
fn skip_note() {
|
||
eprintln!(
|
||
"skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB"
|
||
);
|
||
}
|
||
|
||
/// A fresh dedicated database per test (LOs are database-level —
|
||
/// the C7 lesson); the drop-guard removes it in every exit path.
|
||
struct TestDatabase {
|
||
dsn: String,
|
||
_janitor: DatabaseJanitor,
|
||
}
|
||
|
||
fn admin_dsn(dsn: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname=postgres")
|
||
}
|
||
|
||
fn db_dsn(dsn: &str, db: &str) -> String {
|
||
let base = dsn.rsplit_once("dbname=").expect("dbname").0;
|
||
format!("{base} dbname={db}")
|
||
}
|
||
|
||
async fn fresh_db(tag: &str) -> Option<TestDatabase> {
|
||
let dsn = harness_dsn()?;
|
||
let db = format!(
|
||
"abtfleetlo_{}_{}_{:x}",
|
||
tag,
|
||
std::process::id(),
|
||
DB_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
);
|
||
let admin = admin_dsn(&dsn);
|
||
{
|
||
let (client, conn) = tokio_postgres::connect(&admin, NoTls)
|
||
.await
|
||
.expect("admin connect");
|
||
tokio::spawn(async move {
|
||
let _ = conn.await;
|
||
});
|
||
client
|
||
.execute(
|
||
&format!(
|
||
"CREATE DATABASE {}",
|
||
crate::pg_common::quote_identifier(&db)
|
||
),
|
||
&[],
|
||
)
|
||
.await
|
||
.expect("create test database");
|
||
}
|
||
Some(TestDatabase {
|
||
dsn: db_dsn(&dsn, &db),
|
||
_janitor: DatabaseJanitor::new(&admin, &db),
|
||
})
|
||
}
|
||
|
||
fn fresh_schema(tag: &str) -> String {
|
||
format!(
|
||
"abtfleetlo_{}_{}_{:x}",
|
||
tag,
|
||
std::process::id(),
|
||
DB_SEQ.fetch_add(1, Ordering::Relaxed)
|
||
)
|
||
}
|
||
|
||
fn pglo_engine(store: &Store) -> &crate::pg_lo::PgLoBackend {
|
||
match store.large() {
|
||
Some(LargeEngine::PgLo(engine)) => engine,
|
||
Some(LargeEngine::Local(_)) | None => panic!("these tests construct pg-lo-tier stores"),
|
||
}
|
||
}
|
||
|
||
fn postgres_engine(store: &Store) -> &PostgresBackend {
|
||
match store.kv() {
|
||
KvEngine::Postgres(engine) => engine,
|
||
#[cfg(feature = "mem")]
|
||
KvEngine::Mem(_) => panic!("these tests construct postgres stores"),
|
||
#[cfg(feature = "sqlite")]
|
||
KvEngine::Sqlite(_) => panic!("these tests construct postgres stores"),
|
||
}
|
||
}
|
||
|
||
/// The consolidation posture's fleet twin constructs on the same
|
||
/// pg instance: the fleet state carries (flag, TTL, owner id), the
|
||
/// kv host's GC schema ran (the fleet host's both-tables row), and
|
||
/// the one-window-host posture composes — the pg-lo window clauses
|
||
/// stay dormant (no `gc_window` in the pg-lo schema; two distinct
|
||
/// schemas so the dormancy assertion observes genuine absence),
|
||
/// `gc_pins` never in the pg-lo schema.
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_pglo_fleet_constructs_same_instance() {
|
||
let Some(site) = fresh_db("fleet-consolidation").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let kv_schema = fresh_schema("kv");
|
||
let lo_schema = fresh_schema("lo");
|
||
let store = Store::dual_tier_postgres_pglo_fleet(
|
||
&site.dsn,
|
||
PgOpts {
|
||
schema: kv_schema.clone(),
|
||
..Default::default()
|
||
},
|
||
&site.dsn,
|
||
PgOpts {
|
||
schema: lo_schema.clone(),
|
||
..Default::default()
|
||
},
|
||
1024,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
)
|
||
.await
|
||
.expect("the same-instance consolidation posture constructs");
|
||
|
||
assert_eq!(store.mode(), DualTier);
|
||
assert!(store.large().is_some());
|
||
let fleet = store
|
||
.fleet()
|
||
.expect("fleet: true declares: the state carries");
|
||
assert_eq!(fleet.pin_ttl(), DEFAULT_PIN_TTL);
|
||
assert_eq!(fleet.owner().len(), 32);
|
||
|
||
for (len, tier) in [
|
||
(0u64, Tier::Kv),
|
||
(1023, Tier::Kv),
|
||
(1024, Tier::Kv),
|
||
(1025, Tier::Large),
|
||
(u64::MAX, Tier::Large),
|
||
] {
|
||
assert_eq!(
|
||
store.tier_for(len),
|
||
tier,
|
||
"the declaration does not disturb the dispatch"
|
||
);
|
||
}
|
||
|
||
let kv = postgres_engine(&store);
|
||
assert!(kv.table_exists("blobs").await.unwrap());
|
||
assert!(
|
||
kv.table_exists("gc_window").await.unwrap(),
|
||
"the window hosts on the kv engine (the one-window-host rule under fleet)"
|
||
);
|
||
assert!(
|
||
kv.table_exists("gc_pins").await.unwrap(),
|
||
"the fleet host's both-tables row: gc_pins in the kv schema"
|
||
);
|
||
|
||
let lo = pglo_engine(&store);
|
||
assert!(lo.table_exists("lo_entries").await.unwrap());
|
||
assert!(
|
||
!lo.table_exists("gc_window").await.unwrap(),
|
||
"the pg-lo window clauses stay dormant on the dual-SQL instance"
|
||
);
|
||
assert!(
|
||
!lo.table_exists("gc_pins").await.unwrap(),
|
||
"gc_pins is never created by pg-lo (fleet pins live on the kv host)"
|
||
);
|
||
drop(store);
|
||
let _ = (kv_schema, lo_schema);
|
||
}
|
||
|
||
/// The probe is instance-level, not database-level: two distinct
|
||
/// *databases* on one pg instance carry one `system_identifier`
|
||
/// (the data directory's), so the fleet twin accepts the kv engine
|
||
/// and the pg-lo engine living in different databases of the same
|
||
/// instance — the textual-DSN-comparison alternative could not
|
||
/// have answered this shape.
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_pglo_fleet_two_databases_one_instance() {
|
||
let Some(kv_site) = fresh_db("fleet-two-db-kv").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let Some(lo_site) = fresh_db("fleet-two-db-lo").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let store = Store::dual_tier_postgres_pglo_fleet(
|
||
&kv_site.dsn,
|
||
PgOpts::default(),
|
||
&lo_site.dsn,
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
)
|
||
.await
|
||
.expect("two databases on one instance are one fleet-valid instance");
|
||
|
||
let fleet = store
|
||
.fleet()
|
||
.expect("fleet: true declares: the state carries");
|
||
assert_eq!(fleet.pin_ttl(), DEFAULT_PIN_TTL);
|
||
}
|
||
|
||
/// The probe errors closed: a large-side DSN that cannot resolve
|
||
/// to the kv engine's instance (a database that does not exist on
|
||
/// the dev harness — the recorded negative, one instance only) is
|
||
/// a constructor error, never a silent pass. The failing point
|
||
/// rides the engine open or the probe — either way the
|
||
/// constructor rejects.
|
||
#[tokio::test]
|
||
async fn dual_tier_postgres_pglo_fleet_unresolvable_instance_errors_closed() {
|
||
let Some(kv_site) = fresh_db("fleet-closed-kv").await else {
|
||
skip_note();
|
||
return;
|
||
};
|
||
let nonexistent = db_dsn(&kv_site.dsn, "abtnosuchdb_fleet_negative");
|
||
let err = Store::dual_tier_postgres_pglo_fleet(
|
||
&kv_site.dsn,
|
||
PgOpts::default(),
|
||
&nonexistent,
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
DEFAULT_PIN_TTL,
|
||
)
|
||
.await
|
||
.err()
|
||
.expect("the cross-instance (unresolvable) negative errors closed");
|
||
assert!(
|
||
err.to_string().contains("postgres"),
|
||
"the rejection is a pg-side failure surfaced as the constructor error: {err}"
|
||
);
|
||
assert!(
|
||
!matches!(
|
||
err,
|
||
Error::Missing | Error::Verification | Error::KeyInvalid | Error::GcRefuse
|
||
),
|
||
"the failure never masquerades as a non-Io member"
|
||
);
|
||
}
|
||
|
||
/// A zero pin TTL errors closed at entry — before any I/O. The
|
||
/// entry check runs server-less.
|
||
#[tokio::test]
|
||
async fn zero_pin_ttl_is_a_constructor_error_at_entry() {
|
||
let err = Store::dual_tier_postgres_pglo_fleet(
|
||
"host=no-such-host.invalid port=1 user=x password=x dbname=x",
|
||
PgOpts::default(),
|
||
"host=no-such-host.invalid port=1 user=x password=x dbname=x",
|
||
PgOpts::default(),
|
||
64,
|
||
DEFAULT_DELETE_WINDOW_DELAY,
|
||
Duration::ZERO,
|
||
)
|
||
.await
|
||
.err()
|
||
.expect("the entry validation rejects before any connection attempt");
|
||
assert!(
|
||
err.to_string().contains("pin_ttl must be positive"),
|
||
"the zero TTL rejects at entry on the consolidation shape too: {err}"
|
||
);
|
||
}
|
||
}
|