- tests/conformance/fleet_column.rs (cfg feature = "postgres"; the consolidation pieces cfg the postgres+pg-lo pair): the fleet-validity rows, the joint entry+pin tx rows (both-or-neither over an independent connection; the dedup re-put's upsert-renew; the second instance's re-put composing a second row — protection by row; the pg-lo consolidation round-trip byte-exact over the shared instance), the TTL/renewal rows (the guard's TTL/3 cadence through the short test TTL; the never-resurrect semantics through the facade; the paused holder's expiry boundary with the delete-then-recover backstop), and the exact-count fleet sweep rows — the wave's gate, INV-4's engine-gated fleet form: the two-instance exact-count shape, the lease contention row (SweeperLock as report data, nothing deleted), the cross-instance arbitration row (GcRefuse until the foreign row's expiry), the race across the window delay closed by the row, the maintenance step's exact reap, and the fleet liveness verb row (the schema-ownership rule asserted; the deregistration correction flow); the kv-only rejection row mirrored verbatim against the fleet kv-only factory - three fleet factories after the postgres column's registry pattern extended to the fleet shape — one minted schema, two Store instances over it, each its own owner id (the pin rows compose by owner); the dual-tier instances carry private local roots riding FleetSharedMedia::asserted; the consolidation factory follows the pg-lo column's LO-isolation corollary (a TrackedDb::new constructor added to pg_lo_column.rs for the private janitor field); teardown drops exactly the minted schemas — idempotent, owned, panicking-row-safe, structurally exercised - the SKIP posture rides the postgres column's reachability probe: server-less runs skip cleanly, acceptance is green against pglo-poc (:15432) under --features postgres with the consolidation rows under --all-features; the test tuning is the window delay 0 and the short test pin TTL (the fleet constructors' parameter) — ordering rides the tunings or the manual-expiry instrument, no sleep-based race assertions - README amended: the fleet rows table (§Columns beyond the reference set), §Column plan (the fleet column is the wave-6 gate's host), §Current state — six columns - task suite-column-fleet completed (Notes + Summary recorded, including the recorded deviations: the structural predicate shapes reach no fleet constructor; the cross-instance mismatch leg is unexercisable on the one-instance harness server — the probe's errors-closed posture asserted instead) Verification: cargo test (default) green; cargo test --features postgres green server-less (skipping) and green against the harness server; cargo test --all-features green against the server (125 conformance + 226 library tests); cargo clippy --all-targets -- -D warnings and cargo fmt --check green on the default, postgres, and all-features variants
5222 lines
212 KiB
Rust
5222 lines
212 KiB
Rust
//! The **pg-lo column** — the suite's second SKIP-posture column: the
|
|
//! reference row set and the durable-schedule sweep rows run against
|
|
//! the pg-lo large tier (feature `pg-lo`, default-off, ADR-009). The
|
|
//! column grows **two factories spanning feature pairs — the first
|
|
//! column whose factories need different feature sets** (a recorded
|
|
//! design point):
|
|
//!
|
|
//! - **[`DualTierMemPgLoFactory`](mem_pglo::DualTierMemPgLoFactory)** —
|
|
//! `kv = mem` + `large = pg-lo`: the one-window-host rule's named
|
|
//! test topology (ADR-012 §5). The durable-schedule sweep rows run
|
|
//! with the window hosted on the **large engine** (`gc_window` in
|
|
//! the pg-lo schema) — the shape no earlier column exercises. Gates
|
|
//! on `all(feature = "mem", feature = "pg-lo")`.
|
|
//! - **[`DualTierPostgresPgLoFactory`](postgres_pglo::DualTierPostgresPgLoFactory)** —
|
|
//! `kv = postgres` + `large = pg-lo` on one pg instance: the
|
|
//! consolidation posture (ADR-009; the fleet-valid engine set's last
|
|
//! constructor). Mirrors the postgres column's inventory with the
|
|
//! large medium swapped to LOs, and carries the **dormancy row**
|
|
//! (the pg-lo schema carries no `gc_window`; the window lives on the
|
|
//! kv schema — ADR-012 §5). Gates on
|
|
//! `all(feature = "postgres", feature = "pg-lo")`.
|
|
//!
|
|
//! The SKIP posture (README §The SKIP posture) is a factory-owner
|
|
//! responsibility: the connection settings ride the environment
|
|
//! (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB` — host required; the rest
|
|
//! default to the harness server's values), every runner gates behind
|
|
//! a **reachability probe**, and when no server is reachable the
|
|
//! column skips cleanly — a skip is not a failure. The acceptance run
|
|
//! is green **against the harness server** (dockerized `pglo-poc`,
|
|
//! `postgres:16-alpine`, :15432, db `blobs`) under `--all-features` —
|
|
//! a skip never substitutes for it.
|
|
//!
|
|
//! The factories (README §The factory contract): `open()` mints a
|
|
//! **fresh, dedicated database per call** carrying the engine-owned
|
|
//! schemas (the alkstore pg-factory pattern — a fresh engine-owned
|
|
//! schema per open, unique per call, tracked per-instance in a
|
|
//! registry — housed on its own database), and vends the concrete
|
|
//! [`alkblobs::Store`] facade over them (the mem-pglo factory's kv
|
|
//! tier is in-process mem, needing no schema of its own; the
|
|
//! postgres-pglo factory mints two — the kv half and the pg-lo half —
|
|
//! so the dormancy row observes genuine absence). **The LO-isolation
|
|
//! corollary carried in the engine task's recorded shape** (the C7
|
|
//! lesson): Large Objects are **database-level**, not schema-level —
|
|
//! the schema pattern alone does not isolate them and a bare
|
|
//! `DROP SCHEMA` strands their pages in `pg_largeobject` — so each
|
|
//! open gets its own database, and teardown reaps the instance's LOs
|
|
//! (`SELECT loid FROM <pg-lo schema>.lo_entries` — the companion
|
|
//! table, the contract authority — then `lo_unlink` each tracked
|
|
//! LOID), drops exactly the minted schemas (`DROP SCHEMA IF EXISTS …
|
|
//! CASCADE`, panicking-row-safe), and drops the dedicated database
|
|
//! (`WITH (FORCE)`), with a drop-guard janitor per open backing every
|
|
//! abandoned path. Zero LO residue on the shared harness database
|
|
//! (`blobs`) is therefore structural — no pg-lo pool ever opens
|
|
//! against it (the engine suite's
|
|
//! `shared_harness_db_holds_zero_lo_residue` gate measures exactly
|
|
//! this) — and the structural teardown assertions probe zero orphan
|
|
//! residue inside the dedicated database before it goes.
|
|
//!
|
|
//! The sweep rows are **schedule-relative on this column** (README
|
|
//! §The schedule-relative reading for SQL-backed columns, citing
|
|
//! ADR-012 §6.7): the durable form stages its candidates this sweep
|
|
//! and executes them by the next, so the exact counts are asserted
|
|
//! against the staged schedule — on the pg-lo host for the mem-pglo
|
|
//! factory, on the kv host for the postgres-pglo factory. The
|
|
//! form-independent rows mirror verbatim ([`crate::rows`]: the
|
|
//! round-trip family over the content matrix, the stat-discipline
|
|
//! row, and the `NoLivenessSources`/`ProtectFailed` abort shapes).
|
|
//! **No kv-only rejection row** — no kv-only pg-lo constructor exists
|
|
//! (pg-lo is a large engine; the postgres column hosts that row for
|
|
//! the SQL kv engines).
|
|
//!
|
|
//! The dual-tier dispatch rows mirror the postgres column's family
|
|
//! with the placement oracle adapted to the LO medium: large entries
|
|
//! are **companion rows + LOs** (an independent pg connection counting
|
|
//! `lo_entries` and probing `pg_largeobject` for the row's LOID), kv
|
|
//! entries per the factory's kv engine — the mem-pglo factory's kv
|
|
//! placement is in-process (the oracle's kv half follows the mem
|
|
//! column's introspection shape: served reads and absence from the
|
|
//! LO medium), the postgres-pglo factory's kv half counts the kv
|
|
//! schema's `blobs` rows.
|
|
|
|
#![allow(dead_code)]
|
|
|
|
use std::sync::atomic::{AtomicU64, Ordering};
|
|
use std::time::Duration;
|
|
|
|
use tokio_postgres::NoTls;
|
|
|
|
use alkblobs::Error;
|
|
|
|
/// The shipped LO read-window size — src/pg_lo.rs's crate-internal
|
|
/// `READ_CHUNK` constant (ADR-009 §3, POC #7 C2/C4's window shape; one
|
|
/// `lo_get` statement per window). The column's INV-1 extension row
|
|
/// straddles this edge — duplicated here because the constant is
|
|
/// engine-internal.
|
|
pub(crate) const LO_WINDOW: usize = 512 * 1024;
|
|
|
|
/// The **test tuning of the window delay** (ADR-008 §2's legal
|
|
/// degenerate tuning): `0` — execute-on-next-sweep. The durable
|
|
/// schedule's staging sweep stages the dead set; the *next* sweep is
|
|
/// its executor, which is exactly the schedule the sweep rows assert
|
|
/// against. Both factories run at this tuning.
|
|
pub(crate) const WINDOW_DELAY: Duration = Duration::ZERO;
|
|
|
|
// ------------------------------------------------------------------
|
|
// The harness connection (the SKIP posture's gate) — the settings
|
|
// ride the environment, the runners probe reachability before their
|
|
// rows run, an unreachable server skips the column cleanly.
|
|
// ------------------------------------------------------------------
|
|
|
|
/// The harness connection settings ride the environment
|
|
/// (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB` — the alkstore harness
|
|
/// convention). `ALKBLOBS_PG_HOST` set (non-empty) is the probe's
|
|
/// "configured" answer; port/user/password/db default to the harness
|
|
/// server's values (`pglo-poc`: :15432, `postgres`/`poc`, db
|
|
/// `blobs`). Absent → the caller skips.
|
|
pub(crate) fn harness_dsn() -> Option<String> {
|
|
let host = std::env::var("ALKBLOBS_PG_HOST")
|
|
.ok()
|
|
.filter(|v| !v.is_empty())?;
|
|
let port = std::env::var("ALKBLOBS_PG_PORT")
|
|
.ok()
|
|
.filter(|v| !v.is_empty())
|
|
.unwrap_or_else(|| "15432".to_owned());
|
|
let user = std::env::var("ALKBLOBS_PG_USER")
|
|
.ok()
|
|
.filter(|v| !v.is_empty())
|
|
.unwrap_or_else(|| "postgres".to_owned());
|
|
let password = std::env::var("ALKBLOBS_PG_PASSWORD")
|
|
.ok()
|
|
.filter(|v| !v.is_empty())
|
|
.unwrap_or_else(|| "poc".to_owned());
|
|
let db = std::env::var("ALKBLOBS_PG_DB")
|
|
.ok()
|
|
.filter(|v| !v.is_empty())
|
|
.unwrap_or_else(|| "blobs".to_owned());
|
|
Some(format!(
|
|
"host={host} port={port} user={user} password={password} dbname={db}"
|
|
))
|
|
}
|
|
|
|
/// The skip note — the SKIP posture's clean-skip print, exactly once
|
|
/// per skipped test.
|
|
pub(crate) fn skip_note() {
|
|
eprintln!("skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB");
|
|
}
|
|
|
|
async fn admin_connect(dsn: &str) -> Result<tokio_postgres::Client, Error> {
|
|
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
|
|
.await
|
|
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
|
|
tokio::spawn(async move {
|
|
let _ = connection.await;
|
|
});
|
|
Ok(client)
|
|
}
|
|
|
|
/// The server reachability probe: the SKIP posture's gate the runners
|
|
/// check before their rows run. One cheap admin connection per test;
|
|
/// teardown reconnects independently (no shared admin client
|
|
/// lifetime).
|
|
pub(crate) async fn harness_ready() -> bool {
|
|
let Some(dsn) = harness_dsn() else {
|
|
skip_note();
|
|
return false;
|
|
};
|
|
match tokio::time::timeout(Duration::from_secs(3), admin_connect(&dsn)).await {
|
|
Ok(Ok(_)) => true,
|
|
Ok(Err(e)) => {
|
|
eprintln!("skipping (harness server unreachable): {e}");
|
|
false
|
|
}
|
|
Err(_) => {
|
|
eprintln!("skipping (harness server unreachable: connect timeout)");
|
|
false
|
|
}
|
|
}
|
|
}
|
|
|
|
// ------------------------------------------------------------------
|
|
// The dedicated-database mint + the LO-isolation corollary's teardown
|
|
// arms.
|
|
// ------------------------------------------------------------------
|
|
|
|
/// The maintenance DSN — connections through the `postgres`
|
|
/// database that create and drop dedicated databases.
|
|
pub(crate) fn admin_dsn(base: &str) -> String {
|
|
let base = base
|
|
.rsplit_once("dbname=")
|
|
.expect("the harness DSN names a db")
|
|
.0;
|
|
format!("{base}dbname=postgres")
|
|
}
|
|
|
|
/// The DSN scoped to one dedicated database — the store's and the
|
|
/// oracle's connection settings for that open.
|
|
pub(crate) fn db_dsn(base: &str, db: &str) -> String {
|
|
let base = base
|
|
.rsplit_once("dbname=")
|
|
.expect("the harness DSN names a db")
|
|
.0;
|
|
format!("{base}dbname={db}")
|
|
}
|
|
|
|
/// A fresh dedicated-database name — unique per call, seeded per
|
|
/// (tag, pid, nanos, factory-internal sequence) so two factory
|
|
/// instances never share a database, racing opens included.
|
|
pub(crate) fn fresh_db_name(tag: &str, seq: &AtomicU64) -> String {
|
|
format!(
|
|
"abpglosuite_{}_{}_{:x}_{}",
|
|
tag,
|
|
std::process::id(),
|
|
std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.as_nanos())
|
|
.unwrap_or(0),
|
|
seq.fetch_add(1, Ordering::Relaxed),
|
|
)
|
|
}
|
|
|
|
/// A fresh engine-owned schema name — unique per call, seeded per
|
|
/// (tag, pid, nanos, factory-internal sequence) so two factory
|
|
/// instances never share a schema name, racing opens included. The
|
|
/// mint stays well under postgres' 63-character identifier limit.
|
|
pub(crate) fn fresh_schema_name(tag: &str, seq: &AtomicU64) -> String {
|
|
format!(
|
|
"abpglo_{}_{}_{:x}_{}",
|
|
tag,
|
|
std::process::id(),
|
|
std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.as_nanos())
|
|
.unwrap_or(0),
|
|
seq.fetch_add(1, Ordering::Relaxed),
|
|
)
|
|
}
|
|
|
|
/// Create a dedicated database (through the maintenance DSN).
|
|
pub(crate) async fn create_db(maintenance_dsn: &str, db: &str) -> Result<(), Error> {
|
|
let client = admin_connect(maintenance_dsn).await?;
|
|
client
|
|
.execute(&format!("CREATE DATABASE \"{db}\""), &[])
|
|
.await
|
|
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
|
|
Ok(())
|
|
}
|
|
|
|
/// Whether a dedicated database still exists server-side — the
|
|
/// structural teardown tests' instrument (`pg_database`).
|
|
pub(crate) async fn db_exists(maintenance_dsn: &str, db: &str) -> bool {
|
|
let client = admin_connect(maintenance_dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let count: i64 = client
|
|
.query_one(
|
|
"SELECT count(*) FROM pg_database WHERE datname = $1",
|
|
&[&db],
|
|
)
|
|
.await
|
|
.expect("the oracle query succeeds")
|
|
.get(0);
|
|
count > 0
|
|
}
|
|
|
|
/// Drop a dedicated database — `WITH (FORCE)` reaps any connection
|
|
/// the store's pool has not yet drained (postgres ≥ 13; the
|
|
/// panicking-row-safe arm; the same clause the engine suite's
|
|
/// janitor carries).
|
|
pub(crate) async fn drop_db(maintenance_dsn: &str, db: &str) -> Result<(), Error> {
|
|
let client = admin_connect(maintenance_dsn).await?;
|
|
client
|
|
.execute(&format!("DROP DATABASE \"{db}\" WITH (FORCE)"), &[])
|
|
.await
|
|
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
|
|
Ok(())
|
|
}
|
|
|
|
/// The zero-residue probe inside one dedicated database — the LO
|
|
/// catalog and the companion table agree exactly (no missing pages,
|
|
/// no orphans): the LO-isolation corollary's structural assertion,
|
|
/// the engine suite's C1 exact-count shape.
|
|
pub(crate) async fn zero_lo_residue(dsn: &str, schema: &str) -> bool {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let sql = format!(
|
|
"SELECT count(*) FROM (\
|
|
SELECT loid FROM \"{schema}\".lo_entries \
|
|
EXCEPT SELECT DISTINCT loid FROM pg_largeobject\
|
|
) missing, (\
|
|
SELECT DISTINCT loid FROM pg_largeobject \
|
|
EXCEPT SELECT loid FROM \"{schema}\".lo_entries\
|
|
) orphaned"
|
|
);
|
|
let disagreements: i64 = client
|
|
.query_one(sql.as_str(), &[])
|
|
.await
|
|
.expect("the oracle query succeeds")
|
|
.get(0);
|
|
disagreements == 0
|
|
}
|
|
|
|
/// Reap one schema's committed LOs before the schema goes away,
|
|
/// because Large Objects are database-level and a bare `DROP SCHEMA`
|
|
/// would strand their pages in `pg_largeobject`. The loids are read
|
|
/// from the companion table (the contract authority — never
|
|
/// `pg_largeobject`); each is unlinked best-effort so an idempotent
|
|
/// teardown is never torn by a missing oid; uncommitted staged pages
|
|
/// cannot exist (the held-tx rollback stories). Returns the reaped
|
|
/// LOIDs — the tracked-LOID list the structural tests assert against.
|
|
pub(crate) async fn reap_lo_schema(client: &tokio_postgres::Client, schema: &str) -> Vec<u32> {
|
|
let present = table_exists_conn(client, schema, "lo_entries").await;
|
|
if !present {
|
|
return Vec::new();
|
|
}
|
|
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries");
|
|
let loids: Vec<u32> = client
|
|
.query(sql.as_str(), &[])
|
|
.await
|
|
.expect("the loid enumeration succeeds")
|
|
.iter()
|
|
.map(|row| row.get::<_, u32>(0))
|
|
.collect();
|
|
for loid in &loids {
|
|
let _ = client.execute("SELECT lo_unlink($1)", &[loid]).await;
|
|
}
|
|
loids
|
|
}
|
|
|
|
/// The drop-guard janitor — the engine suite's recorded
|
|
/// `DatabaseJanitor`-shaped pattern (`src/pg_common.rs`'s
|
|
/// test-support janitor, `#[cfg(test)]`-internal there): its drop
|
|
/// removes the dedicated database when *Drop* is the only exit — a
|
|
/// row that panics before teardown, a factory released without one.
|
|
/// Same Drop mechanics: no async teardown exists at Drop, so the drop
|
|
/// executes synchronously from a std thread over a one-shot
|
|
/// current-thread runtime; an unreachable harness server is tolerated.
|
|
pub(crate) struct DbJanitor {
|
|
maintenance_dsn: String,
|
|
db: String,
|
|
}
|
|
|
|
impl DbJanitor {
|
|
/// Register the guard for one dedicated database. Constructed as
|
|
/// part of `open()`'s bookkeeping; it fires only if teardown never
|
|
/// consumed the record (teardown disarms it). Dropping an
|
|
/// already-dropped database is a tolerated no-op server-side.
|
|
pub(crate) fn new(maintenance_dsn: &str, db: &str) -> DbJanitor {
|
|
DbJanitor {
|
|
maintenance_dsn: maintenance_dsn.to_owned(),
|
|
db: db.to_owned(),
|
|
}
|
|
}
|
|
}
|
|
|
|
impl Drop for DbJanitor {
|
|
fn drop(&mut self) {
|
|
let maintenance_dsn = self.maintenance_dsn.clone();
|
|
let db = self.db.clone();
|
|
let _ = std::thread::spawn(move || {
|
|
let Ok(rt) = tokio::runtime::Builder::new_current_thread()
|
|
.enable_all()
|
|
.build()
|
|
else {
|
|
return;
|
|
};
|
|
rt.block_on(async move {
|
|
let Ok((client, conn)) = tokio_postgres::connect(&maintenance_dsn, NoTls).await
|
|
else {
|
|
return;
|
|
};
|
|
tokio::spawn(async move {
|
|
let _ = conn.await;
|
|
});
|
|
let _ = client
|
|
.execute(&format!("DROP DATABASE \"{db}\" WITH (FORCE)"), &[])
|
|
.await;
|
|
});
|
|
})
|
|
.join();
|
|
}
|
|
}
|
|
|
|
/// One open's tracked backing site: the dedicated database this open
|
|
/// minted, the engine-owned schemas minted inside it (the alkstore
|
|
/// pg-factory pattern's schema registry, housed on its own database),
|
|
/// and the drop-guard janitor that disposes of the database if
|
|
/// teardown never gets to it (the abandoned paths — a row that panics
|
|
/// before teardown releases the factory into a Drop that runs the
|
|
/// guards). Teardown consumes the record, reaps + drops explicitly,
|
|
/// and disarms the guard.
|
|
pub(crate) struct TrackedDb {
|
|
/// The dedicated database's name (the structural tests'
|
|
/// `pg_database` probe target).
|
|
pub(crate) name: String,
|
|
/// The db-scoped DSN — the store's connection settings and the
|
|
/// oracle's address into this database.
|
|
pub(crate) dsn: String,
|
|
/// Every engine-owned schema minted inside the database (the kv
|
|
/// and pg-lo halves, in mint order — the pg-factory pattern's
|
|
/// registry).
|
|
pub(crate) schemas: Vec<String>,
|
|
/// The pg-lo halves — the schemas carrying `lo_entries`, whose
|
|
/// LOs teardown reaps.
|
|
pub(crate) lo_schemas: Vec<String>,
|
|
/// The janitor disarmed by teardown; Drop runs it otherwise.
|
|
janitor: Option<DbJanitor>,
|
|
}
|
|
|
|
impl TrackedDb {
|
|
/// Register one tracked database — the constructor the columns'
|
|
/// factories share (the janitor field is private; the factory
|
|
/// records the maintenance DSN and the database's name, and the
|
|
/// janitor backs every abandoned exit path).
|
|
pub(crate) fn new(
|
|
name: String,
|
|
dsn: String,
|
|
schemas: Vec<String>,
|
|
lo_schemas: Vec<String>,
|
|
maintenance_dsn: &str,
|
|
) -> TrackedDb {
|
|
TrackedDb {
|
|
janitor: Some(DbJanitor::new(maintenance_dsn, &name)),
|
|
name,
|
|
dsn,
|
|
schemas,
|
|
lo_schemas,
|
|
}
|
|
}
|
|
|
|
/// Disarm the janitor (teardown took over the disposal).
|
|
pub(crate) fn disarm(&mut self) {
|
|
self.janitor = None;
|
|
}
|
|
}
|
|
|
|
// ------------------------------------------------------------------
|
|
// The medium oracle (the dispatch rows' instrument, adapted to the LO
|
|
// medium): an independent connection per probe — large entries are
|
|
// companion rows + LOs (the `lo_entries` count and the `pg_largeobject`
|
|
// pages behind each row's LOID), kv entries per the factory's kv
|
|
// engine, and the schedule itself is `gc_window` rows in the
|
|
// window-hosting schema (the pg-lo schema on the mem-pglo factory,
|
|
// the kv schema on the postgres-pglo factory).
|
|
// ------------------------------------------------------------------
|
|
|
|
async fn table_exists_conn(client: &tokio_postgres::Client, schema: &str, table: &str) -> bool {
|
|
let qualified = format!("\"{schema}\".\"{table}\"");
|
|
let row = client
|
|
.query_one("SELECT to_regclass($1) IS NOT NULL", &[&qualified])
|
|
.await
|
|
.expect("the table probe succeeds");
|
|
row.get(0)
|
|
}
|
|
|
|
/// Rows in one of the store's schemas' tables — the kv tier's pool
|
|
/// (`blobs`), the durable window's staged candidates (`gc_window`),
|
|
/// or the pg-lo tier's companion table (`lo_entries`) — counted from
|
|
/// the oracle's own independent connection to the shared server.
|
|
pub(crate) async fn pg_table_rows(dsn: &str, schema: &str, table: &str) -> usize {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let sql = format!("SELECT COUNT(*) FROM \"{schema}\".{table}");
|
|
let row = client
|
|
.query_one(sql.as_str(), &[])
|
|
.await
|
|
.expect("the oracle query succeeds");
|
|
let count: i64 = row.get(0);
|
|
count.max(0) as usize
|
|
}
|
|
|
|
/// The pg-lo tier's companion-table residency — exactly one row per
|
|
/// committed large-tier entry (the contract authority).
|
|
pub(crate) async fn lo_entry_rows(dsn: &str, schema: &str) -> usize {
|
|
pg_table_rows(dsn, schema, "lo_entries").await
|
|
}
|
|
|
|
/// The durable window's staged-candidate residency — the schedule
|
|
/// observer the durable rows assert against (staged rows land here;
|
|
/// the executing sweep's take empties them).
|
|
pub(crate) async fn window_rows(dsn: &str, schema: &str) -> usize {
|
|
pg_table_rows(dsn, schema, "gc_window").await
|
|
}
|
|
|
|
/// The companion table's loids — the tracked-LOID list the residue
|
|
/// probes and placement assertions ride.
|
|
pub(crate) async fn companion_loids(dsn: &str, schema: &str) -> Vec<u32> {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries ORDER BY loid");
|
|
client
|
|
.query(sql.as_str(), &[])
|
|
.await
|
|
.expect("the oracle query succeeds")
|
|
.iter()
|
|
.map(|row| row.get::<_, u32>(0))
|
|
.collect()
|
|
}
|
|
|
|
/// The LOID behind one companion row — looked up by the entry's key
|
|
/// bytes.
|
|
pub(crate) async fn loid_for_key(dsn: &str, schema: &str, key_bytes: &[u8]) -> Option<u32> {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries WHERE key = $1");
|
|
client
|
|
.query_opt(sql.as_str(), &[&key_bytes])
|
|
.await
|
|
.expect("the oracle query succeeds")
|
|
.map(|row| row.get::<_, u32>(0))
|
|
}
|
|
|
|
/// The `pg_largeobject` pages behind one LOID — the LO-presence probe
|
|
/// the placement oracle and the zero-residue teardown assertions
|
|
/// ride (the catalog read is address-adjacent, never addressing: the
|
|
/// companion table remains the authority).
|
|
pub(crate) async fn loid_pages(dsn: &str, loid: u32) -> i64 {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let row = client
|
|
.query_one(
|
|
"SELECT count(*) FROM pg_largeobject WHERE loid = $1",
|
|
&[&loid],
|
|
)
|
|
.await
|
|
.expect("the oracle query succeeds");
|
|
row.get(0)
|
|
}
|
|
|
|
/// Whether an engine-owned schema carries one of its tables — the
|
|
/// window-host assertions' instrument (`to_regclass`; the dormancy
|
|
/// row's genuine-absence probe).
|
|
pub(crate) async fn table_exists(dsn: &str, schema: &str, table: &str) -> bool {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
table_exists_conn(&client, schema, table).await
|
|
}
|
|
|
|
/// Whether the engine-owned schema still exists server-side — the
|
|
/// structural teardown tests' instrument (`pg_namespace`, the same
|
|
/// probe the alkstore factory shape uses).
|
|
pub(crate) async fn schema_exists(dsn: &str, schema: &str) -> bool {
|
|
let client = admin_connect(dsn)
|
|
.await
|
|
.expect("the oracle connection opens");
|
|
let count: i64 = client
|
|
.query_one(
|
|
"SELECT count(*) FROM pg_namespace WHERE nspname = $1",
|
|
&[&schema],
|
|
)
|
|
.await
|
|
.expect("the oracle query succeeds")
|
|
.get(0);
|
|
count > 0
|
|
}
|
|
|
|
// ------------------------------------------------------------------
|
|
// The mem-pglo factory — the one-window-host rule's named test
|
|
// topology, feature pair `mem` + `pg-lo`.
|
|
// ------------------------------------------------------------------
|
|
|
|
#[cfg(all(feature = "mem", feature = "pg-lo"))]
|
|
mod mem_pglo {
|
|
use std::io::Cursor;
|
|
use std::sync::atomic::AtomicU64;
|
|
use std::sync::{Arc, Mutex};
|
|
use std::time::{Duration, SystemTime};
|
|
|
|
use tokio_postgres::NoTls;
|
|
|
|
use crate::factory::StoreFactory;
|
|
use crate::pg_lo_column::{
|
|
admin_dsn, companion_loids, create_db, db_dsn, db_exists, drop_db, fresh_db_name,
|
|
harness_dsn, harness_ready, lo_entry_rows, loid_for_key, loid_pages, reap_lo_schema,
|
|
window_rows, DbJanitor, TrackedDb, LO_WINDOW, WINDOW_DELAY,
|
|
};
|
|
use crate::rows::{
|
|
self, patterned, slice_digest, Failing, LatchProtect, SetProtect, SetSource,
|
|
};
|
|
use alkblobs::{Error, Key, PgOpts, Store, StoreMode, DEFAULT_DISPATCH_THRESHOLD};
|
|
|
|
/// The column's declared dual-tier threshold — small, so the
|
|
/// shared content matrix and the race fixtures cross it and both
|
|
/// tiers' bodies are exercised (sub-threshold rows land in the
|
|
/// in-process mem kv tier; over-threshold rows stage-commit into
|
|
/// LOs behind companion rows).
|
|
pub(crate) const TUNED_THRESHOLD: u64 = 1024;
|
|
|
|
/// One vended backing site: the dedicated database this open
|
|
/// minted and the engine-owned pg-lo schema minted inside it. The
|
|
/// kv tier is in-process mem — its placement is facade-observed
|
|
/// (the mem introspection shape), the LO medium is the schema.
|
|
#[derive(Clone)]
|
|
pub(crate) struct Site {
|
|
/// The db-scoped DSN — the oracle's independent connections
|
|
/// address it; each open's database is private and gets
|
|
/// dropped whole at teardown (the LO-isolation corollary: LOs
|
|
/// are database-level).
|
|
dsn: String,
|
|
/// The engine-owned pg-lo schema this open minted — the LO
|
|
/// medium (the dispatch oracle's LO-side counters and the
|
|
/// durable window's row home, per the one-window-host rule).
|
|
schema: String,
|
|
/// The dedicated database's name — the structural teardown
|
|
/// tests' `pg_database` probe target.
|
|
db: String,
|
|
}
|
|
|
|
impl Site {
|
|
/// The minted pg-lo schema — the LO medium's and the window's
|
|
/// schema-qualified probe target.
|
|
pub(crate) fn schema(&self) -> &str {
|
|
&self.schema
|
|
}
|
|
|
|
/// The DSN the open minted — scoped to the dedicated database.
|
|
pub(crate) fn dsn(&self) -> &str {
|
|
&self.dsn
|
|
}
|
|
}
|
|
|
|
/// The dispatch-coverage factory: `kv = mem` + `large = pg-lo` at
|
|
/// the threshold this column declares and the test tuning of the
|
|
/// window delay (see [`TUNED_THRESHOLD`] and `WINDOW_DELAY` — the
|
|
/// one-window-host rule's named test topology, the durable window
|
|
/// hosting on the large engine, ADR-012 §5).
|
|
///
|
|
/// **Why a fresh instance per runner (not a shared static):**
|
|
/// teardown drops the dedicated databases *this instance* minted
|
|
/// (having reaped their LOs and dropped their schemas); a shared
|
|
/// instance's teardown would race concurrently running rows'
|
|
/// sites. Every runner constructs its own and owns its lifecycle;
|
|
/// a row that panics before teardown is still backed by the
|
|
/// per-open drop-guard janitors.
|
|
pub(crate) struct DualTierMemPgLoFactory {
|
|
dsn: String,
|
|
/// The declared dispatch threshold — the routing boundary this
|
|
/// column's rows run at (constructor-tunable; ADR-003).
|
|
/// Declared configuration: the dual-tier store never rejects,
|
|
/// so there is nothing to probe.
|
|
threshold: u64,
|
|
/// The declared window delay — the durable schedule's cadence
|
|
/// knob the sweep rows run at (the test tuning:
|
|
/// execute-on-next-sweep).
|
|
#[allow(dead_code)]
|
|
window_delay: Duration,
|
|
seq: AtomicU64,
|
|
/// Every dedicated database this instance minted, for its own
|
|
/// teardown and its drop-guard janitors.
|
|
dbs: Mutex<Vec<TrackedDb>>,
|
|
/// The LOIDs teardown reaped, for the structural tests'
|
|
/// assertions.
|
|
reaped_loids: Mutex<Vec<u32>>,
|
|
}
|
|
|
|
impl DualTierMemPgLoFactory {
|
|
pub(crate) fn new(threshold: u64) -> DualTierMemPgLoFactory {
|
|
DualTierMemPgLoFactory {
|
|
dsn: harness_dsn().expect("the harness server is configured for the column"),
|
|
threshold,
|
|
window_delay: WINDOW_DELAY,
|
|
seq: AtomicU64::new(
|
|
(std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.subsec_nanos() as u64)
|
|
.unwrap_or(0))
|
|
^ ((std::process::id() as u64) << 8),
|
|
),
|
|
dbs: Mutex::new(Vec::new()),
|
|
reaped_loids: Mutex::new(Vec::new()),
|
|
}
|
|
}
|
|
|
|
/// Open + track the vended site: the shape the column's own
|
|
/// rows use (they observe the LO medium, so they need the site
|
|
/// alongside the store).
|
|
pub(crate) async fn open_tracked(&self) -> Result<(Store, Site), Error> {
|
|
let maintenance = admin_dsn(&self.dsn);
|
|
let db = fresh_db_name("mem", &self.seq);
|
|
create_db(&maintenance, &db).await?;
|
|
let dsn = db_dsn(&self.dsn, &db);
|
|
let schema = crate::pg_lo_column::fresh_schema_name("mem", &self.seq);
|
|
self.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.push(TrackedDb {
|
|
name: db.clone(),
|
|
dsn: dsn.clone(),
|
|
schemas: vec![schema.clone()],
|
|
lo_schemas: vec![schema.clone()],
|
|
janitor: Some(DbJanitor::new(&maintenance, &db)),
|
|
});
|
|
let store =
|
|
Store::dual_tier_mem_pglo(&dsn, opts(schema.clone()), self.threshold, WINDOW_DELAY)
|
|
.await?;
|
|
Ok((store, Site { dsn, schema, db }))
|
|
}
|
|
|
|
/// The dedicated databases this instance minted — names,
|
|
/// db-scoped DSNs, and the schemas minted inside — for the
|
|
/// structural tests' assertions.
|
|
pub(crate) fn tracked_dbs(&self) -> Vec<(String, String, Vec<String>)> {
|
|
self.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.iter()
|
|
.map(|d| (d.name.clone(), d.dsn.clone(), d.schemas.clone()))
|
|
.collect()
|
|
}
|
|
|
|
/// The LOIDs teardown reaped, for the structural tests'
|
|
/// tracked-LOID assertions.
|
|
pub(crate) fn reaped_loids(&self) -> Vec<u32> {
|
|
self.reaped_loids
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.clone()
|
|
}
|
|
|
|
/// The maintenance DSN the factory mints databases through.
|
|
pub(crate) fn dsn(&self) -> &str {
|
|
&self.dsn
|
|
}
|
|
}
|
|
|
|
/// The factories' schema-name policy, in engine-opts form: the
|
|
/// constructor's `PgOpts` with exactly the factory's minted
|
|
/// schema and the ship defaults for the rest (pool size,
|
|
/// `synchronous_commit` — the alkstore factory's shape: only the
|
|
/// schema name is factory policy).
|
|
fn opts(schema: String) -> PgOpts {
|
|
PgOpts {
|
|
schema,
|
|
..PgOpts::default()
|
|
}
|
|
}
|
|
|
|
impl StoreFactory for DualTierMemPgLoFactory {
|
|
fn name(&self) -> &'static str {
|
|
"dual-tier mem+pg-lo"
|
|
}
|
|
|
|
async fn open(&self) -> Result<Store, Error> {
|
|
self.open_tracked().await.map(|(store, _)| store)
|
|
}
|
|
|
|
async fn teardown(&self) -> Result<(), Error> {
|
|
let mut dbs = self
|
|
.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.drain(..)
|
|
.collect::<Vec<_>>();
|
|
for tracked in dbs.iter_mut() {
|
|
// The LO-isolation corollary's arms: reap the tracked
|
|
// LOIDs (via the companion table — the contract
|
|
// authority), drop exactly the minted schemas, then
|
|
// drop the dedicated database whole (`WITH (FORCE)`
|
|
// reaps any connection the abandoned store's pool has
|
|
// not drained).
|
|
let maintenance = admin_dsn(&self.dsn);
|
|
let mut reaped = Vec::new();
|
|
if let Ok(client) = admin_connect_for_teardown(&tracked.dsn).await {
|
|
for schema in &tracked.lo_schemas {
|
|
reaped.extend(reap_lo_schema(&client, schema).await);
|
|
}
|
|
for schema in &tracked.schemas {
|
|
let sql = format!("DROP SCHEMA IF EXISTS \"{schema}\" CASCADE");
|
|
let _ = client.batch_execute(&sql).await;
|
|
}
|
|
}
|
|
if !reaped.is_empty() {
|
|
self.reaped_loids
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.extend(reaped);
|
|
}
|
|
drop_db(&maintenance, &tracked.name).await?;
|
|
TrackedDb::disarm(tracked);
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
async fn admin_connect_for_teardown(dsn: &str) -> Result<tokio_postgres::Client, Error> {
|
|
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
|
|
.await
|
|
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
|
|
tokio::spawn(async move {
|
|
let _ = connection.await;
|
|
});
|
|
Ok(client)
|
|
}
|
|
|
|
/// The skip gate every runner checks (the SKIP posture's
|
|
/// factory-owner responsibility).
|
|
async fn ready() -> bool {
|
|
harness_ready().await
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The column's own rows — the durable-schedule sweep family, the
|
|
// schedule-relative readings, the dispatch forms, the INV-1
|
|
// extension, and the virgin-store row. Every row takes this
|
|
// column's concrete factory (it observes the LO medium), is
|
|
// ADR-stamped per the convention, and traces its family.
|
|
// --------------------------------------------------------------
|
|
|
|
/// INV-2 on the durable schedule, over both tiers: identical
|
|
/// content put twice — through the same put path and across the
|
|
/// two paths — is one pool entry, and the re-put's `Pin` protects
|
|
/// exactly as a fresh put's does. The kv leg's single-pool-entry
|
|
/// is asserted on the facade plus the LO medium's genuine absence
|
|
/// (the mem introspection shape — the oracle's kv half); the large
|
|
/// leg's is one companion row behind one LO. One-pool-entry reads
|
|
/// through durable-schedule outcomes (the INV-5 posture): the pins
|
|
/// held stage nothing; once unpinned, sweep 1 stages exactly the
|
|
/// one entry and removes nothing, and its executor sweep deletes
|
|
/// exactly that staged set (README §The schedule-relative reading
|
|
/// for SQL-backed columns).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 2; ADR-003 (whole-value
|
|
/// put is CAS first-writer-wins; the routing function's tier
|
|
/// placement); ADR-005 (a re-put refcounts the existing entry
|
|
/// under the same arbitration lock); ADR-009 §2/§4 (the companion
|
|
/// table is the contract authority; delete = lo_unlink + row
|
|
/// delete); ADR-008 §2 (the staged re-arbitrated delete); ADR-012
|
|
/// §5 (the window hosts on the pg-lo engine in this topology);
|
|
/// ADR-012 §6.7 (the executor is the next sweep — the schedule
|
|
/// these counts read against).
|
|
pub async fn dedup_identical_content_is_one_pool_entry(factory: &DualTierMemPgLoFactory) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
// The kv leg (in-process mem): both refcounts compose under
|
|
// the same lock, nothing ever lands on the LO medium.
|
|
let content = b"deduplicated content".to_vec();
|
|
let canonical = Key::derive(&content, content.len() as u64);
|
|
let (key, first) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the fresh put publishes");
|
|
let (reput, second) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the same-path re-put publishes");
|
|
assert_eq!(key, canonical);
|
|
assert_eq!(reput, canonical, "one derivation, one address");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.aborted, None);
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"one pool entry, both refcounts composing under the same lock"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the kv-tier entries never land on the LO medium"
|
|
);
|
|
drop(first);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"the re-put's pin protects exactly as a fresh put's"
|
|
);
|
|
drop(second);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one pool entry all along — staged this sweep, not executed"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.deleted, report.cancelled_by_arbitration),
|
|
(1, 0),
|
|
"the executing sweep deleted exactly the staged entry"
|
|
);
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
|
|
// The large leg: identical content through both put paths —
|
|
// one companion row behind one LO, both refcounts composing.
|
|
let cross = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
|
|
let cross_canonical = Key::derive(&cross, cross.len() as u64);
|
|
let (key, first) = store
|
|
.put(Some(cross.len() as u64), Cursor::new(cross.as_slice()))
|
|
.await
|
|
.expect("the known-length put publishes");
|
|
let (reput, second) = store
|
|
.put(None, Cursor::new(cross.as_slice()))
|
|
.await
|
|
.expect("the unknown-length re-put publishes");
|
|
assert_eq!(key, cross_canonical);
|
|
assert_eq!(reput, cross_canonical, "one pool entry across the paths");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.aborted, report.candidates_staged),
|
|
(None, 0),
|
|
"both refcounts compose"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
1,
|
|
"one pool entry across both paths — one companion row"
|
|
);
|
|
let loids = companion_loids(site.dsn(), site.schema()).await;
|
|
assert_eq!(loids.len(), 1, "one LO behind the companion row");
|
|
assert!(
|
|
loid_pages(site.dsn(), loids[0]).await > 0,
|
|
"the LO holds the content's pages (never zero)"
|
|
);
|
|
drop(second);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"the re-put's pin protects exactly as a fresh put's (other drop order)"
|
|
);
|
|
drop(first);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one pool entry all along"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "executed exactly the staged set");
|
|
assert!(!store.has(&cross_canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the executed delete emptied the companion table"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), loids[0]).await,
|
|
0,
|
|
"delete is one tx: the LO's pages went with the row"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the window holds nothing after both executed rounds (the pg-lo host)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-3 on the durable schedule: stage hygiene over the LO
|
|
/// medium's real staged form — every put-path failure leaves zero
|
|
/// residue (no companion rows, no LO pages: the held-put
|
|
/// transaction's every failure path converges on ROLLBACK) and
|
|
/// every success leaves exactly one committed entry, over both put
|
|
/// paths and both injectable failure modes (a mid-stream stream
|
|
/// failure and a declared-length mismatch), observed through
|
|
/// durable-schedule sweep outcomes: zero staged after every
|
|
/// failure, and a success staged exactly once, then executed
|
|
/// exactly once.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 3; ADR-010 §1 (the
|
|
/// staged-put contract: commit publishes, every failure path
|
|
/// discards); ADR-003 (stage-then-commit; the routing places this
|
|
/// content on the large tier); ADR-009 §2 (put = one tx — the
|
|
/// staged form is the held tx, so a failure's LO pages roll back
|
|
/// with it); ADR-012 §6.7 (the schedule-relative reading the
|
|
/// success counts follow).
|
|
pub async fn stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let content = patterned(2 * TUNED_THRESHOLD as usize + 17, 167);
|
|
let len = content.len() as u64;
|
|
let canonical = Key::derive(&content, len);
|
|
|
|
// The known path's mid-stream failure — the held tx rolls
|
|
// back: no companion row, no LO pages.
|
|
let err = store
|
|
.put(Some(len), Failing::new(&content, 40))
|
|
.await
|
|
.expect_err("the failed mid-stream put rejects");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
// The unknown path's buffered failure (before the threshold
|
|
// overflow) and its staged failure (the held tx after the
|
|
// overflow) — both roll back to zero residue.
|
|
let err = store
|
|
.put(None, Failing::new(&content, 40))
|
|
.await
|
|
.expect_err("the failed unknown-length put rejects before staging");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
let err = store
|
|
.put(None, Failing::new(&content, TUNED_THRESHOLD as usize + 600))
|
|
.await
|
|
.expect_err("the failed unknown-length put rejects mid-stage");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
// Declared-length mismatches, over-fed and under-fed — both
|
|
// above the threshold, so the staged form began its LO and
|
|
// every failure path discards it.
|
|
let err = store
|
|
.put(Some(TUNED_THRESHOLD + 10), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect_err("the over-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
let err = store
|
|
.put(Some(len + 17), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect_err("the under-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.aborted, None);
|
|
assert_eq!(
|
|
(
|
|
report.candidates_staged,
|
|
report.deleted,
|
|
report.cancelled_by_arbitration
|
|
),
|
|
(0, 0, 0),
|
|
"zero residue after every failure, both put paths, both failure modes"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the companion table holds zero entries after every failure"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the window staged nothing behind the failures"
|
|
);
|
|
|
|
// Success, known path: exactly one committed entry — one
|
|
// companion row behind one LO — staged once, executed once.
|
|
let (key, pin) = store
|
|
.put(Some(len), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the known-length put publishes");
|
|
assert_eq!(key, canonical);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
1,
|
|
"exactly one committed entry on success (one companion row)"
|
|
);
|
|
let loid = loid_for_key(site.dsn(), site.schema(), &key.as_bytes())
|
|
.await
|
|
.expect("the companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), loid).await > 0,
|
|
"the committed LO holds the content's pages"
|
|
);
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed entry staged on success"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.deleted, 1,
|
|
"the executing sweep removed exactly that one entry"
|
|
);
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(loid_pages(site.dsn(), loid).await, 0);
|
|
|
|
// Success, unknown path: the overflow staged into the held tx,
|
|
// the commit published — exactly one committed entry on the
|
|
// other put path too.
|
|
let (key, pin) = store
|
|
.put(None, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the unknown-length put publishes");
|
|
assert_eq!(key, canonical);
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed entry staged on the other put path too"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
window_rows(site.dsn(), site.schema()).await
|
|
),
|
|
(0, 0),
|
|
"the medium and the window are both back to zero"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, the durable-schedule happy shape over both tiers: the
|
|
/// full protection set spanning both tiers (a source-live kv
|
|
/// entry, a pinned large-tier entry, a protect-named large-tier
|
|
/// entry) plus one dead entry per tier — sweep 1 stages exactly
|
|
/// the two dead candidates into the **pg-lo schema's `gc_window`**
|
|
/// (the window hosted on the large engine — the one-window-host
|
|
/// rule's named topology) and deletes nothing; sweep 2 executes
|
|
/// exactly that staged set, and the protected entries stay
|
|
/// byte-identical. The medium oracle distinguishes the tiers (the
|
|
/// companion table's rows behind LOs vs the kv tier's
|
|
/// facade-observed residency) and watches the schedule itself.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4 (exact counts with
|
|
/// correct liveness) and §Invariants 5 (list correctness through
|
|
/// sweep outcomes); ADR-005 (three liveness marks; delete windows
|
|
/// with arbitrate-at-delete-time); ADR-008 §2 (the staged
|
|
/// re-arbitrated delete; the constructor-tunable delay); ADR-009
|
|
/// §2/§4 (the companion table; GC-participating deletes); ADR-012
|
|
/// §1 (sweeps are never errors); ADR-012 §5 (the one-window-host
|
|
/// rule's named topology — the window on the large engine); ADR-012
|
|
/// §6.7 (the executor is the next sweep).
|
|
pub async fn two_sweep_exact_counts_over_the_full_protection_set(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
let source = SetSource::register(&store);
|
|
let protect = Arc::new(SetProtect::default());
|
|
store.register_protect_callback(protect.clone());
|
|
|
|
let source_live = patterned((threshold - 1) as usize, 191);
|
|
let (key_live, _) = store
|
|
.put(
|
|
Some(source_live.len() as u64),
|
|
Cursor::new(source_live.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the source-live entry publishes");
|
|
source.add(key_live.clone());
|
|
|
|
let pinned = patterned(2 * threshold as usize + 7, 173);
|
|
let (key_pinned, pin_pinned) = store
|
|
.put(None, Cursor::new(pinned.as_slice()))
|
|
.await
|
|
.expect("the pinned entry publishes");
|
|
|
|
let protected = patterned(2 * threshold as usize + 11, 167);
|
|
let (key_protected, _) = store
|
|
.put(
|
|
Some(protected.len() as u64),
|
|
Cursor::new(protected.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the protect-named entry publishes");
|
|
protect.set_keys(vec![key_protected.clone()]);
|
|
|
|
let dead_kv = patterned((threshold - 3) as usize, 149);
|
|
let (key_dead_kv, pin) = store
|
|
.put(Some(dead_kv.len() as u64), Cursor::new(dead_kv.as_slice()))
|
|
.await
|
|
.expect("the kv-tier dead entry publishes");
|
|
drop(pin);
|
|
let dead_large = patterned(2 * threshold as usize + 13, 151);
|
|
let (key_dead_large, pin) = store
|
|
.put(None, Cursor::new(dead_large.as_slice()))
|
|
.await
|
|
.expect("the large-tier dead entry publishes");
|
|
drop(pin);
|
|
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
3,
|
|
"before the sweep, the LO medium holds exactly the three \
|
|
large-tier entries (the kv tier's entries are in-process)"
|
|
);
|
|
let dead_large_loid = loid_for_key(site.dsn(), site.schema(), &key_dead_large.as_bytes())
|
|
.await
|
|
.expect("the dead large entry's companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), dead_large_loid).await > 0,
|
|
"the LO medium holds the dead large entry's pages too"
|
|
);
|
|
assert!(
|
|
store.has(&key_live.as_bytes()).await.unwrap()
|
|
&& store.has(&key_dead_kv.as_bytes()).await.unwrap()
|
|
&& store.has(&key_dead_large.as_bytes()).await.unwrap(),
|
|
"the pool spans both tiers and holds exactly its five entries"
|
|
);
|
|
|
|
let first = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(first.aborted, None);
|
|
assert_eq!(
|
|
(
|
|
first.candidates_staged,
|
|
first.deleted,
|
|
first.cancelled_by_arbitration
|
|
),
|
|
(2, 0, 0),
|
|
"sweep 1 staged exactly the two dead entries and deleted \
|
|
nothing — staging is not executing"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
2,
|
|
"the staged set waits in the durable window — hosted on the \
|
|
pg-lo schema (the one-window-host rule's large-engine arm)"
|
|
);
|
|
for (key, content) in [
|
|
(&key_live, &source_live),
|
|
(&key_pinned, &pinned),
|
|
(&key_protected, &protected),
|
|
] {
|
|
let (len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the protected entry is present");
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the window reads"),
|
|
content.as_slice(),
|
|
"the protected entry is byte-identical after the staging sweep"
|
|
);
|
|
}
|
|
assert!(
|
|
store.has(&key_live.as_bytes()).await.unwrap()
|
|
&& store.has(&key_pinned.as_bytes()).await.unwrap()
|
|
&& store.has(&key_protected.as_bytes()).await.unwrap(),
|
|
"the staged sweep deleted nothing: every entry intact"
|
|
);
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(second.aborted, None);
|
|
assert_eq!(
|
|
second.candidates_staged, 0,
|
|
"nothing newly dead since the staging sweep"
|
|
);
|
|
assert_eq!(
|
|
second.deleted + second.cancelled_by_arbitration,
|
|
2,
|
|
"sweep 2 executed exactly the staged set"
|
|
);
|
|
assert_eq!(
|
|
(second.deleted, second.cancelled_by_arbitration),
|
|
(2, 0),
|
|
"with correct liveness, every executed candidate deleted"
|
|
);
|
|
assert!(!store.has(&key_dead_kv.as_bytes()).await.unwrap());
|
|
assert!(!store.has(&key_dead_large.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
2,
|
|
"the executed deletes landed on both media exactly (the dead \
|
|
large entry's companion row is gone)"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), dead_large_loid).await,
|
|
0,
|
|
"the dead large entry's LO pages are gone with it"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the executed rows left the durable window"
|
|
);
|
|
|
|
drop(pin_pinned);
|
|
let third = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(third.candidates_staged, third.deleted),
|
|
(1, 0),
|
|
"with the pin dropped, the formerly pinned entry stages"
|
|
);
|
|
let fourth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
fourth.deleted, 1,
|
|
"and the next sweep executes it — exactly one deletion"
|
|
);
|
|
assert!(!store.has(&key_pinned.as_bytes()).await.unwrap());
|
|
assert!(store.has(&key_live.as_bytes()).await.unwrap());
|
|
assert!(store.has(&key_protected.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
1,
|
|
"exactly the protect-named entry's companion row remains"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, the race, in its durable-schedule form: a put landing
|
|
/// *during the window* survives the executing sweep — the re-put's
|
|
/// pin is minted (pin-before-publish) between the staging sweep
|
|
/// and the executing sweep, so the execution-time re-arbitration
|
|
/// cancels the pinned candidate while the unpinned twin deletes;
|
|
/// once the pin drops, the twin re-stages and executes exactly
|
|
/// once. Both put paths race in-body over both tiers (a kv twin
|
|
/// and a large twin), across the window delay the pg-lo host
|
|
/// carries.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4; ADR-005 (pin-before-
|
|
/// publish ordering; the delete window closes the sweep-vs-put race);
|
|
/// ADR-008 §2 (the staged re-arbitrated delete — the race holds
|
|
/// across the window delay); ADR-009 §4 (GC-participating deletes
|
|
/// over the companion table); ADR-012 §5 (the window on the large
|
|
/// engine in this topology); ADR-012 §6.7 (execution-time
|
|
/// re-arbitration is the executor sweep's posture).
|
|
pub async fn a_put_landing_during_the_window_survives_the_executing_sweep(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let rounds: [(String, Vec<u8>, bool, bool, bool); 2] = [
|
|
(
|
|
"the kv twin (known-path re-put)".to_owned(),
|
|
patterned((threshold - 1) as usize, 191),
|
|
true,
|
|
true,
|
|
false,
|
|
),
|
|
(
|
|
"the large twin (unknown-path re-put)".to_owned(),
|
|
patterned(2 * threshold as usize + 5, 173),
|
|
false,
|
|
false,
|
|
true,
|
|
),
|
|
];
|
|
for (index, (tag, content, original_known, reput_known, lands_large)) in
|
|
rounds.iter().enumerate()
|
|
{
|
|
let len = content.len() as u64;
|
|
let twin = Key::derive(content, len);
|
|
let witness = format!("the dead witness of round {index}").into_bytes();
|
|
let witness_key = Key::derive(&witness, witness.len() as u64);
|
|
|
|
let original = if *original_known { Some(len) } else { None };
|
|
let (_, pin) = store
|
|
.put(original, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the twin publishes");
|
|
drop(pin);
|
|
let (_, pin) = store
|
|
.put(None, Cursor::new(witness.as_slice()))
|
|
.await
|
|
.expect("the witness publishes");
|
|
drop(pin);
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(
|
|
staged.aborted,
|
|
staged.candidates_staged,
|
|
staged.deleted,
|
|
staged.cancelled_by_arbitration
|
|
),
|
|
(None, 2, 0, 0),
|
|
"{tag}: the staging sweep stages exactly the two dead \
|
|
twins and deletes nothing"
|
|
);
|
|
|
|
// The put lands during the window (the re-put dedups into
|
|
// the existing twin and refcounts its pin): protected
|
|
// across the delay, pin-before-publish.
|
|
let reput = if *reput_known { Some(len) } else { None };
|
|
let (_, pin) = store
|
|
.put(reput, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the re-put publishes");
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(executed.aborted, None, "{tag}");
|
|
assert_eq!(
|
|
executed.candidates_staged, 0,
|
|
"{tag}: nothing newly dead since the staging sweep"
|
|
);
|
|
assert_eq!(
|
|
executed.deleted + executed.cancelled_by_arbitration,
|
|
2,
|
|
"{tag}: the executing sweep ran exactly the staged set"
|
|
);
|
|
assert_eq!(executed.deleted, 1, "{tag}: the unpinned witness deleted");
|
|
assert_eq!(
|
|
executed.cancelled_by_arbitration, 1,
|
|
"{tag}: the pinned twin's candidate cancelled at execution time"
|
|
);
|
|
assert!(
|
|
!store.has(&witness_key.as_bytes()).await.unwrap(),
|
|
"{tag}: the unpinned twin is gone"
|
|
);
|
|
let (got_len, blob) = store
|
|
.get(&twin.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("{tag}: the put landing during the window survived");
|
|
assert_eq!(got_len, len, "{tag}");
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the window reads"),
|
|
content.as_slice(),
|
|
"{tag}: the surviving twin is byte-identical"
|
|
);
|
|
drop(pin);
|
|
|
|
let restaged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(restaged.candidates_staged, restaged.deleted),
|
|
(1, 0),
|
|
"{tag}: the pin dropped — the twin re-stages"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"{tag}: the re-staged candidate executes exactly once"
|
|
);
|
|
assert!(!store.has(&twin.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"{tag}: the window holds nothing after the executed round \
|
|
(the pg-lo host)"
|
|
);
|
|
if *lands_large {
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"{tag}: the executed LO deletes reaped the medium exactly"
|
|
);
|
|
}
|
|
}
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, abort shape: the sweep is single-flight on the durable
|
|
/// form too — the in-process sweep lease (ADR-013 §2's single-node
|
|
/// analog). The contended second sweeper aborts with
|
|
/// `SweeperLock` in a completed `Ok(SweepReport)` — nothing
|
|
/// executed, nothing staged, nothing deleted (abort before
|
|
/// staging: form-independent) — while the lease holder's sweep
|
|
/// completes by **staging only**: its staged set waits in the
|
|
/// pg-lo schema's `gc_window` for the next sweep to execute (the
|
|
/// durable form's schedule-relative replacement for the reference
|
|
/// row's inline holder-completion leg).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4; ADR-012 §1 (abort is
|
|
/// report data, never an error); ADR-013 §2 (the named exclusive
|
|
/// sweep lease — one sweeper per pool, holdership ends at release);
|
|
/// ADR-012 §5 (the window hosts on the large engine in this
|
|
/// topology); ADR-012 §6.7 (the executor is the next sweep — the
|
|
/// holder stages, it does not execute).
|
|
pub async fn the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
|
|
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
|
|
store.register_protect_callback(Arc::new(LatchProtect {
|
|
entered: entered_tx,
|
|
release: parking_lot::Mutex::new(Some(release_rx)),
|
|
}));
|
|
|
|
let content = patterned(2 * threshold as usize + 9, 167);
|
|
let (key, pin) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the dead witness publishes");
|
|
drop(pin);
|
|
|
|
// The lease holder sweeps on its own task (its protect consult
|
|
// blocks that task's thread mid-callback); this task contends
|
|
// and then releases the holder.
|
|
let store = Arc::new(store);
|
|
let holder = tokio::spawn({
|
|
let store = Arc::clone(&store);
|
|
async move { store.sweep().await }
|
|
});
|
|
|
|
while entered_rx
|
|
.recv_timeout(std::time::Duration::from_millis(50))
|
|
.is_err()
|
|
{
|
|
tokio::task::yield_now().await;
|
|
}
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
second.aborted,
|
|
Some(alkblobs::GcAbortCause::SweeperLock),
|
|
"the contended sweeper aborts as report data"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
second.candidates_staged,
|
|
second.deleted,
|
|
second.cancelled_by_arbitration
|
|
),
|
|
(0, 0, 0),
|
|
"the contended sweep executes, stages, and deletes nothing"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"the contended sweep deleted nothing"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the aborted sweep staged nothing into the window"
|
|
);
|
|
release_tx.send(()).ok();
|
|
|
|
let report = holder
|
|
.await
|
|
.expect("the holder task joined")
|
|
.expect("sweeps are never errors");
|
|
assert_eq!(report.aborted, None, "the lease holder completes");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"the holder stages exactly the dead set — it does not execute \
|
|
(the durable form's schedule)"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
1,
|
|
"the holder's staged set waits in the durable window"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"the holder staged, did not execute: the entry is intact"
|
|
);
|
|
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"the next sweep is the executor of the staged set"
|
|
);
|
|
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the executed delete emptied the companion table"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-5 on the durable schedule: `list()` correctness is
|
|
/// observable only through GC — the enumeration is complete and
|
|
/// well-formed in both directions, proven by exact
|
|
/// durable-schedule outcomes: the staging sweep stages exactly the
|
|
/// dead set (neither more nor fewer candidates), the executing
|
|
/// sweep deletes exactly that staged set, and a liveness flip
|
|
/// re-stages exactly the newly dead entry. Never a direct list
|
|
/// comparison.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 5; ADR-003 (`list()` is
|
|
/// complete and well-formed by contract — load-bearing for GC);
|
|
/// ADR-005 (the sweep's live-set diff rides `list()`); ADR-012 §5
|
|
/// (the window hosts on the large engine in this topology); ADR-012
|
|
/// §6.7 (the executor is the next sweep — the schedule these
|
|
/// counts read against).
|
|
pub async fn list_correctness_is_observable_through_durable_schedule_outcomes(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
let source = SetSource::register(&store);
|
|
|
|
let mut keys = Vec::new();
|
|
for (index, content) in [
|
|
b"kept live by the source, entry one".to_vec(),
|
|
b"kept live by the source, entry two".to_vec(),
|
|
b"dead at the first staging, entry three".to_vec(),
|
|
b"dead at the first staging, entry four".to_vec(),
|
|
b"dead at the first staging, entry five".to_vec(),
|
|
]
|
|
.into_iter()
|
|
.enumerate()
|
|
{
|
|
let declared = if index % 2 == 0 {
|
|
Some(content.len() as u64)
|
|
} else {
|
|
None
|
|
};
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the entry publishes");
|
|
drop(pin);
|
|
keys.push(key);
|
|
}
|
|
source.add(keys[0].clone());
|
|
source.add(keys[1].clone());
|
|
|
|
let first = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(first.aborted, None);
|
|
assert_eq!(
|
|
(first.candidates_staged, first.deleted),
|
|
(3, 0),
|
|
"the enumeration staged exactly the three dead entries — \
|
|
complete and well-formed in both directions"
|
|
);
|
|
for key in &keys {
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"staging deleted nothing: every enumerated entry is intact"
|
|
);
|
|
}
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.schema()).await,
|
|
3,
|
|
"the staged schedule holds the three candidates"
|
|
);
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(second.candidates_staged, second.deleted),
|
|
(0, 3),
|
|
"the executing sweep removed exactly the staged set"
|
|
);
|
|
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
|
|
assert!(store.has(&keys[1].as_bytes()).await.unwrap());
|
|
for key in &keys[2..] {
|
|
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
|
}
|
|
|
|
source.remove(&keys[1]);
|
|
let third = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(third.candidates_staged, third.deleted),
|
|
(1, 0),
|
|
"the newly dead entry is staged exactly"
|
|
);
|
|
let fourth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(fourth.deleted, 1, "and executed exactly by the next sweep");
|
|
assert!(!store.has(&keys[1].as_bytes()).await.unwrap());
|
|
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
|
|
|
|
source.remove(&keys[0]);
|
|
let fifth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(fifth.candidates_staged, 1, "the last entry stages exactly");
|
|
let sixth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(sixth.deleted, 1, "and executes exactly");
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **routing is a pure function of length** — at,
|
|
/// below, and over the declared threshold, the same content lands
|
|
/// on exactly the same tier through both put paths (same placement
|
|
/// every time; nothing moves, nothing duplicates), identical
|
|
/// content through both paths is one pool entry, and every read is
|
|
/// byte-identical from wherever that tier holds it.
|
|
///
|
|
/// Tier placement is observed through the LO medium (the oracle's
|
|
/// LO half): large entries hold a companion row behind an LO; kv
|
|
/// entries hold nothing on the medium — the facade's served reads
|
|
/// and `has()` are the mem kv half's introspection shape (the mem
|
|
/// column's form; the kv tier is in-process, facade-invisible by
|
|
/// design).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
|
|
/// ADR-003 §Dispatch policy (routing is a pure function of length;
|
|
/// no migration; CAS first-writer-wins makes re-puts one entry);
|
|
/// ADR-005 (re-put refcounts); ADR-009 §2 (the companion table is
|
|
/// the contract authority); ADR-012 §5 (this topology's named
|
|
/// shape); ADR-012 §6.3 (the dual-tier dispatch-coverage shape);
|
|
/// ADR-012 §6.7 (the schedule-relative cleanup the dedup legs read
|
|
/// against).
|
|
pub async fn routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
for content in [
|
|
patterned((threshold - 1) as usize, 191),
|
|
patterned(threshold as usize, 173),
|
|
patterned((threshold + 1) as usize, 167),
|
|
patterned(4 * threshold as usize + 9, 151),
|
|
] {
|
|
let len = content.len() as u64;
|
|
let canonical = Key::derive(&content, len);
|
|
let expected_companions = if len > threshold { 1 } else { 0 };
|
|
|
|
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)]
|
|
{
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key, canonical,
|
|
"the canonical derivation addresses the put on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
expected_companions,
|
|
"length {len} on {path_tag}: the placement is exactly the \
|
|
routing function's answer and never moves or duplicates \
|
|
(the LO medium holds {expected_companions} companion rows)"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"length {len} on {path_tag}: the pool holds the entry (the \
|
|
mem kv half's introspection shape: served reads, no \
|
|
medium of its own)"
|
|
);
|
|
let (got_len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the entry is present");
|
|
assert_eq!(
|
|
got_len, len,
|
|
"the read serves the entry from its placed tier"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the entry is byte-identical from its placed tier across paths"
|
|
);
|
|
drop(pin);
|
|
}
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
staged.candidates_staged, 1,
|
|
"identical content through both put paths is one pool entry"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"the executing sweep removed exactly the one entry (durable \
|
|
schedule: staged this sweep, executed by the next)"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the LO medium is back to zero"
|
|
);
|
|
}
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **threshold boundary lengths route
|
|
/// deterministically** — at/below/above the factory's declared
|
|
/// threshold, on both put paths, the placement is the routing
|
|
/// function's answer every time (the LO medium's companion rows
|
|
/// against the mem kv half's facade shape); and the
|
|
/// default-threshold boundary re-routes with the tuning (the
|
|
/// constructor-tuned threshold deterministically places the
|
|
/// default boundary's length on the tier its own threshold names).
|
|
/// Run against two factories by the runner — the column's tuned
|
|
/// threshold and the shipped default. The durable cleanup: the
|
|
/// staged sweep stages exactly the six entries' dead set and the
|
|
/// next sweep executes it.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
|
|
/// ADR-003 §Dispatch policy (default 128 KiB at-or-below routes kv;
|
|
/// constructor-tunable without API change); ADR-009 §2 (the
|
|
/// companion table); ADR-012 §5 (this topology's named shape);
|
|
/// ADR-012 §6.3 (the dual-tier dispatch-coverage shape); ADR-012
|
|
/// §6.7 (schedule-relative cleanup).
|
|
pub async fn threshold_boundary_lengths_route_deterministically(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let boundary = [
|
|
(threshold - 1, 191u32, false),
|
|
(threshold, 173, false),
|
|
(threshold, 167, false),
|
|
(threshold + 1, 151, true),
|
|
(threshold + 1, 139, true),
|
|
];
|
|
let mut expected_companions = 0u64;
|
|
for (len, modulus, lands_large) in boundary {
|
|
let content = patterned(len as usize, modulus);
|
|
for declared in [Some(content.len() as u64), None] {
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&content, len),
|
|
"the canonical derivation addresses the put"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await as u64,
|
|
expected_companions + if lands_large { 1 } else { 0 },
|
|
"boundary length {len}: at/below the threshold lands kv \
|
|
(nothing on the LO medium), over it lands large (one \
|
|
companion row behind an LO), both put paths"
|
|
);
|
|
drop(pin);
|
|
}
|
|
expected_companions += if lands_large { 1 } else { 0 };
|
|
}
|
|
|
|
// The default boundary re-routes with the constructor's
|
|
// tuning: the shipped default's length belongs to the tier
|
|
// this configuration's own threshold puts it on.
|
|
let at_default = patterned(DEFAULT_DISPATCH_THRESHOLD as usize, 199);
|
|
let default_lands_large = DEFAULT_DISPATCH_THRESHOLD > threshold;
|
|
let (key, pin) = store
|
|
.put(
|
|
Some(DEFAULT_DISPATCH_THRESHOLD),
|
|
Cursor::new(at_default.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the default-boundary put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&at_default, DEFAULT_DISPATCH_THRESHOLD),
|
|
"the canonical derivation addresses the put"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await as u64,
|
|
expected_companions + if default_lands_large { 1 } else { 0 },
|
|
"the default boundary's length routes to the tier this \
|
|
configuration's tuning names (the LO medium's answer)"
|
|
);
|
|
drop(pin);
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
staged.candidates_staged, 6,
|
|
"each leg's content was exactly one pool entry"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 6,
|
|
"the executing sweep removed every entry, tier notwithstanding"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the LO medium is back to zero"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **a kv-tier miss falls through to the large tier
|
|
/// on every key-addressed op** — content the routing function
|
|
/// placed on the large tier (one companion row behind an LO; the
|
|
/// LO medium and `pg_largeobject` hold the pages while the kv tier
|
|
/// misses — the kv half is in-process mem, facade-served) serves
|
|
/// `get`, `stat`, `has`, and `read_range` whole, exact, and
|
|
/// digest-correct; the mirror-position leg proves the kv tier is
|
|
/// the *first* consult (sub-threshold content, nothing on the LO
|
|
/// medium, reads still served); and both-tier misses resolve per
|
|
/// the error model's per-op shapes (reached here by the durable
|
|
/// schedule's staged-then-executed deletion of the subject entry).
|
|
///
|
|
/// Contract stamp: store-api.md §Read path (fall-through consult
|
|
/// order); ADR-012 §6.1 (every key-addressed op consults kv first,
|
|
/// then large; the general rule); ADR-012 §6.5 ruling 5 (both-tier
|
|
/// miss shapes: `None` on get/stat, `Err(Missing)` on read_range);
|
|
/// ADR-006 (the slice digest); ADR-009 §2/§3 (the companion table
|
|
/// and the `lo_get` windows serving the fall-through);
|
|
/// ADR-012 §6.3 (the dispatch-coverage shape).
|
|
pub async fn kv_tier_misses_fall_through_to_the_large_tier_per_op(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
// Over-threshold content — the kv tier misses, the large tier
|
|
// serves every op. Both put paths hold the shape.
|
|
for (content, declared) in [
|
|
(
|
|
patterned(3 * threshold as usize + 41, 181),
|
|
Some(3 * threshold + 41),
|
|
),
|
|
(patterned(3 * threshold as usize + 43, 197), None),
|
|
] {
|
|
let len = content.len() as u64;
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
1,
|
|
"the content lives on exactly the large tier — the kv tier \
|
|
misses"
|
|
);
|
|
let loid = loid_for_key(site.dsn(), site.schema(), &key.as_bytes())
|
|
.await
|
|
.expect("the companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), loid).await > 0,
|
|
"the LO medium holds the entry's pages"
|
|
);
|
|
|
|
let (got_len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the large tier served the kv miss");
|
|
assert_eq!(got_len, len);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the fall-through read is byte-identical"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
|
|
Some(alkblobs::EntryMeta {
|
|
len,
|
|
algorithm: key.algorithm(),
|
|
}),
|
|
"stat falls through and carries the length plus algorithm"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"has falls through"
|
|
);
|
|
let mid = len / 2;
|
|
let (slice, digest) = store
|
|
.read_range(&key.as_bytes(), mid, 17)
|
|
.await
|
|
.expect("read_range falls through");
|
|
assert_eq!(
|
|
slice,
|
|
content[(mid as usize)..(mid as usize + 17)],
|
|
"the range slice is exact"
|
|
);
|
|
assert_eq!(
|
|
digest,
|
|
slice_digest(&slice),
|
|
"the slice digest is SHA-256 over exactly the returned bytes"
|
|
);
|
|
|
|
// Sweep the entry away (the pin dropped): the miss shapes
|
|
// now resolve per op — the large tier misses too. The
|
|
// durable schedule: staged this sweep, executed by the next.
|
|
drop(pin);
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!((staged.candidates_staged, staged.deleted), (1, 0));
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"exactly the one entry died across the two sweeps"
|
|
);
|
|
assert!(
|
|
store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"a both-tier miss on get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"a both-tier miss on stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&key.as_bytes()).await.unwrap(),
|
|
"a both-tier miss on has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&key.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("the both-tier miss errs"),
|
|
Error::Missing,
|
|
"a both-tier miss on read_range is Err(Missing)"
|
|
);
|
|
}
|
|
|
|
// The mirror-position leg: sub-threshold content — the kv tier
|
|
// is the first consult and serves before the large tier is
|
|
// reached (nothing ever lands on the LO medium).
|
|
let small = patterned((threshold - 1) as usize, 149);
|
|
let (key, pin) = store
|
|
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect("the sub-threshold put publishes");
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"the kv tier holds the sub-threshold entry — the large tier \
|
|
was never consulted on the write, never reached on the read"
|
|
);
|
|
let (len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the kv tier served");
|
|
assert_eq!(len, small.len() as u64);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
small
|
|
);
|
|
|
|
drop(pin);
|
|
|
|
// Both-tier miss shapes for a key that never existed at all,
|
|
// per op, over the fresh both-tier store.
|
|
let absent = b"never put here".to_vec();
|
|
let absent = Key::derive(&absent, absent.len() as u64);
|
|
assert!(
|
|
store
|
|
.get(&absent.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"never-put: get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"never-put: stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&absent.as_bytes()).await.unwrap(),
|
|
"never-put: has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&absent.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("never-put read_range errs"),
|
|
Error::Missing,
|
|
"never-put: read_range is Err(Missing)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-1's LO-medium extension: **the `lo_get` windows read
|
|
/// across window boundaries byte-exact, and a body short of its
|
|
/// claimed length fails cleanly** — the round-trip family's ranged
|
|
/// reads extended to the LO medium. Entries on the large tier
|
|
/// (both put paths) read whole through the multi-window assembly,
|
|
/// straddle the shipped window edge (one read crossing two
|
|
/// `lo_get` statements), read ranged slices with the ADR-006
|
|
/// digest over exactly the returned bytes, clamp at EOF (tail,
|
|
/// at-EOF empty, a `u64::MAX` offset clamping in `u64` arithmetic,
|
|
/// zero-length, over-read), and — the beyond-length clean failure
|
|
/// — a companion row overclaiming its size (a corrupted half-state
|
|
/// healthy stores never hold, planted through the oracle)
|
|
/// surfaces as the unexpected-EOF-shaped `Error::Io`, never a
|
|
/// partial read or a panic.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 1 (the round-trip
|
|
/// family's ranged-read legs) and §Read path (the streaming
|
|
/// large-tier windows, pread-style clamping); ADR-009 §2/§3 (the
|
|
/// companion table is the authority — the cursor closes over the
|
|
/// row; descriptorless `lo_get` windows, the shipped read shape
|
|
/// and its unexpected-EOF failure); ADR-006 (the slice digest over
|
|
/// exactly the returned bytes); ADR-010 §1 (the large-tier cursor
|
|
/// body family).
|
|
pub async fn lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(
|
|
factory: &DualTierMemPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let content = patterned(2 * LO_WINDOW + 100, 167);
|
|
let len = content.len() as u64;
|
|
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)] {
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&content, len),
|
|
"the canonical derivation addresses the put on {path_tag}"
|
|
);
|
|
let (_, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the entry is present");
|
|
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the multi-window assembly reads whole and byte-exact on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(LO_WINDOW as u64 - 100, 300)
|
|
.await
|
|
.expect("the straddling read succeeds"),
|
|
content[(LO_WINDOW - 100)..(LO_WINDOW + 200)],
|
|
"a read straddling the window edge crosses two lo_get \
|
|
statements byte-exact on {path_tag}"
|
|
);
|
|
let (slice, digest) = blob
|
|
.read_range(LO_WINDOW as u64, 7)
|
|
.await
|
|
.expect("the ranged slice reads");
|
|
assert_eq!(
|
|
slice,
|
|
content[LO_WINDOW..LO_WINDOW + 7],
|
|
"the ranged slice is exact on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
digest,
|
|
slice_digest(&slice),
|
|
"the slice digest is SHA-256 over exactly the returned bytes"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(len - 5, 100).await.expect("the tail reads"),
|
|
content[(len as usize - 5)..],
|
|
"the tail read clamps at the entry's end"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(len, 8).await.expect("the EOF read succeeds"),
|
|
Vec::<u8>::new(),
|
|
"a read at EOF is empty, never an error"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(u64::MAX, 5).await.expect("the offset reads"),
|
|
Vec::<u8>::new(),
|
|
"an overflowing u64 offset clamps in u64 arithmetic — the \
|
|
C7 int4 trap not inherited"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, 0).await.expect("the empty read succeeds"),
|
|
Vec::<u8>::new(),
|
|
"a zero-length read is empty"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, len + 64)
|
|
.await
|
|
.expect("the over-read reads"),
|
|
content,
|
|
"an over-read clamps to the whole entry"
|
|
);
|
|
|
|
drop(pin);
|
|
}
|
|
|
|
// The beyond-length clean failure: a companion row claiming
|
|
// more bytes than its LO holds (the corrupted half-state
|
|
// healthy stores never hold) planted through the oracle — the
|
|
// read window trusts the row (the contract authority,
|
|
// metadata-only) and fails cleanly when the LO comes up short.
|
|
let planted = b"the planted short body".to_vec();
|
|
let planted_key = Key::derive(&planted, planted.len() as u64);
|
|
let claimed = (planted.len() + 4096) as i64;
|
|
let loid = {
|
|
let client = admin_connect_plant(site.dsn()).await;
|
|
client
|
|
.batch_execute("BEGIN")
|
|
.await
|
|
.expect("the plant opens");
|
|
let loid: u32 = client
|
|
.query_one("SELECT lo_create(0) AS loid", &[])
|
|
.await
|
|
.expect("lo_create mints the planted LO")
|
|
.get(0);
|
|
let fd: i32 = client
|
|
.query_one("SELECT lo_open($1, $2)", &[&loid, &INV_WRITE])
|
|
.await
|
|
.expect("the planted LO opens for write")
|
|
.get(0);
|
|
client
|
|
.query_one("SELECT lowrite($1, $2)", &[&fd, &planted])
|
|
.await
|
|
.expect("the planted partial body writes");
|
|
let committed_at = SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.as_nanos() as i64)
|
|
.unwrap_or(0);
|
|
client
|
|
.execute(
|
|
&format!(
|
|
"INSERT INTO \"{}\".lo_entries (key, loid, size, committed_at) \
|
|
VALUES ($1, $2, $3, $4)",
|
|
site.schema()
|
|
),
|
|
&[&planted_key.as_bytes(), &loid, &claimed, &committed_at],
|
|
)
|
|
.await
|
|
.expect("the overclaiming companion row plants");
|
|
client
|
|
.batch_execute("COMMIT")
|
|
.await
|
|
.expect("the plant commits");
|
|
drop(client);
|
|
loid
|
|
};
|
|
let (got_len, planted_blob) = store
|
|
.get(&planted_key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect(
|
|
"the planted entry is present (the companion row is the \
|
|
authority, metadata-only)",
|
|
);
|
|
assert_eq!(
|
|
got_len, claimed as u64,
|
|
"the cursor's length is the companion row's overclaim — never \
|
|
an LO content probe"
|
|
);
|
|
let err = planted_blob
|
|
.read_at(0, got_len)
|
|
.await
|
|
.expect_err("the short body fails cleanly beyond its length");
|
|
assert!(
|
|
matches!(&err, Error::Io(m) if m.contains("short of")),
|
|
"the beyond-length read surfaces the unexpected-EOF clean \
|
|
failure, never a partial read: {err}"
|
|
);
|
|
|
|
// The durable cleanup: the round-trip legs' identical content
|
|
// was exactly one pool entry (CAS across the paths), so that
|
|
// entry and the planted one stage exactly, and the executing
|
|
// sweep deletes exactly that staged set — the planted LO's
|
|
// pages too.
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(staged.aborted, staged.candidates_staged),
|
|
(None, 2),
|
|
"the round-trip entry (one across both paths) and the planted \
|
|
one stage exactly"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(executed.deleted, 2, "the staged set executed exactly");
|
|
assert_eq!(
|
|
(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
window_rows(site.dsn(), site.schema()).await
|
|
),
|
|
(0, 0),
|
|
"the medium and the window are back to zero"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), loid).await,
|
|
0,
|
|
"the executing sweep's lo_unlink reaped the planted LO"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// The plant's raw-LO flag — postgres Large Objects' `INV_WRITE`
|
|
/// mode bits, the same constant the engine's orphan test plants
|
|
/// with.
|
|
const INV_WRITE: i32 = 0x0002_0000;
|
|
|
|
async fn admin_connect_plant(dsn: &str) -> tokio_postgres::Client {
|
|
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
|
|
.await
|
|
.expect("the plant connection opens");
|
|
tokio::spawn(async move {
|
|
let _ = connection.await;
|
|
});
|
|
client
|
|
}
|
|
|
|
/// Virgin-store row: **reads are no-ops on a fresh schema** — an
|
|
/// empty `list`, absent reads per op (`None` on get/stat, `false`
|
|
/// on has, `Err(Missing)` on read_range), and nothing on the fresh
|
|
/// medium itself (zero companion rows, and the window state per
|
|
/// the host rule: `gc_window` lives in the pg-lo schema — the
|
|
/// one-window-host rule's large-host arm — and holds zero rows;
|
|
/// `gc_pins` is never created by pg-lo). The constructor always
|
|
/// ensures the pg-lo schema, this row being the facade form.
|
|
///
|
|
/// Contract stamp: store-api.md §Error model (virgin-store
|
|
/// semantics: reads on a fresh store see absent/empty, never
|
|
/// errors); ADR-012 §6.5 ruling 5 (the per-op miss shapes); ADR-009
|
|
/// §2 (the companion-table bootstrap at the constructor); ADR-012
|
|
/// §5 (the one-window-host rule's named topology — the window on
|
|
/// the pg-lo host, and pg-lo never carries fleet pins).
|
|
pub async fn virgin_store_reads_are_no_ops_on_a_fresh_schema(factory: &DualTierMemPgLoFactory) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
assert_eq!(
|
|
store.mode(),
|
|
StoreMode::DualTier,
|
|
"the column's factory is the dispatch-coverage shape"
|
|
);
|
|
let dsn = site.dsn();
|
|
let schema = site.schema();
|
|
|
|
let absent = b"nothing was ever put".to_vec();
|
|
let absent = Key::derive(&absent, absent.len() as u64);
|
|
|
|
assert_eq!(
|
|
store.list().await.expect("list succeeds"),
|
|
Vec::<Key>::new(),
|
|
"the virgin pool lists empty"
|
|
);
|
|
assert!(
|
|
store
|
|
.get(&absent.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"absent get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"absent stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&absent.as_bytes()).await.unwrap(),
|
|
"absent has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&absent.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("absent read_range errs"),
|
|
Error::Missing,
|
|
"absent read_range is Err(Missing)"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(dsn, schema).await,
|
|
0,
|
|
"the fresh LO medium holds nothing (zero companion rows)"
|
|
);
|
|
assert_eq!(
|
|
window_rows(dsn, schema).await,
|
|
0,
|
|
"the fresh window holds no staged candidates"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::table_exists(dsn, schema, "gc_window").await,
|
|
"the window lives on the pg-lo host — the one-window-host \
|
|
rule's large-host arm (ADR-012 §5)"
|
|
);
|
|
assert!(
|
|
!crate::pg_lo_column::table_exists(dsn, schema, "gc_pins").await,
|
|
"gc_pins is never created by pg-lo (fleet pins live on the kv \
|
|
host, ADR-012 §5)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The runners — the column's tests: the form-independent reference
|
|
// rows mirrored verbatim against the dual-tier factory (INV-1, INV-6,
|
|
// and the two no-op abort shapes), the durable-schedule rows above,
|
|
// and the factory contract's structural shapes. Every runner gates
|
|
// behind the reachability probe — the SKIP posture: server-less
|
|
// runs skip cleanly, a skip is not a failure; acceptance is the
|
|
// green run against the harness server.
|
|
// --------------------------------------------------------------
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv1_round_trip_puts_and_gets_are_byte_identical() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::round_trip_puts_and_gets_are_byte_identical(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv2_dedup_identical_content_is_one_pool_entry() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
dedup_identical_content_is_one_pool_entry(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv3_stage_hygiene_zero_residue_one_entry_on_success() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv4_two_sweep_exact_counts_over_the_full_protection_set() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
two_sweep_exact_counts_over_the_full_protection_set(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv4_a_put_landing_during_the_window_survives_the_executing_sweep() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
a_put_landing_during_the_window_survives_the_executing_sweep(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv4_sweep_without_liveness_sources_aborts_and_deletes_nothing() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::sweep_without_liveness_sources_aborts_and_deletes_nothing(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv4_sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing(&factory).await;
|
|
}
|
|
|
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
|
async fn mem_pglo_inv4_the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only()
|
|
{
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv5_list_correctness_through_durable_schedule_outcomes() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
list_correctness_is_observable_through_durable_schedule_outcomes(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_inv6_stat_reports_len_and_algorithm_without_fetching_content() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::stat_reports_len_and_algorithm_without_fetching_content(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn mem_pglo_virgin_store_reads_are_no_ops_on_a_fresh_schema() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
virgin_store_reads_are_no_ops_on_a_fresh_schema(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_mem_pglo_routing_is_a_pure_function_of_length_across_put_paths_and_re_puts() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_mem_pglo_threshold_boundary_lengths_route_deterministically() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let tuned = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
threshold_boundary_lengths_route_deterministically(&tuned).await;
|
|
let default = DualTierMemPgLoFactory::new(DEFAULT_DISPATCH_THRESHOLD);
|
|
threshold_boundary_lengths_route_deterministically(&default).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_mem_pglo_kv_tier_misses_fall_through_to_the_large_tier_per_op() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
kv_tier_misses_fall_through_to_the_large_tier_per_op(&factory).await;
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The factory contract's structural shapes (the postgres column's
|
|
// pattern, carried to the LO medium with the LO-isolation
|
|
// corollary's arms): fresh isolated opens (a fresh dedicated
|
|
// database with a fresh schema per open, nothing shared — two
|
|
// opens never observe one another) and teardown against a
|
|
// panicking row (the schemas really dropped, the LOs really
|
|
// unlinked — zero orphan residue probed inside the dedicated
|
|
// database before it goes, and the shared harness database
|
|
// untouched by construction). Not conformance rows — no backlog
|
|
// entry — the factory contract's own wiring-level assertions.
|
|
// --------------------------------------------------------------
|
|
|
|
/// Every `open()` mints a fresh, isolated database + schema pair:
|
|
/// no shared state between islands, observable on the facade *and*
|
|
/// on the fresh medium (each open's database and schema are
|
|
/// private and start empty).
|
|
#[tokio::test]
|
|
async fn mem_pglo_factory_open_vends_fresh_isolated_stores() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
|
|
|
|
let (a, site_a) = factory.open_tracked().await.expect("open vends a store");
|
|
let (b, site_b) = factory.open_tracked().await.expect("open vends another");
|
|
assert_ne!(
|
|
site_a.schema(),
|
|
site_b.schema(),
|
|
"two opens never share a schema"
|
|
);
|
|
assert_ne!(site_a.db, site_b.db, "two opens never share a database");
|
|
|
|
let content_a = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
|
|
let content_b = patterned(TUNED_THRESHOLD as usize - 5, 191);
|
|
let (key_a, _) = a
|
|
.put(
|
|
Some(content_a.len() as u64),
|
|
Cursor::new(content_a.as_slice()),
|
|
)
|
|
.await
|
|
.expect("island a's put publishes");
|
|
let (key_b, _) = b
|
|
.put(None, Cursor::new(content_b.as_slice()))
|
|
.await
|
|
.expect("island b's put publishes");
|
|
|
|
assert!(a.has(&key_a.as_bytes()).await.unwrap());
|
|
assert!(
|
|
!a.has(&key_b.as_bytes()).await.unwrap(),
|
|
"island a sees nothing of island b"
|
|
);
|
|
assert!(b.has(&key_b.as_bytes()).await.unwrap());
|
|
assert!(
|
|
!b.has(&key_a.as_bytes()).await.unwrap(),
|
|
"island b sees nothing of island a"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site_a.dsn(), site_a.schema()).await,
|
|
1,
|
|
"island a's medium holds exactly its own entry (a's content routes large)"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site_b.dsn(), site_b.schema()).await,
|
|
0,
|
|
"island b's medium holds exactly its own entry (b's content routes kv)"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::zero_lo_residue(site_a.dsn(), site_a.schema()).await,
|
|
"island a's committed content has zero orphan LO residue (the \
|
|
LO catalog and the companion table agree exactly)"
|
|
);
|
|
|
|
let (c, site_c) = factory.open_tracked().await.expect("a third vends fresh");
|
|
assert_eq!(
|
|
lo_entry_rows(site_c.dsn(), site_c.schema()).await,
|
|
0,
|
|
"a third open's medium is virgin"
|
|
);
|
|
assert!(
|
|
!c.has(&key_a.as_bytes()).await.unwrap(),
|
|
"a third open shares nothing with the first"
|
|
);
|
|
assert!(
|
|
!c.has(&key_b.as_bytes()).await.unwrap(),
|
|
"a third open shares nothing with the second"
|
|
);
|
|
|
|
let dbs = factory.tracked_dbs();
|
|
assert_eq!(dbs.len(), 3, "one minted database per open");
|
|
assert_eq!(
|
|
dbs.iter()
|
|
.map(|(name, _, _)| name.clone())
|
|
.collect::<std::collections::BTreeSet<_>>()
|
|
.len(),
|
|
3,
|
|
"no two opens share a database"
|
|
);
|
|
|
|
// The engine-owned schemas really exist inside their dedicated
|
|
// databases (the registry's minted list is not bookkeeping
|
|
// fiction).
|
|
for (_, dsn, schemas) in &dbs {
|
|
for schema in schemas {
|
|
assert!(
|
|
crate::pg_lo_column::schema_exists(dsn, schema).await,
|
|
"the minted schema exists inside its dedicated database: {schema}"
|
|
);
|
|
}
|
|
}
|
|
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown reaps the LOs, drops the schemas, drops the databases");
|
|
factory.teardown().await.expect("teardown repeats cleanly");
|
|
let maintenance = admin_dsn(factory.dsn());
|
|
for (name, _, _) in &dbs {
|
|
assert!(
|
|
!db_exists(&maintenance, name).await,
|
|
"teardown dropped the dedicated database: {name} (schemas \
|
|
and all — the whole LO state goes with it)"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// A failing assertion panics the row before teardown runs — and
|
|
/// teardown must make sense over the abandoned instance anyway: it
|
|
/// is idempotent against the panicked row's path, the LOs are
|
|
/// really unlinked (the tracked-LOID reap arm — the recorded
|
|
/// LOIDs match the pre-captured set) and the minted schemas really
|
|
/// dropped (gone with the dedicated database, **zero orphan
|
|
/// `pg_largeobject` residue** probed inside before it goes), and
|
|
/// the next open mints a fresh database on a fresh (empty) medium
|
|
/// (README §The factory contract). The shared harness database is
|
|
/// untouched by construction — no pg-lo pool ever opens against
|
|
/// it.
|
|
#[tokio::test]
|
|
async fn mem_pglo_factory_teardown_is_safe_against_a_panicking_row() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let content = patterned(2 * TUNED_THRESHOLD as usize + 7, 173);
|
|
let abandoned = Key::derive(&content, content.len() as u64);
|
|
let factory = Arc::new(DualTierMemPgLoFactory::new(TUNED_THRESHOLD));
|
|
|
|
let task = {
|
|
let factory = Arc::clone(&factory);
|
|
tokio::spawn(async move {
|
|
let (store, _) = factory.open_tracked().await.expect("open vends the store");
|
|
let (_, pin) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the put publishes");
|
|
drop(pin);
|
|
panic!("a failing assertion panics the row (the factory contract's early exit)");
|
|
})
|
|
};
|
|
assert!(
|
|
task.await.is_err(),
|
|
"the row's panic surfaced as its failure"
|
|
);
|
|
|
|
let dbs = factory.tracked_dbs();
|
|
assert_eq!(dbs.len(), 1, "the panicked row minted one database");
|
|
let (name, dsn, schemas) = &dbs[0];
|
|
let loids = companion_loids(dsn, &schemas[0]).await;
|
|
assert!(!loids.is_empty(), "the panicked row's LO is tracked");
|
|
assert!(
|
|
crate::pg_lo_column::zero_lo_residue(dsn, &schemas[0]).await,
|
|
"the abandoned run's committed content has zero orphan LO \
|
|
residue (the LO catalog and the companion table agree \
|
|
exactly)"
|
|
);
|
|
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown makes sense over the abandoned instance");
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
assert!(
|
|
!db_exists(&admin_dsn(factory.dsn()), name).await,
|
|
"teardown dropped the panicked row's database: {name} (the \
|
|
schemas and the whole LO state go with it)"
|
|
);
|
|
let mut reaped = factory.reaped_loids();
|
|
reaped.sort_unstable();
|
|
let mut captured = loids.clone();
|
|
captured.sort_unstable();
|
|
assert_eq!(
|
|
reaped, captured,
|
|
"the teardown's reap arm unlinked exactly the tracked LOIDs"
|
|
);
|
|
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("the next open still vends a store");
|
|
assert!(
|
|
!store.has(&abandoned.as_bytes()).await.unwrap(),
|
|
"no residue from the panicked row — isolation is factory-owned"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.schema()).await,
|
|
0,
|
|
"and its medium carries nothing of the abandoned run"
|
|
);
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown drops the last schema");
|
|
}
|
|
}
|
|
// ------------------------------------------------------------------
|
|
// The postgres-pglo factory — the consolidation posture, feature pair
|
|
// `postgres` + `pg-lo`: kv = postgres + large = pg-lo on one pg
|
|
// instance, the durable window hosted on the kv schema (the pg-lo
|
|
// window clauses dormant), and the placement oracle's kv half counted
|
|
// as rows in the kv schema's `blobs` table.
|
|
// ------------------------------------------------------------------
|
|
|
|
#[cfg(all(feature = "postgres", feature = "pg-lo"))]
|
|
mod postgres_pglo {
|
|
use std::io::Cursor;
|
|
use std::sync::atomic::AtomicU64;
|
|
use std::sync::{Arc, Mutex};
|
|
use std::time::{Duration, SystemTime};
|
|
|
|
use tokio_postgres::NoTls;
|
|
|
|
use crate::factory::StoreFactory;
|
|
use crate::pg_lo_column::{
|
|
admin_dsn, companion_loids, create_db, db_dsn, db_exists, drop_db, fresh_db_name,
|
|
harness_dsn, harness_ready, lo_entry_rows, loid_for_key, loid_pages, pg_table_rows,
|
|
reap_lo_schema, window_rows, DbJanitor, TrackedDb, LO_WINDOW, WINDOW_DELAY,
|
|
};
|
|
use crate::rows::{
|
|
self, patterned, slice_digest, Failing, LatchProtect, SetProtect, SetSource,
|
|
};
|
|
use alkblobs::{Error, Key, PgOpts, Store, StoreMode, DEFAULT_DISPATCH_THRESHOLD};
|
|
|
|
/// The column's declared dual-tier threshold — small, so the
|
|
/// shared content matrix and the race fixtures cross it and both
|
|
/// tiers' bodies are exercised (sub-threshold rows land as rows in
|
|
/// the kv schema's `blobs` table; over-threshold rows land as
|
|
/// companion rows + LOs in the pg-lo schema).
|
|
pub(crate) const TUNED_THRESHOLD: u64 = 1024;
|
|
|
|
/// One vended backing site: the dedicated database this open
|
|
/// minted and the two engine-owned schemas minted inside it — the
|
|
/// kv half (`blobs` + the GC tables) and the pg-lo half
|
|
/// (`lo_entries`, window-clauses dormant) — on one pg instance.
|
|
#[derive(Clone)]
|
|
pub(crate) struct Site {
|
|
/// The db-scoped DSN — the oracle's independent connections
|
|
/// address it; each open's database is private and gets
|
|
/// dropped whole at teardown (the LO-isolation corollary: LOs
|
|
/// are database-level).
|
|
dsn: String,
|
|
/// The kv engine's schema (the pool table and, per the
|
|
/// one-window-host rule, the `gc_window` + `gc_pins` tables).
|
|
kv_schema: String,
|
|
/// The pg-lo engine's schema (the companion table; no window
|
|
/// clauses — the dormancy shape).
|
|
pglo_schema: String,
|
|
/// The dedicated database's name — the structural teardown
|
|
/// tests' `pg_database` probe target.
|
|
db: String,
|
|
}
|
|
|
|
impl Site {
|
|
/// The kv tier's schema — the dispatch oracle's kv-side
|
|
/// counter (`blobs`) and the window host's `gc_window`.
|
|
pub(crate) fn kv_schema(&self) -> &str {
|
|
&self.kv_schema
|
|
}
|
|
|
|
/// The pg-lo tier's schema — the LO medium's companion table
|
|
/// and the dormancy row's genuine-absence target.
|
|
pub(crate) fn pglo_schema(&self) -> &str {
|
|
&self.pglo_schema
|
|
}
|
|
|
|
/// The DSN the open minted — scoped to the dedicated database.
|
|
pub(crate) fn dsn(&self) -> &str {
|
|
&self.dsn
|
|
}
|
|
}
|
|
|
|
/// The dispatch-coverage factory: `kv = postgres` + `large =
|
|
/// pg-lo` on one pg instance (the consolidation posture, ADR-009)
|
|
/// at the threshold this column declares and the test tuning of
|
|
/// the window delay (see [`TUNED_THRESHOLD`] and `WINDOW_DELAY` —
|
|
/// the window lives on the kv schema; the pg-lo schema carries no
|
|
/// `gc_window`, the dormancy shape the column's row asserts).
|
|
///
|
|
/// **Why a fresh instance per runner (not a shared static):**
|
|
/// teardown drops the dedicated databases *this instance* minted
|
|
/// (having reaped their LOs and dropped their schemas); a shared
|
|
/// instance's teardown would race concurrently running rows'
|
|
/// sites. Every runner constructs its own and owns its lifecycle;
|
|
/// a row that panics before teardown is still backed by the
|
|
/// per-open drop-guard janitors.
|
|
pub(crate) struct DualTierPostgresPgLoFactory {
|
|
dsn: String,
|
|
/// The declared dispatch threshold — the routing boundary this
|
|
/// column's rows run at (constructor-tunable; ADR-003).
|
|
/// Declared configuration: the dual-tier store never rejects,
|
|
/// so there is nothing to probe.
|
|
threshold: u64,
|
|
/// The declared window delay — the durable schedule's cadence
|
|
/// knob the sweep rows run at (the test tuning:
|
|
/// execute-on-next-sweep).
|
|
#[allow(dead_code)]
|
|
window_delay: Duration,
|
|
seq: AtomicU64,
|
|
/// Every dedicated database this instance minted, for its own
|
|
/// teardown and its drop-guard janitors.
|
|
dbs: Mutex<Vec<TrackedDb>>,
|
|
/// The LOIDs teardown reaped, for the structural tests'
|
|
/// assertions.
|
|
reaped_loids: Mutex<Vec<u32>>,
|
|
}
|
|
|
|
impl DualTierPostgresPgLoFactory {
|
|
pub(crate) fn new(threshold: u64) -> DualTierPostgresPgLoFactory {
|
|
DualTierPostgresPgLoFactory {
|
|
dsn: harness_dsn().expect("the harness server is configured for the column"),
|
|
threshold,
|
|
window_delay: WINDOW_DELAY,
|
|
seq: AtomicU64::new(
|
|
(std::time::SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.subsec_nanos() as u64)
|
|
.unwrap_or(0))
|
|
^ ((std::process::id() as u64) << 8),
|
|
),
|
|
dbs: Mutex::new(Vec::new()),
|
|
reaped_loids: Mutex::new(Vec::new()),
|
|
}
|
|
}
|
|
|
|
/// Open + track the vended site: the shape the column's own
|
|
/// rows use (they observe both mediums, so they need the site
|
|
/// alongside the store). One dedicated database per open with
|
|
/// two minted schemas — the kv half and the pg-lo half — so
|
|
/// the dormancy row observes genuine absence.
|
|
pub(crate) async fn open_tracked(&self) -> Result<(Store, Site), Error> {
|
|
let maintenance = admin_dsn(&self.dsn);
|
|
let db = fresh_db_name("consolidation", &self.seq);
|
|
create_db(&maintenance, &db).await?;
|
|
let dsn = db_dsn(&self.dsn, &db);
|
|
let kv_schema = crate::pg_lo_column::fresh_schema_name("kv", &self.seq);
|
|
let pglo_schema = crate::pg_lo_column::fresh_schema_name("lo", &self.seq);
|
|
self.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.push(TrackedDb {
|
|
name: db.clone(),
|
|
dsn: dsn.clone(),
|
|
schemas: vec![kv_schema.clone(), pglo_schema.clone()],
|
|
lo_schemas: vec![pglo_schema.clone()],
|
|
janitor: Some(DbJanitor::new(&maintenance, &db)),
|
|
});
|
|
let store = Store::dual_tier_postgres_pglo(
|
|
&dsn,
|
|
opts(kv_schema.clone()),
|
|
&dsn,
|
|
opts(pglo_schema.clone()),
|
|
self.threshold,
|
|
WINDOW_DELAY,
|
|
)
|
|
.await?;
|
|
Ok((
|
|
store,
|
|
Site {
|
|
dsn,
|
|
kv_schema,
|
|
pglo_schema,
|
|
db,
|
|
},
|
|
))
|
|
}
|
|
|
|
/// The dedicated databases this instance minted — names,
|
|
/// db-scoped DSNs, and the schemas minted inside — for the
|
|
/// structural tests' assertions.
|
|
pub(crate) fn tracked_dbs(&self) -> Vec<(String, String, Vec<String>)> {
|
|
self.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.iter()
|
|
.map(|d| (d.name.clone(), d.dsn.clone(), d.schemas.clone()))
|
|
.collect()
|
|
}
|
|
|
|
/// The LOIDs teardown reaped, for the structural tests'
|
|
/// tracked-LOID assertions.
|
|
pub(crate) fn reaped_loids(&self) -> Vec<u32> {
|
|
self.reaped_loids
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.clone()
|
|
}
|
|
|
|
/// The maintenance DSN the factory mints databases through.
|
|
pub(crate) fn dsn(&self) -> &str {
|
|
&self.dsn
|
|
}
|
|
}
|
|
|
|
/// The factories' schema-name policy, in engine-opts form: the
|
|
/// constructor's `PgOpts` with exactly the factory's minted schema
|
|
/// and the ship defaults for the rest (pool size,
|
|
/// `synchronous_commit` — the alkstore factory's shape: only the
|
|
/// schema name is factory policy).
|
|
fn opts(schema: String) -> PgOpts {
|
|
PgOpts {
|
|
schema,
|
|
..PgOpts::default()
|
|
}
|
|
}
|
|
|
|
impl StoreFactory for DualTierPostgresPgLoFactory {
|
|
fn name(&self) -> &'static str {
|
|
"dual-tier postgres+pg-lo"
|
|
}
|
|
|
|
async fn open(&self) -> Result<Store, Error> {
|
|
self.open_tracked().await.map(|(store, _)| store)
|
|
}
|
|
|
|
async fn teardown(&self) -> Result<(), Error> {
|
|
let mut dbs = self
|
|
.dbs
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.drain(..)
|
|
.collect::<Vec<_>>();
|
|
for tracked in dbs.iter_mut() {
|
|
// The LO-isolation corollary's arms: reap the tracked
|
|
// LOIDs (via the companion table — the contract
|
|
// authority), drop exactly the minted schemas (kv and
|
|
// pg-lo halves), then drop the dedicated database
|
|
// whole (`WITH (FORCE)` reaps any connection the
|
|
// abandoned store's pool has not drained).
|
|
let maintenance = admin_dsn(&self.dsn);
|
|
let mut reaped = Vec::new();
|
|
if let Ok(client) = admin_connect_for_teardown(&tracked.dsn).await {
|
|
for schema in &tracked.lo_schemas {
|
|
reaped.extend(reap_lo_schema(&client, schema).await);
|
|
}
|
|
for schema in &tracked.schemas {
|
|
let sql = format!("DROP SCHEMA IF EXISTS \"{schema}\" CASCADE");
|
|
let _ = client.batch_execute(&sql).await;
|
|
}
|
|
}
|
|
if !reaped.is_empty() {
|
|
self.reaped_loids
|
|
.lock()
|
|
.unwrap_or_else(|e| e.into_inner())
|
|
.extend(reaped);
|
|
}
|
|
drop_db(&maintenance, &tracked.name).await?;
|
|
TrackedDb::disarm(tracked);
|
|
}
|
|
Ok(())
|
|
}
|
|
}
|
|
|
|
async fn admin_connect_for_teardown(dsn: &str) -> Result<tokio_postgres::Client, Error> {
|
|
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
|
|
.await
|
|
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
|
|
tokio::spawn(async move {
|
|
let _ = connection.await;
|
|
});
|
|
Ok(client)
|
|
}
|
|
|
|
/// The skip gate every runner checks (the SKIP posture's
|
|
/// factory-owner responsibility).
|
|
async fn ready() -> bool {
|
|
harness_ready().await
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The column's own rows — the durable-schedule sweep family, the
|
|
// schedule-relative readings, the dispatch forms, the INV-1
|
|
// extension, the dormancy row, and the virgin-store row. Every
|
|
// row takes this column's concrete factory (it observes the
|
|
// mediums), is ADR-stamped per the convention, and traces its
|
|
// family.
|
|
// --------------------------------------------------------------
|
|
|
|
/// INV-2 on the durable schedule, over both tiers: identical
|
|
/// content put twice — through the same put path and across the
|
|
/// two paths — is one pool entry, and the re-put's `Pin` protects
|
|
/// exactly as a fresh put's does. The kv leg's single-pool-entry
|
|
/// is one row in the kv schema; the large leg's is one companion
|
|
/// row behind one LO in the pg-lo schema. One-pool-entry reads
|
|
/// through durable-schedule outcomes (the INV-5 posture): the pins
|
|
/// held stage nothing; once unpinned, sweep 1 stages exactly the
|
|
/// one entry and removes nothing, and its executor sweep deletes
|
|
/// exactly that staged set (README §The schedule-relative reading
|
|
/// for SQL-backed columns).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 2; ADR-003 (whole-value
|
|
/// put is CAS first-writer-wins; the routing function's tier
|
|
/// placement); ADR-005 (a re-put refcounts the existing entry
|
|
/// under the same arbitration lock); ADR-007 §5 (postgres, the
|
|
/// consolidation shape's kv engine); ADR-009 §2/§4 (the companion
|
|
/// table is the contract authority; delete = lo_unlink + row
|
|
/// delete); ADR-008 §2 (the staged re-arbitrated delete); ADR-012
|
|
/// §4/§5 (the consolidation posture; the kv host wins the window);
|
|
/// ADR-012 §6.7 (the executor is the next sweep — the schedule
|
|
/// these counts read against).
|
|
pub async fn dedup_identical_content_is_one_pool_entry(factory: &DualTierPostgresPgLoFactory) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
// The kv leg: one row in the kv schema, both refcounts
|
|
// composing under the same lock.
|
|
let content = patterned((threshold - 1) as usize, 191);
|
|
let canonical = Key::derive(&content, content.len() as u64);
|
|
let (key, first) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the fresh put publishes");
|
|
let (reput, second) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the same-path re-put publishes");
|
|
assert_eq!(key, canonical);
|
|
assert_eq!(reput, canonical, "one derivation, one address");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.aborted, None);
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"one pool entry, both refcounts composing under the same lock"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
1,
|
|
"one pool entry, both refcounts — one row in the kv schema"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
0,
|
|
"the kv-tier entries never land on the LO medium"
|
|
);
|
|
drop(first);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"the re-put's pin protects exactly as a fresh put's"
|
|
);
|
|
drop(second);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one pool entry all along — staged this sweep, not executed"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.deleted, report.cancelled_by_arbitration),
|
|
(1, 0),
|
|
"the executing sweep deleted exactly the staged entry"
|
|
);
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
|
|
// The large leg: identical content through both put paths —
|
|
// one companion row behind one LO, both refcounts composing.
|
|
let cross = patterned(2 * threshold as usize + 5, 167);
|
|
let cross_canonical = Key::derive(&cross, cross.len() as u64);
|
|
let (key, first) = store
|
|
.put(Some(cross.len() as u64), Cursor::new(cross.as_slice()))
|
|
.await
|
|
.expect("the known-length put publishes");
|
|
let (reput, second) = store
|
|
.put(None, Cursor::new(cross.as_slice()))
|
|
.await
|
|
.expect("the unknown-length re-put publishes");
|
|
assert_eq!(key, cross_canonical);
|
|
assert_eq!(reput, cross_canonical, "one pool entry across the paths");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.aborted, report.candidates_staged),
|
|
(None, 0),
|
|
"both refcounts compose"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
1,
|
|
"one pool entry across both paths — one companion row"
|
|
);
|
|
let loids = companion_loids(site.dsn(), site.pglo_schema()).await;
|
|
assert_eq!(loids.len(), 1, "one LO behind the companion row");
|
|
assert!(
|
|
loid_pages(site.dsn(), loids[0]).await > 0,
|
|
"the LO holds the content's pages (never zero)"
|
|
);
|
|
drop(second);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.candidates_staged, 0,
|
|
"the re-put's pin protects exactly as a fresh put's (other drop order)"
|
|
);
|
|
drop(first);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one pool entry all along"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "executed exactly the staged set");
|
|
assert!(!store.has(&cross_canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
0,
|
|
"the executed delete emptied the companion table"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), loids[0]).await,
|
|
0,
|
|
"delete is one tx: the LO's pages went with the row"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
0,
|
|
"the window holds nothing after both executed rounds (the kv host)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-3 on the durable schedule: stage hygiene over both mediums'
|
|
/// forms — every put-path failure leaves zero residue (the kv tier
|
|
/// never touches its pool; the large tier's held-put transaction
|
|
/// converges on ROLLBACK: no companion rows, no LO pages) and every
|
|
/// success leaves exactly one committed entry, over both put paths
|
|
/// and both injectable failure modes (a mid-stream stream failure
|
|
/// and a declared-length mismatch), observed through
|
|
/// durable-schedule sweep outcomes: zero staged after every
|
|
/// failure, and a success staged exactly once, then executed
|
|
/// exactly once.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 3; ADR-010 §1 (the
|
|
/// staged-put contract: commit publishes, every failure path
|
|
/// discards); ADR-003 (stage-then-commit; the routing places each
|
|
/// leg's content on its tier); ADR-007 §5 (postgres, the kv
|
|
/// engine); ADR-009 §2 (put = one tx — the staged form is the held
|
|
/// tx, so a failure's LO pages roll back with it); ADR-012 §6.7
|
|
/// (the schedule-relative reading the success counts follow).
|
|
pub async fn stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
// The kv tier's degenerate stage form (nothing stages outside
|
|
// a flush) — residue is the pool itself, the kv schema's rows.
|
|
let small = patterned((threshold - 1) as usize, 167);
|
|
let err = store
|
|
.put(Some(small.len() as u64), Failing::new(&small, 40))
|
|
.await
|
|
.expect_err("the failed mid-stream put rejects");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
let err = store
|
|
.put(None, Failing::new(&small, 40))
|
|
.await
|
|
.expect_err("the failed unknown-length put rejects");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
let err = store
|
|
.put(Some(100), Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect_err("the over-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
let err = store
|
|
.put(Some(400), Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect_err("the under-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(
|
|
report.candidates_staged,
|
|
report.deleted,
|
|
report.cancelled_by_arbitration
|
|
),
|
|
(0, 0, 0),
|
|
"zero residue after every kv failure, both put paths, both \
|
|
failure modes"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
0,
|
|
"the pool table holds zero entries after every kv failure"
|
|
);
|
|
|
|
let (key, pin) = store
|
|
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect("the known-length put publishes");
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed kv entry staged on success"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
|
|
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
|
|
|
let (_, pin) = store
|
|
.put(None, Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect("the unknown-length put publishes");
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed kv entry staged on the other put path too"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
|
|
|
|
// The LO tier's real staged form — the held tx's failure paths
|
|
// roll back to no companion row, no LO pages.
|
|
let content = patterned(2 * threshold as usize + 17, 173);
|
|
let len = content.len() as u64;
|
|
let canonical = Key::derive(&content, len);
|
|
let err = store
|
|
.put(Some(len), Failing::new(&content, 40))
|
|
.await
|
|
.expect_err("the failed mid-stream put rejects");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
let err = store
|
|
.put(None, Failing::new(&content, threshold as usize + 600))
|
|
.await
|
|
.expect_err("the failed unknown-length put rejects mid-stage");
|
|
assert!(
|
|
matches!(err, Error::Io(_)),
|
|
"the stream failure surfaces as Io: {err}"
|
|
);
|
|
let err = store
|
|
.put(Some(threshold + 10), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect_err("the over-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
let err = store
|
|
.put(Some(len + 17), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect_err("the under-fed mismatch rejects");
|
|
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(
|
|
report.candidates_staged,
|
|
report.deleted,
|
|
report.cancelled_by_arbitration
|
|
),
|
|
(0, 0, 0),
|
|
"zero residue after every large failure, both put paths, both \
|
|
failure modes"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
0,
|
|
"the companion table holds zero entries after every failure"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
0,
|
|
"the window staged nothing behind the failures"
|
|
);
|
|
|
|
// Success, known path on the LO tier: exactly one committed
|
|
// entry — one companion row behind one LO — staged once,
|
|
// executed once.
|
|
let (key, pin) = store
|
|
.put(Some(len), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the known-length put publishes");
|
|
assert_eq!(key, canonical);
|
|
let loid = loid_for_key(site.dsn(), site.pglo_schema(), &key.as_bytes())
|
|
.await
|
|
.expect("the companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), loid).await > 0,
|
|
"the committed LO holds the content's pages"
|
|
);
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed entry staged on success"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
report.deleted, 1,
|
|
"the executing sweep removed exactly that one entry"
|
|
);
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(loid_pages(site.dsn(), loid).await, 0);
|
|
|
|
// Success, unknown path: the overflow staged into the held tx,
|
|
// the commit published — exactly one committed entry on the
|
|
// other put path too.
|
|
let (key, pin) = store
|
|
.put(None, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the unknown-length put publishes");
|
|
assert_eq!(key, canonical);
|
|
drop(pin);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"exactly one committed entry staged on the other put path too"
|
|
);
|
|
let report = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
|
|
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
window_rows(site.dsn(), site.kv_schema()).await
|
|
),
|
|
(0, 0),
|
|
"both mediums and the window are back to zero"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, the durable-schedule happy shape over both tiers: the
|
|
/// full protection set spanning both tiers (a source-live kv
|
|
/// entry, a pinned large-tier entry, a protect-named large-tier
|
|
/// entry) plus one dead entry per tier — sweep 1 stages exactly
|
|
/// the two dead candidates into the **kv schema's `gc_window`**
|
|
/// (the window hosted on the kv engine per the one-window-host
|
|
/// rule) and deletes nothing; sweep 2 executes exactly that staged
|
|
/// set, and the protected entries stay byte-identical. The medium
|
|
/// oracle distinguishes the tiers (the kv schema's pool rows vs
|
|
/// the pg-lo schema's companion rows behind LOs) and watches the
|
|
/// schedule itself.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4 (exact counts with
|
|
/// correct liveness) and §Invariants 5 (list correctness through
|
|
/// sweep outcomes); ADR-005 (three liveness marks; delete windows
|
|
/// with arbitrate-at-delete-time); ADR-008 §2 (the staged
|
|
/// re-arbitrated delete; the constructor-tunable delay); ADR-009
|
|
/// §2/§4 (the companion table; GC-participating deletes); ADR-007
|
|
/// §5 (postgres, the kv engine); ADR-012 §1 (sweeps are never
|
|
/// errors); ADR-012 §4/§5 (the consolidation posture; the kv host
|
|
/// wins the window); ADR-012 §6.7 (the executor is the next
|
|
/// sweep).
|
|
pub async fn two_sweep_exact_counts_over_the_full_protection_set(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
let source = SetSource::register(&store);
|
|
let protect = Arc::new(SetProtect::default());
|
|
store.register_protect_callback(protect.clone());
|
|
|
|
let source_live = patterned((threshold - 1) as usize, 191);
|
|
let (key_live, _) = store
|
|
.put(
|
|
Some(source_live.len() as u64),
|
|
Cursor::new(source_live.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the source-live entry publishes");
|
|
source.add(key_live.clone());
|
|
|
|
let pinned = patterned(2 * threshold as usize + 7, 173);
|
|
let (key_pinned, pin_pinned) = store
|
|
.put(None, Cursor::new(pinned.as_slice()))
|
|
.await
|
|
.expect("the pinned entry publishes");
|
|
|
|
let protected = patterned(2 * threshold as usize + 11, 167);
|
|
let (key_protected, _) = store
|
|
.put(
|
|
Some(protected.len() as u64),
|
|
Cursor::new(protected.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the protect-named entry publishes");
|
|
protect.set_keys(vec![key_protected.clone()]);
|
|
|
|
let dead_kv = patterned((threshold - 3) as usize, 149);
|
|
let (key_dead_kv, pin) = store
|
|
.put(Some(dead_kv.len() as u64), Cursor::new(dead_kv.as_slice()))
|
|
.await
|
|
.expect("the kv-tier dead entry publishes");
|
|
drop(pin);
|
|
let dead_large = patterned(2 * threshold as usize + 13, 151);
|
|
let (key_dead_large, pin) = store
|
|
.put(None, Cursor::new(dead_large.as_slice()))
|
|
.await
|
|
.expect("the large-tier dead entry publishes");
|
|
drop(pin);
|
|
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(2, 3),
|
|
"before the sweep, both tiers hold their entries exactly (two \
|
|
pool rows in the kv schema, three companion rows in the \
|
|
pg-lo schema)"
|
|
);
|
|
let dead_large_loid =
|
|
loid_for_key(site.dsn(), site.pglo_schema(), &key_dead_large.as_bytes())
|
|
.await
|
|
.expect("the dead large entry's companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), dead_large_loid).await > 0,
|
|
"the LO medium holds the dead large entry's pages too"
|
|
);
|
|
|
|
let first = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(first.aborted, None);
|
|
assert_eq!(
|
|
(
|
|
first.candidates_staged,
|
|
first.deleted,
|
|
first.cancelled_by_arbitration
|
|
),
|
|
(2, 0, 0),
|
|
"sweep 1 staged exactly the two dead entries and deleted \
|
|
nothing — staging is not executing"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
2,
|
|
"the staged set waits in the durable window — hosted on the \
|
|
kv schema (the one-window-host rule's kv host)"
|
|
);
|
|
for (key, content) in [
|
|
(&key_live, &source_live),
|
|
(&key_pinned, &pinned),
|
|
(&key_protected, &protected),
|
|
] {
|
|
let (len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the protected entry is present");
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the window reads"),
|
|
content.as_slice(),
|
|
"the protected entry is byte-identical after the staging sweep"
|
|
);
|
|
}
|
|
assert!(
|
|
store.has(&key_live.as_bytes()).await.unwrap()
|
|
&& store.has(&key_pinned.as_bytes()).await.unwrap()
|
|
&& store.has(&key_protected.as_bytes()).await.unwrap(),
|
|
"the staged sweep deleted nothing: every entry intact"
|
|
);
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(second.aborted, None);
|
|
assert_eq!(
|
|
second.candidates_staged, 0,
|
|
"nothing newly dead since the staging sweep"
|
|
);
|
|
assert_eq!(
|
|
second.deleted + second.cancelled_by_arbitration,
|
|
2,
|
|
"sweep 2 executed exactly the staged set"
|
|
);
|
|
assert_eq!(
|
|
(second.deleted, second.cancelled_by_arbitration),
|
|
(2, 0),
|
|
"with correct liveness, every executed candidate deleted"
|
|
);
|
|
assert!(!store.has(&key_dead_kv.as_bytes()).await.unwrap());
|
|
assert!(!store.has(&key_dead_large.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(1, 2),
|
|
"the executed deletes landed on both mediums exactly"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), dead_large_loid).await,
|
|
0,
|
|
"the dead large entry's LO pages are gone with it"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
0,
|
|
"the executed rows left the durable window"
|
|
);
|
|
|
|
drop(pin_pinned);
|
|
let third = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(third.candidates_staged, third.deleted),
|
|
(1, 0),
|
|
"with the pin dropped, the formerly pinned entry stages"
|
|
);
|
|
let fourth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
fourth.deleted, 1,
|
|
"and the next sweep executes it — exactly one deletion"
|
|
);
|
|
assert!(!store.has(&key_pinned.as_bytes()).await.unwrap());
|
|
assert!(store.has(&key_live.as_bytes()).await.unwrap());
|
|
assert!(store.has(&key_protected.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
1,
|
|
"exactly the protect-named entry's companion row remains"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, the race, in its durable-schedule form: a put landing
|
|
/// *during the window* survives the executing sweep — the re-put's
|
|
/// pin is minted (pin-before-publish) between the staging sweep
|
|
/// and the executing sweep, so the execution-time re-arbitration
|
|
/// cancels the pinned candidate while the unpinned twin deletes;
|
|
/// once the pin drops, the twin re-stages and executes exactly
|
|
/// once. Both put paths race in-body over both tiers (a kv twin
|
|
/// and a large twin), across the window delay the kv host carries.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4; ADR-005 (pin-before-
|
|
/// publish ordering; the delete window closes the sweep-vs-put race);
|
|
/// ADR-008 §2 (the staged re-arbitrated delete — the race holds
|
|
/// across the window delay); ADR-009 §4 (GC-participating deletes
|
|
/// over the companion table); ADR-012 §5 (the window on the kv
|
|
/// host in this consolidation shape); ADR-012 §6.7 (execution-time
|
|
/// re-arbitration is the executor sweep's posture).
|
|
pub async fn a_put_landing_during_the_window_survives_the_executing_sweep(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let rounds: [(String, Vec<u8>, bool, bool, bool); 2] = [
|
|
(
|
|
"the kv twin (known-path re-put)".to_owned(),
|
|
patterned((threshold - 1) as usize, 191),
|
|
true,
|
|
true,
|
|
false,
|
|
),
|
|
(
|
|
"the large twin (unknown-path re-put)".to_owned(),
|
|
patterned(2 * threshold as usize + 5, 173),
|
|
false,
|
|
false,
|
|
true,
|
|
),
|
|
];
|
|
for (index, (tag, content, original_known, reput_known, lands_large)) in
|
|
rounds.iter().enumerate()
|
|
{
|
|
let len = content.len() as u64;
|
|
let twin = Key::derive(content, len);
|
|
let witness = format!("the dead witness of round {index}").into_bytes();
|
|
let witness_key = Key::derive(&witness, witness.len() as u64);
|
|
|
|
let original = if *original_known { Some(len) } else { None };
|
|
let (_, pin) = store
|
|
.put(original, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the twin publishes");
|
|
drop(pin);
|
|
let (_, pin) = store
|
|
.put(None, Cursor::new(witness.as_slice()))
|
|
.await
|
|
.expect("the witness publishes");
|
|
drop(pin);
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(
|
|
staged.aborted,
|
|
staged.candidates_staged,
|
|
staged.deleted,
|
|
staged.cancelled_by_arbitration
|
|
),
|
|
(None, 2, 0, 0),
|
|
"{tag}: the staging sweep stages exactly the two dead \
|
|
twins and deletes nothing"
|
|
);
|
|
|
|
// The put lands during the window (the re-put dedups into
|
|
// the existing twin and refcounts its pin): protected
|
|
// across the delay, pin-before-publish.
|
|
let reput = if *reput_known { Some(len) } else { None };
|
|
let (_, pin) = store
|
|
.put(reput, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the re-put publishes");
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(executed.aborted, None, "{tag}");
|
|
assert_eq!(
|
|
executed.candidates_staged, 0,
|
|
"{tag}: nothing newly dead since the staging sweep"
|
|
);
|
|
assert_eq!(
|
|
executed.deleted + executed.cancelled_by_arbitration,
|
|
2,
|
|
"{tag}: the executing sweep ran exactly the staged set"
|
|
);
|
|
assert_eq!(executed.deleted, 1, "{tag}: the unpinned witness deleted");
|
|
assert_eq!(
|
|
executed.cancelled_by_arbitration, 1,
|
|
"{tag}: the pinned twin's candidate cancelled at execution time"
|
|
);
|
|
assert!(
|
|
!store.has(&witness_key.as_bytes()).await.unwrap(),
|
|
"{tag}: the unpinned twin is gone"
|
|
);
|
|
let (got_len, blob) = store
|
|
.get(&twin.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("{tag}: the put landing during the window survived");
|
|
assert_eq!(got_len, len, "{tag}");
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the window reads"),
|
|
content.as_slice(),
|
|
"{tag}: the surviving twin is byte-identical"
|
|
);
|
|
drop(pin);
|
|
|
|
let restaged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(restaged.candidates_staged, restaged.deleted),
|
|
(1, 0),
|
|
"{tag}: the pin dropped — the twin re-stages"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"{tag}: the re-staged candidate executes exactly once"
|
|
);
|
|
assert!(!store.has(&twin.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
0,
|
|
"{tag}: the window holds nothing after the executed round \
|
|
(the kv host)"
|
|
);
|
|
if *lands_large {
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
0,
|
|
"{tag}: the executed LO deletes reaped the medium exactly"
|
|
);
|
|
}
|
|
}
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-4, abort shape: the sweep is single-flight on the durable
|
|
/// form too — the in-process sweep lease (ADR-013 §2's single-node
|
|
/// analog). The contended second sweeper aborts with
|
|
/// `SweeperLock` in a completed `Ok(SweepReport)` — nothing
|
|
/// executed, nothing staged, nothing deleted (abort before
|
|
/// staging: form-independent) — while the lease holder's sweep
|
|
/// completes by **staging only**: its staged set waits in the
|
|
/// kv schema's `gc_window` for the next sweep to execute (the
|
|
/// durable form's schedule-relative replacement for the reference
|
|
/// row's inline holder-completion leg).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4; ADR-012 §1 (abort is
|
|
/// report data, never an error); ADR-013 §2 (the named exclusive
|
|
/// sweep lease — one sweeper per pool, holdership ends at release);
|
|
/// ADR-012 §5 (the window hosts on the kv engine in this
|
|
/// consolidation shape); ADR-012 §6.7 (the executor is the next
|
|
/// sweep — the holder stages, it does not execute).
|
|
pub async fn the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
|
|
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
|
|
store.register_protect_callback(Arc::new(LatchProtect {
|
|
entered: entered_tx,
|
|
release: parking_lot::Mutex::new(Some(release_rx)),
|
|
}));
|
|
|
|
let content = patterned(2 * threshold as usize + 9, 167);
|
|
let (key, pin) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the dead witness publishes");
|
|
drop(pin);
|
|
|
|
// The lease holder sweeps on its own task (its protect consult
|
|
// blocks that task's thread mid-callback); this task contends
|
|
// and then releases the holder.
|
|
let store = Arc::new(store);
|
|
let holder = tokio::spawn({
|
|
let store = Arc::clone(&store);
|
|
async move { store.sweep().await }
|
|
});
|
|
|
|
while entered_rx
|
|
.recv_timeout(std::time::Duration::from_millis(50))
|
|
.is_err()
|
|
{
|
|
tokio::task::yield_now().await;
|
|
}
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
second.aborted,
|
|
Some(alkblobs::GcAbortCause::SweeperLock),
|
|
"the contended sweeper aborts as report data"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
second.candidates_staged,
|
|
second.deleted,
|
|
second.cancelled_by_arbitration
|
|
),
|
|
(0, 0, 0),
|
|
"the contended sweep executes, stages, and deletes nothing"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"the contended sweep deleted nothing"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
0,
|
|
"the aborted sweep staged nothing into the window"
|
|
);
|
|
release_tx.send(()).ok();
|
|
|
|
let report = holder
|
|
.await
|
|
.expect("the holder task joined")
|
|
.expect("sweeps are never errors");
|
|
assert_eq!(report.aborted, None, "the lease holder completes");
|
|
assert_eq!(
|
|
(report.candidates_staged, report.deleted),
|
|
(1, 0),
|
|
"the holder stages exactly the dead set — it does not execute \
|
|
(the durable form's schedule)"
|
|
);
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
1,
|
|
"the holder's staged set waits in the durable window"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"the holder staged, did not execute: the entry is intact"
|
|
);
|
|
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"the next sweep is the executor of the staged set"
|
|
);
|
|
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
0,
|
|
"the executed delete emptied the companion table"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-5 on the durable schedule: `list()` correctness is
|
|
/// observable only through GC — the enumeration is complete and
|
|
/// well-formed in both directions, proven by exact
|
|
/// durable-schedule outcomes: the staging sweep stages exactly the
|
|
/// dead set (neither more nor fewer candidates), the executing
|
|
/// sweep deletes exactly that staged set, and a liveness flip
|
|
/// re-stages exactly the newly dead entry. Never a direct list
|
|
/// comparison.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 5; ADR-003 (`list()` is
|
|
/// complete and well-formed by contract — load-bearing for GC);
|
|
/// ADR-005 (the sweep's live-set diff rides `list()`); ADR-012 §5
|
|
/// (the window hosts on the kv engine in this consolidation
|
|
/// shape); ADR-012 §6.7 (the executor is the next sweep — the
|
|
/// schedule these counts read against).
|
|
pub async fn list_correctness_is_observable_through_durable_schedule_outcomes(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
let source = SetSource::register(&store);
|
|
|
|
let mut keys = Vec::new();
|
|
for (index, content) in [
|
|
b"kept live by the source, entry one".to_vec(),
|
|
b"kept live by the source, entry two".to_vec(),
|
|
b"dead at the first staging, entry three".to_vec(),
|
|
b"dead at the first staging, entry four".to_vec(),
|
|
b"dead at the first staging, entry five".to_vec(),
|
|
]
|
|
.into_iter()
|
|
.enumerate()
|
|
{
|
|
let declared = if index % 2 == 0 {
|
|
Some(content.len() as u64)
|
|
} else {
|
|
None
|
|
};
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the entry publishes");
|
|
drop(pin);
|
|
keys.push(key);
|
|
}
|
|
source.add(keys[0].clone());
|
|
source.add(keys[1].clone());
|
|
|
|
let first = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(first.aborted, None);
|
|
assert_eq!(
|
|
(first.candidates_staged, first.deleted),
|
|
(3, 0),
|
|
"the enumeration staged exactly the three dead entries — \
|
|
complete and well-formed in both directions"
|
|
);
|
|
for key in &keys {
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"staging deleted nothing: every enumerated entry is intact"
|
|
);
|
|
}
|
|
assert_eq!(
|
|
window_rows(site.dsn(), site.kv_schema()).await,
|
|
3,
|
|
"the staged schedule holds the three candidates"
|
|
);
|
|
|
|
let second = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(second.candidates_staged, second.deleted),
|
|
(0, 3),
|
|
"the executing sweep removed exactly the staged set"
|
|
);
|
|
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
|
|
assert!(store.has(&keys[1].as_bytes()).await.unwrap());
|
|
for key in &keys[2..] {
|
|
assert!(!store.has(&key.as_bytes()).await.unwrap());
|
|
}
|
|
|
|
source.remove(&keys[1]);
|
|
let third = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(third.candidates_staged, third.deleted),
|
|
(1, 0),
|
|
"the newly dead entry is staged exactly"
|
|
);
|
|
let fourth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(fourth.deleted, 1, "and executed exactly by the next sweep");
|
|
assert!(!store.has(&keys[1].as_bytes()).await.unwrap());
|
|
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
|
|
|
|
source.remove(&keys[0]);
|
|
let fifth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(fifth.candidates_staged, 1, "the last entry stages exactly");
|
|
let sixth = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(sixth.deleted, 1, "and executes exactly");
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **routing is a pure function of length** — at,
|
|
/// below, and over the declared threshold, the same content lands
|
|
/// on exactly the same tier through both put paths (same placement
|
|
/// every time; nothing moves, nothing duplicates), identical
|
|
/// content through both paths is one pool entry, and every read is
|
|
/// byte-identical from wherever that tier holds it.
|
|
///
|
|
/// Tier placement is observed through both mediums (the oracle
|
|
/// adapted to the LO medium): kv entries hold a row in the kv
|
|
/// schema's `blobs` table and nothing in the pg-lo schema; large
|
|
/// entries hold a companion row behind an LO and no pool row — the
|
|
/// facade cannot and must not expose tier residency (it is
|
|
/// engine-blind).
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
|
|
/// ADR-003 §Dispatch policy (routing is a pure function of length;
|
|
/// no migration; CAS first-writer-wins makes re-puts one entry);
|
|
/// ADR-005 (re-put refcounts); ADR-007 §5 (postgres, the kv
|
|
/// engine); ADR-009 §2 (the companion table is the contract
|
|
/// authority); ADR-012 §4/§5 (the consolidation posture); ADR-012
|
|
/// §6.3 (the dual-tier dispatch-coverage shape); ADR-012 §6.7 (the
|
|
/// schedule-relative cleanup the dedup legs read against).
|
|
pub async fn routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
for content in [
|
|
patterned((threshold - 1) as usize, 191),
|
|
patterned(threshold as usize, 173),
|
|
patterned((threshold + 1) as usize, 167),
|
|
patterned(4 * threshold as usize + 9, 151),
|
|
] {
|
|
let len = content.len() as u64;
|
|
let canonical = Key::derive(&content, len);
|
|
let expected_rows = if len > threshold { 0 } else { 1 };
|
|
let expected_companions = if len > threshold { 1 } else { 0 };
|
|
|
|
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)]
|
|
{
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key, canonical,
|
|
"the canonical derivation addresses the put on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
expected_rows,
|
|
"length {len} on {path_tag}: the placement is exactly the \
|
|
routing function's answer and never moves or duplicates \
|
|
(the kv medium holds {expected_rows} pool rows)"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
expected_companions,
|
|
"length {len} on {path_tag}: the LO medium holds \
|
|
{expected_companions} companion rows"
|
|
);
|
|
let (got_len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the entry is present");
|
|
assert_eq!(
|
|
got_len, len,
|
|
"the read serves the entry from its placed tier"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the entry is byte-identical from its placed tier across paths"
|
|
);
|
|
drop(pin);
|
|
}
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
staged.candidates_staged, 1,
|
|
"identical content through both put paths is one pool entry"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"the executing sweep removed exactly the one entry (durable \
|
|
schedule: staged this sweep, executed by the next)"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(0, 0),
|
|
"both mediums are back to zero"
|
|
);
|
|
}
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **threshold boundary lengths route
|
|
/// deterministically** — at/below/above the factory's declared
|
|
/// threshold, on both put paths, the placement is the routing
|
|
/// function's answer every time (the mediums distinguish the kv
|
|
/// schema's rows from the pg-lo schema's companion rows); and the
|
|
/// default-threshold boundary re-routes with the tuning (the
|
|
/// constructor-tuned threshold deterministically places the
|
|
/// default boundary's length on the tier its own threshold names).
|
|
/// Run against two factories by the runner — the column's tuned
|
|
/// threshold and the shipped default. The durable cleanup: the
|
|
/// staged sweep stages exactly the six entries' dead set and the
|
|
/// next sweep executes it.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
|
|
/// ADR-003 §Dispatch policy (default 128 KiB at-or-below routes kv;
|
|
/// constructor-tunable without API change); ADR-007 §5 (postgres,
|
|
/// the kv engine); ADR-009 §2 (the companion table); ADR-012 §6.3
|
|
/// (the dual-tier dispatch-coverage shape); ADR-012 §6.7
|
|
/// (schedule-relative cleanup).
|
|
pub async fn threshold_boundary_lengths_route_deterministically(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let boundary = [
|
|
(threshold - 1, 191u32, false),
|
|
(threshold, 173, false),
|
|
(threshold, 167, false),
|
|
(threshold + 1, 151, true),
|
|
(threshold + 1, 139, true),
|
|
];
|
|
let mut expected_rows = 0u64;
|
|
let mut expected_companions = 0u64;
|
|
for (len, modulus, lands_large) in boundary {
|
|
let content = patterned(len as usize, modulus);
|
|
for declared in [Some(content.len() as u64), None] {
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&content, len),
|
|
"the canonical derivation addresses the put"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await as u64,
|
|
expected_rows + if lands_large { 0 } else { 1 },
|
|
"boundary length {len}: at/below the threshold lands kv \
|
|
(one row in the kv schema), over it lands large (a \
|
|
companion row instead), both put paths"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await as u64,
|
|
expected_companions + if lands_large { 1 } else { 0 },
|
|
"boundary length {len}: the LO medium matches the routing answer"
|
|
);
|
|
drop(pin);
|
|
}
|
|
if lands_large {
|
|
expected_companions += 1;
|
|
} else {
|
|
expected_rows += 1;
|
|
}
|
|
}
|
|
|
|
// The default boundary re-routes with the constructor's
|
|
// tuning: the shipped default's length belongs to the tier
|
|
// this configuration's own threshold puts it on.
|
|
let at_default = patterned(DEFAULT_DISPATCH_THRESHOLD as usize, 199);
|
|
let default_lands_large = DEFAULT_DISPATCH_THRESHOLD > threshold;
|
|
let (key, pin) = store
|
|
.put(
|
|
Some(DEFAULT_DISPATCH_THRESHOLD),
|
|
Cursor::new(at_default.as_slice()),
|
|
)
|
|
.await
|
|
.expect("the default-boundary put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&at_default, DEFAULT_DISPATCH_THRESHOLD),
|
|
"the canonical derivation addresses the put"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await as u64,
|
|
expected_rows + if default_lands_large { 0 } else { 1 },
|
|
"the default boundary's length routes to the tier this \
|
|
configuration's tuning names (the kv medium's answer)"
|
|
);
|
|
assert_eq!(
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await as u64,
|
|
expected_companions + if default_lands_large { 1 } else { 0 },
|
|
"the default boundary's length routes to the tier this \
|
|
configuration's tuning names (the LO medium's answer)"
|
|
);
|
|
drop(pin);
|
|
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
staged.candidates_staged, 6,
|
|
"each leg's content was exactly one pool entry"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 6,
|
|
"the executing sweep removed every entry, tier notwithstanding"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(0, 0),
|
|
"both mediums are back to zero"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Dispatch row: **a kv-tier miss falls through to the large tier
|
|
/// on every key-addressed op** — content the routing function
|
|
/// placed on the large tier (one companion row behind an LO; the
|
|
/// kv medium holds no pool row — the kv tier misses) serves `get`,
|
|
/// `stat`, `has`, and `read_range` whole, exact, and
|
|
/// digest-correct; the mirror-position leg proves the kv tier is
|
|
/// the *first* consult (sub-threshold content, one pool row and no
|
|
/// companion row, reads still served); and both-tier misses
|
|
/// resolve per the error model's per-op shapes (reached here by
|
|
/// the durable schedule's staged-then-executed deletion of the
|
|
/// subject entry).
|
|
///
|
|
/// Contract stamp: store-api.md §Read path (fall-through consult
|
|
/// order); ADR-012 §6.1 (every key-addressed op consults kv first,
|
|
/// then large; the general rule); ADR-012 §6.5 ruling 5 (both-tier
|
|
/// miss shapes: `None` on get/stat, `Err(Missing)` on read_range);
|
|
/// ADR-006 (the slice digest); ADR-007 §5 (postgres, the kv
|
|
/// engine); ADR-009 §2/§3 (the companion table and the `lo_get`
|
|
/// windows serving the fall-through); ADR-012 §6.3 (the
|
|
/// dispatch-coverage shape).
|
|
pub async fn kv_tier_misses_fall_through_to_the_large_tier_per_op(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let threshold = factory.threshold;
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
// Over-threshold content — the kv tier misses, the large tier
|
|
// serves every op. Both put paths hold the shape.
|
|
for (content, declared) in [
|
|
(
|
|
patterned(3 * threshold as usize + 41, 181),
|
|
Some(3 * threshold + 41),
|
|
),
|
|
(patterned(3 * threshold as usize + 43, 197), None),
|
|
] {
|
|
let len = content.len() as u64;
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(0, 1),
|
|
"the content lives on exactly the large tier — the kv tier misses"
|
|
);
|
|
let loid = loid_for_key(site.dsn(), site.pglo_schema(), &key.as_bytes())
|
|
.await
|
|
.expect("the companion row carries its LOID");
|
|
assert!(
|
|
loid_pages(site.dsn(), loid).await > 0,
|
|
"the LO medium holds the entry's pages"
|
|
);
|
|
|
|
let (got_len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the large tier served the kv miss");
|
|
assert_eq!(got_len, len);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the fall-through read is byte-identical"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
|
|
Some(alkblobs::EntryMeta {
|
|
len,
|
|
algorithm: key.algorithm(),
|
|
}),
|
|
"stat falls through and carries the length plus algorithm"
|
|
);
|
|
assert!(
|
|
store.has(&key.as_bytes()).await.unwrap(),
|
|
"has falls through"
|
|
);
|
|
let mid = len / 2;
|
|
let (slice, digest) = store
|
|
.read_range(&key.as_bytes(), mid, 17)
|
|
.await
|
|
.expect("read_range falls through");
|
|
assert_eq!(
|
|
slice,
|
|
content[(mid as usize)..(mid as usize + 17)],
|
|
"the range slice is exact"
|
|
);
|
|
assert_eq!(
|
|
digest,
|
|
slice_digest(&slice),
|
|
"the slice digest is SHA-256 over exactly the returned bytes"
|
|
);
|
|
|
|
// Sweep the entry away (the pin dropped): the miss shapes
|
|
// now resolve per op — the large tier misses too. The
|
|
// durable schedule: staged this sweep, executed by the next.
|
|
drop(pin);
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!((staged.candidates_staged, staged.deleted), (1, 0));
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
executed.deleted, 1,
|
|
"exactly the one entry died across the two sweeps"
|
|
);
|
|
assert!(
|
|
store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"a both-tier miss on get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"a both-tier miss on stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&key.as_bytes()).await.unwrap(),
|
|
"a both-tier miss on has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&key.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("the both-tier miss errs"),
|
|
Error::Missing,
|
|
"a both-tier miss on read_range is Err(Missing)"
|
|
);
|
|
}
|
|
|
|
// The mirror-position leg: sub-threshold content — the kv tier
|
|
// is the first consult and serves before the large tier is
|
|
// reached (one pool row; nothing ever lands on the LO medium).
|
|
let small = patterned((threshold - 1) as usize, 149);
|
|
let (key, pin) = store
|
|
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
|
|
.await
|
|
.expect("the sub-threshold put publishes");
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(1, 0),
|
|
"the kv tier holds the sub-threshold entry — the large tier \
|
|
was never consulted on the write, never reached on the read"
|
|
);
|
|
let (len, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the kv tier served");
|
|
assert_eq!(len, small.len() as u64);
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
small
|
|
);
|
|
|
|
drop(pin);
|
|
|
|
// Both-tier miss shapes for a key that never existed at all,
|
|
// per op, over the fresh both-tier store.
|
|
let absent = b"never put here".to_vec();
|
|
let absent = Key::derive(&absent, absent.len() as u64);
|
|
assert!(
|
|
store
|
|
.get(&absent.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"never-put: get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"never-put: stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&absent.as_bytes()).await.unwrap(),
|
|
"never-put: has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&absent.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("never-put read_range errs"),
|
|
Error::Missing,
|
|
"never-put: read_range is Err(Missing)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// INV-1's LO-medium extension (the mem-pglo factory's twin row,
|
|
/// over the consolidation shape's kv+LO mediums): **the `lo_get`
|
|
/// windows read across window boundaries byte-exact, and a body
|
|
/// short of its claimed length fails cleanly** — entries on the
|
|
/// large tier (both put paths) read whole through the
|
|
/// multi-window assembly, straddle the shipped window edge, read
|
|
/// ranged slices with the ADR-006 digest, clamp at EOF (tail,
|
|
/// at-EOF empty, a `u64::MAX` offset clamping in `u64` arithmetic,
|
|
/// zero-length, over-read), and a companion row overclaiming its
|
|
/// size (planted through the oracle) surfaces as the
|
|
/// unexpected-EOF-shaped `Error::Io`, never a partial read or a
|
|
/// panic.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 1 (the round-trip
|
|
/// family's ranged-read legs) and §Read path (the streaming
|
|
/// large-tier windows, pread-style clamping); ADR-009 §2/§3 (the
|
|
/// companion table is the authority — the cursor closes over the
|
|
/// row; descriptorless `lo_get` windows, the shipped read shape
|
|
/// and its unexpected-EOF failure); ADR-006 (the slice digest over
|
|
/// exactly the returned bytes); ADR-010 §1 (the large-tier cursor
|
|
/// body family).
|
|
pub async fn lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
SetSource::register(&store);
|
|
|
|
let content = patterned(2 * LO_WINDOW + 100, 167);
|
|
let len = content.len() as u64;
|
|
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)] {
|
|
let (key, pin) = store
|
|
.put(declared, Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("put publishes");
|
|
assert_eq!(
|
|
key,
|
|
Key::derive(&content, len),
|
|
"the canonical derivation addresses the put on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
0,
|
|
"the entries live on the LO medium — the kv tier misses"
|
|
);
|
|
let (_, blob) = store
|
|
.get(&key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect("the entry is present");
|
|
|
|
assert_eq!(
|
|
blob.read_at(0, len).await.expect("the whole window reads"),
|
|
content,
|
|
"the multi-window assembly reads whole and byte-exact on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(LO_WINDOW as u64 - 100, 300)
|
|
.await
|
|
.expect("the straddling read succeeds"),
|
|
content[(LO_WINDOW - 100)..(LO_WINDOW + 200)],
|
|
"a read straddling the window edge crosses two lo_get \
|
|
statements byte-exact on {path_tag}"
|
|
);
|
|
let (slice, digest) = blob
|
|
.read_range(LO_WINDOW as u64, 7)
|
|
.await
|
|
.expect("the ranged slice reads");
|
|
assert_eq!(
|
|
slice,
|
|
content[LO_WINDOW..LO_WINDOW + 7],
|
|
"the ranged slice is exact on {path_tag}"
|
|
);
|
|
assert_eq!(
|
|
digest,
|
|
slice_digest(&slice),
|
|
"the slice digest is SHA-256 over exactly the returned bytes"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(len - 5, 100).await.expect("the tail reads"),
|
|
content[(len as usize - 5)..],
|
|
"the tail read clamps at the entry's end"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(len, 8).await.expect("the EOF read succeeds"),
|
|
Vec::<u8>::new(),
|
|
"a read at EOF is empty, never an error"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(u64::MAX, 5).await.expect("the offset reads"),
|
|
Vec::<u8>::new(),
|
|
"an overflowing u64 offset clamps in u64 arithmetic — the \
|
|
C7 int4 trap not inherited"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, 0).await.expect("the empty read succeeds"),
|
|
Vec::<u8>::new(),
|
|
"a zero-length read is empty"
|
|
);
|
|
assert_eq!(
|
|
blob.read_at(0, len + 64)
|
|
.await
|
|
.expect("the over-read reads"),
|
|
content,
|
|
"an over-read clamps to the whole entry"
|
|
);
|
|
|
|
drop(pin);
|
|
}
|
|
|
|
// The beyond-length clean failure: a companion row claiming
|
|
// more bytes than its LO holds (the corrupted half-state
|
|
// healthy stores never hold) planted through the oracle — the
|
|
// read window trusts the row (the contract authority,
|
|
// metadata-only) and fails cleanly when the LO comes up short.
|
|
let planted = b"the planted short body".to_vec();
|
|
let planted_key = Key::derive(&planted, planted.len() as u64);
|
|
let claimed = (planted.len() + 4096) as i64;
|
|
let loid = {
|
|
let client = admin_connect_plant(site.dsn()).await;
|
|
client
|
|
.batch_execute("BEGIN")
|
|
.await
|
|
.expect("the plant opens");
|
|
let loid: u32 = client
|
|
.query_one("SELECT lo_create(0) AS loid", &[])
|
|
.await
|
|
.expect("lo_create mints the planted LO")
|
|
.get(0);
|
|
let fd: i32 = client
|
|
.query_one("SELECT lo_open($1, $2)", &[&loid, &INV_WRITE])
|
|
.await
|
|
.expect("the planted LO opens for write")
|
|
.get(0);
|
|
client
|
|
.query_one("SELECT lowrite($1, $2)", &[&fd, &planted])
|
|
.await
|
|
.expect("the planted partial body writes");
|
|
let committed_at = SystemTime::now()
|
|
.duration_since(std::time::UNIX_EPOCH)
|
|
.map(|d| d.as_nanos() as i64)
|
|
.unwrap_or(0);
|
|
client
|
|
.execute(
|
|
&format!(
|
|
"INSERT INTO \"{}\".lo_entries (key, loid, size, committed_at) \
|
|
VALUES ($1, $2, $3, $4)",
|
|
site.pglo_schema()
|
|
),
|
|
&[&planted_key.as_bytes(), &loid, &claimed, &committed_at],
|
|
)
|
|
.await
|
|
.expect("the overclaiming companion row plants");
|
|
client
|
|
.batch_execute("COMMIT")
|
|
.await
|
|
.expect("the plant commits");
|
|
drop(client);
|
|
loid
|
|
};
|
|
let (got_len, planted_blob) = store
|
|
.get(&planted_key.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.expect(
|
|
"the planted entry is present (the companion row is the \
|
|
authority, metadata-only)",
|
|
);
|
|
assert_eq!(
|
|
got_len, claimed as u64,
|
|
"the cursor's length is the companion row's overclaim — never \
|
|
an LO content probe"
|
|
);
|
|
let err = planted_blob
|
|
.read_at(0, got_len)
|
|
.await
|
|
.expect_err("the short body fails cleanly beyond its length");
|
|
assert!(
|
|
matches!(&err, Error::Io(m) if m.contains("short of")),
|
|
"the beyond-length read surfaces the unexpected-EOF clean \
|
|
failure, never a partial read: {err}"
|
|
);
|
|
|
|
// The durable cleanup: the round-trip legs' identical content
|
|
// was exactly one pool entry (CAS across the paths), so that
|
|
// entry and the planted one stage exactly, and the executing
|
|
// sweep deletes exactly that staged set — the planted LO's
|
|
// pages too.
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(staged.aborted, staged.candidates_staged),
|
|
(None, 2),
|
|
"the round-trip entry (one across both paths) and the planted \
|
|
one stage exactly"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(executed.deleted, 2, "the staged set executed exactly");
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
|
|
window_rows(site.dsn(), site.kv_schema()).await
|
|
),
|
|
(0, 0, 0),
|
|
"both mediums and the window are back to zero"
|
|
);
|
|
assert_eq!(
|
|
loid_pages(site.dsn(), loid).await,
|
|
0,
|
|
"the executing sweep's lo_unlink reaped the planted LO"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// The plant's raw-LO flag — postgres Large Objects' `INV_WRITE`
|
|
/// mode bits, the same constant the engine's orphan test plants
|
|
/// with.
|
|
const INV_WRITE: i32 = 0x0002_0000;
|
|
|
|
async fn admin_connect_plant(dsn: &str) -> tokio_postgres::Client {
|
|
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
|
|
.await
|
|
.expect("the plant connection opens");
|
|
tokio::spawn(async move {
|
|
let _ = connection.await;
|
|
});
|
|
client
|
|
}
|
|
|
|
/// The **dormancy row** — the consolidation posture's
|
|
/// schema-placement shape (the one-window-host rule's kv-host arm,
|
|
/// ADR-012 §5): the kv schema carries `blobs` and both GC tables
|
|
/// (`gc_pins` + `gc_window` — the fleet host's both-tables row),
|
|
/// the pg-lo schema carries only the companion table and **no
|
|
/// `gc_window`** (the pg-lo window clauses are dormant on the
|
|
/// dual-SQL instance), and the schedule itself is observable on
|
|
/// the kv host: a dead entry stages into the kv schema's
|
|
/// `gc_window`, never anywhere in the pg-lo schema.
|
|
///
|
|
/// Contract stamp: store-api.md §Invariants 4 (the durable
|
|
/// schedule's host, read through its own staging outcome);
|
|
/// ADR-008 §2 (the durable window; the staged re-arbitrated
|
|
/// delete); ADR-012 §5 (the one-window-host rule: the kv engine
|
|
/// hosts the window on dual-SQL instances — the pg-lo window
|
|
/// clauses dormant; and pg-lo never carries fleet pins); ADR-012
|
|
/// §6.7 (the executor is the next sweep); ADR-009 §2 (the
|
|
/// companion table's bootstrap).
|
|
pub async fn the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
assert_eq!(
|
|
store.mode(),
|
|
StoreMode::DualTier,
|
|
"the column's factory is the dispatch-coverage shape"
|
|
);
|
|
let dsn = site.dsn();
|
|
let kv = site.kv_schema();
|
|
let lo = site.pglo_schema();
|
|
|
|
assert!(
|
|
crate::pg_lo_column::table_exists(dsn, kv, "blobs").await,
|
|
"the kv tier's pool table exists in the kv schema"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::table_exists(dsn, kv, "gc_window").await,
|
|
"the kv host wins the window: gc_window in the kv schema"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::table_exists(dsn, kv, "gc_pins").await,
|
|
"the fleet host's both-tables row: gc_pins in the kv schema"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::table_exists(dsn, lo, "lo_entries").await,
|
|
"the pg-lo tier's companion table exists in the pg-lo schema"
|
|
);
|
|
assert!(
|
|
!crate::pg_lo_column::table_exists(dsn, lo, "gc_window").await,
|
|
"the pg-lo window clauses are dormant on the dual-SQL instance: \
|
|
the pg-lo schema carries no gc_window"
|
|
);
|
|
assert!(
|
|
!crate::pg_lo_column::table_exists(dsn, lo, "gc_pins").await,
|
|
"gc_pins is never created by pg-lo (fleet pins live on the kv host)"
|
|
);
|
|
|
|
// The schedule itself rides the kv host: a dead entry stages
|
|
// into the kv schema's gc_window and executes by the next
|
|
// sweep — nothing ever stages into the pg-lo schema (which
|
|
// carries no window at all).
|
|
SetSource::register(&store);
|
|
let content = patterned(2 * TUNED_THRESHOLD as usize + 21, 173);
|
|
let (key, pin) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the dead candidate publishes");
|
|
drop(pin);
|
|
let staged = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(
|
|
(staged.aborted, staged.candidates_staged, staged.deleted),
|
|
(None, 1, 0),
|
|
"the staging sweep staged the dead candidate"
|
|
);
|
|
assert_eq!(
|
|
window_rows(dsn, kv).await,
|
|
1,
|
|
"the staged candidate waits in the kv schema's gc_window"
|
|
);
|
|
let executed = store.sweep().await.expect("sweeps are never errors");
|
|
assert_eq!(executed.deleted, 1, "the next sweep executed it");
|
|
assert!(
|
|
!store.has(&key.as_bytes()).await.unwrap(),
|
|
"the executed delete landed"
|
|
);
|
|
assert_eq!(
|
|
(window_rows(dsn, kv).await, lo_entry_rows(dsn, lo).await),
|
|
(0, 0),
|
|
"the window emptied and the companion table carries nothing of it"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
/// Virgin-store row: **reads are no-ops on a fresh schema** — an
|
|
/// empty `list`, absent reads per op (`None` on get/stat, `false`
|
|
/// on has, `Err(Missing)` on read_range), and nothing on the fresh
|
|
/// mediums themselves (zero pool rows in the kv schema, zero
|
|
/// companion rows in the pg-lo schema, and the window state per
|
|
/// the host rule: `gc_window` in the kv schema holds zero rows).
|
|
/// The constructors always ensure their schemas, this row being
|
|
/// the facade form.
|
|
///
|
|
/// Contract stamp: store-api.md §Error model (virgin-store
|
|
/// semantics: reads on a fresh store see absent/empty, never
|
|
/// errors); ADR-012 §6.5 ruling 5 (the per-op miss shapes); ADR-007
|
|
/// §5 (postgres, the schema-ensuring construction posture);
|
|
/// ADR-009 §2 (the companion-table bootstrap); ADR-012 §5 (the
|
|
/// one-window-host rule's kv host on dual-SQL instances).
|
|
pub async fn virgin_store_reads_are_no_ops_on_a_fresh_schema(
|
|
factory: &DualTierPostgresPgLoFactory,
|
|
) {
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("open vends a fresh, isolated store");
|
|
assert_eq!(
|
|
store.mode(),
|
|
StoreMode::DualTier,
|
|
"the column's factory is the dispatch-coverage shape"
|
|
);
|
|
let dsn = site.dsn();
|
|
|
|
let absent = b"nothing was ever put".to_vec();
|
|
let absent = Key::derive(&absent, absent.len() as u64);
|
|
|
|
assert_eq!(
|
|
store.list().await.expect("list succeeds"),
|
|
Vec::<Key>::new(),
|
|
"the virgin pool lists empty"
|
|
);
|
|
assert!(
|
|
store
|
|
.get(&absent.as_bytes())
|
|
.await
|
|
.expect("get succeeds")
|
|
.is_none(),
|
|
"absent get is Ok(None)"
|
|
);
|
|
assert_eq!(
|
|
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
|
|
None,
|
|
"absent stat is Ok(None)"
|
|
);
|
|
assert!(
|
|
!store.has(&absent.as_bytes()).await.unwrap(),
|
|
"absent has is false"
|
|
);
|
|
assert_eq!(
|
|
store
|
|
.read_range(&absent.as_bytes(), 0, 4)
|
|
.await
|
|
.expect_err("absent read_range errs"),
|
|
Error::Missing,
|
|
"absent read_range is Err(Missing)"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(dsn, site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(dsn, site.pglo_schema()).await,
|
|
window_rows(dsn, site.kv_schema()).await
|
|
),
|
|
(0, 0, 0),
|
|
"the fresh mediums hold nothing (pool, companion table, and \
|
|
the kv-hosted window all empty)"
|
|
);
|
|
|
|
drop(store);
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The runners — the column's tests: the form-independent reference
|
|
// rows mirrored verbatim against the dual-tier factory (INV-1, INV-6,
|
|
// and the two no-op abort shapes), the durable-schedule rows above,
|
|
// and the factory contract's structural shapes. Every runner gates
|
|
// behind the reachability probe — the SKIP posture: server-less
|
|
// runs skip cleanly, a skip is not a failure; acceptance is the
|
|
// green run against the harness server.
|
|
// --------------------------------------------------------------
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv1_round_trip_puts_and_gets_are_byte_identical() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::round_trip_puts_and_gets_are_byte_identical(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv2_dedup_identical_content_is_one_pool_entry() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
dedup_identical_content_is_one_pool_entry(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv3_stage_hygiene_zero_residue_one_entry_on_success() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv4_two_sweep_exact_counts_over_the_full_protection_set() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
two_sweep_exact_counts_over_the_full_protection_set(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv4_a_put_landing_during_the_window_survives_the_executing_sweep() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
a_put_landing_during_the_window_survives_the_executing_sweep(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv4_sweep_without_liveness_sources_aborts_and_deletes_nothing() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::sweep_without_liveness_sources_aborts_and_deletes_nothing(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv4_sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing(&factory).await;
|
|
}
|
|
|
|
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
|
async fn postgres_pglo_inv4_the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
|
|
) {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv5_list_correctness_through_durable_schedule_outcomes() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
list_correctness_is_observable_through_durable_schedule_outcomes(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_inv6_stat_reports_len_and_algorithm_without_fetching_content() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
rows::stat_reports_len_and_algorithm_without_fetching_content(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly()
|
|
{
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn postgres_pglo_virgin_store_reads_are_no_ops_on_a_fresh_schema() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
virgin_store_reads_are_no_ops_on_a_fresh_schema(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_postgres_pglo_routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
|
|
) {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(&factory).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_postgres_pglo_threshold_boundary_lengths_route_deterministically() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let tuned = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
threshold_boundary_lengths_route_deterministically(&tuned).await;
|
|
let default = DualTierPostgresPgLoFactory::new(DEFAULT_DISPATCH_THRESHOLD);
|
|
threshold_boundary_lengths_route_deterministically(&default).await;
|
|
}
|
|
|
|
#[tokio::test]
|
|
async fn dispatch_postgres_pglo_kv_tier_misses_fall_through_to_the_large_tier_per_op() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
kv_tier_misses_fall_through_to_the_large_tier_per_op(&factory).await;
|
|
}
|
|
|
|
// --------------------------------------------------------------
|
|
// The factory contract's structural shapes (the postgres column's
|
|
// pattern, carried to the LO medium with the LO-isolation
|
|
// corollary's arms): fresh isolated opens (a fresh dedicated
|
|
// database with two fresh schemas per open, nothing shared — two
|
|
// opens never observe one another) and teardown against a
|
|
// panicking row (the schemas really dropped, the LOs really
|
|
// unlinked — zero orphan residue probed inside the dedicated
|
|
// database before it goes, and the shared harness database
|
|
// untouched by construction). Not conformance rows — no backlog
|
|
// entry — the factory contract's own wiring-level assertions.
|
|
// --------------------------------------------------------------
|
|
|
|
/// Every `open()` mints a fresh, isolated database + schema pair:
|
|
/// no shared state between islands, observable on the facade *and*
|
|
/// on the fresh mediums (each open's kv and pg-lo schemas are
|
|
/// private and start empty).
|
|
#[tokio::test]
|
|
async fn postgres_pglo_factory_open_vends_fresh_isolated_stores() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
|
|
|
|
let (a, site_a) = factory.open_tracked().await.expect("open vends a store");
|
|
let (b, site_b) = factory.open_tracked().await.expect("open vends another");
|
|
assert_ne!(
|
|
site_a.kv_schema(),
|
|
site_b.kv_schema(),
|
|
"two opens never share a kv schema"
|
|
);
|
|
assert_ne!(
|
|
site_a.pglo_schema(),
|
|
site_b.pglo_schema(),
|
|
"two opens never share a pg-lo schema"
|
|
);
|
|
assert_ne!(site_a.db, site_b.db, "two opens never share a database");
|
|
|
|
let content_a = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
|
|
let content_b = patterned(TUNED_THRESHOLD as usize - 5, 191);
|
|
let (key_a, _) = a
|
|
.put(
|
|
Some(content_a.len() as u64),
|
|
Cursor::new(content_a.as_slice()),
|
|
)
|
|
.await
|
|
.expect("island a's put publishes");
|
|
let (key_b, _) = b
|
|
.put(None, Cursor::new(content_b.as_slice()))
|
|
.await
|
|
.expect("island b's put publishes");
|
|
|
|
assert!(a.has(&key_a.as_bytes()).await.unwrap());
|
|
assert!(
|
|
!a.has(&key_b.as_bytes()).await.unwrap(),
|
|
"island a sees nothing of island b"
|
|
);
|
|
assert!(b.has(&key_b.as_bytes()).await.unwrap());
|
|
assert!(
|
|
!b.has(&key_a.as_bytes()).await.unwrap(),
|
|
"island b sees nothing of island a"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site_a.dsn(), site_a.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site_a.dsn(), site_a.pglo_schema()).await
|
|
),
|
|
(0, 1),
|
|
"island a's mediums hold exactly its own entry (a's content routes large)"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site_b.dsn(), site_b.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site_b.dsn(), site_b.pglo_schema()).await
|
|
),
|
|
(1, 0),
|
|
"island b's mediums hold exactly its own entry (b's content routes kv)"
|
|
);
|
|
assert!(
|
|
crate::pg_lo_column::zero_lo_residue(site_a.dsn(), site_a.pglo_schema()).await,
|
|
"island a's committed content has zero orphan LO residue (the \
|
|
LO catalog and the companion table agree exactly)"
|
|
);
|
|
|
|
let (c, site_c) = factory.open_tracked().await.expect("a third vends fresh");
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site_c.dsn(), site_c.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site_c.dsn(), site_c.pglo_schema()).await
|
|
),
|
|
(0, 0),
|
|
"a third open's mediums are virgin"
|
|
);
|
|
assert!(
|
|
!c.has(&key_a.as_bytes()).await.unwrap(),
|
|
"a third open shares nothing with the first"
|
|
);
|
|
assert!(
|
|
!c.has(&key_b.as_bytes()).await.unwrap(),
|
|
"a third open shares nothing with the second"
|
|
);
|
|
|
|
let dbs = factory.tracked_dbs();
|
|
assert_eq!(dbs.len(), 3, "one minted database per open");
|
|
assert_eq!(
|
|
dbs.iter()
|
|
.map(|(name, _, _)| name.clone())
|
|
.collect::<std::collections::BTreeSet<_>>()
|
|
.len(),
|
|
3,
|
|
"no two opens share a database"
|
|
);
|
|
for (_, _, schemas) in &dbs {
|
|
assert_eq!(
|
|
schemas.len(),
|
|
2,
|
|
"one kv + one pg-lo minted schema per database"
|
|
);
|
|
}
|
|
|
|
// The engine-owned schemas really exist inside their dedicated
|
|
// databases (the registry's minted list is not bookkeeping
|
|
// fiction).
|
|
for (name, dsn, schemas) in &dbs {
|
|
for schema in schemas {
|
|
assert!(
|
|
crate::pg_lo_column::schema_exists(dsn, schema).await,
|
|
"the minted schema exists inside {name}: {schema}"
|
|
);
|
|
}
|
|
}
|
|
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown reaps the LOs, drops the schemas, drops the databases");
|
|
factory.teardown().await.expect("teardown repeats cleanly");
|
|
let maintenance = admin_dsn(factory.dsn());
|
|
for (name, _, _) in &dbs {
|
|
assert!(
|
|
!db_exists(&maintenance, name).await,
|
|
"teardown dropped the dedicated database: {name} (schemas \
|
|
and all — the whole LO state goes with it)"
|
|
);
|
|
}
|
|
}
|
|
|
|
/// A failing assertion panics the row before teardown runs — and
|
|
/// teardown must make sense over the abandoned instance anyway: it
|
|
/// is idempotent against the panicked row's path, the LOs are
|
|
/// really unlinked (the tracked-LOID reap arm — the recorded
|
|
/// LOIDs match the pre-captured set) and the minted schemas really
|
|
/// dropped (gone with the dedicated database, **zero orphan
|
|
/// `pg_largeobject` residue** probed inside before it goes), and
|
|
/// the next open mints a fresh database on fresh (empty) mediums
|
|
/// (README §The factory contract). The shared harness database is
|
|
/// untouched by construction — no pg-lo pool ever opens against it.
|
|
#[tokio::test]
|
|
async fn postgres_pglo_factory_teardown_is_safe_against_a_panicking_row() {
|
|
if !ready().await {
|
|
return;
|
|
}
|
|
let content = patterned(2 * TUNED_THRESHOLD as usize + 7, 173);
|
|
let abandoned = Key::derive(&content, content.len() as u64);
|
|
let factory = Arc::new(DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD));
|
|
|
|
let task = {
|
|
let factory = Arc::clone(&factory);
|
|
tokio::spawn(async move {
|
|
let (store, _) = factory.open_tracked().await.expect("open vends the store");
|
|
let (_, pin) = store
|
|
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
|
|
.await
|
|
.expect("the put publishes");
|
|
drop(pin);
|
|
panic!("a failing assertion panics the row (the factory contract's early exit)");
|
|
})
|
|
};
|
|
assert!(
|
|
task.await.is_err(),
|
|
"the row's panic surfaced as its failure"
|
|
);
|
|
|
|
let dbs = factory.tracked_dbs();
|
|
assert_eq!(dbs.len(), 1, "the panicked row minted one database");
|
|
let (name, dsn, schemas) = &dbs[0];
|
|
let lo_schema = {
|
|
let mut found = None;
|
|
for schema in schemas {
|
|
if crate::pg_lo_column::table_exists(dsn, schema, "lo_entries").await {
|
|
found = Some(schema);
|
|
break;
|
|
}
|
|
}
|
|
found.expect("the minted pg-lo half carries the companion table")
|
|
};
|
|
let loids = companion_loids(dsn, lo_schema).await;
|
|
assert!(!loids.is_empty(), "the panicked row's LO is tracked");
|
|
assert!(
|
|
crate::pg_lo_column::zero_lo_residue(dsn, lo_schema).await,
|
|
"the abandoned run's committed content has zero orphan LO \
|
|
residue (the LO catalog and the companion table agree exactly)"
|
|
);
|
|
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown makes sense over the abandoned instance");
|
|
factory.teardown().await.expect("teardown is idempotent");
|
|
assert!(
|
|
!db_exists(&admin_dsn(factory.dsn()), name).await,
|
|
"teardown dropped the panicked row's database: {name} (the \
|
|
schemas and the whole LO state go with it)"
|
|
);
|
|
let mut reaped = factory.reaped_loids();
|
|
reaped.sort_unstable();
|
|
let mut captured = loids.clone();
|
|
captured.sort_unstable();
|
|
assert_eq!(
|
|
reaped, captured,
|
|
"the teardown's reap arm unlinked exactly the tracked LOIDs"
|
|
);
|
|
|
|
let (store, site) = factory
|
|
.open_tracked()
|
|
.await
|
|
.expect("the next open still vends a store");
|
|
assert!(
|
|
!store.has(&abandoned.as_bytes()).await.unwrap(),
|
|
"no residue from the panicked row — isolation is factory-owned"
|
|
);
|
|
assert_eq!(
|
|
(
|
|
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
|
|
lo_entry_rows(site.dsn(), site.pglo_schema()).await
|
|
),
|
|
(0, 0),
|
|
"and its mediums carry nothing of the abandoned run"
|
|
);
|
|
factory
|
|
.teardown()
|
|
.await
|
|
.expect("teardown drops the last database");
|
|
}
|
|
}
|