Files
alkblobs/tests/conformance/pg_lo_column.rs
T
glm-5.3-flash aa96f8b981 feat(tests): the conformance suite's pg-lo column — the LO-medium SKIP-posture column green (wave 5)
- tests/conformance/pg_lo_column.rs (new): the two pair-gated factories
  (DualTierMemPgLoFactory on all(mem, pg-lo) — the one-window-host rule's
  named test topology; DualTierPostgresPgLoFactory on
  all(postgres, pg-lo) — the consolidation posture, two schemas per open,
  carrying the dormancy row), a fresh dedicated database per open with
  the engine-owned schemas minted inside, teardown reaping tracked LOIDs
  (lo_unlink) then dropping the minted schemas then the database
  (WITH (FORCE)) with per-open drop-guard janitors for the abandoned
  paths, and the rows: the verbatim form-independent mirror (INV-1, INV-6,
  the two no-op abort shapes), the schedule-relative INV-2/3, the
  two-sweep exact counts over the full protection set across each
  factory's observed window host, the window-delay race, the
  holder-stages-only lease, INV-5's durable row, the three dispatch rows
  over the LO-medium placement oracle, the INV-1 extension (window-edge
  ranged reads + the planted-overclaim beyond-length clean failure), the
  virgin-store rows, and the dormancy row; structural isolation +
  panicking-row tests probing zero orphan residue (no kv-only rejection
  row — recorded; pg-lo is a large engine)
- the LO-isolation corollary carried in the engine task's recorded
  per-test-database shape (the C7 lesson): zero LO residue on the shared
  harness database is structural, so the engine suite's
  shared_harness_db_holds_zero_lo_residue gate stays green alongside the
  column under --all-features
- main.rs: the pg_lo_column gate (feature pg-lo, default-off) + the
  columns' doc paragraph
- README: the pg-lo rows table in §Columns beyond the reference set, the
  SKIP-posture note naming the second landed column and the per-factory
  feature-pair gating, current state = five columns
- store.rs: one mechanical gate fix — the PgOpts import was unused in the
  sqlite+pg-lo config (tripping clippy --features pg-lo -D warnings);
  the gate now names the configs that consume it

Verification (harness server live 2026-10-12): --all-features against the
server fully green (188 lib + 106 conformance + 4 git-interop; zero
LO/schema/database residue after the run); default test green (69/19/3);
--features "pg-lo,mem" and "postgres,pg-lo" green server-less
(skipping cleanly) and --features pg-lo alone; clippy -D warnings green
on default, pg-lo, pg-lo+mem, postgres+pg-lo, all-features; fmt --check
green; cargo doc --no-deps clean.
2026-10-12 08:47:55 +00:00

5202 lines
212 KiB
Rust

//! The **pg-lo column** — the suite's second SKIP-posture column: the
//! reference row set and the durable-schedule sweep rows run against
//! the pg-lo large tier (feature `pg-lo`, default-off, ADR-009). The
//! column grows **two factories spanning feature pairs — the first
//! column whose factories need different feature sets** (a recorded
//! design point):
//!
//! - **[`DualTierMemPgLoFactory`](mem_pglo::DualTierMemPgLoFactory)** —
//! `kv = mem` + `large = pg-lo`: the one-window-host rule's named
//! test topology (ADR-012 §5). The durable-schedule sweep rows run
//! with the window hosted on the **large engine** (`gc_window` in
//! the pg-lo schema) — the shape no earlier column exercises. Gates
//! on `all(feature = "mem", feature = "pg-lo")`.
//! - **[`DualTierPostgresPgLoFactory`](postgres_pglo::DualTierPostgresPgLoFactory)** —
//! `kv = postgres` + `large = pg-lo` on one pg instance: the
//! consolidation posture (ADR-009; the fleet-valid engine set's last
//! constructor). Mirrors the postgres column's inventory with the
//! large medium swapped to LOs, and carries the **dormancy row**
//! (the pg-lo schema carries no `gc_window`; the window lives on the
//! kv schema — ADR-012 §5). Gates on
//! `all(feature = "postgres", feature = "pg-lo")`.
//!
//! The SKIP posture (README §The SKIP posture) is a factory-owner
//! responsibility: the connection settings ride the environment
//! (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB` — host required; the rest
//! default to the harness server's values), every runner gates behind
//! a **reachability probe**, and when no server is reachable the
//! column skips cleanly — a skip is not a failure. The acceptance run
//! is green **against the harness server** (dockerized `pglo-poc`,
//! `postgres:16-alpine`, :15432, db `blobs`) under `--all-features` —
//! a skip never substitutes for it.
//!
//! The factories (README §The factory contract): `open()` mints a
//! **fresh, dedicated database per call** carrying the engine-owned
//! schemas (the alkstore pg-factory pattern — a fresh engine-owned
//! schema per open, unique per call, tracked per-instance in a
//! registry — housed on its own database), and vends the concrete
//! [`alkblobs::Store`] facade over them (the mem-pglo factory's kv
//! tier is in-process mem, needing no schema of its own; the
//! postgres-pglo factory mints two — the kv half and the pg-lo half —
//! so the dormancy row observes genuine absence). **The LO-isolation
//! corollary carried in the engine task's recorded shape** (the C7
//! lesson): Large Objects are **database-level**, not schema-level —
//! the schema pattern alone does not isolate them and a bare
//! `DROP SCHEMA` strands their pages in `pg_largeobject` — so each
//! open gets its own database, and teardown reaps the instance's LOs
//! (`SELECT loid FROM <pg-lo schema>.lo_entries` — the companion
//! table, the contract authority — then `lo_unlink` each tracked
//! LOID), drops exactly the minted schemas (`DROP SCHEMA IF EXISTS …
//! CASCADE`, panicking-row-safe), and drops the dedicated database
//! (`WITH (FORCE)`), with a drop-guard janitor per open backing every
//! abandoned path. Zero LO residue on the shared harness database
//! (`blobs`) is therefore structural — no pg-lo pool ever opens
//! against it (the engine suite's
//! `shared_harness_db_holds_zero_lo_residue` gate measures exactly
//! this) — and the structural teardown assertions probe zero orphan
//! residue inside the dedicated database before it goes.
//!
//! The sweep rows are **schedule-relative on this column** (README
//! §The schedule-relative reading for SQL-backed columns, citing
//! ADR-012 §6.7): the durable form stages its candidates this sweep
//! and executes them by the next, so the exact counts are asserted
//! against the staged schedule — on the pg-lo host for the mem-pglo
//! factory, on the kv host for the postgres-pglo factory. The
//! form-independent rows mirror verbatim ([`crate::rows`]: the
//! round-trip family over the content matrix, the stat-discipline
//! row, and the `NoLivenessSources`/`ProtectFailed` abort shapes).
//! **No kv-only rejection row** — no kv-only pg-lo constructor exists
//! (pg-lo is a large engine; the postgres column hosts that row for
//! the SQL kv engines).
//!
//! The dual-tier dispatch rows mirror the postgres column's family
//! with the placement oracle adapted to the LO medium: large entries
//! are **companion rows + LOs** (an independent pg connection counting
//! `lo_entries` and probing `pg_largeobject` for the row's LOID), kv
//! entries per the factory's kv engine — the mem-pglo factory's kv
//! placement is in-process (the oracle's kv half follows the mem
//! column's introspection shape: served reads and absence from the
//! LO medium), the postgres-pglo factory's kv half counts the kv
//! schema's `blobs` rows.
#![allow(dead_code)]
use std::sync::atomic::{AtomicU64, Ordering};
use std::time::Duration;
use tokio_postgres::NoTls;
use alkblobs::Error;
/// The shipped LO read-window size — src/pg_lo.rs's crate-internal
/// `READ_CHUNK` constant (ADR-009 §3, POC #7 C2/C4's window shape; one
/// `lo_get` statement per window). The column's INV-1 extension row
/// straddles this edge — duplicated here because the constant is
/// engine-internal.
pub(crate) const LO_WINDOW: usize = 512 * 1024;
/// The **test tuning of the window delay** (ADR-008 §2's legal
/// degenerate tuning): `0` — execute-on-next-sweep. The durable
/// schedule's staging sweep stages the dead set; the *next* sweep is
/// its executor, which is exactly the schedule the sweep rows assert
/// against. Both factories run at this tuning.
pub(crate) const WINDOW_DELAY: Duration = Duration::ZERO;
// ------------------------------------------------------------------
// The harness connection (the SKIP posture's gate) — the settings
// ride the environment, the runners probe reachability before their
// rows run, an unreachable server skips the column cleanly.
// ------------------------------------------------------------------
/// The harness connection settings ride the environment
/// (`ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB` — the alkstore harness
/// convention). `ALKBLOBS_PG_HOST` set (non-empty) is the probe's
/// "configured" answer; port/user/password/db default to the harness
/// server's values (`pglo-poc`: :15432, `postgres`/`poc`, db
/// `blobs`). Absent → the caller skips.
pub(crate) fn harness_dsn() -> Option<String> {
let host = std::env::var("ALKBLOBS_PG_HOST")
.ok()
.filter(|v| !v.is_empty())?;
let port = std::env::var("ALKBLOBS_PG_PORT")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| "15432".to_owned());
let user = std::env::var("ALKBLOBS_PG_USER")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| "postgres".to_owned());
let password = std::env::var("ALKBLOBS_PG_PASSWORD")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| "poc".to_owned());
let db = std::env::var("ALKBLOBS_PG_DB")
.ok()
.filter(|v| !v.is_empty())
.unwrap_or_else(|| "blobs".to_owned());
Some(format!(
"host={host} port={port} user={user} password={password} dbname={db}"
))
}
/// The skip note — the SKIP posture's clean-skip print, exactly once
/// per skipped test.
pub(crate) fn skip_note() {
eprintln!("skipping (no postgres harness server): set ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB");
}
async fn admin_connect(dsn: &str) -> Result<tokio_postgres::Client, Error> {
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
.await
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
tokio::spawn(async move {
let _ = connection.await;
});
Ok(client)
}
/// The server reachability probe: the SKIP posture's gate the runners
/// check before their rows run. One cheap admin connection per test;
/// teardown reconnects independently (no shared admin client
/// lifetime).
pub(crate) async fn harness_ready() -> bool {
let Some(dsn) = harness_dsn() else {
skip_note();
return false;
};
match tokio::time::timeout(Duration::from_secs(3), admin_connect(&dsn)).await {
Ok(Ok(_)) => true,
Ok(Err(e)) => {
eprintln!("skipping (harness server unreachable): {e}");
false
}
Err(_) => {
eprintln!("skipping (harness server unreachable: connect timeout)");
false
}
}
}
// ------------------------------------------------------------------
// The dedicated-database mint + the LO-isolation corollary's teardown
// arms.
// ------------------------------------------------------------------
/// The maintenance DSN — connections through the `postgres`
/// database that create and drop dedicated databases.
pub(crate) fn admin_dsn(base: &str) -> String {
let base = base
.rsplit_once("dbname=")
.expect("the harness DSN names a db")
.0;
format!("{base}dbname=postgres")
}
/// The DSN scoped to one dedicated database — the store's and the
/// oracle's connection settings for that open.
pub(crate) fn db_dsn(base: &str, db: &str) -> String {
let base = base
.rsplit_once("dbname=")
.expect("the harness DSN names a db")
.0;
format!("{base}dbname={db}")
}
/// A fresh dedicated-database name — unique per call, seeded per
/// (tag, pid, nanos, factory-internal sequence) so two factory
/// instances never share a database, racing opens included.
pub(crate) fn fresh_db_name(tag: &str, seq: &AtomicU64) -> String {
format!(
"abpglosuite_{}_{}_{:x}_{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0),
seq.fetch_add(1, Ordering::Relaxed),
)
}
/// A fresh engine-owned schema name — unique per call, seeded per
/// (tag, pid, nanos, factory-internal sequence) so two factory
/// instances never share a schema name, racing opens included. The
/// mint stays well under postgres' 63-character identifier limit.
pub(crate) fn fresh_schema_name(tag: &str, seq: &AtomicU64) -> String {
format!(
"abpglo_{}_{}_{:x}_{}",
tag,
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or(0),
seq.fetch_add(1, Ordering::Relaxed),
)
}
/// Create a dedicated database (through the maintenance DSN).
pub(crate) async fn create_db(maintenance_dsn: &str, db: &str) -> Result<(), Error> {
let client = admin_connect(maintenance_dsn).await?;
client
.execute(&format!("CREATE DATABASE \"{db}\""), &[])
.await
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
Ok(())
}
/// Whether a dedicated database still exists server-side — the
/// structural teardown tests' instrument (`pg_database`).
pub(crate) async fn db_exists(maintenance_dsn: &str, db: &str) -> bool {
let client = admin_connect(maintenance_dsn)
.await
.expect("the oracle connection opens");
let count: i64 = client
.query_one(
"SELECT count(*) FROM pg_database WHERE datname = $1",
&[&db],
)
.await
.expect("the oracle query succeeds")
.get(0);
count > 0
}
/// Drop a dedicated database — `WITH (FORCE)` reaps any connection
/// the store's pool has not yet drained (postgres ≥ 13; the
/// panicking-row-safe arm; the same clause the engine suite's
/// janitor carries).
pub(crate) async fn drop_db(maintenance_dsn: &str, db: &str) -> Result<(), Error> {
let client = admin_connect(maintenance_dsn).await?;
client
.execute(&format!("DROP DATABASE \"{db}\" WITH (FORCE)"), &[])
.await
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
Ok(())
}
/// The zero-residue probe inside one dedicated database — the LO
/// catalog and the companion table agree exactly (no missing pages,
/// no orphans): the LO-isolation corollary's structural assertion,
/// the engine suite's C1 exact-count shape.
pub(crate) async fn zero_lo_residue(dsn: &str, schema: &str) -> bool {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let sql = format!(
"SELECT count(*) FROM (\
SELECT loid FROM \"{schema}\".lo_entries \
EXCEPT SELECT DISTINCT loid FROM pg_largeobject\
) missing, (\
SELECT DISTINCT loid FROM pg_largeobject \
EXCEPT SELECT loid FROM \"{schema}\".lo_entries\
) orphaned"
);
let disagreements: i64 = client
.query_one(sql.as_str(), &[])
.await
.expect("the oracle query succeeds")
.get(0);
disagreements == 0
}
/// Reap one schema's committed LOs before the schema goes away,
/// because Large Objects are database-level and a bare `DROP SCHEMA`
/// would strand their pages in `pg_largeobject`. The loids are read
/// from the companion table (the contract authority — never
/// `pg_largeobject`); each is unlinked best-effort so an idempotent
/// teardown is never torn by a missing oid; uncommitted staged pages
/// cannot exist (the held-tx rollback stories). Returns the reaped
/// LOIDs — the tracked-LOID list the structural tests assert against.
pub(crate) async fn reap_lo_schema(client: &tokio_postgres::Client, schema: &str) -> Vec<u32> {
let present = table_exists_conn(client, schema, "lo_entries").await;
if !present {
return Vec::new();
}
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries");
let loids: Vec<u32> = client
.query(sql.as_str(), &[])
.await
.expect("the loid enumeration succeeds")
.iter()
.map(|row| row.get::<_, u32>(0))
.collect();
for loid in &loids {
let _ = client.execute("SELECT lo_unlink($1)", &[loid]).await;
}
loids
}
/// The drop-guard janitor — the engine suite's recorded
/// `DatabaseJanitor`-shaped pattern (`src/pg_common.rs`'s
/// test-support janitor, `#[cfg(test)]`-internal there): its drop
/// removes the dedicated database when *Drop* is the only exit — a
/// row that panics before teardown, a factory released without one.
/// Same Drop mechanics: no async teardown exists at Drop, so the drop
/// executes synchronously from a std thread over a one-shot
/// current-thread runtime; an unreachable harness server is tolerated.
pub(crate) struct DbJanitor {
maintenance_dsn: String,
db: String,
}
impl DbJanitor {
/// Register the guard for one dedicated database. Constructed as
/// part of `open()`'s bookkeeping; it fires only if teardown never
/// consumed the record (teardown disarms it). Dropping an
/// already-dropped database is a tolerated no-op server-side.
pub(crate) fn new(maintenance_dsn: &str, db: &str) -> DbJanitor {
DbJanitor {
maintenance_dsn: maintenance_dsn.to_owned(),
db: db.to_owned(),
}
}
}
impl Drop for DbJanitor {
fn drop(&mut self) {
let maintenance_dsn = self.maintenance_dsn.clone();
let db = self.db.clone();
let _ = std::thread::spawn(move || {
let Ok(rt) = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
else {
return;
};
rt.block_on(async move {
let Ok((client, conn)) = tokio_postgres::connect(&maintenance_dsn, NoTls).await
else {
return;
};
tokio::spawn(async move {
let _ = conn.await;
});
let _ = client
.execute(&format!("DROP DATABASE \"{db}\" WITH (FORCE)"), &[])
.await;
});
})
.join();
}
}
/// One open's tracked backing site: the dedicated database this open
/// minted, the engine-owned schemas minted inside it (the alkstore
/// pg-factory pattern's schema registry, housed on its own database),
/// and the drop-guard janitor that disposes of the database if
/// teardown never gets to it (the abandoned paths — a row that panics
/// before teardown releases the factory into a Drop that runs the
/// guards). Teardown consumes the record, reaps + drops explicitly,
/// and disarms the guard.
pub(crate) struct TrackedDb {
/// The dedicated database's name (the structural tests'
/// `pg_database` probe target).
pub(crate) name: String,
/// The db-scoped DSN — the store's connection settings and the
/// oracle's address into this database.
pub(crate) dsn: String,
/// Every engine-owned schema minted inside the database (the kv
/// and pg-lo halves, in mint order — the pg-factory pattern's
/// registry).
pub(crate) schemas: Vec<String>,
/// The pg-lo halves — the schemas carrying `lo_entries`, whose
/// LOs teardown reaps.
pub(crate) lo_schemas: Vec<String>,
/// The janitor disarmed by teardown; Drop runs it otherwise.
janitor: Option<DbJanitor>,
}
impl TrackedDb {
/// Disarm the janitor (teardown took over the disposal).
pub(crate) fn disarm(&mut self) {
self.janitor = None;
}
}
// ------------------------------------------------------------------
// The medium oracle (the dispatch rows' instrument, adapted to the LO
// medium): an independent connection per probe — large entries are
// companion rows + LOs (the `lo_entries` count and the `pg_largeobject`
// pages behind each row's LOID), kv entries per the factory's kv
// engine, and the schedule itself is `gc_window` rows in the
// window-hosting schema (the pg-lo schema on the mem-pglo factory,
// the kv schema on the postgres-pglo factory).
// ------------------------------------------------------------------
async fn table_exists_conn(client: &tokio_postgres::Client, schema: &str, table: &str) -> bool {
let qualified = format!("\"{schema}\".\"{table}\"");
let row = client
.query_one("SELECT to_regclass($1) IS NOT NULL", &[&qualified])
.await
.expect("the table probe succeeds");
row.get(0)
}
/// Rows in one of the store's schemas' tables — the kv tier's pool
/// (`blobs`), the durable window's staged candidates (`gc_window`),
/// or the pg-lo tier's companion table (`lo_entries`) — counted from
/// the oracle's own independent connection to the shared server.
pub(crate) async fn pg_table_rows(dsn: &str, schema: &str, table: &str) -> usize {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let sql = format!("SELECT COUNT(*) FROM \"{schema}\".{table}");
let row = client
.query_one(sql.as_str(), &[])
.await
.expect("the oracle query succeeds");
let count: i64 = row.get(0);
count.max(0) as usize
}
/// The pg-lo tier's companion-table residency — exactly one row per
/// committed large-tier entry (the contract authority).
pub(crate) async fn lo_entry_rows(dsn: &str, schema: &str) -> usize {
pg_table_rows(dsn, schema, "lo_entries").await
}
/// The durable window's staged-candidate residency — the schedule
/// observer the durable rows assert against (staged rows land here;
/// the executing sweep's take empties them).
pub(crate) async fn window_rows(dsn: &str, schema: &str) -> usize {
pg_table_rows(dsn, schema, "gc_window").await
}
/// The companion table's loids — the tracked-LOID list the residue
/// probes and placement assertions ride.
pub(crate) async fn companion_loids(dsn: &str, schema: &str) -> Vec<u32> {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries ORDER BY loid");
client
.query(sql.as_str(), &[])
.await
.expect("the oracle query succeeds")
.iter()
.map(|row| row.get::<_, u32>(0))
.collect()
}
/// The LOID behind one companion row — looked up by the entry's key
/// bytes.
pub(crate) async fn loid_for_key(dsn: &str, schema: &str, key_bytes: &[u8]) -> Option<u32> {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let sql = format!("SELECT loid FROM \"{schema}\".lo_entries WHERE key = $1");
client
.query_opt(sql.as_str(), &[&key_bytes])
.await
.expect("the oracle query succeeds")
.map(|row| row.get::<_, u32>(0))
}
/// The `pg_largeobject` pages behind one LOID — the LO-presence probe
/// the placement oracle and the zero-residue teardown assertions
/// ride (the catalog read is address-adjacent, never addressing: the
/// companion table remains the authority).
pub(crate) async fn loid_pages(dsn: &str, loid: u32) -> i64 {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let row = client
.query_one(
"SELECT count(*) FROM pg_largeobject WHERE loid = $1",
&[&loid],
)
.await
.expect("the oracle query succeeds");
row.get(0)
}
/// Whether an engine-owned schema carries one of its tables — the
/// window-host assertions' instrument (`to_regclass`; the dormancy
/// row's genuine-absence probe).
pub(crate) async fn table_exists(dsn: &str, schema: &str, table: &str) -> bool {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
table_exists_conn(&client, schema, table).await
}
/// Whether the engine-owned schema still exists server-side — the
/// structural teardown tests' instrument (`pg_namespace`, the same
/// probe the alkstore factory shape uses).
pub(crate) async fn schema_exists(dsn: &str, schema: &str) -> bool {
let client = admin_connect(dsn)
.await
.expect("the oracle connection opens");
let count: i64 = client
.query_one(
"SELECT count(*) FROM pg_namespace WHERE nspname = $1",
&[&schema],
)
.await
.expect("the oracle query succeeds")
.get(0);
count > 0
}
// ------------------------------------------------------------------
// The mem-pglo factory — the one-window-host rule's named test
// topology, feature pair `mem` + `pg-lo`.
// ------------------------------------------------------------------
#[cfg(all(feature = "mem", feature = "pg-lo"))]
mod mem_pglo {
use std::io::Cursor;
use std::sync::atomic::AtomicU64;
use std::sync::{Arc, Mutex};
use std::time::{Duration, SystemTime};
use tokio_postgres::NoTls;
use crate::factory::StoreFactory;
use crate::pg_lo_column::{
admin_dsn, companion_loids, create_db, db_dsn, db_exists, drop_db, fresh_db_name,
harness_dsn, harness_ready, lo_entry_rows, loid_for_key, loid_pages, reap_lo_schema,
window_rows, DbJanitor, TrackedDb, LO_WINDOW, WINDOW_DELAY,
};
use crate::rows::{
self, patterned, slice_digest, Failing, LatchProtect, SetProtect, SetSource,
};
use alkblobs::{Error, Key, PgOpts, Store, StoreMode, DEFAULT_DISPATCH_THRESHOLD};
/// The column's declared dual-tier threshold — small, so the
/// shared content matrix and the race fixtures cross it and both
/// tiers' bodies are exercised (sub-threshold rows land in the
/// in-process mem kv tier; over-threshold rows stage-commit into
/// LOs behind companion rows).
pub(crate) const TUNED_THRESHOLD: u64 = 1024;
/// One vended backing site: the dedicated database this open
/// minted and the engine-owned pg-lo schema minted inside it. The
/// kv tier is in-process mem — its placement is facade-observed
/// (the mem introspection shape), the LO medium is the schema.
#[derive(Clone)]
pub(crate) struct Site {
/// The db-scoped DSN — the oracle's independent connections
/// address it; each open's database is private and gets
/// dropped whole at teardown (the LO-isolation corollary: LOs
/// are database-level).
dsn: String,
/// The engine-owned pg-lo schema this open minted — the LO
/// medium (the dispatch oracle's LO-side counters and the
/// durable window's row home, per the one-window-host rule).
schema: String,
/// The dedicated database's name — the structural teardown
/// tests' `pg_database` probe target.
db: String,
}
impl Site {
/// The minted pg-lo schema — the LO medium's and the window's
/// schema-qualified probe target.
pub(crate) fn schema(&self) -> &str {
&self.schema
}
/// The DSN the open minted — scoped to the dedicated database.
pub(crate) fn dsn(&self) -> &str {
&self.dsn
}
}
/// The dispatch-coverage factory: `kv = mem` + `large = pg-lo` at
/// the threshold this column declares and the test tuning of the
/// window delay (see [`TUNED_THRESHOLD`] and `WINDOW_DELAY` — the
/// one-window-host rule's named test topology, the durable window
/// hosting on the large engine, ADR-012 §5).
///
/// **Why a fresh instance per runner (not a shared static):**
/// teardown drops the dedicated databases *this instance* minted
/// (having reaped their LOs and dropped their schemas); a shared
/// instance's teardown would race concurrently running rows'
/// sites. Every runner constructs its own and owns its lifecycle;
/// a row that panics before teardown is still backed by the
/// per-open drop-guard janitors.
pub(crate) struct DualTierMemPgLoFactory {
dsn: String,
/// The declared dispatch threshold — the routing boundary this
/// column's rows run at (constructor-tunable; ADR-003).
/// Declared configuration: the dual-tier store never rejects,
/// so there is nothing to probe.
threshold: u64,
/// The declared window delay — the durable schedule's cadence
/// knob the sweep rows run at (the test tuning:
/// execute-on-next-sweep).
#[allow(dead_code)]
window_delay: Duration,
seq: AtomicU64,
/// Every dedicated database this instance minted, for its own
/// teardown and its drop-guard janitors.
dbs: Mutex<Vec<TrackedDb>>,
/// The LOIDs teardown reaped, for the structural tests'
/// assertions.
reaped_loids: Mutex<Vec<u32>>,
}
impl DualTierMemPgLoFactory {
pub(crate) fn new(threshold: u64) -> DualTierMemPgLoFactory {
DualTierMemPgLoFactory {
dsn: harness_dsn().expect("the harness server is configured for the column"),
threshold,
window_delay: WINDOW_DELAY,
seq: AtomicU64::new(
(std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.subsec_nanos() as u64)
.unwrap_or(0))
^ ((std::process::id() as u64) << 8),
),
dbs: Mutex::new(Vec::new()),
reaped_loids: Mutex::new(Vec::new()),
}
}
/// Open + track the vended site: the shape the column's own
/// rows use (they observe the LO medium, so they need the site
/// alongside the store).
pub(crate) async fn open_tracked(&self) -> Result<(Store, Site), Error> {
let maintenance = admin_dsn(&self.dsn);
let db = fresh_db_name("mem", &self.seq);
create_db(&maintenance, &db).await?;
let dsn = db_dsn(&self.dsn, &db);
let schema = crate::pg_lo_column::fresh_schema_name("mem", &self.seq);
self.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(TrackedDb {
name: db.clone(),
dsn: dsn.clone(),
schemas: vec![schema.clone()],
lo_schemas: vec![schema.clone()],
janitor: Some(DbJanitor::new(&maintenance, &db)),
});
let store =
Store::dual_tier_mem_pglo(&dsn, opts(schema.clone()), self.threshold, WINDOW_DELAY)
.await?;
Ok((store, Site { dsn, schema, db }))
}
/// The dedicated databases this instance minted — names,
/// db-scoped DSNs, and the schemas minted inside — for the
/// structural tests' assertions.
pub(crate) fn tracked_dbs(&self) -> Vec<(String, String, Vec<String>)> {
self.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.iter()
.map(|d| (d.name.clone(), d.dsn.clone(), d.schemas.clone()))
.collect()
}
/// The LOIDs teardown reaped, for the structural tests'
/// tracked-LOID assertions.
pub(crate) fn reaped_loids(&self) -> Vec<u32> {
self.reaped_loids
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
/// The maintenance DSN the factory mints databases through.
pub(crate) fn dsn(&self) -> &str {
&self.dsn
}
}
/// The factories' schema-name policy, in engine-opts form: the
/// constructor's `PgOpts` with exactly the factory's minted
/// schema and the ship defaults for the rest (pool size,
/// `synchronous_commit` — the alkstore factory's shape: only the
/// schema name is factory policy).
fn opts(schema: String) -> PgOpts {
PgOpts {
schema,
..PgOpts::default()
}
}
impl StoreFactory for DualTierMemPgLoFactory {
fn name(&self) -> &'static str {
"dual-tier mem+pg-lo"
}
async fn open(&self) -> Result<Store, Error> {
self.open_tracked().await.map(|(store, _)| store)
}
async fn teardown(&self) -> Result<(), Error> {
let mut dbs = self
.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.drain(..)
.collect::<Vec<_>>();
for tracked in dbs.iter_mut() {
// The LO-isolation corollary's arms: reap the tracked
// LOIDs (via the companion table — the contract
// authority), drop exactly the minted schemas, then
// drop the dedicated database whole (`WITH (FORCE)`
// reaps any connection the abandoned store's pool has
// not drained).
let maintenance = admin_dsn(&self.dsn);
let mut reaped = Vec::new();
if let Ok(client) = admin_connect_for_teardown(&tracked.dsn).await {
for schema in &tracked.lo_schemas {
reaped.extend(reap_lo_schema(&client, schema).await);
}
for schema in &tracked.schemas {
let sql = format!("DROP SCHEMA IF EXISTS \"{schema}\" CASCADE");
let _ = client.batch_execute(&sql).await;
}
}
if !reaped.is_empty() {
self.reaped_loids
.lock()
.unwrap_or_else(|e| e.into_inner())
.extend(reaped);
}
drop_db(&maintenance, &tracked.name).await?;
TrackedDb::disarm(tracked);
}
Ok(())
}
}
async fn admin_connect_for_teardown(dsn: &str) -> Result<tokio_postgres::Client, Error> {
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
.await
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
tokio::spawn(async move {
let _ = connection.await;
});
Ok(client)
}
/// The skip gate every runner checks (the SKIP posture's
/// factory-owner responsibility).
async fn ready() -> bool {
harness_ready().await
}
// --------------------------------------------------------------
// The column's own rows — the durable-schedule sweep family, the
// schedule-relative readings, the dispatch forms, the INV-1
// extension, and the virgin-store row. Every row takes this
// column's concrete factory (it observes the LO medium), is
// ADR-stamped per the convention, and traces its family.
// --------------------------------------------------------------
/// INV-2 on the durable schedule, over both tiers: identical
/// content put twice — through the same put path and across the
/// two paths — is one pool entry, and the re-put's `Pin` protects
/// exactly as a fresh put's does. The kv leg's single-pool-entry
/// is asserted on the facade plus the LO medium's genuine absence
/// (the mem introspection shape — the oracle's kv half); the large
/// leg's is one companion row behind one LO. One-pool-entry reads
/// through durable-schedule outcomes (the INV-5 posture): the pins
/// held stage nothing; once unpinned, sweep 1 stages exactly the
/// one entry and removes nothing, and its executor sweep deletes
/// exactly that staged set (README §The schedule-relative reading
/// for SQL-backed columns).
///
/// Contract stamp: store-api.md §Invariants 2; ADR-003 (whole-value
/// put is CAS first-writer-wins; the routing function's tier
/// placement); ADR-005 (a re-put refcounts the existing entry
/// under the same arbitration lock); ADR-009 §2/§4 (the companion
/// table is the contract authority; delete = lo_unlink + row
/// delete); ADR-008 §2 (the staged re-arbitrated delete); ADR-012
/// §5 (the window hosts on the pg-lo engine in this topology);
/// ADR-012 §6.7 (the executor is the next sweep — the schedule
/// these counts read against).
pub async fn dedup_identical_content_is_one_pool_entry(factory: &DualTierMemPgLoFactory) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
// The kv leg (in-process mem): both refcounts compose under
// the same lock, nothing ever lands on the LO medium.
let content = b"deduplicated content".to_vec();
let canonical = Key::derive(&content, content.len() as u64);
let (key, first) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the fresh put publishes");
let (reput, second) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the same-path re-put publishes");
assert_eq!(key, canonical);
assert_eq!(reput, canonical, "one derivation, one address");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.aborted, None);
assert_eq!(
report.candidates_staged, 0,
"one pool entry, both refcounts composing under the same lock"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the kv-tier entries never land on the LO medium"
);
drop(first);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.candidates_staged, 0,
"the re-put's pin protects exactly as a fresh put's"
);
drop(second);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one pool entry all along — staged this sweep, not executed"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.deleted, report.cancelled_by_arbitration),
(1, 0),
"the executing sweep deleted exactly the staged entry"
);
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
// The large leg: identical content through both put paths —
// one companion row behind one LO, both refcounts composing.
let cross = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
let cross_canonical = Key::derive(&cross, cross.len() as u64);
let (key, first) = store
.put(Some(cross.len() as u64), Cursor::new(cross.as_slice()))
.await
.expect("the known-length put publishes");
let (reput, second) = store
.put(None, Cursor::new(cross.as_slice()))
.await
.expect("the unknown-length re-put publishes");
assert_eq!(key, cross_canonical);
assert_eq!(reput, cross_canonical, "one pool entry across the paths");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.aborted, report.candidates_staged),
(None, 0),
"both refcounts compose"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
1,
"one pool entry across both paths — one companion row"
);
let loids = companion_loids(site.dsn(), site.schema()).await;
assert_eq!(loids.len(), 1, "one LO behind the companion row");
assert!(
loid_pages(site.dsn(), loids[0]).await > 0,
"the LO holds the content's pages (never zero)"
);
drop(second);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.candidates_staged, 0,
"the re-put's pin protects exactly as a fresh put's (other drop order)"
);
drop(first);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one pool entry all along"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "executed exactly the staged set");
assert!(!store.has(&cross_canonical.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the executed delete emptied the companion table"
);
assert_eq!(
loid_pages(site.dsn(), loids[0]).await,
0,
"delete is one tx: the LO's pages went with the row"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
0,
"the window holds nothing after both executed rounds (the pg-lo host)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-3 on the durable schedule: stage hygiene over the LO
/// medium's real staged form — every put-path failure leaves zero
/// residue (no companion rows, no LO pages: the held-put
/// transaction's every failure path converges on ROLLBACK) and
/// every success leaves exactly one committed entry, over both put
/// paths and both injectable failure modes (a mid-stream stream
/// failure and a declared-length mismatch), observed through
/// durable-schedule sweep outcomes: zero staged after every
/// failure, and a success staged exactly once, then executed
/// exactly once.
///
/// Contract stamp: store-api.md §Invariants 3; ADR-010 §1 (the
/// staged-put contract: commit publishes, every failure path
/// discards); ADR-003 (stage-then-commit; the routing places this
/// content on the large tier); ADR-009 §2 (put = one tx — the
/// staged form is the held tx, so a failure's LO pages roll back
/// with it); ADR-012 §6.7 (the schedule-relative reading the
/// success counts follow).
pub async fn stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(
factory: &DualTierMemPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let content = patterned(2 * TUNED_THRESHOLD as usize + 17, 167);
let len = content.len() as u64;
let canonical = Key::derive(&content, len);
// The known path's mid-stream failure — the held tx rolls
// back: no companion row, no LO pages.
let err = store
.put(Some(len), Failing::new(&content, 40))
.await
.expect_err("the failed mid-stream put rejects");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
// The unknown path's buffered failure (before the threshold
// overflow) and its staged failure (the held tx after the
// overflow) — both roll back to zero residue.
let err = store
.put(None, Failing::new(&content, 40))
.await
.expect_err("the failed unknown-length put rejects before staging");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
let err = store
.put(None, Failing::new(&content, TUNED_THRESHOLD as usize + 600))
.await
.expect_err("the failed unknown-length put rejects mid-stage");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
// Declared-length mismatches, over-fed and under-fed — both
// above the threshold, so the staged form began its LO and
// every failure path discards it.
let err = store
.put(Some(TUNED_THRESHOLD + 10), Cursor::new(content.as_slice()))
.await
.expect_err("the over-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let err = store
.put(Some(len + 17), Cursor::new(content.as_slice()))
.await
.expect_err("the under-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.aborted, None);
assert_eq!(
(
report.candidates_staged,
report.deleted,
report.cancelled_by_arbitration
),
(0, 0, 0),
"zero residue after every failure, both put paths, both failure modes"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the companion table holds zero entries after every failure"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
0,
"the window staged nothing behind the failures"
);
// Success, known path: exactly one committed entry — one
// companion row behind one LO — staged once, executed once.
let (key, pin) = store
.put(Some(len), Cursor::new(content.as_slice()))
.await
.expect("the known-length put publishes");
assert_eq!(key, canonical);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
1,
"exactly one committed entry on success (one companion row)"
);
let loid = loid_for_key(site.dsn(), site.schema(), &key.as_bytes())
.await
.expect("the companion row carries its LOID");
assert!(
loid_pages(site.dsn(), loid).await > 0,
"the committed LO holds the content's pages"
);
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed entry staged on success"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.deleted, 1,
"the executing sweep removed exactly that one entry"
);
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
assert_eq!(loid_pages(site.dsn(), loid).await, 0);
// Success, unknown path: the overflow staged into the held tx,
// the commit published — exactly one committed entry on the
// other put path too.
let (key, pin) = store
.put(None, Cursor::new(content.as_slice()))
.await
.expect("the unknown-length put publishes");
assert_eq!(key, canonical);
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed entry staged on the other put path too"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
assert_eq!(
(
lo_entry_rows(site.dsn(), site.schema()).await,
window_rows(site.dsn(), site.schema()).await
),
(0, 0),
"the medium and the window are both back to zero"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, the durable-schedule happy shape over both tiers: the
/// full protection set spanning both tiers (a source-live kv
/// entry, a pinned large-tier entry, a protect-named large-tier
/// entry) plus one dead entry per tier — sweep 1 stages exactly
/// the two dead candidates into the **pg-lo schema's `gc_window`**
/// (the window hosted on the large engine — the one-window-host
/// rule's named topology) and deletes nothing; sweep 2 executes
/// exactly that staged set, and the protected entries stay
/// byte-identical. The medium oracle distinguishes the tiers (the
/// companion table's rows behind LOs vs the kv tier's
/// facade-observed residency) and watches the schedule itself.
///
/// Contract stamp: store-api.md §Invariants 4 (exact counts with
/// correct liveness) and §Invariants 5 (list correctness through
/// sweep outcomes); ADR-005 (three liveness marks; delete windows
/// with arbitrate-at-delete-time); ADR-008 §2 (the staged
/// re-arbitrated delete; the constructor-tunable delay); ADR-009
/// §2/§4 (the companion table; GC-participating deletes); ADR-012
/// §1 (sweeps are never errors); ADR-012 §5 (the one-window-host
/// rule's named topology — the window on the large engine); ADR-012
/// §6.7 (the executor is the next sweep).
pub async fn two_sweep_exact_counts_over_the_full_protection_set(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
let source = SetSource::register(&store);
let protect = Arc::new(SetProtect::default());
store.register_protect_callback(protect.clone());
let source_live = patterned((threshold - 1) as usize, 191);
let (key_live, _) = store
.put(
Some(source_live.len() as u64),
Cursor::new(source_live.as_slice()),
)
.await
.expect("the source-live entry publishes");
source.add(key_live.clone());
let pinned = patterned(2 * threshold as usize + 7, 173);
let (key_pinned, pin_pinned) = store
.put(None, Cursor::new(pinned.as_slice()))
.await
.expect("the pinned entry publishes");
let protected = patterned(2 * threshold as usize + 11, 167);
let (key_protected, _) = store
.put(
Some(protected.len() as u64),
Cursor::new(protected.as_slice()),
)
.await
.expect("the protect-named entry publishes");
protect.set_keys(vec![key_protected.clone()]);
let dead_kv = patterned((threshold - 3) as usize, 149);
let (key_dead_kv, pin) = store
.put(Some(dead_kv.len() as u64), Cursor::new(dead_kv.as_slice()))
.await
.expect("the kv-tier dead entry publishes");
drop(pin);
let dead_large = patterned(2 * threshold as usize + 13, 151);
let (key_dead_large, pin) = store
.put(None, Cursor::new(dead_large.as_slice()))
.await
.expect("the large-tier dead entry publishes");
drop(pin);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
3,
"before the sweep, the LO medium holds exactly the three \
large-tier entries (the kv tier's entries are in-process)"
);
let dead_large_loid = loid_for_key(site.dsn(), site.schema(), &key_dead_large.as_bytes())
.await
.expect("the dead large entry's companion row carries its LOID");
assert!(
loid_pages(site.dsn(), dead_large_loid).await > 0,
"the LO medium holds the dead large entry's pages too"
);
assert!(
store.has(&key_live.as_bytes()).await.unwrap()
&& store.has(&key_dead_kv.as_bytes()).await.unwrap()
&& store.has(&key_dead_large.as_bytes()).await.unwrap(),
"the pool spans both tiers and holds exactly its five entries"
);
let first = store.sweep().await.expect("sweeps are never errors");
assert_eq!(first.aborted, None);
assert_eq!(
(
first.candidates_staged,
first.deleted,
first.cancelled_by_arbitration
),
(2, 0, 0),
"sweep 1 staged exactly the two dead entries and deleted \
nothing — staging is not executing"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
2,
"the staged set waits in the durable window — hosted on the \
pg-lo schema (the one-window-host rule's large-engine arm)"
);
for (key, content) in [
(&key_live, &source_live),
(&key_pinned, &pinned),
(&key_protected, &protected),
] {
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the protected entry is present");
assert_eq!(
blob.read_at(0, len).await.expect("the window reads"),
content.as_slice(),
"the protected entry is byte-identical after the staging sweep"
);
}
assert!(
store.has(&key_live.as_bytes()).await.unwrap()
&& store.has(&key_pinned.as_bytes()).await.unwrap()
&& store.has(&key_protected.as_bytes()).await.unwrap(),
"the staged sweep deleted nothing: every entry intact"
);
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(second.aborted, None);
assert_eq!(
second.candidates_staged, 0,
"nothing newly dead since the staging sweep"
);
assert_eq!(
second.deleted + second.cancelled_by_arbitration,
2,
"sweep 2 executed exactly the staged set"
);
assert_eq!(
(second.deleted, second.cancelled_by_arbitration),
(2, 0),
"with correct liveness, every executed candidate deleted"
);
assert!(!store.has(&key_dead_kv.as_bytes()).await.unwrap());
assert!(!store.has(&key_dead_large.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
2,
"the executed deletes landed on both media exactly (the dead \
large entry's companion row is gone)"
);
assert_eq!(
loid_pages(site.dsn(), dead_large_loid).await,
0,
"the dead large entry's LO pages are gone with it"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
0,
"the executed rows left the durable window"
);
drop(pin_pinned);
let third = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(third.candidates_staged, third.deleted),
(1, 0),
"with the pin dropped, the formerly pinned entry stages"
);
let fourth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
fourth.deleted, 1,
"and the next sweep executes it — exactly one deletion"
);
assert!(!store.has(&key_pinned.as_bytes()).await.unwrap());
assert!(store.has(&key_live.as_bytes()).await.unwrap());
assert!(store.has(&key_protected.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
1,
"exactly the protect-named entry's companion row remains"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, the race, in its durable-schedule form: a put landing
/// *during the window* survives the executing sweep — the re-put's
/// pin is minted (pin-before-publish) between the staging sweep
/// and the executing sweep, so the execution-time re-arbitration
/// cancels the pinned candidate while the unpinned twin deletes;
/// once the pin drops, the twin re-stages and executes exactly
/// once. Both put paths race in-body over both tiers (a kv twin
/// and a large twin), across the window delay the pg-lo host
/// carries.
///
/// Contract stamp: store-api.md §Invariants 4; ADR-005 (pin-before-
/// publish ordering; the delete window closes the sweep-vs-put race);
/// ADR-008 §2 (the staged re-arbitrated delete — the race holds
/// across the window delay); ADR-009 §4 (GC-participating deletes
/// over the companion table); ADR-012 §5 (the window on the large
/// engine in this topology); ADR-012 §6.7 (execution-time
/// re-arbitration is the executor sweep's posture).
pub async fn a_put_landing_during_the_window_survives_the_executing_sweep(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let rounds: [(String, Vec<u8>, bool, bool, bool); 2] = [
(
"the kv twin (known-path re-put)".to_owned(),
patterned((threshold - 1) as usize, 191),
true,
true,
false,
),
(
"the large twin (unknown-path re-put)".to_owned(),
patterned(2 * threshold as usize + 5, 173),
false,
false,
true,
),
];
for (index, (tag, content, original_known, reput_known, lands_large)) in
rounds.iter().enumerate()
{
let len = content.len() as u64;
let twin = Key::derive(content, len);
let witness = format!("the dead witness of round {index}").into_bytes();
let witness_key = Key::derive(&witness, witness.len() as u64);
let original = if *original_known { Some(len) } else { None };
let (_, pin) = store
.put(original, Cursor::new(content.as_slice()))
.await
.expect("the twin publishes");
drop(pin);
let (_, pin) = store
.put(None, Cursor::new(witness.as_slice()))
.await
.expect("the witness publishes");
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(
staged.aborted,
staged.candidates_staged,
staged.deleted,
staged.cancelled_by_arbitration
),
(None, 2, 0, 0),
"{tag}: the staging sweep stages exactly the two dead \
twins and deletes nothing"
);
// The put lands during the window (the re-put dedups into
// the existing twin and refcounts its pin): protected
// across the delay, pin-before-publish.
let reput = if *reput_known { Some(len) } else { None };
let (_, pin) = store
.put(reput, Cursor::new(content.as_slice()))
.await
.expect("the re-put publishes");
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(executed.aborted, None, "{tag}");
assert_eq!(
executed.candidates_staged, 0,
"{tag}: nothing newly dead since the staging sweep"
);
assert_eq!(
executed.deleted + executed.cancelled_by_arbitration,
2,
"{tag}: the executing sweep ran exactly the staged set"
);
assert_eq!(executed.deleted, 1, "{tag}: the unpinned witness deleted");
assert_eq!(
executed.cancelled_by_arbitration, 1,
"{tag}: the pinned twin's candidate cancelled at execution time"
);
assert!(
!store.has(&witness_key.as_bytes()).await.unwrap(),
"{tag}: the unpinned twin is gone"
);
let (got_len, blob) = store
.get(&twin.as_bytes())
.await
.expect("get succeeds")
.expect("{tag}: the put landing during the window survived");
assert_eq!(got_len, len, "{tag}");
assert_eq!(
blob.read_at(0, len).await.expect("the window reads"),
content.as_slice(),
"{tag}: the surviving twin is byte-identical"
);
drop(pin);
let restaged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(restaged.candidates_staged, restaged.deleted),
(1, 0),
"{tag}: the pin dropped — the twin re-stages"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"{tag}: the re-staged candidate executes exactly once"
);
assert!(!store.has(&twin.as_bytes()).await.unwrap());
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
0,
"{tag}: the window holds nothing after the executed round \
(the pg-lo host)"
);
if *lands_large {
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"{tag}: the executed LO deletes reaped the medium exactly"
);
}
}
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, abort shape: the sweep is single-flight on the durable
/// form too — the in-process sweep lease (ADR-013 §2's single-node
/// analog). The contended second sweeper aborts with
/// `SweeperLock` in a completed `Ok(SweepReport)` — nothing
/// executed, nothing staged, nothing deleted (abort before
/// staging: form-independent) — while the lease holder's sweep
/// completes by **staging only**: its staged set waits in the
/// pg-lo schema's `gc_window` for the next sweep to execute (the
/// durable form's schedule-relative replacement for the reference
/// row's inline holder-completion leg).
///
/// Contract stamp: store-api.md §Invariants 4; ADR-012 §1 (abort is
/// report data, never an error); ADR-013 §2 (the named exclusive
/// sweep lease — one sweeper per pool, holdership ends at release);
/// ADR-012 §5 (the window hosts on the large engine in this
/// topology); ADR-012 §6.7 (the executor is the next sweep — the
/// holder stages, it does not execute).
pub async fn the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
store.register_protect_callback(Arc::new(LatchProtect {
entered: entered_tx,
release: parking_lot::Mutex::new(Some(release_rx)),
}));
let content = patterned(2 * threshold as usize + 9, 167);
let (key, pin) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the dead witness publishes");
drop(pin);
// The lease holder sweeps on its own task (its protect consult
// blocks that task's thread mid-callback); this task contends
// and then releases the holder.
let store = Arc::new(store);
let holder = tokio::spawn({
let store = Arc::clone(&store);
async move { store.sweep().await }
});
while entered_rx
.recv_timeout(std::time::Duration::from_millis(50))
.is_err()
{
tokio::task::yield_now().await;
}
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
second.aborted,
Some(alkblobs::GcAbortCause::SweeperLock),
"the contended sweeper aborts as report data"
);
assert_eq!(
(
second.candidates_staged,
second.deleted,
second.cancelled_by_arbitration
),
(0, 0, 0),
"the contended sweep executes, stages, and deletes nothing"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"the contended sweep deleted nothing"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
0,
"the aborted sweep staged nothing into the window"
);
release_tx.send(()).ok();
let report = holder
.await
.expect("the holder task joined")
.expect("sweeps are never errors");
assert_eq!(report.aborted, None, "the lease holder completes");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"the holder stages exactly the dead set — it does not execute \
(the durable form's schedule)"
);
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
1,
"the holder's staged set waits in the durable window"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"the holder staged, did not execute: the entry is intact"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"the next sweep is the executor of the staged set"
);
assert!(!store.has(&key.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the executed delete emptied the companion table"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-5 on the durable schedule: `list()` correctness is
/// observable only through GC — the enumeration is complete and
/// well-formed in both directions, proven by exact
/// durable-schedule outcomes: the staging sweep stages exactly the
/// dead set (neither more nor fewer candidates), the executing
/// sweep deletes exactly that staged set, and a liveness flip
/// re-stages exactly the newly dead entry. Never a direct list
/// comparison.
///
/// Contract stamp: store-api.md §Invariants 5; ADR-003 (`list()` is
/// complete and well-formed by contract — load-bearing for GC);
/// ADR-005 (the sweep's live-set diff rides `list()`); ADR-012 §5
/// (the window hosts on the large engine in this topology); ADR-012
/// §6.7 (the executor is the next sweep — the schedule these
/// counts read against).
pub async fn list_correctness_is_observable_through_durable_schedule_outcomes(
factory: &DualTierMemPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
let source = SetSource::register(&store);
let mut keys = Vec::new();
for (index, content) in [
b"kept live by the source, entry one".to_vec(),
b"kept live by the source, entry two".to_vec(),
b"dead at the first staging, entry three".to_vec(),
b"dead at the first staging, entry four".to_vec(),
b"dead at the first staging, entry five".to_vec(),
]
.into_iter()
.enumerate()
{
let declared = if index % 2 == 0 {
Some(content.len() as u64)
} else {
None
};
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("the entry publishes");
drop(pin);
keys.push(key);
}
source.add(keys[0].clone());
source.add(keys[1].clone());
let first = store.sweep().await.expect("sweeps are never errors");
assert_eq!(first.aborted, None);
assert_eq!(
(first.candidates_staged, first.deleted),
(3, 0),
"the enumeration staged exactly the three dead entries — \
complete and well-formed in both directions"
);
for key in &keys {
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"staging deleted nothing: every enumerated entry is intact"
);
}
assert_eq!(
window_rows(site.dsn(), site.schema()).await,
3,
"the staged schedule holds the three candidates"
);
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(second.candidates_staged, second.deleted),
(0, 3),
"the executing sweep removed exactly the staged set"
);
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
assert!(store.has(&keys[1].as_bytes()).await.unwrap());
for key in &keys[2..] {
assert!(!store.has(&key.as_bytes()).await.unwrap());
}
source.remove(&keys[1]);
let third = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(third.candidates_staged, third.deleted),
(1, 0),
"the newly dead entry is staged exactly"
);
let fourth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(fourth.deleted, 1, "and executed exactly by the next sweep");
assert!(!store.has(&keys[1].as_bytes()).await.unwrap());
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
source.remove(&keys[0]);
let fifth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(fifth.candidates_staged, 1, "the last entry stages exactly");
let sixth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(sixth.deleted, 1, "and executes exactly");
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **routing is a pure function of length** — at,
/// below, and over the declared threshold, the same content lands
/// on exactly the same tier through both put paths (same placement
/// every time; nothing moves, nothing duplicates), identical
/// content through both paths is one pool entry, and every read is
/// byte-identical from wherever that tier holds it.
///
/// Tier placement is observed through the LO medium (the oracle's
/// LO half): large entries hold a companion row behind an LO; kv
/// entries hold nothing on the medium — the facade's served reads
/// and `has()` are the mem kv half's introspection shape (the mem
/// column's form; the kv tier is in-process, facade-invisible by
/// design).
///
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
/// ADR-003 §Dispatch policy (routing is a pure function of length;
/// no migration; CAS first-writer-wins makes re-puts one entry);
/// ADR-005 (re-put refcounts); ADR-009 §2 (the companion table is
/// the contract authority); ADR-012 §5 (this topology's named
/// shape); ADR-012 §6.3 (the dual-tier dispatch-coverage shape);
/// ADR-012 §6.7 (the schedule-relative cleanup the dedup legs read
/// against).
pub async fn routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
for content in [
patterned((threshold - 1) as usize, 191),
patterned(threshold as usize, 173),
patterned((threshold + 1) as usize, 167),
patterned(4 * threshold as usize + 9, 151),
] {
let len = content.len() as u64;
let canonical = Key::derive(&content, len);
let expected_companions = if len > threshold { 1 } else { 0 };
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)]
{
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key, canonical,
"the canonical derivation addresses the put on {path_tag}"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
expected_companions,
"length {len} on {path_tag}: the placement is exactly the \
routing function's answer and never moves or duplicates \
(the LO medium holds {expected_companions} companion rows)"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"length {len} on {path_tag}: the pool holds the entry (the \
mem kv half's introspection shape: served reads, no \
medium of its own)"
);
let (got_len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the entry is present");
assert_eq!(
got_len, len,
"the read serves the entry from its placed tier"
);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the entry is byte-identical from its placed tier across paths"
);
drop(pin);
}
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
staged.candidates_staged, 1,
"identical content through both put paths is one pool entry"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"the executing sweep removed exactly the one entry (durable \
schedule: staged this sweep, executed by the next)"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the LO medium is back to zero"
);
}
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **threshold boundary lengths route
/// deterministically** — at/below/above the factory's declared
/// threshold, on both put paths, the placement is the routing
/// function's answer every time (the LO medium's companion rows
/// against the mem kv half's facade shape); and the
/// default-threshold boundary re-routes with the tuning (the
/// constructor-tuned threshold deterministically places the
/// default boundary's length on the tier its own threshold names).
/// Run against two factories by the runner — the column's tuned
/// threshold and the shipped default. The durable cleanup: the
/// staged sweep stages exactly the six entries' dead set and the
/// next sweep executes it.
///
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
/// ADR-003 §Dispatch policy (default 128 KiB at-or-below routes kv;
/// constructor-tunable without API change); ADR-009 §2 (the
/// companion table); ADR-012 §5 (this topology's named shape);
/// ADR-012 §6.3 (the dual-tier dispatch-coverage shape); ADR-012
/// §6.7 (schedule-relative cleanup).
pub async fn threshold_boundary_lengths_route_deterministically(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let boundary = [
(threshold - 1, 191u32, false),
(threshold, 173, false),
(threshold, 167, false),
(threshold + 1, 151, true),
(threshold + 1, 139, true),
];
let mut expected_companions = 0u64;
for (len, modulus, lands_large) in boundary {
let content = patterned(len as usize, modulus);
for declared in [Some(content.len() as u64), None] {
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key,
Key::derive(&content, len),
"the canonical derivation addresses the put"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await as u64,
expected_companions + if lands_large { 1 } else { 0 },
"boundary length {len}: at/below the threshold lands kv \
(nothing on the LO medium), over it lands large (one \
companion row behind an LO), both put paths"
);
drop(pin);
}
expected_companions += if lands_large { 1 } else { 0 };
}
// The default boundary re-routes with the constructor's
// tuning: the shipped default's length belongs to the tier
// this configuration's own threshold puts it on.
let at_default = patterned(DEFAULT_DISPATCH_THRESHOLD as usize, 199);
let default_lands_large = DEFAULT_DISPATCH_THRESHOLD > threshold;
let (key, pin) = store
.put(
Some(DEFAULT_DISPATCH_THRESHOLD),
Cursor::new(at_default.as_slice()),
)
.await
.expect("the default-boundary put publishes");
assert_eq!(
key,
Key::derive(&at_default, DEFAULT_DISPATCH_THRESHOLD),
"the canonical derivation addresses the put"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await as u64,
expected_companions + if default_lands_large { 1 } else { 0 },
"the default boundary's length routes to the tier this \
configuration's tuning names (the LO medium's answer)"
);
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
staged.candidates_staged, 6,
"each leg's content was exactly one pool entry"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 6,
"the executing sweep removed every entry, tier notwithstanding"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the LO medium is back to zero"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **a kv-tier miss falls through to the large tier
/// on every key-addressed op** — content the routing function
/// placed on the large tier (one companion row behind an LO; the
/// LO medium and `pg_largeobject` hold the pages while the kv tier
/// misses — the kv half is in-process mem, facade-served) serves
/// `get`, `stat`, `has`, and `read_range` whole, exact, and
/// digest-correct; the mirror-position leg proves the kv tier is
/// the *first* consult (sub-threshold content, nothing on the LO
/// medium, reads still served); and both-tier misses resolve per
/// the error model's per-op shapes (reached here by the durable
/// schedule's staged-then-executed deletion of the subject entry).
///
/// Contract stamp: store-api.md §Read path (fall-through consult
/// order); ADR-012 §6.1 (every key-addressed op consults kv first,
/// then large; the general rule); ADR-012 §6.5 ruling 5 (both-tier
/// miss shapes: `None` on get/stat, `Err(Missing)` on read_range);
/// ADR-006 (the slice digest); ADR-009 §2/§3 (the companion table
/// and the `lo_get` windows serving the fall-through);
/// ADR-012 §6.3 (the dispatch-coverage shape).
pub async fn kv_tier_misses_fall_through_to_the_large_tier_per_op(
factory: &DualTierMemPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
// Over-threshold content — the kv tier misses, the large tier
// serves every op. Both put paths hold the shape.
for (content, declared) in [
(
patterned(3 * threshold as usize + 41, 181),
Some(3 * threshold + 41),
),
(patterned(3 * threshold as usize + 43, 197), None),
] {
let len = content.len() as u64;
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
1,
"the content lives on exactly the large tier — the kv tier \
misses"
);
let loid = loid_for_key(site.dsn(), site.schema(), &key.as_bytes())
.await
.expect("the companion row carries its LOID");
assert!(
loid_pages(site.dsn(), loid).await > 0,
"the LO medium holds the entry's pages"
);
let (got_len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the large tier served the kv miss");
assert_eq!(got_len, len);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the fall-through read is byte-identical"
);
assert_eq!(
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
Some(alkblobs::EntryMeta {
len,
algorithm: key.algorithm(),
}),
"stat falls through and carries the length plus algorithm"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"has falls through"
);
let mid = len / 2;
let (slice, digest) = store
.read_range(&key.as_bytes(), mid, 17)
.await
.expect("read_range falls through");
assert_eq!(
slice,
content[(mid as usize)..(mid as usize + 17)],
"the range slice is exact"
);
assert_eq!(
digest,
slice_digest(&slice),
"the slice digest is SHA-256 over exactly the returned bytes"
);
// Sweep the entry away (the pin dropped): the miss shapes
// now resolve per op — the large tier misses too. The
// durable schedule: staged this sweep, executed by the next.
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!((staged.candidates_staged, staged.deleted), (1, 0));
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"exactly the one entry died across the two sweeps"
);
assert!(
store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"a both-tier miss on get is Ok(None)"
);
assert_eq!(
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
None,
"a both-tier miss on stat is Ok(None)"
);
assert!(
!store.has(&key.as_bytes()).await.unwrap(),
"a both-tier miss on has is false"
);
assert_eq!(
store
.read_range(&key.as_bytes(), 0, 4)
.await
.expect_err("the both-tier miss errs"),
Error::Missing,
"a both-tier miss on read_range is Err(Missing)"
);
}
// The mirror-position leg: sub-threshold content — the kv tier
// is the first consult and serves before the large tier is
// reached (nothing ever lands on the LO medium).
let small = patterned((threshold - 1) as usize, 149);
let (key, pin) = store
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
.await
.expect("the sub-threshold put publishes");
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"the kv tier holds the sub-threshold entry — the large tier \
was never consulted on the write, never reached on the read"
);
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the kv tier served");
assert_eq!(len, small.len() as u64);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
small
);
drop(pin);
// Both-tier miss shapes for a key that never existed at all,
// per op, over the fresh both-tier store.
let absent = b"never put here".to_vec();
let absent = Key::derive(&absent, absent.len() as u64);
assert!(
store
.get(&absent.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"never-put: get is Ok(None)"
);
assert_eq!(
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
None,
"never-put: stat is Ok(None)"
);
assert!(
!store.has(&absent.as_bytes()).await.unwrap(),
"never-put: has is false"
);
assert_eq!(
store
.read_range(&absent.as_bytes(), 0, 4)
.await
.expect_err("never-put read_range errs"),
Error::Missing,
"never-put: read_range is Err(Missing)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-1's LO-medium extension: **the `lo_get` windows read
/// across window boundaries byte-exact, and a body short of its
/// claimed length fails cleanly** — the round-trip family's ranged
/// reads extended to the LO medium. Entries on the large tier
/// (both put paths) read whole through the multi-window assembly,
/// straddle the shipped window edge (one read crossing two
/// `lo_get` statements), read ranged slices with the ADR-006
/// digest over exactly the returned bytes, clamp at EOF (tail,
/// at-EOF empty, a `u64::MAX` offset clamping in `u64` arithmetic,
/// zero-length, over-read), and — the beyond-length clean failure
/// — a companion row overclaiming its size (a corrupted half-state
/// healthy stores never hold, planted through the oracle)
/// surfaces as the unexpected-EOF-shaped `Error::Io`, never a
/// partial read or a panic.
///
/// Contract stamp: store-api.md §Invariants 1 (the round-trip
/// family's ranged-read legs) and §Read path (the streaming
/// large-tier windows, pread-style clamping); ADR-009 §2/§3 (the
/// companion table is the authority — the cursor closes over the
/// row; descriptorless `lo_get` windows, the shipped read shape
/// and its unexpected-EOF failure); ADR-006 (the slice digest over
/// exactly the returned bytes); ADR-010 §1 (the large-tier cursor
/// body family).
pub async fn lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(
factory: &DualTierMemPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let content = patterned(2 * LO_WINDOW + 100, 167);
let len = content.len() as u64;
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)] {
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key,
Key::derive(&content, len),
"the canonical derivation addresses the put on {path_tag}"
);
let (_, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the entry is present");
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the multi-window assembly reads whole and byte-exact on {path_tag}"
);
assert_eq!(
blob.read_at(LO_WINDOW as u64 - 100, 300)
.await
.expect("the straddling read succeeds"),
content[(LO_WINDOW - 100)..(LO_WINDOW + 200)],
"a read straddling the window edge crosses two lo_get \
statements byte-exact on {path_tag}"
);
let (slice, digest) = blob
.read_range(LO_WINDOW as u64, 7)
.await
.expect("the ranged slice reads");
assert_eq!(
slice,
content[LO_WINDOW..LO_WINDOW + 7],
"the ranged slice is exact on {path_tag}"
);
assert_eq!(
digest,
slice_digest(&slice),
"the slice digest is SHA-256 over exactly the returned bytes"
);
assert_eq!(
blob.read_at(len - 5, 100).await.expect("the tail reads"),
content[(len as usize - 5)..],
"the tail read clamps at the entry's end"
);
assert_eq!(
blob.read_at(len, 8).await.expect("the EOF read succeeds"),
Vec::<u8>::new(),
"a read at EOF is empty, never an error"
);
assert_eq!(
blob.read_at(u64::MAX, 5).await.expect("the offset reads"),
Vec::<u8>::new(),
"an overflowing u64 offset clamps in u64 arithmetic — the \
C7 int4 trap not inherited"
);
assert_eq!(
blob.read_at(0, 0).await.expect("the empty read succeeds"),
Vec::<u8>::new(),
"a zero-length read is empty"
);
assert_eq!(
blob.read_at(0, len + 64)
.await
.expect("the over-read reads"),
content,
"an over-read clamps to the whole entry"
);
drop(pin);
}
// The beyond-length clean failure: a companion row claiming
// more bytes than its LO holds (the corrupted half-state
// healthy stores never hold) planted through the oracle — the
// read window trusts the row (the contract authority,
// metadata-only) and fails cleanly when the LO comes up short.
let planted = b"the planted short body".to_vec();
let planted_key = Key::derive(&planted, planted.len() as u64);
let claimed = (planted.len() + 4096) as i64;
let loid = {
let client = admin_connect_plant(site.dsn()).await;
client
.batch_execute("BEGIN")
.await
.expect("the plant opens");
let loid: u32 = client
.query_one("SELECT lo_create(0) AS loid", &[])
.await
.expect("lo_create mints the planted LO")
.get(0);
let fd: i32 = client
.query_one("SELECT lo_open($1, $2)", &[&loid, &INV_WRITE])
.await
.expect("the planted LO opens for write")
.get(0);
client
.query_one("SELECT lowrite($1, $2)", &[&fd, &planted])
.await
.expect("the planted partial body writes");
let committed_at = SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as i64)
.unwrap_or(0);
client
.execute(
&format!(
"INSERT INTO \"{}\".lo_entries (key, loid, size, committed_at) \
VALUES ($1, $2, $3, $4)",
site.schema()
),
&[&planted_key.as_bytes(), &loid, &claimed, &committed_at],
)
.await
.expect("the overclaiming companion row plants");
client
.batch_execute("COMMIT")
.await
.expect("the plant commits");
drop(client);
loid
};
let (got_len, planted_blob) = store
.get(&planted_key.as_bytes())
.await
.expect("get succeeds")
.expect(
"the planted entry is present (the companion row is the \
authority, metadata-only)",
);
assert_eq!(
got_len, claimed as u64,
"the cursor's length is the companion row's overclaim — never \
an LO content probe"
);
let err = planted_blob
.read_at(0, got_len)
.await
.expect_err("the short body fails cleanly beyond its length");
assert!(
matches!(&err, Error::Io(m) if m.contains("short of")),
"the beyond-length read surfaces the unexpected-EOF clean \
failure, never a partial read: {err}"
);
// The durable cleanup: the round-trip legs' identical content
// was exactly one pool entry (CAS across the paths), so that
// entry and the planted one stage exactly, and the executing
// sweep deletes exactly that staged set — the planted LO's
// pages too.
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(staged.aborted, staged.candidates_staged),
(None, 2),
"the round-trip entry (one across both paths) and the planted \
one stage exactly"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(executed.deleted, 2, "the staged set executed exactly");
assert_eq!(
(
lo_entry_rows(site.dsn(), site.schema()).await,
window_rows(site.dsn(), site.schema()).await
),
(0, 0),
"the medium and the window are back to zero"
);
assert_eq!(
loid_pages(site.dsn(), loid).await,
0,
"the executing sweep's lo_unlink reaped the planted LO"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// The plant's raw-LO flag — postgres Large Objects' `INV_WRITE`
/// mode bits, the same constant the engine's orphan test plants
/// with.
const INV_WRITE: i32 = 0x0002_0000;
async fn admin_connect_plant(dsn: &str) -> tokio_postgres::Client {
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
.await
.expect("the plant connection opens");
tokio::spawn(async move {
let _ = connection.await;
});
client
}
/// Virgin-store row: **reads are no-ops on a fresh schema** — an
/// empty `list`, absent reads per op (`None` on get/stat, `false`
/// on has, `Err(Missing)` on read_range), and nothing on the fresh
/// medium itself (zero companion rows, and the window state per
/// the host rule: `gc_window` lives in the pg-lo schema — the
/// one-window-host rule's large-host arm — and holds zero rows;
/// `gc_pins` is never created by pg-lo). The constructor always
/// ensures the pg-lo schema, this row being the facade form.
///
/// Contract stamp: store-api.md §Error model (virgin-store
/// semantics: reads on a fresh store see absent/empty, never
/// errors); ADR-012 §6.5 ruling 5 (the per-op miss shapes); ADR-009
/// §2 (the companion-table bootstrap at the constructor); ADR-012
/// §5 (the one-window-host rule's named topology — the window on
/// the pg-lo host, and pg-lo never carries fleet pins).
pub async fn virgin_store_reads_are_no_ops_on_a_fresh_schema(factory: &DualTierMemPgLoFactory) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
assert_eq!(
store.mode(),
StoreMode::DualTier,
"the column's factory is the dispatch-coverage shape"
);
let dsn = site.dsn();
let schema = site.schema();
let absent = b"nothing was ever put".to_vec();
let absent = Key::derive(&absent, absent.len() as u64);
assert_eq!(
store.list().await.expect("list succeeds"),
Vec::<Key>::new(),
"the virgin pool lists empty"
);
assert!(
store
.get(&absent.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"absent get is Ok(None)"
);
assert_eq!(
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
None,
"absent stat is Ok(None)"
);
assert!(
!store.has(&absent.as_bytes()).await.unwrap(),
"absent has is false"
);
assert_eq!(
store
.read_range(&absent.as_bytes(), 0, 4)
.await
.expect_err("absent read_range errs"),
Error::Missing,
"absent read_range is Err(Missing)"
);
assert_eq!(
lo_entry_rows(dsn, schema).await,
0,
"the fresh LO medium holds nothing (zero companion rows)"
);
assert_eq!(
window_rows(dsn, schema).await,
0,
"the fresh window holds no staged candidates"
);
assert!(
crate::pg_lo_column::table_exists(dsn, schema, "gc_window").await,
"the window lives on the pg-lo host — the one-window-host \
rule's large-host arm (ADR-012 §5)"
);
assert!(
!crate::pg_lo_column::table_exists(dsn, schema, "gc_pins").await,
"gc_pins is never created by pg-lo (fleet pins live on the kv \
host, ADR-012 §5)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
// --------------------------------------------------------------
// The runners — the column's tests: the form-independent reference
// rows mirrored verbatim against the dual-tier factory (INV-1, INV-6,
// and the two no-op abort shapes), the durable-schedule rows above,
// and the factory contract's structural shapes. Every runner gates
// behind the reachability probe — the SKIP posture: server-less
// runs skip cleanly, a skip is not a failure; acceptance is the
// green run against the harness server.
// --------------------------------------------------------------
#[tokio::test]
async fn mem_pglo_inv1_round_trip_puts_and_gets_are_byte_identical() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
rows::round_trip_puts_and_gets_are_byte_identical(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv2_dedup_identical_content_is_one_pool_entry() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
dedup_identical_content_is_one_pool_entry(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv3_stage_hygiene_zero_residue_one_entry_on_success() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv4_two_sweep_exact_counts_over_the_full_protection_set() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
two_sweep_exact_counts_over_the_full_protection_set(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv4_a_put_landing_during_the_window_survives_the_executing_sweep() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
a_put_landing_during_the_window_survives_the_executing_sweep(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv4_sweep_without_liveness_sources_aborts_and_deletes_nothing() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
rows::sweep_without_liveness_sources_aborts_and_deletes_nothing(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv4_sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
rows::sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing(&factory).await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn mem_pglo_inv4_the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only()
{
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv5_list_correctness_through_durable_schedule_outcomes() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
list_correctness_is_observable_through_durable_schedule_outcomes(&factory).await;
}
#[tokio::test]
async fn mem_pglo_inv6_stat_reports_len_and_algorithm_without_fetching_content() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
rows::stat_reports_len_and_algorithm_without_fetching_content(&factory).await;
}
#[tokio::test]
async fn mem_pglo_lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(&factory).await;
}
#[tokio::test]
async fn mem_pglo_virgin_store_reads_are_no_ops_on_a_fresh_schema() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
virgin_store_reads_are_no_ops_on_a_fresh_schema(&factory).await;
}
#[tokio::test]
async fn dispatch_mem_pglo_routing_is_a_pure_function_of_length_across_put_paths_and_re_puts() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(&factory).await;
}
#[tokio::test]
async fn dispatch_mem_pglo_threshold_boundary_lengths_route_deterministically() {
if !ready().await {
return;
}
let tuned = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
threshold_boundary_lengths_route_deterministically(&tuned).await;
let default = DualTierMemPgLoFactory::new(DEFAULT_DISPATCH_THRESHOLD);
threshold_boundary_lengths_route_deterministically(&default).await;
}
#[tokio::test]
async fn dispatch_mem_pglo_kv_tier_misses_fall_through_to_the_large_tier_per_op() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
kv_tier_misses_fall_through_to_the_large_tier_per_op(&factory).await;
}
// --------------------------------------------------------------
// The factory contract's structural shapes (the postgres column's
// pattern, carried to the LO medium with the LO-isolation
// corollary's arms): fresh isolated opens (a fresh dedicated
// database with a fresh schema per open, nothing shared — two
// opens never observe one another) and teardown against a
// panicking row (the schemas really dropped, the LOs really
// unlinked — zero orphan residue probed inside the dedicated
// database before it goes, and the shared harness database
// untouched by construction). Not conformance rows — no backlog
// entry — the factory contract's own wiring-level assertions.
// --------------------------------------------------------------
/// Every `open()` mints a fresh, isolated database + schema pair:
/// no shared state between islands, observable on the facade *and*
/// on the fresh medium (each open's database and schema are
/// private and start empty).
#[tokio::test]
async fn mem_pglo_factory_open_vends_fresh_isolated_stores() {
if !ready().await {
return;
}
let factory = DualTierMemPgLoFactory::new(TUNED_THRESHOLD);
let (a, site_a) = factory.open_tracked().await.expect("open vends a store");
let (b, site_b) = factory.open_tracked().await.expect("open vends another");
assert_ne!(
site_a.schema(),
site_b.schema(),
"two opens never share a schema"
);
assert_ne!(site_a.db, site_b.db, "two opens never share a database");
let content_a = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
let content_b = patterned(TUNED_THRESHOLD as usize - 5, 191);
let (key_a, _) = a
.put(
Some(content_a.len() as u64),
Cursor::new(content_a.as_slice()),
)
.await
.expect("island a's put publishes");
let (key_b, _) = b
.put(None, Cursor::new(content_b.as_slice()))
.await
.expect("island b's put publishes");
assert!(a.has(&key_a.as_bytes()).await.unwrap());
assert!(
!a.has(&key_b.as_bytes()).await.unwrap(),
"island a sees nothing of island b"
);
assert!(b.has(&key_b.as_bytes()).await.unwrap());
assert!(
!b.has(&key_a.as_bytes()).await.unwrap(),
"island b sees nothing of island a"
);
assert_eq!(
lo_entry_rows(site_a.dsn(), site_a.schema()).await,
1,
"island a's medium holds exactly its own entry (a's content routes large)"
);
assert_eq!(
lo_entry_rows(site_b.dsn(), site_b.schema()).await,
0,
"island b's medium holds exactly its own entry (b's content routes kv)"
);
assert!(
crate::pg_lo_column::zero_lo_residue(site_a.dsn(), site_a.schema()).await,
"island a's committed content has zero orphan LO residue (the \
LO catalog and the companion table agree exactly)"
);
let (c, site_c) = factory.open_tracked().await.expect("a third vends fresh");
assert_eq!(
lo_entry_rows(site_c.dsn(), site_c.schema()).await,
0,
"a third open's medium is virgin"
);
assert!(
!c.has(&key_a.as_bytes()).await.unwrap(),
"a third open shares nothing with the first"
);
assert!(
!c.has(&key_b.as_bytes()).await.unwrap(),
"a third open shares nothing with the second"
);
let dbs = factory.tracked_dbs();
assert_eq!(dbs.len(), 3, "one minted database per open");
assert_eq!(
dbs.iter()
.map(|(name, _, _)| name.clone())
.collect::<std::collections::BTreeSet<_>>()
.len(),
3,
"no two opens share a database"
);
// The engine-owned schemas really exist inside their dedicated
// databases (the registry's minted list is not bookkeeping
// fiction).
for (_, dsn, schemas) in &dbs {
for schema in schemas {
assert!(
crate::pg_lo_column::schema_exists(dsn, schema).await,
"the minted schema exists inside its dedicated database: {schema}"
);
}
}
factory
.teardown()
.await
.expect("teardown reaps the LOs, drops the schemas, drops the databases");
factory.teardown().await.expect("teardown repeats cleanly");
let maintenance = admin_dsn(factory.dsn());
for (name, _, _) in &dbs {
assert!(
!db_exists(&maintenance, name).await,
"teardown dropped the dedicated database: {name} (schemas \
and all — the whole LO state goes with it)"
);
}
}
/// A failing assertion panics the row before teardown runs — and
/// teardown must make sense over the abandoned instance anyway: it
/// is idempotent against the panicked row's path, the LOs are
/// really unlinked (the tracked-LOID reap arm — the recorded
/// LOIDs match the pre-captured set) and the minted schemas really
/// dropped (gone with the dedicated database, **zero orphan
/// `pg_largeobject` residue** probed inside before it goes), and
/// the next open mints a fresh database on a fresh (empty) medium
/// (README §The factory contract). The shared harness database is
/// untouched by construction — no pg-lo pool ever opens against
/// it.
#[tokio::test]
async fn mem_pglo_factory_teardown_is_safe_against_a_panicking_row() {
if !ready().await {
return;
}
let content = patterned(2 * TUNED_THRESHOLD as usize + 7, 173);
let abandoned = Key::derive(&content, content.len() as u64);
let factory = Arc::new(DualTierMemPgLoFactory::new(TUNED_THRESHOLD));
let task = {
let factory = Arc::clone(&factory);
tokio::spawn(async move {
let (store, _) = factory.open_tracked().await.expect("open vends the store");
let (_, pin) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the put publishes");
drop(pin);
panic!("a failing assertion panics the row (the factory contract's early exit)");
})
};
assert!(
task.await.is_err(),
"the row's panic surfaced as its failure"
);
let dbs = factory.tracked_dbs();
assert_eq!(dbs.len(), 1, "the panicked row minted one database");
let (name, dsn, schemas) = &dbs[0];
let loids = companion_loids(dsn, &schemas[0]).await;
assert!(!loids.is_empty(), "the panicked row's LO is tracked");
assert!(
crate::pg_lo_column::zero_lo_residue(dsn, &schemas[0]).await,
"the abandoned run's committed content has zero orphan LO \
residue (the LO catalog and the companion table agree \
exactly)"
);
factory
.teardown()
.await
.expect("teardown makes sense over the abandoned instance");
factory.teardown().await.expect("teardown is idempotent");
assert!(
!db_exists(&admin_dsn(factory.dsn()), name).await,
"teardown dropped the panicked row's database: {name} (the \
schemas and the whole LO state go with it)"
);
let mut reaped = factory.reaped_loids();
reaped.sort_unstable();
let mut captured = loids.clone();
captured.sort_unstable();
assert_eq!(
reaped, captured,
"the teardown's reap arm unlinked exactly the tracked LOIDs"
);
let (store, site) = factory
.open_tracked()
.await
.expect("the next open still vends a store");
assert!(
!store.has(&abandoned.as_bytes()).await.unwrap(),
"no residue from the panicked row — isolation is factory-owned"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.schema()).await,
0,
"and its medium carries nothing of the abandoned run"
);
factory
.teardown()
.await
.expect("teardown drops the last schema");
}
}
// ------------------------------------------------------------------
// The postgres-pglo factory — the consolidation posture, feature pair
// `postgres` + `pg-lo`: kv = postgres + large = pg-lo on one pg
// instance, the durable window hosted on the kv schema (the pg-lo
// window clauses dormant), and the placement oracle's kv half counted
// as rows in the kv schema's `blobs` table.
// ------------------------------------------------------------------
#[cfg(all(feature = "postgres", feature = "pg-lo"))]
mod postgres_pglo {
use std::io::Cursor;
use std::sync::atomic::AtomicU64;
use std::sync::{Arc, Mutex};
use std::time::{Duration, SystemTime};
use tokio_postgres::NoTls;
use crate::factory::StoreFactory;
use crate::pg_lo_column::{
admin_dsn, companion_loids, create_db, db_dsn, db_exists, drop_db, fresh_db_name,
harness_dsn, harness_ready, lo_entry_rows, loid_for_key, loid_pages, pg_table_rows,
reap_lo_schema, window_rows, DbJanitor, TrackedDb, LO_WINDOW, WINDOW_DELAY,
};
use crate::rows::{
self, patterned, slice_digest, Failing, LatchProtect, SetProtect, SetSource,
};
use alkblobs::{Error, Key, PgOpts, Store, StoreMode, DEFAULT_DISPATCH_THRESHOLD};
/// The column's declared dual-tier threshold — small, so the
/// shared content matrix and the race fixtures cross it and both
/// tiers' bodies are exercised (sub-threshold rows land as rows in
/// the kv schema's `blobs` table; over-threshold rows land as
/// companion rows + LOs in the pg-lo schema).
pub(crate) const TUNED_THRESHOLD: u64 = 1024;
/// One vended backing site: the dedicated database this open
/// minted and the two engine-owned schemas minted inside it — the
/// kv half (`blobs` + the GC tables) and the pg-lo half
/// (`lo_entries`, window-clauses dormant) — on one pg instance.
#[derive(Clone)]
pub(crate) struct Site {
/// The db-scoped DSN — the oracle's independent connections
/// address it; each open's database is private and gets
/// dropped whole at teardown (the LO-isolation corollary: LOs
/// are database-level).
dsn: String,
/// The kv engine's schema (the pool table and, per the
/// one-window-host rule, the `gc_window` + `gc_pins` tables).
kv_schema: String,
/// The pg-lo engine's schema (the companion table; no window
/// clauses — the dormancy shape).
pglo_schema: String,
/// The dedicated database's name — the structural teardown
/// tests' `pg_database` probe target.
db: String,
}
impl Site {
/// The kv tier's schema — the dispatch oracle's kv-side
/// counter (`blobs`) and the window host's `gc_window`.
pub(crate) fn kv_schema(&self) -> &str {
&self.kv_schema
}
/// The pg-lo tier's schema — the LO medium's companion table
/// and the dormancy row's genuine-absence target.
pub(crate) fn pglo_schema(&self) -> &str {
&self.pglo_schema
}
/// The DSN the open minted — scoped to the dedicated database.
pub(crate) fn dsn(&self) -> &str {
&self.dsn
}
}
/// The dispatch-coverage factory: `kv = postgres` + `large =
/// pg-lo` on one pg instance (the consolidation posture, ADR-009)
/// at the threshold this column declares and the test tuning of
/// the window delay (see [`TUNED_THRESHOLD`] and `WINDOW_DELAY` —
/// the window lives on the kv schema; the pg-lo schema carries no
/// `gc_window`, the dormancy shape the column's row asserts).
///
/// **Why a fresh instance per runner (not a shared static):**
/// teardown drops the dedicated databases *this instance* minted
/// (having reaped their LOs and dropped their schemas); a shared
/// instance's teardown would race concurrently running rows'
/// sites. Every runner constructs its own and owns its lifecycle;
/// a row that panics before teardown is still backed by the
/// per-open drop-guard janitors.
pub(crate) struct DualTierPostgresPgLoFactory {
dsn: String,
/// The declared dispatch threshold — the routing boundary this
/// column's rows run at (constructor-tunable; ADR-003).
/// Declared configuration: the dual-tier store never rejects,
/// so there is nothing to probe.
threshold: u64,
/// The declared window delay — the durable schedule's cadence
/// knob the sweep rows run at (the test tuning:
/// execute-on-next-sweep).
#[allow(dead_code)]
window_delay: Duration,
seq: AtomicU64,
/// Every dedicated database this instance minted, for its own
/// teardown and its drop-guard janitors.
dbs: Mutex<Vec<TrackedDb>>,
/// The LOIDs teardown reaped, for the structural tests'
/// assertions.
reaped_loids: Mutex<Vec<u32>>,
}
impl DualTierPostgresPgLoFactory {
pub(crate) fn new(threshold: u64) -> DualTierPostgresPgLoFactory {
DualTierPostgresPgLoFactory {
dsn: harness_dsn().expect("the harness server is configured for the column"),
threshold,
window_delay: WINDOW_DELAY,
seq: AtomicU64::new(
(std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.subsec_nanos() as u64)
.unwrap_or(0))
^ ((std::process::id() as u64) << 8),
),
dbs: Mutex::new(Vec::new()),
reaped_loids: Mutex::new(Vec::new()),
}
}
/// Open + track the vended site: the shape the column's own
/// rows use (they observe both mediums, so they need the site
/// alongside the store). One dedicated database per open with
/// two minted schemas — the kv half and the pg-lo half — so
/// the dormancy row observes genuine absence.
pub(crate) async fn open_tracked(&self) -> Result<(Store, Site), Error> {
let maintenance = admin_dsn(&self.dsn);
let db = fresh_db_name("consolidation", &self.seq);
create_db(&maintenance, &db).await?;
let dsn = db_dsn(&self.dsn, &db);
let kv_schema = crate::pg_lo_column::fresh_schema_name("kv", &self.seq);
let pglo_schema = crate::pg_lo_column::fresh_schema_name("lo", &self.seq);
self.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.push(TrackedDb {
name: db.clone(),
dsn: dsn.clone(),
schemas: vec![kv_schema.clone(), pglo_schema.clone()],
lo_schemas: vec![pglo_schema.clone()],
janitor: Some(DbJanitor::new(&maintenance, &db)),
});
let store = Store::dual_tier_postgres_pglo(
&dsn,
opts(kv_schema.clone()),
&dsn,
opts(pglo_schema.clone()),
self.threshold,
WINDOW_DELAY,
)
.await?;
Ok((
store,
Site {
dsn,
kv_schema,
pglo_schema,
db,
},
))
}
/// The dedicated databases this instance minted — names,
/// db-scoped DSNs, and the schemas minted inside — for the
/// structural tests' assertions.
pub(crate) fn tracked_dbs(&self) -> Vec<(String, String, Vec<String>)> {
self.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.iter()
.map(|d| (d.name.clone(), d.dsn.clone(), d.schemas.clone()))
.collect()
}
/// The LOIDs teardown reaped, for the structural tests'
/// tracked-LOID assertions.
pub(crate) fn reaped_loids(&self) -> Vec<u32> {
self.reaped_loids
.lock()
.unwrap_or_else(|e| e.into_inner())
.clone()
}
/// The maintenance DSN the factory mints databases through.
pub(crate) fn dsn(&self) -> &str {
&self.dsn
}
}
/// The factories' schema-name policy, in engine-opts form: the
/// constructor's `PgOpts` with exactly the factory's minted schema
/// and the ship defaults for the rest (pool size,
/// `synchronous_commit` — the alkstore factory's shape: only the
/// schema name is factory policy).
fn opts(schema: String) -> PgOpts {
PgOpts {
schema,
..PgOpts::default()
}
}
impl StoreFactory for DualTierPostgresPgLoFactory {
fn name(&self) -> &'static str {
"dual-tier postgres+pg-lo"
}
async fn open(&self) -> Result<Store, Error> {
self.open_tracked().await.map(|(store, _)| store)
}
async fn teardown(&self) -> Result<(), Error> {
let mut dbs = self
.dbs
.lock()
.unwrap_or_else(|e| e.into_inner())
.drain(..)
.collect::<Vec<_>>();
for tracked in dbs.iter_mut() {
// The LO-isolation corollary's arms: reap the tracked
// LOIDs (via the companion table — the contract
// authority), drop exactly the minted schemas (kv and
// pg-lo halves), then drop the dedicated database
// whole (`WITH (FORCE)` reaps any connection the
// abandoned store's pool has not drained).
let maintenance = admin_dsn(&self.dsn);
let mut reaped = Vec::new();
if let Ok(client) = admin_connect_for_teardown(&tracked.dsn).await {
for schema in &tracked.lo_schemas {
reaped.extend(reap_lo_schema(&client, schema).await);
}
for schema in &tracked.schemas {
let sql = format!("DROP SCHEMA IF EXISTS \"{schema}\" CASCADE");
let _ = client.batch_execute(&sql).await;
}
}
if !reaped.is_empty() {
self.reaped_loids
.lock()
.unwrap_or_else(|e| e.into_inner())
.extend(reaped);
}
drop_db(&maintenance, &tracked.name).await?;
TrackedDb::disarm(tracked);
}
Ok(())
}
}
async fn admin_connect_for_teardown(dsn: &str) -> Result<tokio_postgres::Client, Error> {
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
.await
.map_err(|e| Error::Io(format!("postgres harness: {e}")))?;
tokio::spawn(async move {
let _ = connection.await;
});
Ok(client)
}
/// The skip gate every runner checks (the SKIP posture's
/// factory-owner responsibility).
async fn ready() -> bool {
harness_ready().await
}
// --------------------------------------------------------------
// The column's own rows — the durable-schedule sweep family, the
// schedule-relative readings, the dispatch forms, the INV-1
// extension, the dormancy row, and the virgin-store row. Every
// row takes this column's concrete factory (it observes the
// mediums), is ADR-stamped per the convention, and traces its
// family.
// --------------------------------------------------------------
/// INV-2 on the durable schedule, over both tiers: identical
/// content put twice — through the same put path and across the
/// two paths — is one pool entry, and the re-put's `Pin` protects
/// exactly as a fresh put's does. The kv leg's single-pool-entry
/// is one row in the kv schema; the large leg's is one companion
/// row behind one LO in the pg-lo schema. One-pool-entry reads
/// through durable-schedule outcomes (the INV-5 posture): the pins
/// held stage nothing; once unpinned, sweep 1 stages exactly the
/// one entry and removes nothing, and its executor sweep deletes
/// exactly that staged set (README §The schedule-relative reading
/// for SQL-backed columns).
///
/// Contract stamp: store-api.md §Invariants 2; ADR-003 (whole-value
/// put is CAS first-writer-wins; the routing function's tier
/// placement); ADR-005 (a re-put refcounts the existing entry
/// under the same arbitration lock); ADR-007 §5 (postgres, the
/// consolidation shape's kv engine); ADR-009 §2/§4 (the companion
/// table is the contract authority; delete = lo_unlink + row
/// delete); ADR-008 §2 (the staged re-arbitrated delete); ADR-012
/// §4/§5 (the consolidation posture; the kv host wins the window);
/// ADR-012 §6.7 (the executor is the next sweep — the schedule
/// these counts read against).
pub async fn dedup_identical_content_is_one_pool_entry(factory: &DualTierPostgresPgLoFactory) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
// The kv leg: one row in the kv schema, both refcounts
// composing under the same lock.
let content = patterned((threshold - 1) as usize, 191);
let canonical = Key::derive(&content, content.len() as u64);
let (key, first) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the fresh put publishes");
let (reput, second) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the same-path re-put publishes");
assert_eq!(key, canonical);
assert_eq!(reput, canonical, "one derivation, one address");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.aborted, None);
assert_eq!(
report.candidates_staged, 0,
"one pool entry, both refcounts composing under the same lock"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
1,
"one pool entry, both refcounts — one row in the kv schema"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
0,
"the kv-tier entries never land on the LO medium"
);
drop(first);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.candidates_staged, 0,
"the re-put's pin protects exactly as a fresh put's"
);
drop(second);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one pool entry all along — staged this sweep, not executed"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.deleted, report.cancelled_by_arbitration),
(1, 0),
"the executing sweep deleted exactly the staged entry"
);
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
// The large leg: identical content through both put paths —
// one companion row behind one LO, both refcounts composing.
let cross = patterned(2 * threshold as usize + 5, 167);
let cross_canonical = Key::derive(&cross, cross.len() as u64);
let (key, first) = store
.put(Some(cross.len() as u64), Cursor::new(cross.as_slice()))
.await
.expect("the known-length put publishes");
let (reput, second) = store
.put(None, Cursor::new(cross.as_slice()))
.await
.expect("the unknown-length re-put publishes");
assert_eq!(key, cross_canonical);
assert_eq!(reput, cross_canonical, "one pool entry across the paths");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.aborted, report.candidates_staged),
(None, 0),
"both refcounts compose"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
1,
"one pool entry across both paths — one companion row"
);
let loids = companion_loids(site.dsn(), site.pglo_schema()).await;
assert_eq!(loids.len(), 1, "one LO behind the companion row");
assert!(
loid_pages(site.dsn(), loids[0]).await > 0,
"the LO holds the content's pages (never zero)"
);
drop(second);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.candidates_staged, 0,
"the re-put's pin protects exactly as a fresh put's (other drop order)"
);
drop(first);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one pool entry all along"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "executed exactly the staged set");
assert!(!store.has(&cross_canonical.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
0,
"the executed delete emptied the companion table"
);
assert_eq!(
loid_pages(site.dsn(), loids[0]).await,
0,
"delete is one tx: the LO's pages went with the row"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
0,
"the window holds nothing after both executed rounds (the kv host)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-3 on the durable schedule: stage hygiene over both mediums'
/// forms — every put-path failure leaves zero residue (the kv tier
/// never touches its pool; the large tier's held-put transaction
/// converges on ROLLBACK: no companion rows, no LO pages) and every
/// success leaves exactly one committed entry, over both put paths
/// and both injectable failure modes (a mid-stream stream failure
/// and a declared-length mismatch), observed through
/// durable-schedule sweep outcomes: zero staged after every
/// failure, and a success staged exactly once, then executed
/// exactly once.
///
/// Contract stamp: store-api.md §Invariants 3; ADR-010 §1 (the
/// staged-put contract: commit publishes, every failure path
/// discards); ADR-003 (stage-then-commit; the routing places each
/// leg's content on its tier); ADR-007 §5 (postgres, the kv
/// engine); ADR-009 §2 (put = one tx — the staged form is the held
/// tx, so a failure's LO pages roll back with it); ADR-012 §6.7
/// (the schedule-relative reading the success counts follow).
pub async fn stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
// The kv tier's degenerate stage form (nothing stages outside
// a flush) — residue is the pool itself, the kv schema's rows.
let small = patterned((threshold - 1) as usize, 167);
let err = store
.put(Some(small.len() as u64), Failing::new(&small, 40))
.await
.expect_err("the failed mid-stream put rejects");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
let err = store
.put(None, Failing::new(&small, 40))
.await
.expect_err("the failed unknown-length put rejects");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
let err = store
.put(Some(100), Cursor::new(small.as_slice()))
.await
.expect_err("the over-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let err = store
.put(Some(400), Cursor::new(small.as_slice()))
.await
.expect_err("the under-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(
report.candidates_staged,
report.deleted,
report.cancelled_by_arbitration
),
(0, 0, 0),
"zero residue after every kv failure, both put paths, both \
failure modes"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
0,
"the pool table holds zero entries after every kv failure"
);
let (key, pin) = store
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
.await
.expect("the known-length put publishes");
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed kv entry staged on success"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
assert!(!store.has(&key.as_bytes()).await.unwrap());
let (_, pin) = store
.put(None, Cursor::new(small.as_slice()))
.await
.expect("the unknown-length put publishes");
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed kv entry staged on the other put path too"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
// The LO tier's real staged form — the held tx's failure paths
// roll back to no companion row, no LO pages.
let content = patterned(2 * threshold as usize + 17, 173);
let len = content.len() as u64;
let canonical = Key::derive(&content, len);
let err = store
.put(Some(len), Failing::new(&content, 40))
.await
.expect_err("the failed mid-stream put rejects");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
let err = store
.put(None, Failing::new(&content, threshold as usize + 600))
.await
.expect_err("the failed unknown-length put rejects mid-stage");
assert!(
matches!(err, Error::Io(_)),
"the stream failure surfaces as Io: {err}"
);
let err = store
.put(Some(threshold + 10), Cursor::new(content.as_slice()))
.await
.expect_err("the over-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let err = store
.put(Some(len + 17), Cursor::new(content.as_slice()))
.await
.expect_err("the under-fed mismatch rejects");
assert_eq!(err, Error::Verification, "declared-len mismatch rejects");
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(
report.candidates_staged,
report.deleted,
report.cancelled_by_arbitration
),
(0, 0, 0),
"zero residue after every large failure, both put paths, both \
failure modes"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
0,
"the companion table holds zero entries after every failure"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
0,
"the window staged nothing behind the failures"
);
// Success, known path on the LO tier: exactly one committed
// entry — one companion row behind one LO — staged once,
// executed once.
let (key, pin) = store
.put(Some(len), Cursor::new(content.as_slice()))
.await
.expect("the known-length put publishes");
assert_eq!(key, canonical);
let loid = loid_for_key(site.dsn(), site.pglo_schema(), &key.as_bytes())
.await
.expect("the companion row carries its LOID");
assert!(
loid_pages(site.dsn(), loid).await > 0,
"the committed LO holds the content's pages"
);
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed entry staged on success"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
report.deleted, 1,
"the executing sweep removed exactly that one entry"
);
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
assert_eq!(loid_pages(site.dsn(), loid).await, 0);
// Success, unknown path: the overflow staged into the held tx,
// the commit published — exactly one committed entry on the
// other put path too.
let (key, pin) = store
.put(None, Cursor::new(content.as_slice()))
.await
.expect("the unknown-length put publishes");
assert_eq!(key, canonical);
drop(pin);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"exactly one committed entry staged on the other put path too"
);
let report = store.sweep().await.expect("sweeps are never errors");
assert_eq!(report.deleted, 1, "and executed exactly that staged set");
assert!(!store.has(&canonical.as_bytes()).await.unwrap());
assert_eq!(
(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
window_rows(site.dsn(), site.kv_schema()).await
),
(0, 0),
"both mediums and the window are back to zero"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, the durable-schedule happy shape over both tiers: the
/// full protection set spanning both tiers (a source-live kv
/// entry, a pinned large-tier entry, a protect-named large-tier
/// entry) plus one dead entry per tier — sweep 1 stages exactly
/// the two dead candidates into the **kv schema's `gc_window`**
/// (the window hosted on the kv engine per the one-window-host
/// rule) and deletes nothing; sweep 2 executes exactly that staged
/// set, and the protected entries stay byte-identical. The medium
/// oracle distinguishes the tiers (the kv schema's pool rows vs
/// the pg-lo schema's companion rows behind LOs) and watches the
/// schedule itself.
///
/// Contract stamp: store-api.md §Invariants 4 (exact counts with
/// correct liveness) and §Invariants 5 (list correctness through
/// sweep outcomes); ADR-005 (three liveness marks; delete windows
/// with arbitrate-at-delete-time); ADR-008 §2 (the staged
/// re-arbitrated delete; the constructor-tunable delay); ADR-009
/// §2/§4 (the companion table; GC-participating deletes); ADR-007
/// §5 (postgres, the kv engine); ADR-012 §1 (sweeps are never
/// errors); ADR-012 §4/§5 (the consolidation posture; the kv host
/// wins the window); ADR-012 §6.7 (the executor is the next
/// sweep).
pub async fn two_sweep_exact_counts_over_the_full_protection_set(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
let source = SetSource::register(&store);
let protect = Arc::new(SetProtect::default());
store.register_protect_callback(protect.clone());
let source_live = patterned((threshold - 1) as usize, 191);
let (key_live, _) = store
.put(
Some(source_live.len() as u64),
Cursor::new(source_live.as_slice()),
)
.await
.expect("the source-live entry publishes");
source.add(key_live.clone());
let pinned = patterned(2 * threshold as usize + 7, 173);
let (key_pinned, pin_pinned) = store
.put(None, Cursor::new(pinned.as_slice()))
.await
.expect("the pinned entry publishes");
let protected = patterned(2 * threshold as usize + 11, 167);
let (key_protected, _) = store
.put(
Some(protected.len() as u64),
Cursor::new(protected.as_slice()),
)
.await
.expect("the protect-named entry publishes");
protect.set_keys(vec![key_protected.clone()]);
let dead_kv = patterned((threshold - 3) as usize, 149);
let (key_dead_kv, pin) = store
.put(Some(dead_kv.len() as u64), Cursor::new(dead_kv.as_slice()))
.await
.expect("the kv-tier dead entry publishes");
drop(pin);
let dead_large = patterned(2 * threshold as usize + 13, 151);
let (key_dead_large, pin) = store
.put(None, Cursor::new(dead_large.as_slice()))
.await
.expect("the large-tier dead entry publishes");
drop(pin);
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(2, 3),
"before the sweep, both tiers hold their entries exactly (two \
pool rows in the kv schema, three companion rows in the \
pg-lo schema)"
);
let dead_large_loid =
loid_for_key(site.dsn(), site.pglo_schema(), &key_dead_large.as_bytes())
.await
.expect("the dead large entry's companion row carries its LOID");
assert!(
loid_pages(site.dsn(), dead_large_loid).await > 0,
"the LO medium holds the dead large entry's pages too"
);
let first = store.sweep().await.expect("sweeps are never errors");
assert_eq!(first.aborted, None);
assert_eq!(
(
first.candidates_staged,
first.deleted,
first.cancelled_by_arbitration
),
(2, 0, 0),
"sweep 1 staged exactly the two dead entries and deleted \
nothing — staging is not executing"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
2,
"the staged set waits in the durable window — hosted on the \
kv schema (the one-window-host rule's kv host)"
);
for (key, content) in [
(&key_live, &source_live),
(&key_pinned, &pinned),
(&key_protected, &protected),
] {
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the protected entry is present");
assert_eq!(
blob.read_at(0, len).await.expect("the window reads"),
content.as_slice(),
"the protected entry is byte-identical after the staging sweep"
);
}
assert!(
store.has(&key_live.as_bytes()).await.unwrap()
&& store.has(&key_pinned.as_bytes()).await.unwrap()
&& store.has(&key_protected.as_bytes()).await.unwrap(),
"the staged sweep deleted nothing: every entry intact"
);
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(second.aborted, None);
assert_eq!(
second.candidates_staged, 0,
"nothing newly dead since the staging sweep"
);
assert_eq!(
second.deleted + second.cancelled_by_arbitration,
2,
"sweep 2 executed exactly the staged set"
);
assert_eq!(
(second.deleted, second.cancelled_by_arbitration),
(2, 0),
"with correct liveness, every executed candidate deleted"
);
assert!(!store.has(&key_dead_kv.as_bytes()).await.unwrap());
assert!(!store.has(&key_dead_large.as_bytes()).await.unwrap());
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(1, 2),
"the executed deletes landed on both mediums exactly"
);
assert_eq!(
loid_pages(site.dsn(), dead_large_loid).await,
0,
"the dead large entry's LO pages are gone with it"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
0,
"the executed rows left the durable window"
);
drop(pin_pinned);
let third = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(third.candidates_staged, third.deleted),
(1, 0),
"with the pin dropped, the formerly pinned entry stages"
);
let fourth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
fourth.deleted, 1,
"and the next sweep executes it — exactly one deletion"
);
assert!(!store.has(&key_pinned.as_bytes()).await.unwrap());
assert!(store.has(&key_live.as_bytes()).await.unwrap());
assert!(store.has(&key_protected.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
1,
"exactly the protect-named entry's companion row remains"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, the race, in its durable-schedule form: a put landing
/// *during the window* survives the executing sweep — the re-put's
/// pin is minted (pin-before-publish) between the staging sweep
/// and the executing sweep, so the execution-time re-arbitration
/// cancels the pinned candidate while the unpinned twin deletes;
/// once the pin drops, the twin re-stages and executes exactly
/// once. Both put paths race in-body over both tiers (a kv twin
/// and a large twin), across the window delay the kv host carries.
///
/// Contract stamp: store-api.md §Invariants 4; ADR-005 (pin-before-
/// publish ordering; the delete window closes the sweep-vs-put race);
/// ADR-008 §2 (the staged re-arbitrated delete — the race holds
/// across the window delay); ADR-009 §4 (GC-participating deletes
/// over the companion table); ADR-012 §5 (the window on the kv
/// host in this consolidation shape); ADR-012 §6.7 (execution-time
/// re-arbitration is the executor sweep's posture).
pub async fn a_put_landing_during_the_window_survives_the_executing_sweep(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let rounds: [(String, Vec<u8>, bool, bool, bool); 2] = [
(
"the kv twin (known-path re-put)".to_owned(),
patterned((threshold - 1) as usize, 191),
true,
true,
false,
),
(
"the large twin (unknown-path re-put)".to_owned(),
patterned(2 * threshold as usize + 5, 173),
false,
false,
true,
),
];
for (index, (tag, content, original_known, reput_known, lands_large)) in
rounds.iter().enumerate()
{
let len = content.len() as u64;
let twin = Key::derive(content, len);
let witness = format!("the dead witness of round {index}").into_bytes();
let witness_key = Key::derive(&witness, witness.len() as u64);
let original = if *original_known { Some(len) } else { None };
let (_, pin) = store
.put(original, Cursor::new(content.as_slice()))
.await
.expect("the twin publishes");
drop(pin);
let (_, pin) = store
.put(None, Cursor::new(witness.as_slice()))
.await
.expect("the witness publishes");
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(
staged.aborted,
staged.candidates_staged,
staged.deleted,
staged.cancelled_by_arbitration
),
(None, 2, 0, 0),
"{tag}: the staging sweep stages exactly the two dead \
twins and deletes nothing"
);
// The put lands during the window (the re-put dedups into
// the existing twin and refcounts its pin): protected
// across the delay, pin-before-publish.
let reput = if *reput_known { Some(len) } else { None };
let (_, pin) = store
.put(reput, Cursor::new(content.as_slice()))
.await
.expect("the re-put publishes");
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(executed.aborted, None, "{tag}");
assert_eq!(
executed.candidates_staged, 0,
"{tag}: nothing newly dead since the staging sweep"
);
assert_eq!(
executed.deleted + executed.cancelled_by_arbitration,
2,
"{tag}: the executing sweep ran exactly the staged set"
);
assert_eq!(executed.deleted, 1, "{tag}: the unpinned witness deleted");
assert_eq!(
executed.cancelled_by_arbitration, 1,
"{tag}: the pinned twin's candidate cancelled at execution time"
);
assert!(
!store.has(&witness_key.as_bytes()).await.unwrap(),
"{tag}: the unpinned twin is gone"
);
let (got_len, blob) = store
.get(&twin.as_bytes())
.await
.expect("get succeeds")
.expect("{tag}: the put landing during the window survived");
assert_eq!(got_len, len, "{tag}");
assert_eq!(
blob.read_at(0, len).await.expect("the window reads"),
content.as_slice(),
"{tag}: the surviving twin is byte-identical"
);
drop(pin);
let restaged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(restaged.candidates_staged, restaged.deleted),
(1, 0),
"{tag}: the pin dropped — the twin re-stages"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"{tag}: the re-staged candidate executes exactly once"
);
assert!(!store.has(&twin.as_bytes()).await.unwrap());
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
0,
"{tag}: the window holds nothing after the executed round \
(the kv host)"
);
if *lands_large {
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
0,
"{tag}: the executed LO deletes reaped the medium exactly"
);
}
}
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-4, abort shape: the sweep is single-flight on the durable
/// form too — the in-process sweep lease (ADR-013 §2's single-node
/// analog). The contended second sweeper aborts with
/// `SweeperLock` in a completed `Ok(SweepReport)` — nothing
/// executed, nothing staged, nothing deleted (abort before
/// staging: form-independent) — while the lease holder's sweep
/// completes by **staging only**: its staged set waits in the
/// kv schema's `gc_window` for the next sweep to execute (the
/// durable form's schedule-relative replacement for the reference
/// row's inline holder-completion leg).
///
/// Contract stamp: store-api.md §Invariants 4; ADR-012 §1 (abort is
/// report data, never an error); ADR-013 §2 (the named exclusive
/// sweep lease — one sweeper per pool, holdership ends at release);
/// ADR-012 §5 (the window hosts on the kv engine in this
/// consolidation shape); ADR-012 §6.7 (the executor is the next
/// sweep — the holder stages, it does not execute).
pub async fn the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let (entered_tx, entered_rx) = std::sync::mpsc::channel::<()>();
let (release_tx, release_rx) = std::sync::mpsc::channel::<()>();
store.register_protect_callback(Arc::new(LatchProtect {
entered: entered_tx,
release: parking_lot::Mutex::new(Some(release_rx)),
}));
let content = patterned(2 * threshold as usize + 9, 167);
let (key, pin) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the dead witness publishes");
drop(pin);
// The lease holder sweeps on its own task (its protect consult
// blocks that task's thread mid-callback); this task contends
// and then releases the holder.
let store = Arc::new(store);
let holder = tokio::spawn({
let store = Arc::clone(&store);
async move { store.sweep().await }
});
while entered_rx
.recv_timeout(std::time::Duration::from_millis(50))
.is_err()
{
tokio::task::yield_now().await;
}
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
second.aborted,
Some(alkblobs::GcAbortCause::SweeperLock),
"the contended sweeper aborts as report data"
);
assert_eq!(
(
second.candidates_staged,
second.deleted,
second.cancelled_by_arbitration
),
(0, 0, 0),
"the contended sweep executes, stages, and deletes nothing"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"the contended sweep deleted nothing"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
0,
"the aborted sweep staged nothing into the window"
);
release_tx.send(()).ok();
let report = holder
.await
.expect("the holder task joined")
.expect("sweeps are never errors");
assert_eq!(report.aborted, None, "the lease holder completes");
assert_eq!(
(report.candidates_staged, report.deleted),
(1, 0),
"the holder stages exactly the dead set — it does not execute \
(the durable form's schedule)"
);
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
1,
"the holder's staged set waits in the durable window"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"the holder staged, did not execute: the entry is intact"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"the next sweep is the executor of the staged set"
);
assert!(!store.has(&key.as_bytes()).await.unwrap());
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
0,
"the executed delete emptied the companion table"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-5 on the durable schedule: `list()` correctness is
/// observable only through GC — the enumeration is complete and
/// well-formed in both directions, proven by exact
/// durable-schedule outcomes: the staging sweep stages exactly the
/// dead set (neither more nor fewer candidates), the executing
/// sweep deletes exactly that staged set, and a liveness flip
/// re-stages exactly the newly dead entry. Never a direct list
/// comparison.
///
/// Contract stamp: store-api.md §Invariants 5; ADR-003 (`list()` is
/// complete and well-formed by contract — load-bearing for GC);
/// ADR-005 (the sweep's live-set diff rides `list()`); ADR-012 §5
/// (the window hosts on the kv engine in this consolidation
/// shape); ADR-012 §6.7 (the executor is the next sweep — the
/// schedule these counts read against).
pub async fn list_correctness_is_observable_through_durable_schedule_outcomes(
factory: &DualTierPostgresPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
let source = SetSource::register(&store);
let mut keys = Vec::new();
for (index, content) in [
b"kept live by the source, entry one".to_vec(),
b"kept live by the source, entry two".to_vec(),
b"dead at the first staging, entry three".to_vec(),
b"dead at the first staging, entry four".to_vec(),
b"dead at the first staging, entry five".to_vec(),
]
.into_iter()
.enumerate()
{
let declared = if index % 2 == 0 {
Some(content.len() as u64)
} else {
None
};
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("the entry publishes");
drop(pin);
keys.push(key);
}
source.add(keys[0].clone());
source.add(keys[1].clone());
let first = store.sweep().await.expect("sweeps are never errors");
assert_eq!(first.aborted, None);
assert_eq!(
(first.candidates_staged, first.deleted),
(3, 0),
"the enumeration staged exactly the three dead entries — \
complete and well-formed in both directions"
);
for key in &keys {
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"staging deleted nothing: every enumerated entry is intact"
);
}
assert_eq!(
window_rows(site.dsn(), site.kv_schema()).await,
3,
"the staged schedule holds the three candidates"
);
let second = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(second.candidates_staged, second.deleted),
(0, 3),
"the executing sweep removed exactly the staged set"
);
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
assert!(store.has(&keys[1].as_bytes()).await.unwrap());
for key in &keys[2..] {
assert!(!store.has(&key.as_bytes()).await.unwrap());
}
source.remove(&keys[1]);
let third = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(third.candidates_staged, third.deleted),
(1, 0),
"the newly dead entry is staged exactly"
);
let fourth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(fourth.deleted, 1, "and executed exactly by the next sweep");
assert!(!store.has(&keys[1].as_bytes()).await.unwrap());
assert!(store.has(&keys[0].as_bytes()).await.unwrap());
source.remove(&keys[0]);
let fifth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(fifth.candidates_staged, 1, "the last entry stages exactly");
let sixth = store.sweep().await.expect("sweeps are never errors");
assert_eq!(sixth.deleted, 1, "and executes exactly");
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **routing is a pure function of length** — at,
/// below, and over the declared threshold, the same content lands
/// on exactly the same tier through both put paths (same placement
/// every time; nothing moves, nothing duplicates), identical
/// content through both paths is one pool entry, and every read is
/// byte-identical from wherever that tier holds it.
///
/// Tier placement is observed through both mediums (the oracle
/// adapted to the LO medium): kv entries hold a row in the kv
/// schema's `blobs` table and nothing in the pg-lo schema; large
/// entries hold a companion row behind an LO and no pool row — the
/// facade cannot and must not expose tier residency (it is
/// engine-blind).
///
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
/// ADR-003 §Dispatch policy (routing is a pure function of length;
/// no migration; CAS first-writer-wins makes re-puts one entry);
/// ADR-005 (re-put refcounts); ADR-007 §5 (postgres, the kv
/// engine); ADR-009 §2 (the companion table is the contract
/// authority); ADR-012 §4/§5 (the consolidation posture); ADR-012
/// §6.3 (the dual-tier dispatch-coverage shape); ADR-012 §6.7 (the
/// schedule-relative cleanup the dedup legs read against).
pub async fn routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
for content in [
patterned((threshold - 1) as usize, 191),
patterned(threshold as usize, 173),
patterned((threshold + 1) as usize, 167),
patterned(4 * threshold as usize + 9, 151),
] {
let len = content.len() as u64;
let canonical = Key::derive(&content, len);
let expected_rows = if len > threshold { 0 } else { 1 };
let expected_companions = if len > threshold { 1 } else { 0 };
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)]
{
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key, canonical,
"the canonical derivation addresses the put on {path_tag}"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
expected_rows,
"length {len} on {path_tag}: the placement is exactly the \
routing function's answer and never moves or duplicates \
(the kv medium holds {expected_rows} pool rows)"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
expected_companions,
"length {len} on {path_tag}: the LO medium holds \
{expected_companions} companion rows"
);
let (got_len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the entry is present");
assert_eq!(
got_len, len,
"the read serves the entry from its placed tier"
);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the entry is byte-identical from its placed tier across paths"
);
drop(pin);
}
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
staged.candidates_staged, 1,
"identical content through both put paths is one pool entry"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"the executing sweep removed exactly the one entry (durable \
schedule: staged this sweep, executed by the next)"
);
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(0, 0),
"both mediums are back to zero"
);
}
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **threshold boundary lengths route
/// deterministically** — at/below/above the factory's declared
/// threshold, on both put paths, the placement is the routing
/// function's answer every time (the mediums distinguish the kv
/// schema's rows from the pg-lo schema's companion rows); and the
/// default-threshold boundary re-routes with the tuning (the
/// constructor-tuned threshold deterministically places the
/// default boundary's length on the tier its own threshold names).
/// Run against two factories by the runner — the column's tuned
/// threshold and the shipped default. The durable cleanup: the
/// staged sweep stages exactly the six entries' dead set and the
/// next sweep executes it.
///
/// Contract stamp: store-api.md §Invariants 7 (facade-contract row);
/// ADR-003 §Dispatch policy (default 128 KiB at-or-below routes kv;
/// constructor-tunable without API change); ADR-007 §5 (postgres,
/// the kv engine); ADR-009 §2 (the companion table); ADR-012 §6.3
/// (the dual-tier dispatch-coverage shape); ADR-012 §6.7
/// (schedule-relative cleanup).
pub async fn threshold_boundary_lengths_route_deterministically(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let boundary = [
(threshold - 1, 191u32, false),
(threshold, 173, false),
(threshold, 167, false),
(threshold + 1, 151, true),
(threshold + 1, 139, true),
];
let mut expected_rows = 0u64;
let mut expected_companions = 0u64;
for (len, modulus, lands_large) in boundary {
let content = patterned(len as usize, modulus);
for declared in [Some(content.len() as u64), None] {
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key,
Key::derive(&content, len),
"the canonical derivation addresses the put"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await as u64,
expected_rows + if lands_large { 0 } else { 1 },
"boundary length {len}: at/below the threshold lands kv \
(one row in the kv schema), over it lands large (a \
companion row instead), both put paths"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await as u64,
expected_companions + if lands_large { 1 } else { 0 },
"boundary length {len}: the LO medium matches the routing answer"
);
drop(pin);
}
if lands_large {
expected_companions += 1;
} else {
expected_rows += 1;
}
}
// The default boundary re-routes with the constructor's
// tuning: the shipped default's length belongs to the tier
// this configuration's own threshold puts it on.
let at_default = patterned(DEFAULT_DISPATCH_THRESHOLD as usize, 199);
let default_lands_large = DEFAULT_DISPATCH_THRESHOLD > threshold;
let (key, pin) = store
.put(
Some(DEFAULT_DISPATCH_THRESHOLD),
Cursor::new(at_default.as_slice()),
)
.await
.expect("the default-boundary put publishes");
assert_eq!(
key,
Key::derive(&at_default, DEFAULT_DISPATCH_THRESHOLD),
"the canonical derivation addresses the put"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await as u64,
expected_rows + if default_lands_large { 0 } else { 1 },
"the default boundary's length routes to the tier this \
configuration's tuning names (the kv medium's answer)"
);
assert_eq!(
lo_entry_rows(site.dsn(), site.pglo_schema()).await as u64,
expected_companions + if default_lands_large { 1 } else { 0 },
"the default boundary's length routes to the tier this \
configuration's tuning names (the LO medium's answer)"
);
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
staged.candidates_staged, 6,
"each leg's content was exactly one pool entry"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 6,
"the executing sweep removed every entry, tier notwithstanding"
);
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(0, 0),
"both mediums are back to zero"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Dispatch row: **a kv-tier miss falls through to the large tier
/// on every key-addressed op** — content the routing function
/// placed on the large tier (one companion row behind an LO; the
/// kv medium holds no pool row — the kv tier misses) serves `get`,
/// `stat`, `has`, and `read_range` whole, exact, and
/// digest-correct; the mirror-position leg proves the kv tier is
/// the *first* consult (sub-threshold content, one pool row and no
/// companion row, reads still served); and both-tier misses
/// resolve per the error model's per-op shapes (reached here by
/// the durable schedule's staged-then-executed deletion of the
/// subject entry).
///
/// Contract stamp: store-api.md §Read path (fall-through consult
/// order); ADR-012 §6.1 (every key-addressed op consults kv first,
/// then large; the general rule); ADR-012 §6.5 ruling 5 (both-tier
/// miss shapes: `None` on get/stat, `Err(Missing)` on read_range);
/// ADR-006 (the slice digest); ADR-007 §5 (postgres, the kv
/// engine); ADR-009 §2/§3 (the companion table and the `lo_get`
/// windows serving the fall-through); ADR-012 §6.3 (the
/// dispatch-coverage shape).
pub async fn kv_tier_misses_fall_through_to_the_large_tier_per_op(
factory: &DualTierPostgresPgLoFactory,
) {
let threshold = factory.threshold;
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
// Over-threshold content — the kv tier misses, the large tier
// serves every op. Both put paths hold the shape.
for (content, declared) in [
(
patterned(3 * threshold as usize + 41, 181),
Some(3 * threshold + 41),
),
(patterned(3 * threshold as usize + 43, 197), None),
] {
let len = content.len() as u64;
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(0, 1),
"the content lives on exactly the large tier — the kv tier misses"
);
let loid = loid_for_key(site.dsn(), site.pglo_schema(), &key.as_bytes())
.await
.expect("the companion row carries its LOID");
assert!(
loid_pages(site.dsn(), loid).await > 0,
"the LO medium holds the entry's pages"
);
let (got_len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the large tier served the kv miss");
assert_eq!(got_len, len);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the fall-through read is byte-identical"
);
assert_eq!(
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
Some(alkblobs::EntryMeta {
len,
algorithm: key.algorithm(),
}),
"stat falls through and carries the length plus algorithm"
);
assert!(
store.has(&key.as_bytes()).await.unwrap(),
"has falls through"
);
let mid = len / 2;
let (slice, digest) = store
.read_range(&key.as_bytes(), mid, 17)
.await
.expect("read_range falls through");
assert_eq!(
slice,
content[(mid as usize)..(mid as usize + 17)],
"the range slice is exact"
);
assert_eq!(
digest,
slice_digest(&slice),
"the slice digest is SHA-256 over exactly the returned bytes"
);
// Sweep the entry away (the pin dropped): the miss shapes
// now resolve per op — the large tier misses too. The
// durable schedule: staged this sweep, executed by the next.
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!((staged.candidates_staged, staged.deleted), (1, 0));
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
executed.deleted, 1,
"exactly the one entry died across the two sweeps"
);
assert!(
store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"a both-tier miss on get is Ok(None)"
);
assert_eq!(
store.stat(&key.as_bytes()).await.expect("stat succeeds"),
None,
"a both-tier miss on stat is Ok(None)"
);
assert!(
!store.has(&key.as_bytes()).await.unwrap(),
"a both-tier miss on has is false"
);
assert_eq!(
store
.read_range(&key.as_bytes(), 0, 4)
.await
.expect_err("the both-tier miss errs"),
Error::Missing,
"a both-tier miss on read_range is Err(Missing)"
);
}
// The mirror-position leg: sub-threshold content — the kv tier
// is the first consult and serves before the large tier is
// reached (one pool row; nothing ever lands on the LO medium).
let small = patterned((threshold - 1) as usize, 149);
let (key, pin) = store
.put(Some(small.len() as u64), Cursor::new(small.as_slice()))
.await
.expect("the sub-threshold put publishes");
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(1, 0),
"the kv tier holds the sub-threshold entry — the large tier \
was never consulted on the write, never reached on the read"
);
let (len, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the kv tier served");
assert_eq!(len, small.len() as u64);
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
small
);
drop(pin);
// Both-tier miss shapes for a key that never existed at all,
// per op, over the fresh both-tier store.
let absent = b"never put here".to_vec();
let absent = Key::derive(&absent, absent.len() as u64);
assert!(
store
.get(&absent.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"never-put: get is Ok(None)"
);
assert_eq!(
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
None,
"never-put: stat is Ok(None)"
);
assert!(
!store.has(&absent.as_bytes()).await.unwrap(),
"never-put: has is false"
);
assert_eq!(
store
.read_range(&absent.as_bytes(), 0, 4)
.await
.expect_err("never-put read_range errs"),
Error::Missing,
"never-put: read_range is Err(Missing)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// INV-1's LO-medium extension (the mem-pglo factory's twin row,
/// over the consolidation shape's kv+LO mediums): **the `lo_get`
/// windows read across window boundaries byte-exact, and a body
/// short of its claimed length fails cleanly** — entries on the
/// large tier (both put paths) read whole through the
/// multi-window assembly, straddle the shipped window edge, read
/// ranged slices with the ADR-006 digest, clamp at EOF (tail,
/// at-EOF empty, a `u64::MAX` offset clamping in `u64` arithmetic,
/// zero-length, over-read), and a companion row overclaiming its
/// size (planted through the oracle) surfaces as the
/// unexpected-EOF-shaped `Error::Io`, never a partial read or a
/// panic.
///
/// Contract stamp: store-api.md §Invariants 1 (the round-trip
/// family's ranged-read legs) and §Read path (the streaming
/// large-tier windows, pread-style clamping); ADR-009 §2/§3 (the
/// companion table is the authority — the cursor closes over the
/// row; descriptorless `lo_get` windows, the shipped read shape
/// and its unexpected-EOF failure); ADR-006 (the slice digest over
/// exactly the returned bytes); ADR-010 §1 (the large-tier cursor
/// body family).
pub async fn lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(
factory: &DualTierPostgresPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
SetSource::register(&store);
let content = patterned(2 * LO_WINDOW + 100, 167);
let len = content.len() as u64;
for (path_tag, declared) in [("the known path", Some(len)), ("the unknown path", None)] {
let (key, pin) = store
.put(declared, Cursor::new(content.as_slice()))
.await
.expect("put publishes");
assert_eq!(
key,
Key::derive(&content, len),
"the canonical derivation addresses the put on {path_tag}"
);
assert_eq!(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
0,
"the entries live on the LO medium — the kv tier misses"
);
let (_, blob) = store
.get(&key.as_bytes())
.await
.expect("get succeeds")
.expect("the entry is present");
assert_eq!(
blob.read_at(0, len).await.expect("the whole window reads"),
content,
"the multi-window assembly reads whole and byte-exact on {path_tag}"
);
assert_eq!(
blob.read_at(LO_WINDOW as u64 - 100, 300)
.await
.expect("the straddling read succeeds"),
content[(LO_WINDOW - 100)..(LO_WINDOW + 200)],
"a read straddling the window edge crosses two lo_get \
statements byte-exact on {path_tag}"
);
let (slice, digest) = blob
.read_range(LO_WINDOW as u64, 7)
.await
.expect("the ranged slice reads");
assert_eq!(
slice,
content[LO_WINDOW..LO_WINDOW + 7],
"the ranged slice is exact on {path_tag}"
);
assert_eq!(
digest,
slice_digest(&slice),
"the slice digest is SHA-256 over exactly the returned bytes"
);
assert_eq!(
blob.read_at(len - 5, 100).await.expect("the tail reads"),
content[(len as usize - 5)..],
"the tail read clamps at the entry's end"
);
assert_eq!(
blob.read_at(len, 8).await.expect("the EOF read succeeds"),
Vec::<u8>::new(),
"a read at EOF is empty, never an error"
);
assert_eq!(
blob.read_at(u64::MAX, 5).await.expect("the offset reads"),
Vec::<u8>::new(),
"an overflowing u64 offset clamps in u64 arithmetic — the \
C7 int4 trap not inherited"
);
assert_eq!(
blob.read_at(0, 0).await.expect("the empty read succeeds"),
Vec::<u8>::new(),
"a zero-length read is empty"
);
assert_eq!(
blob.read_at(0, len + 64)
.await
.expect("the over-read reads"),
content,
"an over-read clamps to the whole entry"
);
drop(pin);
}
// The beyond-length clean failure: a companion row claiming
// more bytes than its LO holds (the corrupted half-state
// healthy stores never hold) planted through the oracle — the
// read window trusts the row (the contract authority,
// metadata-only) and fails cleanly when the LO comes up short.
let planted = b"the planted short body".to_vec();
let planted_key = Key::derive(&planted, planted.len() as u64);
let claimed = (planted.len() + 4096) as i64;
let loid = {
let client = admin_connect_plant(site.dsn()).await;
client
.batch_execute("BEGIN")
.await
.expect("the plant opens");
let loid: u32 = client
.query_one("SELECT lo_create(0) AS loid", &[])
.await
.expect("lo_create mints the planted LO")
.get(0);
let fd: i32 = client
.query_one("SELECT lo_open($1, $2)", &[&loid, &INV_WRITE])
.await
.expect("the planted LO opens for write")
.get(0);
client
.query_one("SELECT lowrite($1, $2)", &[&fd, &planted])
.await
.expect("the planted partial body writes");
let committed_at = SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos() as i64)
.unwrap_or(0);
client
.execute(
&format!(
"INSERT INTO \"{}\".lo_entries (key, loid, size, committed_at) \
VALUES ($1, $2, $3, $4)",
site.pglo_schema()
),
&[&planted_key.as_bytes(), &loid, &claimed, &committed_at],
)
.await
.expect("the overclaiming companion row plants");
client
.batch_execute("COMMIT")
.await
.expect("the plant commits");
drop(client);
loid
};
let (got_len, planted_blob) = store
.get(&planted_key.as_bytes())
.await
.expect("get succeeds")
.expect(
"the planted entry is present (the companion row is the \
authority, metadata-only)",
);
assert_eq!(
got_len, claimed as u64,
"the cursor's length is the companion row's overclaim — never \
an LO content probe"
);
let err = planted_blob
.read_at(0, got_len)
.await
.expect_err("the short body fails cleanly beyond its length");
assert!(
matches!(&err, Error::Io(m) if m.contains("short of")),
"the beyond-length read surfaces the unexpected-EOF clean \
failure, never a partial read: {err}"
);
// The durable cleanup: the round-trip legs' identical content
// was exactly one pool entry (CAS across the paths), so that
// entry and the planted one stage exactly, and the executing
// sweep deletes exactly that staged set — the planted LO's
// pages too.
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(staged.aborted, staged.candidates_staged),
(None, 2),
"the round-trip entry (one across both paths) and the planted \
one stage exactly"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(executed.deleted, 2, "the staged set executed exactly");
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await,
window_rows(site.dsn(), site.kv_schema()).await
),
(0, 0, 0),
"both mediums and the window are back to zero"
);
assert_eq!(
loid_pages(site.dsn(), loid).await,
0,
"the executing sweep's lo_unlink reaped the planted LO"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// The plant's raw-LO flag — postgres Large Objects' `INV_WRITE`
/// mode bits, the same constant the engine's orphan test plants
/// with.
const INV_WRITE: i32 = 0x0002_0000;
async fn admin_connect_plant(dsn: &str) -> tokio_postgres::Client {
let (client, connection) = tokio_postgres::connect(dsn, NoTls)
.await
.expect("the plant connection opens");
tokio::spawn(async move {
let _ = connection.await;
});
client
}
/// The **dormancy row** — the consolidation posture's
/// schema-placement shape (the one-window-host rule's kv-host arm,
/// ADR-012 §5): the kv schema carries `blobs` and both GC tables
/// (`gc_pins` + `gc_window` — the fleet host's both-tables row),
/// the pg-lo schema carries only the companion table and **no
/// `gc_window`** (the pg-lo window clauses are dormant on the
/// dual-SQL instance), and the schedule itself is observable on
/// the kv host: a dead entry stages into the kv schema's
/// `gc_window`, never anywhere in the pg-lo schema.
///
/// Contract stamp: store-api.md §Invariants 4 (the durable
/// schedule's host, read through its own staging outcome);
/// ADR-008 §2 (the durable window; the staged re-arbitrated
/// delete); ADR-012 §5 (the one-window-host rule: the kv engine
/// hosts the window on dual-SQL instances — the pg-lo window
/// clauses dormant; and pg-lo never carries fleet pins); ADR-012
/// §6.7 (the executor is the next sweep); ADR-009 §2 (the
/// companion table's bootstrap).
pub async fn the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant(
factory: &DualTierPostgresPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
assert_eq!(
store.mode(),
StoreMode::DualTier,
"the column's factory is the dispatch-coverage shape"
);
let dsn = site.dsn();
let kv = site.kv_schema();
let lo = site.pglo_schema();
assert!(
crate::pg_lo_column::table_exists(dsn, kv, "blobs").await,
"the kv tier's pool table exists in the kv schema"
);
assert!(
crate::pg_lo_column::table_exists(dsn, kv, "gc_window").await,
"the kv host wins the window: gc_window in the kv schema"
);
assert!(
crate::pg_lo_column::table_exists(dsn, kv, "gc_pins").await,
"the fleet host's both-tables row: gc_pins in the kv schema"
);
assert!(
crate::pg_lo_column::table_exists(dsn, lo, "lo_entries").await,
"the pg-lo tier's companion table exists in the pg-lo schema"
);
assert!(
!crate::pg_lo_column::table_exists(dsn, lo, "gc_window").await,
"the pg-lo window clauses are dormant on the dual-SQL instance: \
the pg-lo schema carries no gc_window"
);
assert!(
!crate::pg_lo_column::table_exists(dsn, lo, "gc_pins").await,
"gc_pins is never created by pg-lo (fleet pins live on the kv host)"
);
// The schedule itself rides the kv host: a dead entry stages
// into the kv schema's gc_window and executes by the next
// sweep — nothing ever stages into the pg-lo schema (which
// carries no window at all).
SetSource::register(&store);
let content = patterned(2 * TUNED_THRESHOLD as usize + 21, 173);
let (key, pin) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the dead candidate publishes");
drop(pin);
let staged = store.sweep().await.expect("sweeps are never errors");
assert_eq!(
(staged.aborted, staged.candidates_staged, staged.deleted),
(None, 1, 0),
"the staging sweep staged the dead candidate"
);
assert_eq!(
window_rows(dsn, kv).await,
1,
"the staged candidate waits in the kv schema's gc_window"
);
let executed = store.sweep().await.expect("sweeps are never errors");
assert_eq!(executed.deleted, 1, "the next sweep executed it");
assert!(
!store.has(&key.as_bytes()).await.unwrap(),
"the executed delete landed"
);
assert_eq!(
(window_rows(dsn, kv).await, lo_entry_rows(dsn, lo).await),
(0, 0),
"the window emptied and the companion table carries nothing of it"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
/// Virgin-store row: **reads are no-ops on a fresh schema** — an
/// empty `list`, absent reads per op (`None` on get/stat, `false`
/// on has, `Err(Missing)` on read_range), and nothing on the fresh
/// mediums themselves (zero pool rows in the kv schema, zero
/// companion rows in the pg-lo schema, and the window state per
/// the host rule: `gc_window` in the kv schema holds zero rows).
/// The constructors always ensure their schemas, this row being
/// the facade form.
///
/// Contract stamp: store-api.md §Error model (virgin-store
/// semantics: reads on a fresh store see absent/empty, never
/// errors); ADR-012 §6.5 ruling 5 (the per-op miss shapes); ADR-007
/// §5 (postgres, the schema-ensuring construction posture);
/// ADR-009 §2 (the companion-table bootstrap); ADR-012 §5 (the
/// one-window-host rule's kv host on dual-SQL instances).
pub async fn virgin_store_reads_are_no_ops_on_a_fresh_schema(
factory: &DualTierPostgresPgLoFactory,
) {
let (store, site) = factory
.open_tracked()
.await
.expect("open vends a fresh, isolated store");
assert_eq!(
store.mode(),
StoreMode::DualTier,
"the column's factory is the dispatch-coverage shape"
);
let dsn = site.dsn();
let absent = b"nothing was ever put".to_vec();
let absent = Key::derive(&absent, absent.len() as u64);
assert_eq!(
store.list().await.expect("list succeeds"),
Vec::<Key>::new(),
"the virgin pool lists empty"
);
assert!(
store
.get(&absent.as_bytes())
.await
.expect("get succeeds")
.is_none(),
"absent get is Ok(None)"
);
assert_eq!(
store.stat(&absent.as_bytes()).await.expect("stat succeeds"),
None,
"absent stat is Ok(None)"
);
assert!(
!store.has(&absent.as_bytes()).await.unwrap(),
"absent has is false"
);
assert_eq!(
store
.read_range(&absent.as_bytes(), 0, 4)
.await
.expect_err("absent read_range errs"),
Error::Missing,
"absent read_range is Err(Missing)"
);
assert_eq!(
(
pg_table_rows(dsn, site.kv_schema(), "blobs").await,
lo_entry_rows(dsn, site.pglo_schema()).await,
window_rows(dsn, site.kv_schema()).await
),
(0, 0, 0),
"the fresh mediums hold nothing (pool, companion table, and \
the kv-hosted window all empty)"
);
drop(store);
factory.teardown().await.expect("teardown is idempotent");
}
// --------------------------------------------------------------
// The runners — the column's tests: the form-independent reference
// rows mirrored verbatim against the dual-tier factory (INV-1, INV-6,
// and the two no-op abort shapes), the durable-schedule rows above,
// and the factory contract's structural shapes. Every runner gates
// behind the reachability probe — the SKIP posture: server-less
// runs skip cleanly, a skip is not a failure; acceptance is the
// green run against the harness server.
// --------------------------------------------------------------
#[tokio::test]
async fn postgres_pglo_inv1_round_trip_puts_and_gets_are_byte_identical() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
rows::round_trip_puts_and_gets_are_byte_identical(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv2_dedup_identical_content_is_one_pool_entry() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
dedup_identical_content_is_one_pool_entry(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv3_stage_hygiene_zero_residue_one_entry_on_success() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv4_two_sweep_exact_counts_over_the_full_protection_set() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
two_sweep_exact_counts_over_the_full_protection_set(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv4_a_put_landing_during_the_window_survives_the_executing_sweep() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
a_put_landing_during_the_window_survives_the_executing_sweep(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv4_sweep_without_liveness_sources_aborts_and_deletes_nothing() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
rows::sweep_without_liveness_sources_aborts_and_deletes_nothing(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv4_sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
rows::sweep_with_a_failed_protect_callback_aborts_and_deletes_nothing(&factory).await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn postgres_pglo_inv4_the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(
) {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv5_list_correctness_through_durable_schedule_outcomes() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
list_correctness_is_observable_through_durable_schedule_outcomes(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_inv6_stat_reports_len_and_algorithm_without_fetching_content() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
rows::stat_reports_len_and_algorithm_without_fetching_content(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly()
{
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
lo_windows_read_across_window_boundaries_and_a_short_body_fails_cleanly(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
the_window_lives_on_the_kv_schema_and_the_pg_lo_schema_stays_dormant(&factory).await;
}
#[tokio::test]
async fn postgres_pglo_virgin_store_reads_are_no_ops_on_a_fresh_schema() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
virgin_store_reads_are_no_ops_on_a_fresh_schema(&factory).await;
}
#[tokio::test]
async fn dispatch_postgres_pglo_routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(
) {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
routing_is_a_pure_function_of_length_across_put_paths_and_re_puts(&factory).await;
}
#[tokio::test]
async fn dispatch_postgres_pglo_threshold_boundary_lengths_route_deterministically() {
if !ready().await {
return;
}
let tuned = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
threshold_boundary_lengths_route_deterministically(&tuned).await;
let default = DualTierPostgresPgLoFactory::new(DEFAULT_DISPATCH_THRESHOLD);
threshold_boundary_lengths_route_deterministically(&default).await;
}
#[tokio::test]
async fn dispatch_postgres_pglo_kv_tier_misses_fall_through_to_the_large_tier_per_op() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
kv_tier_misses_fall_through_to_the_large_tier_per_op(&factory).await;
}
// --------------------------------------------------------------
// The factory contract's structural shapes (the postgres column's
// pattern, carried to the LO medium with the LO-isolation
// corollary's arms): fresh isolated opens (a fresh dedicated
// database with two fresh schemas per open, nothing shared — two
// opens never observe one another) and teardown against a
// panicking row (the schemas really dropped, the LOs really
// unlinked — zero orphan residue probed inside the dedicated
// database before it goes, and the shared harness database
// untouched by construction). Not conformance rows — no backlog
// entry — the factory contract's own wiring-level assertions.
// --------------------------------------------------------------
/// Every `open()` mints a fresh, isolated database + schema pair:
/// no shared state between islands, observable on the facade *and*
/// on the fresh mediums (each open's kv and pg-lo schemas are
/// private and start empty).
#[tokio::test]
async fn postgres_pglo_factory_open_vends_fresh_isolated_stores() {
if !ready().await {
return;
}
let factory = DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD);
let (a, site_a) = factory.open_tracked().await.expect("open vends a store");
let (b, site_b) = factory.open_tracked().await.expect("open vends another");
assert_ne!(
site_a.kv_schema(),
site_b.kv_schema(),
"two opens never share a kv schema"
);
assert_ne!(
site_a.pglo_schema(),
site_b.pglo_schema(),
"two opens never share a pg-lo schema"
);
assert_ne!(site_a.db, site_b.db, "two opens never share a database");
let content_a = patterned(2 * TUNED_THRESHOLD as usize + 5, 167);
let content_b = patterned(TUNED_THRESHOLD as usize - 5, 191);
let (key_a, _) = a
.put(
Some(content_a.len() as u64),
Cursor::new(content_a.as_slice()),
)
.await
.expect("island a's put publishes");
let (key_b, _) = b
.put(None, Cursor::new(content_b.as_slice()))
.await
.expect("island b's put publishes");
assert!(a.has(&key_a.as_bytes()).await.unwrap());
assert!(
!a.has(&key_b.as_bytes()).await.unwrap(),
"island a sees nothing of island b"
);
assert!(b.has(&key_b.as_bytes()).await.unwrap());
assert!(
!b.has(&key_a.as_bytes()).await.unwrap(),
"island b sees nothing of island a"
);
assert_eq!(
(
pg_table_rows(site_a.dsn(), site_a.kv_schema(), "blobs").await,
lo_entry_rows(site_a.dsn(), site_a.pglo_schema()).await
),
(0, 1),
"island a's mediums hold exactly its own entry (a's content routes large)"
);
assert_eq!(
(
pg_table_rows(site_b.dsn(), site_b.kv_schema(), "blobs").await,
lo_entry_rows(site_b.dsn(), site_b.pglo_schema()).await
),
(1, 0),
"island b's mediums hold exactly its own entry (b's content routes kv)"
);
assert!(
crate::pg_lo_column::zero_lo_residue(site_a.dsn(), site_a.pglo_schema()).await,
"island a's committed content has zero orphan LO residue (the \
LO catalog and the companion table agree exactly)"
);
let (c, site_c) = factory.open_tracked().await.expect("a third vends fresh");
assert_eq!(
(
pg_table_rows(site_c.dsn(), site_c.kv_schema(), "blobs").await,
lo_entry_rows(site_c.dsn(), site_c.pglo_schema()).await
),
(0, 0),
"a third open's mediums are virgin"
);
assert!(
!c.has(&key_a.as_bytes()).await.unwrap(),
"a third open shares nothing with the first"
);
assert!(
!c.has(&key_b.as_bytes()).await.unwrap(),
"a third open shares nothing with the second"
);
let dbs = factory.tracked_dbs();
assert_eq!(dbs.len(), 3, "one minted database per open");
assert_eq!(
dbs.iter()
.map(|(name, _, _)| name.clone())
.collect::<std::collections::BTreeSet<_>>()
.len(),
3,
"no two opens share a database"
);
for (_, _, schemas) in &dbs {
assert_eq!(
schemas.len(),
2,
"one kv + one pg-lo minted schema per database"
);
}
// The engine-owned schemas really exist inside their dedicated
// databases (the registry's minted list is not bookkeeping
// fiction).
for (name, dsn, schemas) in &dbs {
for schema in schemas {
assert!(
crate::pg_lo_column::schema_exists(dsn, schema).await,
"the minted schema exists inside {name}: {schema}"
);
}
}
factory
.teardown()
.await
.expect("teardown reaps the LOs, drops the schemas, drops the databases");
factory.teardown().await.expect("teardown repeats cleanly");
let maintenance = admin_dsn(factory.dsn());
for (name, _, _) in &dbs {
assert!(
!db_exists(&maintenance, name).await,
"teardown dropped the dedicated database: {name} (schemas \
and all — the whole LO state goes with it)"
);
}
}
/// A failing assertion panics the row before teardown runs — and
/// teardown must make sense over the abandoned instance anyway: it
/// is idempotent against the panicked row's path, the LOs are
/// really unlinked (the tracked-LOID reap arm — the recorded
/// LOIDs match the pre-captured set) and the minted schemas really
/// dropped (gone with the dedicated database, **zero orphan
/// `pg_largeobject` residue** probed inside before it goes), and
/// the next open mints a fresh database on fresh (empty) mediums
/// (README §The factory contract). The shared harness database is
/// untouched by construction — no pg-lo pool ever opens against it.
#[tokio::test]
async fn postgres_pglo_factory_teardown_is_safe_against_a_panicking_row() {
if !ready().await {
return;
}
let content = patterned(2 * TUNED_THRESHOLD as usize + 7, 173);
let abandoned = Key::derive(&content, content.len() as u64);
let factory = Arc::new(DualTierPostgresPgLoFactory::new(TUNED_THRESHOLD));
let task = {
let factory = Arc::clone(&factory);
tokio::spawn(async move {
let (store, _) = factory.open_tracked().await.expect("open vends the store");
let (_, pin) = store
.put(Some(content.len() as u64), Cursor::new(content.as_slice()))
.await
.expect("the put publishes");
drop(pin);
panic!("a failing assertion panics the row (the factory contract's early exit)");
})
};
assert!(
task.await.is_err(),
"the row's panic surfaced as its failure"
);
let dbs = factory.tracked_dbs();
assert_eq!(dbs.len(), 1, "the panicked row minted one database");
let (name, dsn, schemas) = &dbs[0];
let lo_schema = {
let mut found = None;
for schema in schemas {
if crate::pg_lo_column::table_exists(dsn, schema, "lo_entries").await {
found = Some(schema);
break;
}
}
found.expect("the minted pg-lo half carries the companion table")
};
let loids = companion_loids(dsn, lo_schema).await;
assert!(!loids.is_empty(), "the panicked row's LO is tracked");
assert!(
crate::pg_lo_column::zero_lo_residue(dsn, lo_schema).await,
"the abandoned run's committed content has zero orphan LO \
residue (the LO catalog and the companion table agree exactly)"
);
factory
.teardown()
.await
.expect("teardown makes sense over the abandoned instance");
factory.teardown().await.expect("teardown is idempotent");
assert!(
!db_exists(&admin_dsn(factory.dsn()), name).await,
"teardown dropped the panicked row's database: {name} (the \
schemas and the whole LO state go with it)"
);
let mut reaped = factory.reaped_loids();
reaped.sort_unstable();
let mut captured = loids.clone();
captured.sort_unstable();
assert_eq!(
reaped, captured,
"the teardown's reap arm unlinked exactly the tracked LOIDs"
);
let (store, site) = factory
.open_tracked()
.await
.expect("the next open still vends a store");
assert!(
!store.has(&abandoned.as_bytes()).await.unwrap(),
"no residue from the panicked row — isolation is factory-owned"
);
assert_eq!(
(
pg_table_rows(site.dsn(), site.kv_schema(), "blobs").await,
lo_entry_rows(site.dsn(), site.pglo_schema()).await
),
(0, 0),
"and its mediums carry nothing of the abandoned run"
);
factory
.teardown()
.await
.expect("teardown drops the last database");
}
}