feat(tests): the conformance suite's sqlite column — the durable-schedule SQL-backed column green (wave 3)

- The first SQL-backed column: the default-build production shape
  (kv = sqlite + large = local, feature sqlite default-on) runs the
  form-independent reference rows verbatim (INV-1, INV-6, the
  NoLivenessSources/ProtectFailed abort shapes, the kv-only rejection
  row against kv-only(sqlite)) and grows the durable-schedule sweep
  rows — exact counts asserted relative to the staged schedule
  (ADR-012 §6.7): two-sweep counts over the full protection set, the
  race across the window delay (a re-put landing during the window
  survives the executing sweep), the holder-stages-only lease shape,
  INV-5 through durable-schedule outcomes, and the schedule-relative
  INV-2/INV-3 readings
- Dispatch rows mirror the local column's family with the placement
  oracle adapted by medium: the sqlite file's pool/window rows vs the
  root's committed files; virgin-store reads as no-ops on a fresh file
  (the no-tables-yet tolerance engine-instrumented, this row its
  facade form)
- Factories: dual-tier sqlite+local and kv-only(sqlite), instance-
  tracked temp sites (fresh sqlite file + private root per open;
  idempotent panicking-row-safe teardown), the column-tuned threshold
  and the test window-delay tuning (Duration::ZERO — execute-on-next-
  sweep); the module is cfg(feature = "sqlite"), default-on — the
  column runs in every default cargo test, server-less, never skips
- Shared fixtures moved to rows.rs (committed_file_count,
  staging_file_count, slice_digest, LatchProtect pub(crate)) so the
  columns share them without the mem gate
- README: the schedule-relative reading for SQL-backed columns
  recorded (citing ADR-012 §6.7), the sqlite rows table inventory,
  Current state (three columns)

Verification: cargo test (default) green — 69 lib + 19 conformance +
3 git-interop; cargo test --features mem green (124 lib + 54
conformance); cargo check --all-features --tests green; cargo clippy
--all-targets -- -D warnings green; cargo fmt --check green.
This commit is contained in:
glm-5.3-flash committed 2026-10-12 05:14:58 +00:00
1 parent fb8326c4d5
commit 3f8e8158a7
6 files changed
+2049 -67

No files matched your search

+77 -2
View File
@@ -1,7 +1,7 @@
---
id: suite-column-sqlite
name: Conformance suite — the sqlite column (the first SQL-backed column)
status: pending
status: completed
depends_on: [store-sweep-durable-window]
scope: moderate
risk: medium
@@ -103,6 +103,81 @@ the default build (feature `sqlite` is default-on).
> To be filled by implementation agent
The column as it landed (`tests/conformance/sqlite_column.rs`, 17 tests):
- **Factories**: `dual-tier sqlite+local` (the default-build
production shape, `DualTierSqliteFactory`) and `kv-only(sqlite)`
(`KvOnlySqliteFactory`), each instance-tracking the temp sites it
vends (a private temp directory per open holding `pool.db` and the
local root; teardown `remove_dir_all`s it — idempotent, NotFound
tolerated, panicking-row-safe, one runner per instance), with the
column-declared threshold (1024) and the **test tuning of the
window delay (`Duration::ZERO` — execute-on-next-sweep)**. The
module is `#[cfg(feature = "sqlite")]` — **default-on**, so the
column runs in every default `cargo test`.
- **The medium oracle distinguishes by medium**: the dual-tier rows
count the sqlite file's `blobs` rows (the kv tier) vs the local
root's committed files (the large tier), plus `gc_window` rows (the
durable schedule observer) — via the test's own independent
connection. The shared helpers (`committed_file_count`,
`staging_file_count`, `slice_digest`, `LatchProtect`) moved to
`rows.rs` (`pub(crate)`, the columns' shared fixtures) so the
sqlite column reuses them without the mem gate.
- **Mirrored verbatim** (form-independent, from `rows.rs`): INV-1
round-trips, INV-6 stat discipline, the `NoLivenessSources`/
`ProtectFailed` abort shapes, and the kv-only rejection row
(against `kv-only(sqlite)`; unchanged, threshold-agnostic by
discovery).
- **Durable-schedule rows** (the column's own, concrete-factory,
ADR-stamped, traced): INV-2 dedup and INV-3 stage hygiene with
counts read against the staged schedule (the sweep-counting rows'
amendment), the two-sweep exact-count row over the full protection
set (both tiers, the window observed), the race across the window
delay (a re-put landing during the window survives the executing
sweep via execution-time re-arbitration; the unpinned twin
deletes; both put paths in-body over both tiers), the
holder-stages-only lease shape (the reference `SweeperLock` row's
schedule-relative form), and INV-5 list correctness through
durable-schedule outcomes.
- **Dispatch rows** mirror the local column's family (pure-function
routing, threshold boundaries at the tuned + shipped-default
thresholds, fall-through per op) with the placement oracle adapted
by medium. **Virgin-store row**: reads are no-ops on a fresh file
(pool, window, and committed tree all empty); the no-tables-yet
tolerance itself is engine-instrumented (src/sqlite.rs — the
constructor always ensures the schema), this row being the facade
form. **Structural factory tests**: fresh isolated opens
(distinct private sqlite files and roots, medium-observable) and
teardown-against-a-panicking-row (files really removed).
- **README amendments**: §The schedule-relative reading for
SQL-backed columns (the mirror convention's amendment, citing
ADR-012 §6.7 — staged-this-sweep / executed-next-sweep, which rows
carry it, which mirror verbatim), the sqlite rows table in §Columns
beyond the reference set, and Current state (three columns; the
default-build column). main.rs's module doc names the column.
## Summary
> To be filled on completion
> To be filled on completion
The sqlite column — the suite's first SQL-backed column — is green:
the default-build production shape (`kv = sqlite` + `large = local`,
feature `sqlite` default-on) runs the form-independent reference rows
verbatim against its factories, grows the durable-schedule sweep rows
whose exact counts are asserted relative to the staged schedule
(ADR-012 §6.7: staged this sweep, executed by the next — two-sweep
counts over the full protection set, the race across the window delay,
the holder-stages-only lease shape, INV-5 through outcomes, and the
schedule-relative INV-2/INV-3 readings), and the dispatch rows with
the medium oracle distinguishing the sqlite file's rows from the
root's committed files. Every row ADR-stamped and traced; the
schedule-relative INV-4/INV-5 reading is recorded in the README (the
mirror convention's amendment); factory isolation (fresh site per
open) and idempotent panicking-row-safe teardown hold structurally.
The column never skips and runs in the default build (17 tests).
Verification: `cargo test` green (default: 69 lib + 19 conformance +
3 git-interop), `cargo test --features mem` green (124 lib + 54
conformance), `cargo check --all-features --tests` green,
`cargo clippy --all-targets -- -D warnings` green, `cargo fmt --check`
green (fmt applied).
+101 -10
View File
@@ -185,6 +185,38 @@ families' first shapes (the `SweeperLock` abort) complete the three
`GcAbortCause` variants within the single-node exact-count family
(ADR-013 §2's lease, its in-process ADR-012 §5 form).
### The schedule-relative reading for SQL-backed columns (the mirror convention's amendment)
The reference set's single-sweep exact-count rows are the **inline-form
host's** (mem): there a completed sweep's
`candidates_staged == deleted + cancelled_by_arbitration` holds over
its same-sweep staged set. The durable form's semantics differ —
**staged this sweep, executed by the next** (ADR-012 §6.7) — so
SQL-backed columns read every exact-count assertion **relative to the
staged schedule**: the staging sweep reports `candidates_staged` as
the newly dead set and deletes nothing; its executor (the next sweep)
reports `deleted + cancelled_by_arbitration` summing exactly to that
staged set. Recorded, citing ADR-012 §6.7:
- **INV-4 and INV-5** run the **durable-schedule equivalents** on
SQL-backed columns — the two-sweep exact-count rows over the full
protection set, the sweep-vs-put race across the window delay, and
list correctness through durable-schedule outcomes (README §Columns
beyond the reference set grows the rows table). The `SweeperLock`
abort shape is form-independent (abort before staging) but its
reference holder-completion leg is inline-form; SQL-backed columns
run the holder-stages-only form (its staged set waits for the next
sweep to execute).
- **INV-2 and INV-3** — whose mirror rows observe their invariants
through sweep outcomes — likewise assert their counts relative to
the schedule: zero residue after failures is still staged-0; a
success's exactly-one-entry is staged-1 then deleted-1 across the
two sweeps. INV-1, INV-6, and the constructor-mode rows consume no
entry-dependent sweep counts and mirror verbatim.
- **Aborts are form-independent**: `NoLivenessSources` and
`ProtectFailed` abort before staging or executing — counts all zero
on the durable form too — so those rows mirror directly.
## Columns beyond the reference set (INV-7's mirror + extension)
The constructor-mode rows of the reference set run wherever the mode
@@ -228,6 +260,41 @@ and kv-only mode rows do not mirror onto the dual-tier factories (the
precondition `mode()` shapes they assert are not this column's mode);
the dispatch rows are the dual-tier mode's facade-contract coverage.
**The sqlite column's own rows** (tests/conformance/sqlite_column.rs,
the first SQL-backed column — the default-build production shape,
`kv = sqlite` + `large = local`, feature `sqlite` default-on,
ADR-007/013 §4): the dispatch-coverage rows with the **placement
oracle adapted by medium** — both tiers are on disk now, kv entries
live in the sqlite file (rows in the `blobs` table), large entries
under the local root — and the **durable-schedule sweep rows** (§The
schedule-relative reading for SQL-backed columns; ADR-012 §6.7; the
window's `gc_window` rows are the schedule observer). The
form-independent reference rows mirror verbatim: the INV-1 round-trip
family and the INV-6 stat-discipline row against the dual-tier
factory, the `NoLivenessSources`/`ProtectFailed` abort shapes
(form-independent — abort before staging), and the kv-only rejection
row against a `kv-only(sqlite)` factory (threshold-agnostic by
discovery, unchanged; the threshold materializable per the
configuration contract).
| Row (`sqlite_column.rs`) | Traces to |
|---|---|
| `dedup_identical_content_is_one_pool_entry` | INV-2 (same-path and cross-path legs; counts read against the durable schedule, one row-oracle leg) |
| `stage_hygiene_zero_residue_on_failure_exactly_one_entry_on_success` | INV-3 (the degenerate kv stage form through the injectable failure modes; schedule-relative counts) |
| `two_sweep_exact_counts_over_the_full_protection_set` | INV-4/INV-5 (the full protection set across two sweeps — staged this sweep, executed by the next; the both-tiers form) |
| `a_put_landing_during_the_window_survives_the_executing_sweep` | INV-4 (the sweep-vs-put race across the window delay; execution-time re-arbitration) |
| `the_contended_second_sweeper_aborts_as_data_and_the_holder_stages_only` | INV-4 (the `SweeperLock` abort shape — abort before staging; the holder's durable-form completion: stages only) |
| `list_correctness_is_observable_through_durable_schedule_outcomes` | INV-5 (exact counts, never list comparison) |
| `routing_is_a_pure_function_of_length_across_put_paths_and_re_puts` | ADR-003 §Dispatch policy (purity/no-migration); ADR-012 §6.3; ADR-012 §6.7 (schedule-relative dedup legs); ADR-007; INV-7 facade-contract row |
| `threshold_boundary_lengths_route_deterministically` | ADR-003 §Dispatch policy (at/below/above, tuned + shipped-default thresholds); ADR-012 §6.3; ADR-007 |
| `kv_tier_misses_fall_through_to_the_large_tier_per_op` | ADR-012 §6.1 (fall-through per op); ADR-012 §6.5 ruling 5 (miss shapes); ADR-006; ADR-012 §6.3; ADR-007 |
| `virgin_store_reads_are_no_ops_on_a_fresh_file` | store-api.md §Error model (virgin-store semantics); ADR-012 §6.5; ADR-007 (the no-tables-yet read tolerance — engine-instrumented, the facade's fresh-file form) |
The sqlite column's window delay is the **test tuning** (ADR-008 §2's
legal degenerate tuning, `Duration::ZERO` — execute-on-next-sweep):
the schedule the durable rows assert against is exactly the
next-sweep executor form.
## Column plan
**mem is the contract-reference engine** (ADR-011 §2): wave 2 lands
@@ -240,15 +307,15 @@ and first run in wave 6.
## Current state
The harness is green on **two columns** (wave 2). The **mem column**
is the contract-reference column: the factory binding is the concrete
`alkblobs::Store` facade (see §The factory contract — the binding
demonstrator remains in `binding.rs`), the reference row set is landed
(§The reference row set — `rows.rs`, every row ADR-stamped and traced
to the backlog), the column's factories (`mem-only`, the reference;
`kv-only(mem)` at a column-declared threshold) vend fresh isolated
stores with idempotent teardown (`mem_column.rs`, the structural
factory-contract tests included), and mem never skips.
The harness is green on **three columns** (waves 2–3). The **mem
column** is the contract-reference column: the factory binding is the
concrete `alkblobs::Store` facade (see §The factory contract — the
binding demonstrator remains in `binding.rs`), the reference row set
is landed (§The reference row set — `rows.rs`, every row ADR-stamped
and traced to the backlog), the column's factories (`mem-only`, the
reference; `kv-only(mem)` at a column-declared threshold) vend fresh
isolated stores with idempotent teardown (`mem_column.rs`, the
structural factory-contract tests included), and mem never skips.
The **local column** (the suite's first dual-tier column, the
dispatch-coverage shape ADR-012 §6.3 names) mirrors the reference INV
@@ -269,4 +336,28 @@ constructor rides feature `mem` because mem is the only constructible
kv engine until wave 3 (the gate is the kv pairing's, not the local
engine's, which is ungated in `src/`) — the column runs under `cargo
test --features mem`, with the reference row set compiling ungated so
it stays type-checked in default builds.
it stays type-checked in default builds.
The **sqlite column** (the suite's first SQL-backed column — the
default-build production shape, `kv = sqlite` + `large = local`,
ADR-007/013 §4) mirrors the form-independent reference rows (INV-1,
INV-6, the `NoLivenessSources`/`ProtectFailed` abort shapes, and the
kv-only rejection row against `kv-only(sqlite)`) and grows the
**durable-schedule sweep rows** (§The schedule-relative reading for
SQL-backed columns — the two-sweep exact counts, the race across the
window delay, the holder-stages-only lease shape, list correctness
through durable-schedule outcomes, and the schedule-relative INV-2/3
readings) plus the dispatch rows with the placement oracle adapted by
medium (both tiers on disk: the sqlite file's rows vs the root's
committed files) and the virgin-store row — the recorded inventory is
in §Columns beyond the reference set. Its factories declare the
column-tuned threshold and the test tuning of the window delay
(`Duration::ZERO` — execute-on-next-sweep), vend a fresh temp site
per open (a private directory holding the sqlite file and the local
root; idempotent teardown removing the site, panicking-row-safe, each
runner owning its instance), and never skip. The column is
feature-gated on `sqlite` — **default-on**, so the column runs in the
default build: every `cargo test` run exercises it. The sqlite
engine's no-tables-yet read tolerance is engine-instrumented
(src/sqlite.rs's tests; the constructor-vs-lazy timing design point
there), the column's virgin-store row being its facade form.
+9 -51
View File
@@ -46,13 +46,14 @@
#![allow(dead_code)]
use std::io::Cursor;
use std::path::{Path, PathBuf};
use std::path::PathBuf;
use std::sync::{Arc, Mutex};
use sha2::{Digest as _, Sha256};
use crate::factory::StoreFactory;
use crate::rows::{self, assert_exact_counts, patterned, Chunked, Failing, SetProtect, SetSource};
use crate::rows::{
self, assert_exact_counts, committed_file_count, patterned, slice_digest, staging_file_count,
Chunked, Failing, SetProtect, SetSource,
};
use alkblobs::{Error, Key, Store, StoreMode, DEFAULT_DISPATCH_THRESHOLD};
// ------------------------------------------------------------------
@@ -159,55 +160,12 @@ impl StoreFactory for DualTierLocalFactory {
}
// ------------------------------------------------------------------
// On-disk observation helpers (the dispatch-coverage shape's medium
// oracle; kv = mem never touches this disk, so these counts are
// exactly the large tier's committed/staging residency).
// On-disk observation helpers live in `rows` (the medium-backed
// columns' shared oracle); kv = mem never touches this disk, so the
// counts below are exactly the large tier's committed/staging
// residency.
// ------------------------------------------------------------------
/// Every regular file under `root` outside any `staging` directory —
/// with `kv = mem`, exactly one per committed large-tier entry.
fn committed_file_count(root: &Path) -> usize {
fn walk(dir: &Path, out: &mut Vec<PathBuf>) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let Ok(kind) = entry.file_type() else {
continue;
};
let path = entry.path();
if kind.is_dir() {
if path.file_name().and_then(|n| n.to_str()) == Some("staging") {
continue;
}
walk(&path, out);
} else if kind.is_file() {
out.push(path);
}
}
}
let mut files = Vec::new();
walk(root, &mut files);
files.len()
}
/// Files in the large tier's staging area — the residue counter of the
/// exact-count hygiene row (zero after every failure form and every
/// success form; a staged artifact lives there only mid-put).
fn staging_file_count(root: &Path) -> usize {
match std::fs::read_dir(root.join("staging")) {
Ok(entries) => entries.count(),
Err(_) => 0,
}
}
/// The ADR-006 slice digest, computed independently of the crate —
/// SHA-256 over exactly the returned bytes (the out-of-band
/// verification the convention requires; callers verify the same way).
fn slice_digest(bytes: &[u8]) -> [u8; 32] {
Sha256::digest(bytes).into()
}
// ------------------------------------------------------------------
// The column's own rows — dispatch coverage (the coverage shape's
// reason to exist) and the local-specific properties. Every row takes
+8 -1
View File
@@ -25,7 +25,12 @@
//! [`local_column`] — the suite's first dual-tier column (the
//! `kv = mem` + `large = local` dispatch-coverage shape ADR-012 §6.3
//! names), which mirrors the reference set and grows the
//! dispatch-coverage and local-specific rows.
//! dispatch-coverage and local-specific rows; [`sqlite_column`] — the
//! suite's first SQL-backed column (the default-build production
//! shape, `kv = sqlite` + `large = local`), which mirrors the
//! form-independent reference rows and grows the durable-schedule
//! sweep rows (README §The schedule-relative reading for SQL-backed
//! columns).
//!
//! Items beyond the tests below are the later columns' surface, hence
//! the scoped `#![allow(dead_code)]` — it shrinks as columns land.
@@ -41,6 +46,8 @@ mod local_column;
#[cfg(feature = "mem")]
mod mem_column;
mod rows;
#[cfg(feature = "sqlite")]
mod sqlite_column;
mod stamp;
#[test]
+57 -3
View File
@@ -32,12 +32,14 @@
use std::collections::BTreeSet;
use std::io::Cursor;
use std::path::{Path, PathBuf};
use std::pin::Pin as StdPin;
use std::sync::Arc;
use std::task::{Context, Poll};
use std::time::Duration;
use parking_lot::Mutex;
use sha2::{Digest as _, Sha256};
use tokio::io::{AsyncRead, ReadBuf};
use crate::factory::StoreFactory;
@@ -111,9 +113,10 @@ impl ProtectCallback for SetProtect {
/// A protect callback that blocks mid-consult — the fixture for the
/// sweep-lease contended window: it announces its entry on a channel,
/// waits on a second, then answers with an empty protection set.
struct LatchProtect {
entered: std::sync::mpsc::Sender<()>,
release: Mutex<Option<std::sync::mpsc::Receiver<()>>>,
/// `pub(crate)`: the suite's other columns reuse it.
pub(crate) struct LatchProtect {
pub(crate) entered: std::sync::mpsc::Sender<()>,
pub(crate) release: Mutex<Option<std::sync::mpsc::Receiver<()>>>,
}
impl ProtectCallback for LatchProtect {
@@ -267,6 +270,57 @@ pub(crate) fn assert_exact_counts(report: &alkblobs::SweepReport) {
);
}
/// Every regular file under `root` outside any `staging` directory —
/// the committed-tree counter of the medium-backed columns' on-disk
/// placement oracle. With a kv tier that never touches this disk it is
/// exactly one per committed large-tier entry; a SQL-backed kv tier
/// distinguishes by medium instead (its entries live in the database
/// file, so this count is still exactly the large tier's residency).
/// `pub(crate)`: the suite's other columns reuse it.
pub(crate) fn committed_file_count(root: &Path) -> usize {
fn walk(dir: &Path, out: &mut Vec<PathBuf>) {
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for entry in entries.flatten() {
let Ok(kind) = entry.file_type() else {
continue;
};
let path = entry.path();
if kind.is_dir() {
if path.file_name().and_then(|n| n.to_str()) == Some("staging") {
continue;
}
walk(&path, out);
} else if kind.is_file() {
out.push(path);
}
}
}
let mut files = Vec::new();
walk(root, &mut files);
files.len()
}
/// Files in the large tier's staging area — the residue counter of the
/// on-disk stage-hygiene exact-count form (zero after every failure
/// form and every success form; a staged artifact lives there only
/// mid-put). `pub(crate)`: the suite's other columns reuse it.
pub(crate) fn staging_file_count(root: &Path) -> usize {
match std::fs::read_dir(root.join("staging")) {
Ok(entries) => entries.count(),
Err(_) => 0,
}
}
/// The ADR-006 slice digest, computed independently of the crate —
/// SHA-256 over exactly the returned bytes (the out-of-band
/// verification the convention requires; callers verify the same way).
/// `pub(crate)`: the suite's other columns reuse it.
pub(crate) fn slice_digest(bytes: &[u8]) -> [u8; 32] {
Sha256::digest(bytes).into()
}
/// INV-1: whole-blob put/get round-trips byte-identical under the
/// canonical derivation — the content matrix loops both put paths
/// (known-length, unknown-length) in-body; every put's key is the
File diff suppressed because it is too large. Load diff