Full-tree review before task decomposition found five composition
defects (mechanisms specced correctly in isolation, composition
unruled) and a set of caller-facing gaps. ADR-012 rules each:
- Sweeps are never errors: aborts are GcAbortCause report data in
Ok(SweepReport) (ProtectFailed / SweeperLock / NoLivenessSources);
GcAborted retired from the error enum; direct-delete refusal is
the GcRefuse error covering the full protection set
- Pin token gains its wire shape: blobs/put response {token, digest};
blobs/have's token-renewal form (one digest + token); token
validity domain = the minting serving node, process-lifetime
mapping
- Fleet mode is an explicit constructor declaration (fleet: true),
never inferred from engine choice
- All fleet GC state hosts on the fleet's kv engine (postgres) — one
arbitration domain; large=pg-lo fleet nodes required onto the same
pg instance (composite predicate's new clause); large=local fleet
puts pin-row-first, publish-second
- Engine-state seam reduced: sqlite pin/sweep-lock bodies dropped
(dead machinery); non-SQL engines stage delete-window candidates
in-process; one-window-host rule per instance
- Facade clarifications: fall-through for all key-addressed ops,
kv-only put-time rejection, mem+local dual-tier valid, error-model
member/return-shape ruling (trait/facade family split), has ->
bool, PinState variants, fleet liveness-table registration form,
window executor = the next sweep
Alignment edits across all specs and ADR-005/008/009/010/011
(bracketed corrections per the established pattern); OQ-11 (pg-only-kv
feature graph) added to the parked index for auditability.
Verification: two independent review passes; all findings resolved;
verdict READY for task decomposition.
ADR-010 (new): the Backend trait's I/O seams pinned before any code —
'get' returns a crate-internal read cursor (kv engines materialize;
'local' pread loop; pg-lo lo_get windows), 'put' has two named forms
(whole-value kv / staged-put large); GC state is store-core-owned via
a crate-internal contract-tested engine-state seam on the SQL-backed
engines (backends never learn liveness — ADR-005 verbatim); the store
core holds the joint entry+pin transaction (corrects ADR-009 §2's
ownership statement); kv-tier fleet-validity rule added (mirror of
ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin;
renewal rides have/re-put; pause-past-TTL falls to delete-then-recover);
feature graph pinned (pg-lo does not imply postgres).
ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs
engine (concrete impl, exactly five); store instance / node / fleet
split (fleet = pool-sharing, not node count — fixes requirements.md's
self-contradiction with REQ-2); mem demoted from 'backend'/'testing
tier' to the kv tier's third engine (contract-reference engine);
fs tier renamed 'large' including the feature name (last free moment
before code exists); constructor modes pinned (dual-tier default,
kv-only, mem-only) plus the composite fleet-validity predicate.
Consistency round across all specs and ADR-003/004/007/008/009: stale
pg-lo passages resolved per ADR-009's supersession; ADR-008 owner
'node id' → store-instance id; threshold wording corrected (conservative
edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport
shape specced; mem classification unified; 008/009 ADR files renamed to
match the tier rename; README deferral-policy recap aligned (third
category = decided-but-sequenced work, not a parking kind); POC crate
list completed.
Verification: two independent architecture review rounds (the first
found 4 criticals — unpinned trait I/O shapes, missing fleet-state
seam, node/fleet self-contradiction, pin-token/TTL conflict — all
resolved; final round: zero criticals); all markdown links resolve;
ADR tables complete (11 ADRs).
The ADR-008 pg-lo admission POC ran in a standalone crate
(/workspace/alkblobs-pglo-poc): PgLoBackend over the ADR-003/008 trait
contract (including size), 10/10 exact-count sweep-outcome contract
tests, clippy/fmt clean; dockerized postgres:16-alpine on :15432,
POC #5 driver stack (tokio-postgres + deadpool) via SQL lo_* functions,
no new dependency.
Gate verdict: passed, with named deltas.
- Performance: durable put 60-65 MB/s at >=1 MiB, within 1.5x of — and
below 1 MiB beating — durable local fs on this fsync-slow disk;
cached gets 70-180 MB/s single-stream, ~0.7 GB/s aggregate over 16
readers (20-50x behind page-cache fs — the honest named delta)
- Contract: companion table is the list()/size()/CAS authority (never
the catalogs); stage-then-commit; GC-participating lo_unlink delete
- Handles: the tx-scoped descriptor is real but pool-compatible via
descriptorless lo_get(oid, off, len) windows — window gets keep
handle-acquire p99 at 1-6 ms under readers <= pool; held descriptor
is the fallback posture
- Vacuum: pg_largeobject pages churn-reused, never returned; tracked
by autovacuum; rel-size monitoring named as an ops requirement
- Crash/orphan: LO creation is transactional — kill/terminate
mid-write-tx leaves zero orphan pages; the only orphan class is a
committed LO bypassing the companion table (planted, reaped by the
~7 ms/oid sweep; committed content survives byte-exact)
- Harness lessons: lo_lseek is int4 — the 64 variants are the
>2 GiB discipline; shared-table parallel tests are unsound (per-test
CREATE DATABASE isolation)
Docs: new poc-pglo-findings.md; poc-pglo-spec.md status passed;
register OQ-BL-06 #7 marked passed; ADR-008 pg-lo bullet updated
(duplicate bullet removed) + backends-and-dispatch/open-questions
cross-references.
Verification: cargo test --release (10 passed), clippy -D warnings,
fmt --check in /workspace/alkblobs-pglo-poc.
- requirements.md (new): the three use cases pinned as REQ-1..4 with
the node/pool/fleet/engine vocabulary defined once — ends per-session
re-derivation of consumer facts. REQ-2 (replicator fleet over one
shared pg pool, incl. large-blob serving) is recorded as a planning
fact predating all POCs, on the operator's authority.
- ADR-008: Backend trait gains size(key) length probe (before any
backend ships data — ADR-002 one-way-door discipline); fleet GC
mechanism (DB-backed pin rows committed atomically with entries,
TTL+renewal semantics, liveness = embedder-owned table, protect
callback is single-node-only, advisory-locked single sweeper,
staged re-arbitrated delete window on both engines); fs tier becomes
engine-selectable (local default; pg-lo named candidate) with the
fleet locality contract (shared media or re-routing; mixed tiers are
a documented deployment invariant, not a constructor-provable one).
- poc-pglo-spec.md (new): POC #7 spec — pg Large Objects as the fs
tier's pg-lo engine; instruments, decision gate, registered in
phase-0.md OQ-BL-06.
- ADR-003/005 status amendments point to ADR-008's extensions; specs
ripple (backends/store-api/gc/ops/overview/README).
- open-questions.md: deferral-policy header gains the decisions-vs-
sequenced-work distinction; pg-lo's why-not-parked audit trail
recorded.
- research fixes: postgres POC renumbered #4->#5 to the canonical
register (phase-0 OQ-BL-06), redb cross-refs fixed, thinking-
artifact sentence in B1 replaced with the honest reading.
Verification: docs-only change; reference-integrity sweep across the
tree (ADR/REQ/POC refs resolve); architecture-reviewer pass on the
delta — original 3 criticals addressed, its follow-up (fleet liveness
form, pin TTL, staged-delete semantics, enforceability, shared-media
caveats) fixed in this commit.
Dissolve OQ-10's circular "standing offer" framing and record the
engine decision the POC evidence already supported:
- ADR-007 (new): kv tier ships sqlite (default) + postgres (feature
`postgres`, default-off), constructor-selected per node; records the
deployment economics (classical kv+fs+relational downstream stack
collapses to one relational engine + fs), the pg impl contract
(POC #5 B5 posture deltas), the CI sweep-safety gate under
--all-features, redb ruled out (POC #6), and posture defaults as
deferred cost in the ADR-006 pattern
- ADR-003: amended status (engine pin widened by ADR-007); kv entry
notes the widened engine set
- ADR-004: reconciliation note — the "two backends" count is tiers,
not engines
- backends-and-dispatch.md: tier≠engine distinction, pg engine
section, co-tenancy note (same-file consumer writes share sqlite's
single-writer ceiling), substitution-door precedent
- open-questions.md: OQ-10 resolved-by-ADR-007, with the
why-this-is-not-Schrödinger's-code record (a deployment running
alkblobs cannot pre-exist the pg engine — OQ-09 precedent); parked
index narrowed to OQ-07/OQ-08
- overview.md: layer map + dependency posture reflect the `postgres`
feature; README.md: ADR-007 in tables + corollary applications
- phase-0.md / pg+redb findings: promotion + supersession notes
Trigger framing survives only as deployment guidance (which engine a
node chooses — topology, not throughput), not as a gate on the crate's
own work.
Verified: cargo test, clippy --all-targets -D warnings, fmt --check.
Docs-only change (Phase 1 architecture); no implementation yet.
Records the two-engine story the POC #5/#6 round established, in the
terms this crate's boundary uses (engine, not backend; trait contract
as the admission gate):
- OQ-10 (externally-owned standing offer): open the postgres-kv ADR
when a multi-tenant replicator deployment materializes. Trigger is
topology, not throughput — cross-machine write access to one pool
(throughput pressure / sqlite's flatline is the early warning).
Names the trigger-time work: PgKv impl + engine-posture deltas
(synchronous_commit, pooling/prepared-statement discipline,
autovacuum, ~40x storage overhead) + the sweep-safety proof
(vacuum-stable list() cursor, exact-count sweep tests). Sequenced:
sqlite stays Phase 1 mainline; evidence base is done and waiting.
- poc-postgres-kv-findings.md: engine deployment posture section —
the three use cases (self-hosted git serving, p2p replicator nodes,
agent-workspace pools) mapped onto the measured curves; per-node
choice as the form of the fork; honker parity stack named for the
pg side (LISTEN/NOTIFY + pgboss-rs, deployment layer).
No code changes; engine work parallelizes against the fixed contract.
Post-convergence engine studies against the ADR-003 kv pin, using
POC #3's harness unchanged (sqlite/fs arms byte-identical) plus the
concurrency axis POC #3 never measured:
- #5 postgres (B1-B6): solo pg floor ~1 ms (fsync-dominated, strace-
decomposed), ~40x storage overhead; decisive scale-out — pg PUTs ~12x
at 12 conns / ~37k ops/s while sqlite flatlines at its ~1.2k WAL
single-writer ceiling. Viable future ADR for the multi-client
replicator shape; durability/ops posture change, not a drop-in.
- #6 redb (C1-C6): the "2-7x over sqlite" folklore inverted — sqlite
~430x over redb at crash-consistent puts (redb Immediate = 1
fdatasync/commit, 8-30 ms on this disk; its None tier is not
crash-consistent). Reads ~530k/s but irrelevant. Ruled out at a
durability-tier mismatch: WAL-NORMAL's shipped tier is not
expressible in redb 4.x's two-level API.
- Net: sqlite's pin now has four-way triangulated evidence (fs vs
sqlite #3, pg #5, redb #6, lineage citation); batched commits proven
as the universal lever across all three engines.
Code: /workspace/alkblobs-postgres-poc, /workspace/alkblobs-redb-poc
(standalone crates per the POC placement convention; 14 inherited
tests passing in each, clippy -D warnings + fmt clean).
Verification: no src/ changes — docs only; both POC crates verified
(test/clippy/fmt) with findings recorded above.
The backends spec read as if the Backend trait were a general storage
abstraction and 'two backends, exactly' banned other storage in a
deployment — so a downstream wanting sqlite manifests appeared forced
into implementing a third backend, the exact weld ADR-004 forbids.
- backends-and-dispatch.md: state plainly that the trait is a CAS-tier
contract (immutable digest-keyed entries, GC-shaped list/delete) and
never a host for manifest/index storage; consumer storage is
beside-the-pool. 'Two, exactly' counts trait impls shipped, not
storage per deployment. kv: tier name vs pinned sqlite engine made
explicit.
- ADR-004: add the explicit corollary under the consumer-seams point.
- OQ-09 resolved closed-by-default (deny-unlisted), decided on risk
asymmetry in hand, not on a first deployment the crate itself must
create; named reopen condition instead of a parked wait
- delete oq-09-ops-visibility-tracker.md (watch-task for an event the
crate itself gates — never collapses)
- fix doc-lifecycle circularity: externally-owned OQs no longer block
a spec's `reviewed` promotion
- gc-and-namespaces: "until a consumer names a requirement" reframed
as re-entry via a new ADR, not a parked question
- README: state the corollary explicitly — facts this crate must
create (its own first deployment/consumer) are never deciding inputs
Verified: full-text sweep for stale OQ-09 deferral/tracker references
comes up clean across docs/ and tasks/.
- Verified against alkcall 0.8.1: no networking deps; Connection wraps
a caller-provided established stream (ADR-012 dial/call decoupling)
- 'transport' was being used loosely for 'wire/ops protocol'; now
reserved for endpoints/dialing/byte-moving, which is in neither
crate. Ops/framing/ACL surface = wire/ops protocol (alkcall's layer)
- Vision sentence, scope line, OQ-BL-01 argument, and alkcall
prior-art entry rewritten with the accurate layering; alkcall entry
now states the transport-free fact explicitly
- iroh-blobs-eval.md: 'welded to their transport' -> 'welded to their
protocol stack (endpoints/transport weld on above it)'
Verification: cargo test / clippy -D warnings / fmt --check clean
- The 'alkcall usage undecided with the first wire consumer' posture
was an inherited assumption, never a decision; it conflated iroh-
blobs' actual weld (no Store trait, store inseparable from irpc —
convention 7's rejection target) with having a protocol substrate
under the ops layer at all
- Settled: ops surface is alkcall-shaped — validated JSON control ops
(OperationSpec/JSON Schema, ADR-016 error schemas), binary data
channels via the channel_open marker (ADR-047; Sub ADR-021 fetch,
Pub/Sink ADR-046 put), ACL via AccessControl + ownership (ADR-011)
under the ADR-017 privilege model; an in-band op/auth scheme would
duplicate reviewed family machinery (convention 6 already mandates
the seam)
- Store layer stays substrate-agnostic (principle 1 unchanged); alkcall
enters only at the ops module/sibling boundary
- Remaining Phase 1 ADR scope reduced to placement: feature-gated ops
module (lean) vs sibling crate
- Synced vision sentence, scope line, alkcall prior-art entry,
OQ-BL-05 ACL line, recommended approach, decision table
Verification: cargo test / clippy -D warnings / fmt --check clean
- POC #3 (register OQ-BL-06) executed and passed: streaming LFSObject-shaped
put/get at the store layer; fanout seam (serve-while-persisting) validated
as broadcast above the backends; fs range reads with the verification gap
documented (preamble makes slice-vs-whole verification impossible —
chunk-tree encoding sharpened to a transfer-layer conditional)
- sqlite-vs-fs micro-benchmark: first-party anchor for the small-blob
belief (~9-10x sqlite at 1-16 KiB, crossover ~128-256 KiB, 128 KiB
default lean); harness re-runnable on real media
- Stage-file hygiene tests caught a real leak (one temp file per failed
put) — fixed; all failure paths converge on discard
- Pack tension resolved in shape from the gix-odb read: packs as large
blobs served by range reads; object-level dedup lives in the loose tier
- iroh-blobs-eval.md written against the current checkout (e82cbdc /
v0.103.0): actor/message-protocol pattern, fs backend layout,
verification, do-not-inherit + keep lists
- alknet probe re-check demoted to historical cross-check only: alknet's
code is not a design input (decomposed; substrate is alkcall); its
small-blob perf claim was asserted-never-measured, superseded by A4
- phase-0 register complete: no open POCs; next step is convergence ->
Phase 1
Verification: POC crate (standalone /workspace/alkblobs-largeblob-poc)
14 tests pass, clippy -D warnings clean, fmt clean; digests byte-exact
vs the real git hash-object CLI on both streaming put paths
POC code: /workspace/alkblobs-trait-poc (17 tests passing, clippy -D
warnings + fmt clean; digests validated byte-exact against the real
git hash-object CLI, both object formats)
Findings (poc-trait-dispatch-findings.md, 8):
- lean Backend trait (opaque byte keys, complete list(), put-path
pinning) is sufficient; Key typed wrapper lives at the boundary
- redb traps: list() key-vs-value destructure (silent until GC
catches it — malformed list is as deadly as incomplete), virgin-db
TableDoesNotExist on first read, no in-memory db (temp file used)
- pinning: Pinned RAII guard + refcount map works; batch-scope pins
and the sweep-vs-put race are named Phase 1 requirements (iroh
DeleteSet = prior art)
- git-blob preamble abstraction validated; external-git oids address
pool entries with zero indirection (the OQ-BL-03 resolution,
implementation-confirmed)
- tagged key format ([algo byte][digest]) makes git-sha256 + git-sha1
coexist; sort-stable for sweep diffs
- dual dispatch works as simply as hoped; routing deterministic
- namespace tables: protect callback MUST share live state (the
snapshot-install bug proves the API lesson: register_liveness_source
shape, not install_into)
- GC semantics verified: exact deletion counts, abort-callback,
delete-then-recover byte-identical re-put
phase-0.md: POC #1 marked passed, #2 absorbed (validated), #4 covered
in miniature (concurrency half deferred to Phase 1 impl work);
OQ-BL-02 base settled; OQ-BL-03 implementation-validated; OQ-BL-05
mechanism settled with findings folded in; plan updated (POC #3 is
the last open POC)
- OQ-BL-03 RESOLVED: BLAKE3's presence traced to pure iroh-blobs
inheritance; with that rejected the multi-hash framing dissolves.
Three tiers: canonical git-blob-sha-256 (git oids + every other
consumer share one address domain — cross-consumer dedup free),
tolerated git-blob-sha-1 (protocol necessity, git's hardened-SHA-1
threat model), conditional BLAKE3 (only if a bao-like chunk-tree
encoding is ever adopted; confined to that encoding layer)
- Principle 2 rewritten (one canonical hash, git-family derivation);
alkgit consumer entry updated (the 'hash conflict' was an artifact
of the BLAKE3 inheritance); p2p section notes strengthened dedup
- OQ-BL-04: verification options no longer BLAKE3-mandatory; chunk-
tree encoding note — verified wholes register under the canonical
hash so chunked and whole-file paths dedup together
- POC register: #2 absorbed into #1 (postamble abstraction + SHA-1
tolerance are #1's trait work); #3 gains the sqlite-vs-fs
micro-benchmark pull-out
- AGENTS.md convention 5 aligned (canonical git-family derivation)
- Consumers: add agent workspaces (third consumer); add the p2p shape
section (contract/gossip/replicator sketch) and pin what it implies
here: pooled CAS per node, repos/workspaces as hash-reference sets
- Principles: add 6 (pooling is the point) and 7 (whole-file CAS
default; CDC scoped to large-binary edits; fixed-size trees not
inherited)
- OQ-BL-01: ops surface upgraded from speculative to likely (have/need
+ verified fetch), crate-boundary question remains
- OQ-BL-04: chunking scoped; new sub-question — chunking in store
encoding vs manifest layer
- New OQ-BL-05: pooling/GC — namespace shape, lifetime strategies,
the pack tension (constrains store surface)
- POC register: add #4 (pooled CAS + GC); note pack tension rides #3
- gix-odb: note alternates/object-pools as cross-repo dedup prior art
glm-5.3-flash