alkblobs/docs/architecture

status, last_updated, notes

status	last_updated	notes
draft	2026-10-18	Review-003 recorded (the family wire gap) and ADR-020 accepted the same day — the correction ADR; ops-surface/store-api/ hashing-and-keys flipped `reviewed` pending the ADR-020 implementation wave; ADR-018 §§2/6 superseded

alkblobs — Architecture

Current State

Phase 1 (implementation — release readiness). Phase 0 (Exploration) converged — see docs/research/phase-0.md for the vision, the evidence trail, and the POC register (all POCs passed or were absorbed; POC code lives in standalone crates under /workspace/ — alkblobs-trait-poc, alkblobs-largeblob-poc, alkblobs-postgres-poc, alkblobs-redb-poc, alkblobs-pglo-poc; findings live in docs/research/). Implementation is complete through wave 12 in src/ behind the wave decomposition (docs/plans/implementation.md): waves 1-7 landed the core traits, hashing/keys, all five engines (sqlite/postgres/mem/local/pg-lo), fleet GC, and the full ops surface including put (alkcall 0.9 satisfied ADR-014's amendment-4 gate); review 001 (the post-wave-7 whole-surface pass) inserted the remediation waves: wave 8 (the structural split), wave 9 (correctness), wave 10 (security) — completed at their gates, as was wave 11 (fuzzing — the fuzz/ sidecar workspace, the key_encoding + ops_ingress targets over the remediated parse surfaces, the stable corpus-replay gate joined to AGENTS.md's verification block). Wave 12 (release readiness — unicode hygiene, the docs consistency round, crate docs, the feature-graph verification, changelog + README, publish prep, and the pre-release review gate) completed 2026-10-16 at its review gate: the docs honesty audit, the feature-graph matrix, the full verification block including the fuzz corpus replay, and the publish dry-run are green, and the release-facing specs flipped stable. The crate is publishable (the publish itself is a later, deliberate step). The review's three decision-shaped findings resolved through the OQ/ADR path (2026-10-15): ADR-015/016/017 below — which satisfied waves 9/10's decomposition gates.

Wave 13 (the review-002 round) completed 2026-10-17/18 at its gates: ADR-018 (the git-object derivation family — registry bytes 0x03/0x04, preimage-identity hashing) and ADR-019 (the token-registry admission posture) landed with their implementation halves, and the crate published 0.1.0 (2026-10-18, coordinator-side). The same day, review-003 recorded the wave-13 round's one coherence defect: the ops wire's blob-family scoping (ADR-018 §Decision 2) left the crate a put surface whose stored family entries the protocol cannot address — a half surface, and the coherence bar is the crate's own. The review's addressing conversation decided the correction; ADR-020 (2026-10-18) is its record: family-explicit payloads under the closed op set (the wire speaks tag-prefixed key hex; bare hex dead), the mandatory-claim put offer with the echoed-key response, one derivation-explicit put seam (put_as; put, put_git_object, from_git_oid_hex, and Digest::derive's silent form deleted), reads unchanged by principle, and the invariant statement the next gate can push against. It supersedes ADR-018 §Decision 2/§Decision 6 and ADR-012 §2's put-response field; the affected specs (ops-surface, store-api, hashing-and-keys) flipped reviewed pending the ADR-020 implementation wave (decided-but-sequenced work; the decomposition seams are named in ADR-020 §Decision 9). The store layer takes no changes.

The converged Phase 0 posture has been promoted into the specs and ADRs below. Two framing corrections from the convergence review are captured there — not inherited as hedges:

• ADR-001 makes the substrate posture explicit: the store layer carries no alkcall dependency by design — which is conformance to the family pattern, not deviation from it — and the ops surface rides alkcall, feature-gated.
• ADR-004 fixes the "two backends" framing: the crate ships exactly two tiers (kv small / large — both required by measured scale economics); the path→hash mapping layer (the alkfs/alknet "vfs" shape) is a consumer-layer concern and must never become a third backend.
• ADR-007 widens the kv tier's engine set rather than hedging it: sqlite (default) + postgres (feature-gated) behind one trait. The deployment economics this records: a downstream node (git server, vfs) classically needed three storage systems — kv + large content + relational — and this design collapses the three to two: one relational engine + large-tier storage, with the engine a per-node constructor choice. Resolved from evidence in hand (POCs #3/#5/#6); the would-be "deployment trigger" framing was caught at review as circular hedging (a deployment running alkblobs cannot pre-exist the engine) and closed on the OQ-09 precedent.
• ADR-008 (2026-10-03) repairs the seams ADR-007 stressed, against the pinned consumer requirement REQ-2 (requirements.md — a replicator node may run a fleet of store instances sharing one pg pool, a planning fact predating all POCs): the Backend trait gains a size probe (before any backend ships data); fleet GC gets DB-backed pins, an advisory-locked single sweeper, and SQL-level delete arbitration (single-node postures unchanged); and the large tier becomes engine-selectable (local default; pg-lo named candidate, admission-gated on POC #7). It also adds requirements.md — the REQ anchor that ends per-session re-derivation of consumer facts.
• ADR-009 (2026-10-03) admits pg-lo — postgres Large Objects as the large tier's second engine (feature pg-lo, default-off) — on POC #7's passed gate (docs/research/poc-pglo-findings.md): contract 10/10 exact-count tests, durable put ≈ fs durable put at ≥1 MiB, descriptorless lo_get window gets, transactional LO lifecycle (zero crash orphans), companion-table contract authority. REQ-2's fleet consolidation option is now shipped: a fleet node can run postgres-only. Named deltas: cached-get 20-50x behind page-cache fs single-stream (aggregate serving ~700 MB/s at 16 readers), catalog space reused-but-never-returned, autovacuum inherited. The admission door (ADR-003, exercised by ADR-007 and ADR-009) has now shipped both its tier-extending engines.
• ADR-010 (2026-10-03) pins the Backend trait's I/O seams — the last unpinned signatures before implementation: get returns a crate-internal read cursor (kv engines materialize; large-tier engines stream — local's pread loop, pg-lo's lo_get windows), put has two named forms (whole-value for kv, staged put for large — stage-then-commit-rename promoted from behavior to contract); rules that GC state is store-core-owned (backends never learn liveness — ADR-005 verbatim) hosted by SQL-backed engines via a crate-internal contract-tested engine-state companion seam (schema/pin/window/lock clauses), rules the store core holds the joint entry+pin transaction (correcting ADR-009 §2's ownership statement), adds the kv tier's fleet-validity rule (mirror of ADR-008 §3 — postgres is the fleet engine; a sqlite-fleet posture is not on offer), resolves the ops pin-token TTL race (the token is the pin; renewal rides have/re-put; pause-past-TTL falls to delete-then-recover), and pins the feature graph (pg-lo does not imply postgres).
• ADR-011 (2026-10-03) pins the vocabulary: tier (contract + routing class — exactly two) vs engine (concrete impl — exactly five: kv's sqlite/postgres/mem, large's local/pg-lo); mem demoted from "backend"/"testing tier" to the kv tier's third engine — the contract-reference engine; store instance / node / fleet split so REQ-2's "one node may run multiple instances" phrasing is no longer self-contradictory (fleet = pool-sharing, not node count); and the large tier is renamed large (feature fs → large) — its routing criterion is length, not medium — with constructor modes (dual-tier default, kv-only, mem-only) pinned in the same ADR's table.
• ADR-012 (2026-10-03) is the pre-decomposition consistency round: the full-tree review found five composition defects and this ADR rules each — sweeps are never errors (Ok(SweepReport) with aborted: Option<GcAbortCause>; GcAborted retired from the error enum, direct-delete refusal renamed GcRefuse; NoLivenessSources added as the third cause variant); the pin token gains its wire shape (blobs/put response {token, digest}; blobs/have's token-renewal form; token validity domain = the minting serving node); fleet mode is an explicit constructor declaration (fleet: true, never inferred from engine choice); all fleet GC state hosts on the fleet's kv engine (postgres) — one arbitration domain — with large=pg-lo fleet nodes required onto the same pg instance (the composite predicate's new clause) and large=local fleet puts pinned pin-row-first, publish-second; and the engine-state seam's sqlite rows are repaired (pin/sweep-lock bodies dropped; window staging for non-SQL engines is in-process). The tree is now single-valued on every implementation-facing signature.
• ADR-013 (2026-10-11, the composition round): with the sibling substrate alkstore shipped (v0.1.0 — the reactive coordination crate the first consumers co-deploy), this ADR rules the composition so nothing waits on anything downstream or upstream: the core keeps no alkstore dependency and no reactive machinery (notify/scheduler/queues/invalidation hooks are non-goals — cadence stays embedder-owned per ADR-005, with alkstore the named deployment-layer substrate for it); the fleet sweeper's single-flighter is pinned as named exclusive sweep lease semantics (shipped provider: the pg advisory-lock clause — provider swaps are new-ADR substitution doors); co-tenancy and no-cross-store-atomicity are ruled as the downstream-facing composition contract (consumer liveness tables co-tenant the fleet host; compensation rests on idempotent re-puts, windowed deletes, sweeps-as-report-data); and OQ-11 is resolved — engine features pull their own driver clusters (sqlite default-on carrying rusqlite; kv/large tier features retired; local ungated), so pg-pure builds are supported non-default configurations. REQ-5 records the composition facts; the parked index is now empty.

Architecture Documents

Document	Scope	Status
overview.md	Purpose, consumers, layer map, dependency posture	stable
requirements.md	Consumer requirements REQ-1..5, pinned vocabulary (tier/engine/instance/node/fleet)	stable
hashing-and-keys.md	Canonical hash, key encoding (the one-way door)	reviewed
store-api.md	Store facade: put/get/stat/range/pin/batch/sweep surface, errors, invariants	reviewed
backends-and-dispatch.md	Backend trait contract, tiers + engines table, size-threshold dispatch, constructor modes, fleet-validity	stable
gc-and-namespaces.md	Pooled CAS, liveness seams, mark-and-sweep, delete windows	stable
ops-surface.md	alkcall-backed have/need + fetch/put ops, ACL mapping	reviewed
open-questions.md	Centralized OQ tracker (incl. promoted Phase 0 register)	draft

Decision Records

ADR	Decision	Status
001	Substrate posture & ops-surface placement (conformance, not deviation)	Accepted
002	Canonical hash + key encoding (the wire-format ADR; precedes any consumer)	Accepted
003	Backend contract, shipped backends, dispatch policy	Accepted
004	Exactly two backends; manifest layers stay above	Accepted
005	Pooled CAS, liveness seams, mark-and-sweep, delete windows	Accepted
006	Whole-blob verification; transfer encoding excluded by scoping	Accepted
007	Two kv engines (sqlite + postgres) behind one trait; constructor-selected	Accepted
008	Trait size probe; fleet GC (DB-backed pins, advisory-locked sweeper); large-tier engines	Accepted
009	pg-lo — postgres Large Objects as the large tier's second engine (admitted on POC #7)	Accepted
010	Backend I/O seams (read cursor, staged put); GC state store-core-owned via the engine-state seam; kv fleet-validity; pin-token renewal	Accepted
012	Pre-decomposition rulings: sweep aborts are report data (GcAborted retired); pin-token wire shape; fleet-mode constructor declaration; GC state hosts on the kv engine; sqlite seam reduction	Accepted
011	Vocabulary: tier/engine/instance/node/fleet; mem joins kv; large tier renamed large; constructor modes	Accepted
013	Family composition: no sibling dependency in-core; sweep lease; co-tenancy & no-cross-store-atomicity; OQ-11 resolved (engine features pull own drivers)	Accepted
014	Marked-pub put contract: data plane = the adopted channel (zero call.published); one-response synthesis (ack-in-result); the alkcall amendment-4 gate; no-stub posture	Accepted
015	Local engine durability: fsync-durable publish (file + dir fsync); no-replace publish; error-on-truncation; no constructor knob (OQ-12 resolved)	Accepted
016	pg-lo co-tenancy: one large-object consumer per database; orphan_reap_permitted declaration gates reap_orphans (OQ-13 resolved)	Accepted
017	Ops authority model: CAS global pool — namespace grant is capability-to-ask, not data partitioning; renewal namespace parity; the remote-putter role is read+write (OQ-14 resolved)	Accepted
018	The git-object derivation family: registry 0x03/0x04, preimage-identity hashing over the loose form (ADR-002's annotated exception); the per-family stat semantic; fleet threading part of the contract. Its §Decision 2 (blob-family wire scoping) and §Decision 6 (put_git_object shape) superseded by ADR-020 (OQ-15 resolved)	Accepted
019	Token-registry admission posture: the mint boundary's cap (registry-entry bound, refusal not eviction; Error::RegistryFull; the declared MINT_REFUSED put error over a committed put); the non-fleet custody posture documented with the POC-L attack shape; the non-fleet TTL form rejected (OQ-16 resolved)	Accepted
020	Family-explicit payloads under the closed op set: the wire speaks keys (tag-prefixed key hex, bare hex dead); the delivery-form asymmetry (wire put offer key + len mandatory; put response echoes the key); one derivation-explicit put seam (put_as, no canonical default, no shim); reads unchanged by principle; the invariant statement (OQ-20 resolved)	Accepted

Open Questions

Tracked in open-questions.md. The Phase 0 register (OQ-BL-01..06) is promoted there with its resolutions; two questions remain externally-owned — OQ-07/OQ-08 (alkgit seam mapping, alkfs intake; carried for visibility, gating nothing here) — and the parked index is empty: OQ-09 (namespace-visibility default) resolved closed-by-default; OQ-10 (second kv engine) resolved by ADR-007; OQ-11 (the pg-only-kv feature graph, once deferred(scope) on a "named deployment requirement from outside") resolved by ADR-013 §4 — its deciding fact already existed in-tree (REQ-2's consolidation posture; ADR-009), the caught example of the circular form. The pg-lo large-engine question resolved to sequenced work, not a parked question: REQ-2/ADR-008 named it the candidate large-tier engine and POC #7 (docs/research/poc-pglo-spec.md) was its admission evidence — a task, not a wait; that task completed 2026-10-03 (docs/research/poc-pglo-findings.md, passed) and ADR-009 shipped the engine. The review-002 register resolved 2026-10-17 (OQ-15 → ADR-018; OQ-16 → ADR-019); the review-003 register resolved 2026-10-18 (OQ-20 → ADR-020 — the family wire gap; its implementation half is decided-but- sequenced work, gated on the ADR). Deferral policy (the "Schrödinger's code" rule): a decision this crate needs before shipping may not be deferred on a dependency that is itself waiting for this crate to exist. Legitimate parked kinds: deferred(scope) on a deciding fact that exists independently of this crate, and externally-owned questions (how a consumer maps onto this crate) that gate no decision here — full definitions in the header of open-questions.md. A third category is not a parked kind: decided-but-sequenced work (a resolved decision whose admission gate requires evidence — the pg-lo round's POC #7 is the worked example; a task with an owner, not a parked question). A corollary worth restating, because it recurs: a fact this crate must create (its own first deployment, its own first consumer) is never a deciding input — decisions stand on evidence in hand, and future needs reopen via named requirements, not via waiting. (Corollary applications so far: OQ-09, OQ-10, and OQ-11 — ADR-013's audit round — all resolved on it.)

Lifecycle

Spec docs: draft → reviewed → stable → deprecated. A doc moves reviewed when every crate-owned open question it references resolves and an architecture review pass clears it — externally-owned OQs (OQ-07/OQ-08) gate nothing here by definition and do not block the lifecycle; their outcomes arrive through their owners' processes. stable when implementation verifies against it; deprecated when superseded (kept for reference). ADRs use a separate status set (Accepted | Proposed | Deprecated | Superseded), defined per ADR file. The release-facing spec docs (overview, requirements, hashing-and-keys, store-api, backends-and-dispatch, gc-and-namespaces, ops-surface) flipped stable 2026-10-16 at the wave-12 pre-release review gate — eleven waves of implementation is the verification that rule names. On 2026-10-18 the three specs ADR-020 revises (hashing-and-keys, store-api, ops-surface) re-flipped to reviewed — the ADR is the review; they return to stable when the ADR-020 implementation wave verifies against them. open-questions.md stays draft as a living tracker.