alkgit/docs/architecture/substrate.md

status, last_updated

status	last_updated
reviewed	2026-10-20

Substrate composition: the shipped backend stack

What this is

The architecture of the batteries-included backend — the alkgit-substrate crate's implementation of the five backend traits over alkstore + alkblobs (ADR-019, housed by ADR-025's two-crate workspace; the traits it implements are specified in backend.md). Everything here is the composition: the two substrate crates' surfaces woven into object storage, ref rows, the reactive registry, and GC. The embedder's alternative — implementing the traits over foreign storage — consumes only the protocol core (alkgit, which carries no substrate at any feature level), not this doc.

Evidence base: POC-O (objects/refs/pipelines, poco-findings.md), POC-R (registry/reactivity, pocr-findings.md), POC-X (replication, pocx-findings.md), POC-L (LFS, pocl-findings.md

• lfs.md). The substrate crates in their current published shapes (alkblobs 0.2, the alkstore family 0.1) are the reference.

Feature pin (a wave-6 wiring note, 2026-10-22): alkblobs' alkcall surface rides its optional ops feature (default = [mem, sqlite] — no alkcall by default; alkblobs 0.2.2's ops requires alkcall = "0.10.1", the review-013-fixed release). The composition's reactive halves and the ops wiring (§registry adapters, ADR-019 §3–5) consume that surface, so the substrate manifest enables it when those land; the object-storage core alone (ADR-018's derivation-family puts, §object storage below) stays alkcall-free.

Object storage: alkblobs as the git object store

• Loose objects under the git-object derivation family (alkblobs ADR-018: registry bytes 0x03/0x04; the stored preimage is git's loose-object form — <type> <len>\0 + content, header inside the value; key = tag + git oid). Verification-on-get re-derives under git's own oid function; dedup-by-construction within the family; external-oid addressing is map-free (Key::from_bytes over [tag] + oid bytes — family-explicit, rejects-never-guesses). The odb layer owns the header it writes (parses it back out on read; stat returns the preimage length, the family semantic — alkblobs ADR-018 §3).
• Object graph access via a custom gix Find fronted by the store (the POC-O SubstrateOdb pattern): implements gix_object::Find + Exists and gix_pack::Find; the location/entry/pack-offset legs return nothing (no packed objects) — the pipeline's FindExt blanket impls pick up the rest. No disk odb exists anywhere on the serving path.
• Packs: generation streams through the pack pipeline (ADR-004, Find-generic throughout — the same composition POC-2 proved against the disk odb). Two serving shapes, both validated:
  1. direct — the generated pack streams through sideband, no storage round-trip;
  2. through-store — the pack is put once (the store's own tier dispatch routes big packs to the large tier), then served through read_range windows (per-window slice digests verified mid-flight); a request-digest pack cache lets repeat fetches ride the stored pack; releases/evictions are pins the next sweep reclaims. Large tier required for through-store serving — a kv-only shape rejects over-threshold packs.
• Ingestion per ADR-020 §2: index into a budgeted host working area (thin bases through the substrate Find), unpack-to-loose into the store (verification-on-put = oid re-derivation — corruption cannot enter), pin bridge holding every object until the ref CAS lands, full-graph connectivity walk over the substrate Find (parents included; gix-fsck's Connectivity::check_commit does not walk parents — the owning implementation walks), working area discarded. Pack mirroring / delta synthesis: designed-not-built (optimization bucket).
• The pin bridge's mechanism and failure semantics (what POC-O built; pinned so the composition cannot improvise): the bridge is the set of put-returned pin guards — one Pin per object put, held by the ingest flow across the ingest → CAS span. Releases are explicit and ordered: guards commit (release) only after the ref-CAS transaction commits; abort drops the guards. Crash/failure is safe by construction: neither the ref CAS nor the registry row has landed before the guards release, so a dropped-bridge crash leaves only sweep-eligible orphans (and CAS-shared entries that other refs already cover) — a failed push is a failed push, not partial repo state. The put side uses the store's batch form (put_in_batch — pin-before-publish ordering inherited from the store's own contract) so the working set lands with one commit discipline. The pin's durability is the process's: alkblobs pins are in-memory custody; a crashed ingest's entries are reclaimed by the next sweep after process restart — recovery is GC-decreed, not session-restored.
• Pin custody across the frozen trait seam: the pinned trait signatures carry no guards (prepare returns PreparedPush; the wire layer then calls apply_updates), so the composition owns the custody internally — the substrate impls of GitPackIngest and GitRefs are constructed together and share a composition-internal pending-pin registry: prepare retains the just-put objects' guards there, keyed by repo_id and correlated by content with the prepared RefUpdate list it produced (content-addressed correlation: each apply drains the pending entry whose update list matches — identical update lists guard identical CAS requirements, so which twin drains is immaterial); apply_updates drains the matching entry and commits its guards only after the CAS transaction commits. The normal failure path (prepare returned, apply never arrives — client disconnect, malformed push between the calls) leaves a stale pending entry: it is pruned on the repo's next apply_updates (stale-entry sweep alongside the drain), and its guards otherwise recover the same way the crash case does — process-lifetime custody, reclaimed by the next sweep after restart. The cost of a leaked guard is bounded (sweep-eligible objects held slightly longer), never partial repo state.
• Object ids are sha1 (OQ-05); the store's GitObjectSha256 family byte is registered — the sha256 policy change stays an ecosystem-wait.

Refs: the co-tenant row table

• Table shape: (repo_id, name) → oid, symref_target; HEAD is a symref row (RefLine's oid: None unborn shape). Name validation per git ref rules + the reserved-namespace deny-list is impl-side (ADR-013 §7's gix_validate::reference usage moves from gix-ref files to the table layer).
• HEAD seeding at record creation (composition-internal, pinned so clones are never checkout-less by construction): the substrate's put_repo seeds the repo's HEAD → refs/heads/main symref row in the same row-write transaction as the registry record (git parity — git init --bare creates the (dangling) HEAD at init, not at first push; git's push grammar carries no HEAD, and the op layer is core-side, so the record-write is the one moment the composition owns both surfaces). V1 pins the target refs/heads/main; a constructor knob is additive later (two-way — the row is CAS-rewriteable). The dangling symref is exactly the unborn-HEAD shape ls-refs=unborn serves (OQ-19's rider (a) fixture falls out of this rule).
• apply_updates = all CAS checks + applies inside one transaction (BEGIN IMMEDIATE-class on sqlite): stale/atomic rollback/delete/create/force verified by POC-O against real git push behaviors. This is what makes atomic correct (ADR-013 §7's one-transaction home, now row-implemented).
• No file locks, no gix-ref file storage, no HEAD symlink mechanics — the last disk dependency of the serving path is gone (ADR-019 §2).

The registry: co-tenant tables behind the family adapter pattern

The alkstore-shipped non-goal is consumer row storage (alkstore's TxHandle carries no consumer SQL), so alkgit owns its tables and connections, co-tenant on the same engine instance (alkblobs ADR-013 §3's composition posture) with the family adapter pattern (the alkcall ownership template — POC-R's validated shape):

• Read side (sync): resolve — the advertisement path's hot read — rides the cached provider below; a miss on the full-table snapshot is a definitive unresolved id (the snapshot is the whole table, so there is no fall-through I/O and no partially-populated cache); the unknown-repo ≡ unauthorized rule (ADR-008) is evaluated at resolve's caller, exactly as before.
• Write side (async, #[async_trait]): put/update/remove — each bumps the monotone registry_seq counter inside the row-write transaction (transactionally exact; the watermark's anchor), then announces post-commit (the reactive halves).
• In-memory default adapter: the batteries-included degenerate case — always-populated, always-correct, single-process; the completeness shape in the crate's base (no engine dependency — no mem feature exists; ADR-025's placement rule).
• Engine twins: one trait pair, two implements — the sqlite adapter (sqlite feature, alkstore-sqlite + the co-tenant rusqlite connection) and the postgres twin (postgres feature). The reactive composition code is engine-generic over the alkstore Store trait and ran unchanged over all three engines (POC-R).
• Schema migrations (ADR-024): both engine adapters migrate their co-tenant tables at construction — ordered monomorphic steps per twin, one transaction per step, concurrent-open-safe (the alkstore race-verification pattern), loud failure. One alkgit-owned stamp table versions the co-tenant schema as a unit; additive-first is the post-publish discipline (an unknown-forward stamp is safe to open against; a non-additive step is major-version + its own ADR). The in-memory adapter builds fresh every construction — stamp-free.

The reactive halves (wired by default — ADR-019 §5)

Per record write, after the durable commit, independently and fail-soft:

1. Publish → the durable change feed (alkstore stream, keyed by repo id) — per-consumer durable cursors, monotone saves, replay from head; per-seq idempotency is the consumer's rule (keyed publishes are metadata, not partitioning/dedup — the alkstore contract).
2. Notify → the wake hint (alkstore notify/listen) — the invalidation trigger for every cached provider.
3. Enqueue → the maintenance queue — housekeeping jobs (sweep/pack work) riding the scheduler chain.

The reconcile sweep closes any lost post-commit effect: rows_since(applied_watermark) → snapshot install → watermark advance — idempotent by construction, driven by the scheduler tick (and callable on demand). The crash-window shape (row committed, effects suppressed) closed in one pass when demonstrated (POC-R). The per-push watermark leg: pushes do not modify RepoRecord, so the receive flow anchors the event/watermark legs with a composition-internal per-repo mutation-marker write (the "registry put" of POC-R's receive-flow recipe; ADR-019 §7) — this is what drives diff-driven replication's prompt passes without record-change noise.

The sqlite/postgres asymmetry is designed-in and carried by both the composition and the docs:

• sqlite: the co-tenant file is itself the invalidation bus — alkstore's watcher data_version poll fans a wake to listeners on any commit to the file, including alkgit's own registry writes on their own connection, including cross-process (~3.2 ms measured for an unannounced co-tenant write). A lost hint cannot exist while any write lands; the reconcile sweep is a correctness backstop, not a latency bound.
• postgres: no file-level bump exists; LISTEN/NOTIFY needs the explicit post-commit notify, and a lost/failed hint leaves caches stale until the reconcile tick — the tick's cadence is the real staleness bound on postgres.

The coherence contract (the cached read provider; ADR-019 §4)

• Cached set: the whole record table as one immutable snapshot (an Arc map swapped under a short lock; whole-table by contract — ADR-019 §4).
• Hot path: one lock read + Arc clone + lookup; zero I/O on the steady state.
• Invalidation: alkstore wakes (the postgres notify hint or the sqlite file-wide bump), the provider's own writes (write-through), the reconcile tick.
• Staleness bound: mutation visibility ≤ one wake delivery + one refresh; on sqlite milliseconds (measured); on postgres the reconcile cadence after a lost hint.
• Failure posture: refresh failures leave the previous snapshot serving; the next wake, write, or tick retries.

GC: the wired maintenance chain

The substrate composition wires GC as a chain of alkstore mechanisms feeding alkblobs (POC-R's validated cross-process run):

schedule(@every) → alkstore leader-elected scheduler
  → firing lands in the maintenance queue
  → worker claim (visibility-timeout discipline)
  → named-TTL leader lock ("sweeper" — cross-process exclusion;
       a locked-out sweep is a skip + short retry, a value not an error)
  → alkblobs Store::sweep() -> Result<SweepReport, Error>
       (the report is the GC outcome carrier — stages, deletes,
       aborts; `Err` is substrate-level failure, not a GC verdict)

• Liveness sources (alkgit-owned, in alkgit-substrate's base):
  1. the ref-table closure — every ref's full reachable graph walk over the substrate Find (this is what makes orphaned and delete-uncovered objects reclaimable: liveness, not timestamps, decides);
  2. the LFS media rows under the lfs feature (ADR-022 §4 — cross-reference sharing requires re-protection while any row references a key; the refcount column ships in the initial schema — ADR-024 §5).
• Sweep-window semantics are alkblobs' (stage → delete across the sweep window); pins from an in-flight push hold objects across the window (the pin bridge, ADR-020 §2).
• Repo deletion (the op or replication-side) removes registry + ref rows; the store's sweep reclaims the uncovered objects — no bulk delete exists.

The receive-pack flow, composed

The full push path, as the pieces above compose (each durable surface in its own transaction; no cross-store machinery):

pack stream → budgeted index (thin bases via substrate Find)
  → unpack-to-loose into alkblobs (pinned; ADR-020 §2)
  → connectivity walk (full closure)
  → ref-table CAS (one transaction; ADR-019 §2)
  → per-repo mutation-marker write (the composition-internal row that
      bumps the seq/watermark on every push — pushes do not touch
      `RepoRecord`; ADR-019 §7)
  → post-commit announce: publish + notify + enqueue (fail-soft each)
  → band-1 status report

• Crash windows: pinned objects survive sweeps until the CAS lands and pins release (the object window; POC-O's verified closure); the reconcile sweep closes the event window (POC-R's recipe).
• The bounded pack-header peek (packless deletes — ADR-020 §1) sits at the wire layer, before this flow; packless pushes enter at the ref-CAS step.
• Client-side (the consumer half's push contract): a packless push keeps its write side open until the report — half-closing turns the server's bounded peek into an EOF error instead of the packless timeout (POC-X's measured fact; ADR-020 §3). GitSession's push must hold the stream open across the report wait.

Assembly shapes (the constructor-level matrix)

Engine choice is one assembly-level statement (alkstore ADR-016's deployment honesty) — three shapes, zero assembly code:

Assembly	alkstore engine	alkblobs constructor	registry adapter
Single-host (the minimal real setup)	alkstore-sqlite (open(path, SqliteOpts))	sqlite shapes (kv_only_sqlite; dual_tier_sqlite_local when LFS or through-store packs)	sqlite twin
Fleet / durable-shared	alkstore-postgres (PgStore/PgOpts, schema namespaced)	postgres shapes (kv-only / +dual-tier local / dual_tier_postgres_pglo)	postgres twin
Ephemeral / tests / completeness	alkstore-mem	mem shapes (mem_only, kv_only_mem, dual_tier_mem_*)	in-memory default adapter

• The sqlite+pg-lo combination is unconstructible, not merely discouraged (their constructors have no such shape; a sqlite fleet is a recorded alkblobs non-option).
• :memory: sqlite confers no co-tenancy (per-connection databases; alkblobs' sqlite rejects memory mode — WAL cannot engage): when the whole assembly is ephemeral, compose the mem engines, never sqlite-in-memory (DQ-7's ruling).
• Connection budget (the worst-case assembly datum): alkstore postgres pool + its dedicated LISTEN connection; alkblobs kv pool (+ a second pool in pg-lo dual shapes); alkgit's registry connection — four pools per process, documented here so an assembly sizes max_size with it in view.
• Tuning datums for sqlite assemblies (POC-O/POC-R measurements; assembly-level knobs, not structure): the 1 ms watcher poll cadence wakes on any commit (file-wide fanout; self-traffic machinery ~ 5 refreshes/s at defaults — raise the cadence on busy assemblies); single-writer serialization across all co-tenant connections — receive-pack latency measured indistinguishable with the full machinery co-tenant (5 ms medians); SqliteOpts::{max_readers, poll_interval} are the knobs; busy_timeout is pinned (5000 ms — contracted at both substrate crates since the wave-14 ask landed).

Public API surface (composition-shaped types)

Crate-root re-exports of the alkgit-substrate crate (backend.md §public API carries the per-crate partition) — this list is the freeze inventory ADR-025 names:

• the registry adapter trait pair + engine constructors + the in-memory default adapter;
• the cached read provider; the reactor (announce/wake-runner/stream- consumer/reconcile) — construction wired by a single composition entry that takes the substrate engine handles (the assembly-level one-statement engine choice);
• the five backend-trait implementations — the registry pair's concrete adapters plus the object-side trio (GitRefs/GitPackGen/ GitPackIngest impls): public by necessity, the assembler constructs the backend from them (public API = authorized-surface rule: the op and session layers reach them only through the core's traits); including the substrate Find/odb front (SubstrateOdb-class — constructed once, per-session views; the handle-sharing shape);
• the receive-pack composition (unpack-to-loose + pin bridge + the pending-pin registry) — internal to the trait impls, not a public member;
• the GC liveness sources; the LFS liveness/pointer-lifecycle surface under lfs.

The exact Rust names are the implementation task's to pin under OQ-03's inventory (this doc fixes the members, not the identifiers).

Design Decisions

ADR	Decision	Summary
019	Substrate re-baseline	objects/refs/registry on the substrate; reactive-by-default; coherence contract; GC wiring
020	receive-pack duplex	packless deletes (bounded peek); unpack-to-loose + pin bridge
021	Feature graph	superseded by ADR-025 (graph re-partitioned)
022	LFS	identity-family media keying; alkhttp door; wired liveness rows
025	Two-crate workspace	this composition is alkgit-substrate; the placement rule
023	Replication v1	protocol-native via GitSession; this composition's stream/watermark recipes carry it
004	Pack pipeline	gen/ingest shapes (Find-generic, unchanged in pipeline)
013	receive-pack	the state machine (§3/§6 amended: ADR-020)
018	Trait signatures	the seam this composition implements (zero-drift record)

Open Questions

• OQ-19: implementation-phase wire/odb validations (unborn-HEAD rider; common_haves over the substrate odb — POC-X's NAK-only caveat).
• OQ-05 (sha256) rides alkblobs' registered family byte; unchanged.
• OQ-17 resolved: schema migrations are construction-run, version-stamped, additive-first (ADR-024 — the registry section above carries the composition text).

References

• backend.md (the trait seam + schemas this implements), lfs.md, transport.md, doors.md (assembly section)
• docs/research/poco-findings.md, pocr-findings.md, pocx-findings.md, pocl-findings.md — the four evidence records
• docs/research/substrate-recomposition.md (the re-baseline doc; the DQ ledger)
• ADR-019..023 (the re-baseline ADR package — all rationale)
• alkstore ADR-016 (deployment honesty — the assembly-level engine statement); alkblobs ADR-013 (§3 co-tenancy composition contract, §4 feature-graph retirement), ADR-017 (CAS global-pool caveat), ADR-018 (the git-object derivation family), ADR-019/020 (mint admission cap; pre-keyed wire put) — sibling-crate decisions, always cited with their crate prefix in this doc set