alkblobs/docs/architecture/overview.md

status, last_updated

status	last_updated
draft	2026-10-03 (ADR-010/011 — I/O seams pinned; vocabulary fs→large, mem joins kv)

Overview

What this crate is

alkblobs is content-addressed blob storage in the alk* family: a put/get/verify store keyed under one canonical content hash (the git-family derivation; legacy SHA-1-tolerant), with pluggable tier engines (small blobs in a kv store per ADR-007/011, over-threshold blobs on the large tier per ADR-008/009 — the tier formerly named fs), split from any wire/ops-protocol layer. Network ops ride the alkcall substrate behind a feature-gated ops module (ADR-001).

Deployment-relevant consumer facts — what each use case forces on this crate — are pinned in requirements.md (REQ-1..4, with the node/pool/fleet/engine vocabulary defined there). This tree references REQs by ID; per-session re-derivation from vision prose is exactly what that document exists to prevent.

Why it exists — the extracted shared core

Two downstream projects were converging on the same storage stack, so it is being built once, here:

• alkgit was paused mid-Phase-1 (its architecture is reviewed, docs/architecture/ in the alkgit repo) to stop duplicated storage-layer work. Its object backend needs git's own oid derivation, a better small-object backend than its proposed default file layout, and a git-lfs-shaped large-blob story. Under the canonical-hash resolution (ADR-002) git objects are first-class pool entries with zero indirection.
• alkfs (Phase 0 pending) needs the appfile/workspace shape: small content in a fast kv store, large content on the filesystem, path→hash mapping above. The path-tree/manifest layer is alkgit's and alkfs's, not this crate's (ADR-004).

Both demanding consumers (p2p git replicators, agent workspaces) reduce to the same shape: a pooled CAS per node, where repos and workspaces are sets of hash references, not walled-off object stores — cross-repo dedup by construction (ADR-005). That pooling property is why the storage layer was pulled out of both consumers rather than built twice or three times.

Layer map

┌────────────────────────────────────────────────────────────┐
│ consumers (above this crate)                               │
│   alkgit object storage · alkfs path trees/workspaces      │
│   refs, manifests, path→hash maps, git semantics           │
└────────────┬───────────────────────────────────────────────┘
             ▼
┌────────────────────────────────────────────────────────────┐
│ ops surface   (feature "ops"; ADR-001)                     │
│   have/need + verified fetch + put; JSON control ops,      │
│   binary data channels; ACL via alkcall AccessControl      │
└────────────┬───────────────────────────────────────────────┘
             ▼
┌────────────────────────────────────────────────────────────┐
│ store core    (always on)                                  │
│   hashing & keys (ADR-002) · put/get/stat/range/pin seams  │
│   mark-and-sweep GC + liveness registration (ADR-005)      │
│   size-threshold dispatch (ADR-003); GC state owned        │
│   here, hosted by SQL engines via the seam (ADR-010)       │
└───────┬───────────────────┬───────────────────────────────┘
        ▼                   ▼
┌──────────────────────┐  ┌─────────────────────────────────┐
│ kv tier (ADR-003/    │  │ large tier (ADR-003/008/011;    │
│ 007/011)             │  │  formerly "fs")                 │
│   sqlite (default)   │  │   local (default)               │
│   postgres ("postgres")  │   pg-lo ("pg-lo"; ADR-009)   │
│   mem ("mem"; tests +     │                             │
│    contract reference)    │                             │
└──────────────────────┘  └─────────────────────────────────┘
        ▲                   ▲
        └── one Backend trait (ADR-003/010: read cursor +
            staged put I/O seams); engine = per-node
            constructor choice (ADR-011's table)

   transport: none, anywhere. alkgit/alkfs or the embedder dials;
   an alkcall Connection is handed in at the ops boundary.

Dependency posture

Layer	Dependencies	Notes
base crate (store core + tier engines)	tokio, thiserror, parking_lot, plus rusqlite behind the kv feature and nothing behind large/mem	no alkcall, no serialization frameworks
postgres engine feature (ADR-007)	adds tokio-postgres + deadpool-postgres behind the kv tier's second engine, default-off	depends on kv's contract surface (the Backend trait definition is base-crate code); the engine's impl is additive. Same Backend contract, own sweep-safety CI gate under --all-features
pg-lo engine feature (ADR-009)	Large Objects as the large tier's second engine, default-off	pulls the same tokio-postgres/deadpool-postgres cluster as the postgres kv engine (SQL lo_* functions, no additional dependency); does not imply the postgres feature — engine features pull the driver cluster independently (ADR-010 §1), so a large=pg-lo + kv=sqlite node is coherent and supported; same admission/test-gate shape
mem engine feature (ADR-011)	the kv tier's ephemeral engine, default-off	BTreeMap-shaped; the contract-reference engine; no dependencies
ops feature	adds alkcall (feature-gated, default-off)	JSON op specs + binary channels + AccessControl; ADR-001
consumer crates	depend on alkblobs (and on alkcall directly when they speak ops)	alkgit, alkfs

Tiers are feature-gated; both default-on (batteries-included, the alkgit feature-model pattern — default-on features, opt-out rather than opt-in); the engine features postgres and pg-lo, and mem, are default-off, as are ops. A pg-only kv node (postgres engine without rusqlite) is not a supported configuration this crate optimizes for — feature-graph work for it is deferred until a named requirement exists (the base kv feature pulling rusqlite is the accepted v1 shape; convention 8's lean-base principle applies to default builds, which stay lean since kv is what the feature model controls, not what default-on means downstream). Wasm: the tier engines are not a wasm story; a wasm client is a consumer of the ops surface (which rides alkcall, itself wasm-clean), never an in-process embedder (ADR-001 §Consequences).

Non-goals (each enforced by an ADR)

• No wire framing, op dispatch, ACL, or transport inside the store layer (ADR-001).
• No adopted iroh-blobs wire surface — no tickets, no postcard, no provider protocol (AGENTS convention 7; ADR-001 §Context).
• No namespaces-as-physical-storage; the pool is flat, namespaces are consumer-level reference sets (ADR-005).
• No manifest/path-tree/vfs tables; consumers never implement a Backend to model paths (ADR-004).
• No chunk-tree/CDC storage encoding; whole-file CAS is the store's only granularity (ADR-006).
• No per-repo or per-namespace walled stores — one pooled CAS (ADR-005).

Design Decisions

ADR	Decision	Summary
001	Substrate posture & ops placement	store stays substrate-free; ops ride alkcall feature-gated
002	Canonical hash + key encoding	git-blob-sha-256 canonical; algorithm in-key; one-way door
003	Backend contract & dispatch	two backends, size-threshold routing, no migration
004	Two backends, no third	manifest layers are consumer-side
005	Pooled CAS & GC	flat pool, registered liveness, delete windows
006	Verification posture	whole-blob verification; chunk trees excluded by scoping
007	Two kv engines	sqlite default + postgres behind one trait; one relational engine + large tier per node
008	size probe + fleet GC + large-tier engines	trait length probe; fleet pins/sweeps engine-backed; the large tier (now large) engine-selectable
009	pg-lo large engine	postgres Large Objects as the large tier's second engine, admitted on POC #7; a fleet node can run postgres-only
010	I/O seams + GC-state home	read cursor / staged put; store-core-owned GC state via the engine-state seam; kv fleet-validity rule; pin-token renewal
011	Vocabulary + rename	tier/engine/instance/node/fleet pinned; mem joins the kv tier; large tier renamed large; constructor modes

Open Questions

Tracked in open-questions.md. Ones affecting the crate level:

• OQ-07: how alkgit's object-storage seam consumes alkblobs (externally-owned: alkgit's architecture process)
• OQ-08: alkfs requirement intake (externally-owned: alkfs Phase 0)
• OQ-09: default namespace visibility in network ops — resolved closed-by-default (deny-unlisted; reopen only on a named open-read requirement)
• OQ-10: second kv engine — resolved by ADR-007 (sqlite + postgres shipped behind one trait; per-node constructor choice)
• POC #7 (docs/research/poc-pglo-findings.md): pg Large Objects as the large tier's pg-lo engine — passed 2026-10-03; admitted as a shipped engine by ADR-009 (REQ-2's consolidation option)

References

• docs/research/phase-0.md — the converged Phase 0 record; inputs to every ADR above
• docs/research/poc-trait-dispatch-findings.md, docs/research/poc-largeblob-findings.md, docs/research/iroh-blobs-eval.md — the evidence trail
• docs/sdd_process.md — the process governing this tree
• /workspace/@alkdev/alkgit/docs/architecture/backend.md — the reviewed consumer-side seam that OQ-07 maps onto alkblobs
• AGENTS.md — family conventions (conventions 4–8 gate the decisions)