alkblobs/docs/architecture/backends-and-dispatch.md

status, last_updated

status	last_updated
draft	2026-10-02

Backends and dispatch

What this is

The physical storage layer: the Backend trait contract every backend implements, the two shipped backends (kv, fs), and the size-threshold dispatch that routes between them. The trait is the crate's second stable seam (with the key encoding, ADR-002): backends must survive digest-layer evolution, so the boundary stays dumb.

What the Backend trait is (and is not)

The trait is a CAS-tier contract: it accepts, serves, and enumerates immutable content-addressed entries (hash → bytes), and its list()/ delete() shapes exist to feed GC (ADR-005). That is its whole job, and the contract assumes it: complete-list-equals-sweep-safety only makes sense for a store where every entry is a standalone immutable blob.

It follows that the trait is not a general-purpose storage abstraction, and a downstream must never implement it to host data that is not content-addressed blobs — a manifest table, a queryable index, or any mutable/structured state. Such storage is consumer-owned, beside the pool (a downstream's own sqlite/whatever, ADR-004): it needs no digest-byte addressing, no list()-completeness, and no sweep participation, so the CAS-tier contract fits it worse than it fits any purpose-built option. The wrong turn to rule out is "my data isn't blobs, therefore I implement a Backend for it"; nothing a consumer needs is ever reachable through that door.

The Backend trait contract (ADR-003)

• Opaque byte keys, opaque byte values. Backends never learn what a digest is; typed Key converts at the store boundary (ADR-002). This is also the persistence boundary — a kv row or an fs filename has no schema-migration story, so the byte layout must be self-describing (algorithm-tagged keys).
• Methods: has / get / put / delete / list / name. Streaming put/get shapes ride the store layer's seams (store-api.md); backends provide byte- or handle-level primitives underneath (get on the fs tier yields a file handle, not a loaded buffer).
• list() complete by contract. A malformed list is as deadly as an incomplete one: list correctness is only observable through GC (POC #1 finding 2 — the redb key-vs-value trap produced a sweep that deleted the wrong blobs, silently). Therefore:
  • every backend implementation must prove list correctness through sweep-outcome tests (the invariant's test gate lives in store-api.md);
  • an implementation that cannot enumerate (the rudolfs S3 anti-lesson) must not ship — GC is a structural requirement, not an optional extra (ADR-005).
• Virgin-store reads are no-ops: read paths treat a missing table as absent/empty (POC #1 finding 2).
• Namespace-blind: backends never see namespaces, tenants, or reference structure (ADR-005 — the rudolfs inversion; physical storage is hash → bytes flat).

Shipped backends

Two, exactly — this counts trait implementations this crate ships, the CAS tiers the dispatch routes between; it is the complete set the problem requires, and the "third backend" fear is a category error fixed in ADR-004. It says nothing about other storage existing in a deployment: a downstream runs whatever else it needs on its own media, beside the pool, above the store's seams (see "What the Backend trait is").

kv backend (feature kv, default-on; shipped engine: sqlite)

Small blobs — kv is the tier name; sqlite is the pinned shipped engine (ADR-003's consequences carry that trade: sqlite's on-disk format compat is upstream's guarantee). The shipped trait impl writes plain content-addressed rows into it, nothing else. "Pinned engine, not a kv abstraction" does not open a substitution seam inside this tier: a downstream wanting a different engine there would be new ADRs (a new impl, its own sweep-safety proof); and a downstream's other storage — schemas, manifests, whatever it likes including some other sqlite file — never comes near this tier at all, per "What the Backend trait is".

Evidence (POC #3 finding A4, first-party measured): sqlite is ~9-10× faster than fs at 1-16 KiB (the git small-blob regime — most git objects, workspace files, manifests), with the crossover at ~128-256 KiB where fs stops paying the B-tree row rewrite and wins.

• Bounded reads; has as an EXISTS probe; prepared statements.
• WAL + synchronous=NORMAL as the shipped durability tier (matching what the benchmark measured and what iroh's store ships).
• Read paths tolerate the no-tables-yet database (see contract).

fs backend (feature fs, default-on)

Large blobs. Flat sharded layout: {hex-prefix}/{hex-prefix}/{hash} sharding survives from iroh's conclusion (limits directory size on huge pools); stage-then-commit-rename for the two-pass unknown-length path; pread-based range reads (POC #3 finding A2 — local range serving is sound, e.g. for packfiles).

mem backend (feature mem, default-off)

BTreeMap-shaped ephemeral backend for tests and in-process ephemerality. A testing/utility tier, never a production story.

Size-threshold dispatch (ADR-003)

• Routing is a pure function of content length. Same content ⇒ same length ⇒ same backend; re-puts are deterministic. No content ever migrates between backends — the migration question existed only to patch the unknown-length asymmetry, which ADR-003's pre-threshold buffering eliminates by construction.
• Default threshold: 128 KiB (constructor-tunable). Midpoint of the measured flat zone (POC #3 A4). Re-tuning per deployment media is a constructor parameter, not an API change.
• Get fall-through: small-tier miss queries the large tier (deterministic, cheap — a stat probe).
• Per-namespace or per-tenant backend configuration: rejected (ADR-003 §Consequences — it would re-weld namespacing into the physical layer, the rudolfs anti-pattern ADR-005 inverts).

Where a new backend could come from

The trait is open to future implementations (network stores, S3-like tiers), but nothing in the current consumer set requires one, and the contract is deliberately hostile to half-implementations (complete list(), GC-participating delete). Any future backend is a new ADR carrying its own sweep-safety story. This crate's roadmap is not blocked on one (see open-questions.md — alkfs intake may name needs externally; OQ-08).

Design Decisions

ADR	Decision	Summary
003	Backend contract & dispatch	opaque keys, complete list, pure-function routing, no migration
004	Two backends, no third	scope boundary against manifest-layer absorption
005	Namespace-blindness	backends see hashes only

Open Questions

• OQ-08: alkfs requirement intake may name storage requirements (e.g., durability tiers, sync-friendly layouts) that touch this layer — open, external owner (alkfs Phase 0).

References

• docs/research/poc-trait-dispatch-findings.md findings 1/2/6
• docs/research/poc-largeblob-findings.md findings A2/A4 (+ the re-runnable benchmark harness)
• docs/research/iroh-blobs-eval.md — the fs-layout conclusions borrowed (sharding, crash ordering, inline thresholds rejected as weld)
• rudolfs notes — list() anti-lesson, decorator alternative noted and not adopted (threshold dispatch chose the simpler policy; ADR-003 §Context)
• ADR-003, ADR-004, ADR-005; store-api.md (the invariants backends must satisfy)