alkblobs/docs/research/iroh-blobs-eval.md

---
status: draft
title: "iroh-blobs store eval — against the current checkout"
last_updated: 2026-10-02 (POC #3 reading; checkout e82cbdc, v0.103.0)
---

# iroh-blobs store eval (current checkout)

Evaluated against `/workspace/iroh-blobs` @ `e82cbdc` ("Release 0.103.0",
2026-06-15 — the checkout *is* the 0.103.0 release, so alknet's older
probe claims about "published 0.103" verify directly against it).
Scope: the store layer (`src/store/**`, `src/api/**`, `src/util/**`).
The wire surface (tickets, postcard, provider protocol) is out of scope
per convention 7 — we are not adopting it. This eval feeds OQ-BL-02
(backend shape), OQ-BL-04 (verification/chunking), and the POC #3
large-blob design. GC mechanics were already verified separately
(see phase-0 §Prior art; the gc.rs read predates POC #1).

## The headline: there is no Store trait — and that is instructive

iroh-blobs 0.103 has **no `Store` trait at all**. The store API is a
message protocol: an enum of RPC requests (`src/api/proto.rs:90-139`,
`#[rpc_requests]`-derived `Command`), and each backend (MemStore,
FsStore, ReadonlyMemStore) is an *actor* consuming that command set
over `irpc` channels. The user-facing `Store` struct is just a client
handle; `Store::connect/listen` make the same message enum work over
the network. The abstraction is "same messages locally and remote."

Lessons for alkblobs:

- **The command-enum-as-contract is a working pattern** (it is what
  makes iroh-blobs' remote/local symmetry possible), but it carries the
  whole irpc machinery into the store layer and *is* the reason their
  store is welded to their transport. POC #1's lean `Backend` trait is
  the smaller hammer for our split: the trait stays dumb (opaque bytes),
  and any ops surface above it (alkcall) speaks its own protocol.
- **Re-borrowed conclusion, not mechanism: per-hash serialized actors.**
  FsStore runs one serialized state machine *per hash* (entity manager,
  `src/store/fs/util/entity_manager.rs`) with parallelism across hashes;
  per-hash state is a `watch::Sender<BaoFileStorage>` (cheap snapshots,
  observable progress, idle-recycled actor pool). This is a genuinely
  good concurrency shape for Phase 1's sweep-vs-put story: "one writer
  per content address, readers snapshot-observe" matches CAS semantics
  naturally. iroh's DeleteSet was already our named prior art; the
  per-hash-actor adds a second reusable conclusion.

## What the fs backend actually looks like on disk (0.103)

```
blobs.db                    redb — hash → EntryState (+tags, + inline data/outboard)
data/<hash>.data            complete + partial canonical data
data/<hash>.obao4           outboard (bao pre-order, 64-byte pairs)
data/<hash>.sizes4          per-chunk-group size slots
data/<hash>.bitfield        blake3-checksummed postcard bitfield (coverage)
temp/<uuid>.temp            staging (must be same-device as data/)
```

- **Entry state machine per blob** (`src/store/fs/entry_state.rs`):
  `Complete { data_location: Inline|Owned|External, outboard_location:
  Inline|Owned|NotNeeded }` / `Partial { size: Option<u64> }`. The
  inline/owned split is the small-blob/large-blob dispatch *inside one
  backend*: ≤16 KiB data lives entirely in the redb `inline-data` row;
  ≥~1 MiB is data file + outboard file; between, data file + inline
  outboard. Thresholds are `InlineOptions { max_data_inlined: 16 KiB,
  max_outboard_inlined: 16 KiB }` (`fs/options.rs:71-78`).
- **16 KiB is exactly one bao chunk group** (16 × 1024 B chunks), so
  every blob ≤ 16 KiB has an outboard of size *zero* — a "small blob"
  is one redb row keyed by plain `blake3(content)`, no tree material at
  all (`fs/import.rs:160-221`, `DESIGN.md:71`). iroh couples "hash
  overhead" and "storage tier" at the same boundary; under our git-oid
  derivation there is no outboard to elide (whole-blob digest only) and
the tier boundary is a pure dispatch choice — POC #3's benchmark says
~128-256 KiB, not 16 KiB (see `poc-largeblob-findings.md`, finding A4).
- **Crash-consistency story, borrowed as a conclusion** (`DESIGN.md:
  126-142`, "files are hard"): persist data+outboard+sizes *first*, the
  checksummed bitfield *last*, with truncate-to-zero-on-read marking a
  bitfile dirty under crash. If the bitfield is missing/corrupt, the
  entry re-validates by full local hash check. Lazy persistence (on
  actor idle/shutdown only), never per-chunk fsync. For alkblobs: our
  put path is *stage-temp → hash → one rename*, which needs no coverage
  bitfield at all — we only ever need this machinery if we adopt
  partial/streaming-download entries (range-serve before complete).
  Recorded as a conditional, like bao.
- **Temp-file flow** (<`fs/import.rs`>): byte-stream puts stage to a
  temp file (after a 16 KiB in-memory peek), path-import reflinks-or-
  copies (`reflink-copy` crate) to temp, outboard encodes in a second
  pass, then `fs::rename` atomizes both files before the redb row lands
  *awaiting* the transaction commit. `ExportMode::TryReference` is the
  mirror: rename the store's own file into the caller's tree and mark
  the entry `External`. (TryReference is a neat dedup trick for the
  appfile/workspace consumer — "move into the store without copying" —
  Phase 2 note, not Phase 1.)
- **Metadata batching**: the redb actor batches read commands (10k msgs
  / 1 s window) and write commands (1000 / 500 ms) into single
  transactions (`meta.rs:785-856`) — "the secret to fast download
  speeds is to not touch the metadata database at all" (`DESIGN.md:61`).
  sqlite's WAL gives comparable batching for free at the smaller scale
  our store keys live at; the *conclusion* (batch metadata writes,
  never per-blob-fsync) transfers.

## Verification & chunking (OQ-BL-04 input)

- Fixed 1024 B bao chunks, grouped 16-wide (iroh `IROH_BLOCK_SIZE`,
  `store/mod.rs:17-18`); no content-defined chunking exists in-tree.
- Outboard encoding is a **custom single-pass encoder over blake3
  hazmat APIs** (`src/util.rs:224-336`) — post-order chunk iteration,
  `set_input_offset`/`finalize_non_root` merges, buffered 1 MiB reads.
  iroh wrote their own rather than using bao-tree's `CreateOutboard`;
  the *shape* of "encode outboard while streaming through chunks once"
  is the pattern any chunk-tree encoder follows, bao or CDC.
- **Get verification is subtree-local**: `export_bao(hash, ranges)` runs
  `traverse_ranges_validated` and lazily reads only the parent pairs the
  requested chunks need. Byte-range `export_ranges` is deliberately
  *unvalidated* (bitfield-superset check only; `DataReader` is documented
  "a reader for the unvalidated data file", `fs.rs:337-343`).
- **For our canonical digest this exposes a hard fact** (POC #3 finding
  A2, reproduced independently): the git-blob derivation verifies *the
  whole content* (preamble includes the length). There is no sub-range
  relationship between `blob <len>\0+content` and any slice of it, so
  range reads under our hash cannot be verified against the blob's own
  digest, period. iroh gets range-verification *from the chunk tree*, not
  from the hash algorithm. If ranged transfer ever needs per-range
  verification, the options stay as OQ-BL-04 lists them (caller-carried
  slice digests = transport-integrity only; conditional chunk-tree
  encoding; or backend-native). POC #3's verdict: the local-store case
  doesn't need it (pages fail via EIO, git blobs self-verify on read);
  the *networked large-blob* case is the only real consumer of a chunk
  tree — which matches the "conditional, confined to the encoding layer"
  posture already recorded.

## Small-blob handling & streaming (POC #3 design input)

- **Streaming put exists in three shapes**; none hashes on the fly:
  `add_stream` (bidirectional `ImportByteStream` channel; stages to
  memory then temp file, computes outboard in a **second pass**,
  `fs/import.rs:290-352, 372-419`), `import_bao` (pre-verified items,
  no re-check), and `import_bao_reader` (network decode-verify-push,
  `blobs.rs:446-488` — the provider's push handler). There is **no
  put-from-`AsyncRead` API**; callers adapt to `Stream<io::Result<Bytes>>`.
- POC #3 found the same constraint from our side *before* reading this
  in detail: the git preamble needs the length at hash time, so an
  unknown-length stream *must* stage-then-hash. iroh's architecture
  arrives at the same stage-then-encode shape (for bao-tree reasons
  instead). **The one-pass alternative exists only for known-length
  puts** — which is exactly the git/alkgit dominant case (a staged file
  has a stat; a protocol offer carries a size). Unknown-length remains
  the two-pass shape for both crates; POC #3's finding A1 records why
  (and that the second pass can read from the *staged file*, so it's
  page-cache-cheap, not network-repetitive).
- **Serve-while-persisting is not a bolt-on in iroh — it is the
  download path**: `get_blob_ranges_impl` opens the import *before*
  reading from the network and `tokio::try_join!`s the network decode
  loop with the store import (`src/api/remote.rs:884-944`). POC #3's
  fanout seam (finding A3) rebuilt this at our layer with a broadcast
  channel; the mechanism differs (single reader broadcasting vs their
  per-item verified forwarding) but the conclusion matches: the fetch
  handler should be "decode-verify → broadcast → (store arm + subscriber
  arms)", all driven from one receive loop.
- **`BlobReader`** implements `AsyncRead + AsyncSeek` built on
  `export_ranges` per read call (`api/blobs/reader.rs`); seek-from-end
  is an upstream TODO. Our fs range-read surface covers the same need.

## Performance posture (no in-tree benches; code choices as evidence)

No `benches/`, no criterion in the checkout. The perf story is in
DESIGN.md + code micro-choices: redb write/read batching; partial
entries live entirely outside redb until completion; zero-copy
`Bytes` plumbing end to end; reflink-first import/export; 1 MiB copy
buffers with `yield_now()` fairness; per-hash channel caps bounding
backpressure. The two transferable *conclusions* for us: batch
metadata transactions (never per-blob fsync), and zero-copy/`Bytes`
at the streaming seams. Our own sqlite-vs-fs numbers are in the POC #3
findings (first-party, measured).

## Do-not-inherit list (re-confirmed against 0.103)

- The `Command`-actor + irpc store abstraction (welds store to their
  transport stack).
- redb-specific entry-state modeling if we go sqlite (their tables and
  `EntryState` postcard encoding are theirs; our sqlite schema is
  `key → (value, size)` and our dispatch is store-layer policy).
- Fixed 1024 B chunk trees as the default granularity (principle 7
  already rejects; nothing new in the eval changes it).
- Tickets/postcard/provider — out of scope as always.

## What we keep (conclusions only)

1. Per-hash serialized state machines + snapshot-observation for
   concurrent access (entity-manager pattern).
2. Stage-temp → atomic rename as the universal put commit primitive.
3. Metadata write batching (amortize sync cost).
4. Zero-copy `Bytes` at streaming seams.
5. Serve-while-persisting as the fetch-handler shape (POC #3 validated
   at our layer).
6. Crash-consistency ordering (data first, coverage/index last, with
   dirty-marking) — only if partial entries ever exist.
7. The inline threshold concept, re-sized by our own benchmark
   (~128-256 KiB crossover, git-blob workload) instead of their 16 KiB.