alkblobs/docs/research/iroh-blobs-eval.md

status, title, last_updated

status	title	last_updated
draft	iroh-blobs store eval — against the current checkout	2026-10-02 (POC

iroh-blobs store eval (current checkout)

Evaluated against /workspace/iroh-blobs @ e82cbdc ("Release 0.103.0", 2026-06-15 — the checkout is the 0.103.0 release, so alknet's older probe claims about "published 0.103" verify directly against it). Scope: the store layer (src/store/**, src/api/**, src/util/**). The wire surface (tickets, postcard, provider protocol) is out of scope per convention 7 — we are not adopting it. This eval feeds OQ-BL-02 (backend shape), OQ-BL-04 (verification/chunking), and the POC #3 large-blob design. GC mechanics were already verified separately (see phase-0 §Prior art; the gc.rs read predates POC #1).

The headline: there is no Store trait — and that is instructive

iroh-blobs 0.103 has no Store trait at all. The store API is a message protocol: an enum of RPC requests (src/api/proto.rs:90-139, #[rpc_requests]-derived Command), and each backend (MemStore, FsStore, ReadonlyMemStore) is an actor consuming that command set over irpc channels. The user-facing Store struct is just a client handle; Store::connect/listen make the same message enum work over the network. The abstraction is "same messages locally and remote."

Lessons for alkblobs:

• The command-enum-as-contract is a working pattern (it is what makes iroh-blobs' remote/local symmetry possible), but it carries the whole irpc machinery into the store layer and is the reason their store is welded to their transport. POC #1's lean Backend trait is the smaller hammer for our split: the trait stays dumb (opaque bytes), and any ops surface above it (alkcall) speaks its own protocol.
• Re-borrowed conclusion, not mechanism: per-hash serialized actors. FsStore runs one serialized state machine per hash (entity manager, src/store/fs/util/entity_manager.rs) with parallelism across hashes; per-hash state is a watch::Sender<BaoFileStorage> (cheap snapshots, observable progress, idle-recycled actor pool). This is a genuinely good concurrency shape for Phase 1's sweep-vs-put story: "one writer per content address, readers snapshot-observe" matches CAS semantics naturally. iroh's DeleteSet was already our named prior art; the per-hash-actor adds a second reusable conclusion.

What the fs backend actually looks like on disk (0.103)

blobs.db                    redb — hash → EntryState (+tags, + inline data/outboard)
data/<hash>.data            complete + partial canonical data
data/<hash>.obao4           outboard (bao pre-order, 64-byte pairs)
data/<hash>.sizes4          per-chunk-group size slots
data/<hash>.bitfield        blake3-checksummed postcard bitfield (coverage)
temp/<uuid>.temp            staging (must be same-device as data/)

• Entry state machine per blob (src/store/fs/entry_state.rs): Complete { data_location: Inline|Owned|External, outboard_location: Inline|Owned|NotNeeded } / Partial { size: Option<u64> }. The inline/owned split is the small-blob/large-blob dispatch inside one backend: ≤16 KiB data lives entirely in the redb inline-data row; ≥~1 MiB is data file + outboard file; between, data file + inline outboard. Thresholds are InlineOptions { max_data_inlined: 16 KiB, max_outboard_inlined: 16 KiB } (fs/options.rs:71-78).
• 16 KiB is exactly one bao chunk group (16 × 1024 B chunks), so every blob ≤ 16 KiB has an outboard of size zero — a "small blob" is one redb row keyed by plain blake3(content), no tree material at all (fs/import.rs:160-221, DESIGN.md:71). iroh couples "hash overhead" and "storage tier" at the same boundary; under our git-oid derivation there is no outboard to elide (whole-blob digest only) and the tier boundary is a pure dispatch choice — POC #3's benchmark says ~64-128 KiB, not 16 KiB (see poc-largeblob-findings.md, B-finding).
• Crash-consistency story, borrowed as a conclusion (DESIGN.md: 126-142, "files are hard"): persist data+outboard+sizes first, the checksummed bitfield last, with truncate-to-zero-on-read marking a bitfile dirty under crash. If the bitfield is missing/corrupt, the entry re-validates by full local hash check. Lazy persistence (on actor idle/shutdown only), never per-chunk fsync. For alkblobs: our put path is stage-temp → hash → one rename, which needs no coverage bitfield at all — we only ever need this machinery if we adopt partial/streaming-download entries (range-serve before complete). Recorded as a conditional, like bao.
• Temp-file flow (<fs/import.rs>): byte-stream puts stage to a temp file (after a 16 KiB in-memory peek), path-import reflinks-or- copies (reflink-copy crate) to temp, outboard encodes in a second pass, then fs::rename atomizes both files before the redb row lands awaiting the transaction commit. ExportMode::TryReference is the mirror: rename the store's own file into the caller's tree and mark the entry External. (TryReference is a neat dedup trick for the appfile/workspace consumer — "move into the store without copying" — Phase 2 note, not Phase 1.)
• Metadata batching: the redb actor batches read commands (10k msgs / 1 s window) and write commands (1000 / 500 ms) into single transactions (meta.rs:785-856) — "the secret to fast download speeds is to not touch the metadata database at all" (DESIGN.md:61). sqlite's WAL gives comparable batching for free at the smaller scale our store keys live at; the conclusion (batch metadata writes, never per-blob-fsync) transfers.

Verification & chunking (OQ-BL-04 input)

• Fixed 1024 B bao chunks, grouped 16-wide (iroh IROH_BLOCK_SIZE, store/mod.rs:17-18); no content-defined chunking exists in-tree.
• Outboard encoding is a custom single-pass encoder over blake3 hazmat APIs (src/util.rs:224-336) — post-order chunk iteration, set_input_offset/finalize_non_root merges, buffered 1 MiB reads. iroh wrote their own rather than using bao-tree's CreateOutboard; the shape of "encode outboard while streaming through chunks once" is the pattern any chunk-tree encoder follows, bao or CDC.
• Get verification is subtree-local: export_bao(hash, ranges) runs traverse_ranges_validated and lazily reads only the parent pairs the requested chunks need. Byte-range export_ranges is deliberately unvalidated (bitfield-superset check only; DataReader is documented "a reader for the unvalidated data file", fs.rs:337-343).
• For our canonical digest this exposes a hard fact (POC #3 finding A2, reproduced independently): the git-blob derivation verifies the whole content (preamble includes the length). There is no sub-range relationship between blob <len>\0+content and any slice of it, so range reads under our hash cannot be verified against the blob's own digest, period. iroh gets range-verification from the chunk tree, not from the hash algorithm. If ranged transfer ever needs per-range verification, the options stay as OQ-BL-04 lists them (caller-carried slice digests = transport-integrity only; conditional chunk-tree encoding; or backend-native). POC #3's verdict: the local-store case doesn't need it (pages fail via EIO, git blobs self-verify on read); the networked large-blob case is the only real consumer of a chunk tree — which matches the "conditional, confined to the encoding layer" posture already recorded.

Small-blob handling & streaming (POC #3 design input)

• Streaming put exists in three shapes; none hashes on the fly: add_stream (bidirectional ImportByteStream channel; stages to memory then temp file, computes outboard in a second pass, fs/import.rs:290-352, 372-419), import_bao (pre-verified items, no re-check), and import_bao_reader (network decode-verify-push, blobs.rs:446-488 — the provider's push handler). There is no put-from-AsyncRead API; callers adapt to Stream<io::Result<Bytes>>.
• POC #3 found the same constraint from our side before reading this in detail: the git preamble needs the length at hash time, so an unknown-length stream must stage-then-hash. iroh's architecture arrives at the same stage-then-encode shape (for bao-tree reasons instead). The one-pass alternative exists only for known-length puts — which is exactly the git/alkgit dominant case (a staged file has a stat; a protocol offer carries a size). Unknown-length remains the two-pass shape for both crates; POC #3's finding A1 records why (and that the second pass can read from the staged file, so it's page-cache-cheap, not network-repetitive).
• Serve-while-persisting is not a bolt-on in iroh — it is the download path: get_blob_ranges_impl opens the import before reading from the network and tokio::try_join!s the network decode loop with the store import (src/api/remote.rs:884-944). POC #3's fanout seam (finding A3) rebuilt this at our layer with a broadcast channel; the mechanism differs (single reader broadcasting vs their per-item verified forwarding) but the conclusion matches: the fetch handler should be "decode-verify → broadcast → (store arm + subscriber arms)", all driven from one receive loop.
• BlobReader implements AsyncRead + AsyncSeek built on export_ranges per read call (api/blobs/reader.rs); seek-from-end is an upstream TODO. Our fs range-read surface covers the same need.

Performance posture (no in-tree benches; code choices as evidence)

No benches/, no criterion in the checkout. The perf story is in DESIGN.md + code micro-choices: redb write/read batching; partial entries live entirely outside redb until completion; zero-copy Bytes plumbing end to end; reflink-first import/export; 1 MiB copy buffers with yield_now() fairness; per-hash channel caps bounding backpressure. The two transferable conclusions for us: batch metadata transactions (never per-blob fsync), and zero-copy/Bytes at the streaming seams. Our own sqlite-vs-fs numbers are in the POC #3 findings (first-party, measured).

Do-not-inherit list (re-confirmed against 0.103)

• The Command-actor + irpc store abstraction (welds store to their transport stack).
• redb-specific entry-state modeling if we go sqlite (their tables and EntryState postcard encoding are theirs; our sqlite schema is key → (value, size) and our dispatch is store-layer policy).
• Fixed 1024 B chunk trees as the default granularity (principle 7 already rejects; nothing new in the eval changes it).
• Tickets/postcard/provider — out of scope as always.

What we keep (conclusions only)

1. Per-hash serialized state machines + snapshot-observation for concurrent access (entity-manager pattern).
2. Stage-temp → atomic rename as the universal put commit primitive.
3. Metadata write batching (amortize sync cost).
4. Zero-copy Bytes at streaming seams.
5. Serve-while-persisting as the fetch-handler shape (POC #3 validated at our layer).
6. Crash-consistency ordering (data first, coverage/index last, with dirty-marking) — only if partial entries ever exist.
7. The inline threshold concept, re-sized by our own benchmark (~64-128 KiB crossover, git-blob workload) instead of their 16 KiB.