alkblobs/docs/research/iroh-blobs-eval.md

status, title, last_updated

status	title	last_updated
draft	iroh-blobs store eval — against the current checkout	2026-10-02 (POC

iroh-blobs store eval (current checkout)

Evaluated against /workspace/iroh-blobs @ e82cbdc ("Release 0.103.0", 2026-06-15 — the checkout is the 0.103.0 release, so alknet's older probe claims about "published 0.103" verify directly against it). Scope: the store layer (src/store/**, src/api/**, src/util/**). The wire surface (tickets, postcard, provider protocol) is out of scope per convention 7 — we are not adopting it. This eval feeds OQ-BL-02 (backend shape), OQ-BL-04 (verification/chunking), and the POC #3 large-blob design. GC mechanics were already verified separately (see phase-0 §Prior art; the gc.rs read predates POC #1).

The headline: there is no Store trait — and that is instructive

iroh-blobs 0.103 has no Store trait at all. The store API is a message protocol: an enum of RPC requests (src/api/proto.rs:90-139, #[rpc_requests]-derived Command), and each backend (MemStore, FsStore, ReadonlyMemStore) is an actor consuming that command set over irpc channels. The user-facing Store struct is just a client handle; Store::connect/listen make the same message enum work over the network. The abstraction is "same messages locally and remote."

Lessons for alkblobs:

• The command-enum-as-contract is a working pattern (it is what makes iroh-blobs' remote/local symmetry possible), but it carries the whole irpc machinery into the store layer and is the reason their store is welded to their protocol stack (theirs also welds transport endpoints above that; the store-actor weld alone is what kills backend swappability). POC #1's lean Backend trait is the smaller hammer for our split: the trait stays dumb (opaque bytes), and any ops surface above it (alkcall) speaks its own protocol.
• Re-borrowed conclusion, not mechanism: per-hash serialized actors. FsStore runs one serialized state machine per hash (entity manager, src/store/fs/util/entity_manager.rs) with parallelism across hashes; per-hash state is a watch::Sender<BaoFileStorage> (cheap snapshots, observable progress, idle-recycled actor pool). This is a genuinely good concurrency shape for Phase 1's sweep-vs-put story: "one writer per content address, readers snapshot-observe" matches CAS semantics naturally. iroh's DeleteSet was already our named prior art; the per-hash-actor adds a second reusable conclusion.

What the fs backend actually looks like on disk (0.103)

blobs.db                    redb — hash → EntryState (+tags, + inline data/outboard)
data/<hash>.data            complete + partial canonical data
data/<hash>.obao4           outboard (bao pre-order, 64-byte pairs)
data/<hash>.sizes4          per-chunk-group size slots
data/<hash>.bitfield        blake3-checksummed postcard bitfield (coverage)
temp/<uuid>.temp            staging (must be same-device as data/)

• Entry state machine per blob (src/store/fs/entry_state.rs): Complete { data_location: Inline|Owned|External, outboard_location: Inline|Owned|NotNeeded } / Partial { size: Option<u64> }. The inline/owned split is the small-blob/large-blob dispatch inside one backend: ≤16 KiB data lives entirely in the redb inline-data row; ≥~1 MiB is data file + outboard file; between, data file + inline outboard. Thresholds are InlineOptions { max_data_inlined: 16 KiB, max_outboard_inlined: 16 KiB } (fs/options.rs:71-78).
• 16 KiB is exactly one bao chunk group (16 × 1024 B chunks), so every blob ≤ 16 KiB has an outboard of size zero — a "small blob" is one redb row keyed by plain blake3(content), no tree material at all (fs/import.rs:160-221, DESIGN.md:71). iroh couples "hash overhead" and "storage tier" at the same boundary; under our git-oid derivation there is no outboard to elide (whole-blob digest only) and the tier boundary is a pure dispatch choice — POC #3's benchmark says ~128-256 KiB, not 16 KiB (see poc-largeblob-findings.md, finding A4).
• Crash-consistency story, borrowed as a conclusion (DESIGN.md: 126-142, "files are hard"): persist data+outboard+sizes first, the checksummed bitfield last, with truncate-to-zero-on-read marking a bitfile dirty under crash. If the bitfield is missing/corrupt, the entry re-validates by full local hash check. Lazy persistence (on actor idle/shutdown only), never per-chunk fsync. For alkblobs: our put path is stage-temp → hash → one rename, which needs no coverage bitfield at all — we only ever need this machinery if we adopt partial/streaming-download entries (range-serve before complete). Recorded as a conditional, like bao.
• Temp-file flow (<fs/import.rs>): byte-stream puts stage to a temp file (after a 16 KiB in-memory peek), path-import reflinks-or- copies (reflink-copy crate) to temp, outboard encodes in a second pass, then fs::rename atomizes both files before the redb row lands awaiting the transaction commit. ExportMode::TryReference is the mirror: rename the store's own file into the caller's tree and mark the entry External. (TryReference is a neat dedup trick for the appfile/workspace consumer — "move into the store without copying" — Phase 2 note, not Phase 1.)
• Metadata batching: the redb actor batches read commands (10k msgs / 1 s window) and write commands (1000 / 500 ms) into single transactions (meta.rs:785-856) — "the secret to fast download speeds is to not touch the metadata database at all" (DESIGN.md:61). sqlite's WAL gives comparable batching for free at the smaller scale our store keys live at; the conclusion (batch metadata writes, never per-blob-fsync) transfers.

Verification & chunking (OQ-BL-04 input)

• Fixed 1024 B bao chunks, grouped 16-wide (iroh IROH_BLOCK_SIZE, store/mod.rs:17-18); no content-defined chunking exists in-tree.
• Outboard encoding is a custom single-pass encoder over blake3 hazmat APIs (src/util.rs:224-336) — post-order chunk iteration, set_input_offset/finalize_non_root merges, buffered 1 MiB reads. iroh wrote their own rather than using bao-tree's CreateOutboard; the shape of "encode outboard while streaming through chunks once" is the pattern any chunk-tree encoder follows, bao or CDC.
• Get verification is subtree-local: export_bao(hash, ranges) runs traverse_ranges_validated and lazily reads only the parent pairs the requested chunks need. Byte-range export_ranges is deliberately unvalidated (bitfield-superset check only; DataReader is documented "a reader for the unvalidated data file", fs.rs:337-343).
• For our canonical digest this exposes a hard fact (POC #3 finding A2, reproduced independently): the git-blob derivation verifies the whole content (preamble includes the length). There is no sub-range relationship between blob <len>\0+content and any slice of it, so range reads under our hash cannot be verified against the blob's own digest, period. iroh gets range-verification from the chunk tree, not from the hash algorithm. If ranged transfer ever needs per-range verification, the options stay as OQ-BL-04 lists them (caller-carried slice digests = transport-integrity only; conditional chunk-tree encoding; or backend-native). POC #3's verdict: the local-store case doesn't need it (pages fail via EIO, git blobs self-verify on read); the networked large-blob case is the only real consumer of a chunk tree — which matches the "conditional, confined to the encoding layer" posture already recorded.

Small-blob handling & streaming (POC #3 design input)

• Streaming put exists in three shapes; none hashes on the fly: add_stream (bidirectional ImportByteStream channel; stages to memory then temp file, computes outboard in a second pass, fs/import.rs:290-352, 372-419), import_bao (pre-verified items, no re-check), and import_bao_reader (network decode-verify-push, blobs.rs:446-488 — the provider's push handler). There is no put-from-AsyncRead API; callers adapt to Stream<io::Result<Bytes>>.
• POC #3 found the same constraint from our side before reading this in detail: the git preamble needs the length at hash time, so an unknown-length stream must stage-then-hash. iroh's architecture arrives at the same stage-then-encode shape (for bao-tree reasons instead). The one-pass alternative exists only for known-length puts — which is exactly the git/alkgit dominant case (a staged file has a stat; a protocol offer carries a size). Unknown-length remains the two-pass shape for both crates; POC #3's finding A1 records why (and that the second pass can read from the staged file, so it's page-cache-cheap, not network-repetitive).
• Serve-while-persisting is not a bolt-on in iroh — it is the download path: get_blob_ranges_impl opens the import before reading from the network and tokio::try_join!s the network decode loop with the store import (src/api/remote.rs:884-944). POC #3's fanout seam (finding A3) rebuilt this at our layer with a broadcast channel; the mechanism differs (single reader broadcasting vs their per-item verified forwarding) but the conclusion matches: the fetch handler should be "decode-verify → broadcast → (store arm + subscriber arms)", all driven from one receive loop.
• BlobReader implements AsyncRead + AsyncSeek built on export_ranges per read call (api/blobs/reader.rs); seek-from-end is an upstream TODO. Our fs range-read surface covers the same need.

Performance posture (no in-tree benches; code choices as evidence)

No benches/, no criterion in the checkout. The perf story is in DESIGN.md + code micro-choices: redb write/read batching; partial entries live entirely outside redb until completion; zero-copy Bytes plumbing end to end; reflink-first import/export; 1 MiB copy buffers with yield_now() fairness; per-hash channel caps bounding backpressure. The two transferable conclusions for us: batch metadata transactions (never per-blob fsync), and zero-copy/Bytes at the streaming seams. Our own sqlite-vs-fs numbers are in the POC #3 findings (first-party, measured).

Do-not-inherit list (re-confirmed against 0.103)

• The Command-actor + irpc store abstraction (welds store to their protocol stack; endpoints/transport weld on above it).
• redb-specific entry-state modeling if we go sqlite (their tables and EntryState postcard encoding are theirs; our sqlite schema is key → (value, size) and our dispatch is store-layer policy).
• Fixed 1024 B chunk trees as the default granularity (principle 7 already rejects; nothing new in the eval changes it).
• Tickets/postcard/provider — out of scope as always.

What we keep (conclusions only)

1. Per-hash serialized state machines + snapshot-observation for concurrent access (entity-manager pattern).
2. Stage-temp → atomic rename as the universal put commit primitive.
3. Metadata write batching (amortize sync cost).
4. Zero-copy Bytes at streaming seams.
5. Serve-while-persisting as the fetch-handler shape (POC #3 validated at our layer).
6. Crash-consistency ordering (data first, coverage/index last, with dirty-marking) — only if partial entries ever exist.
7. The inline threshold concept, re-sized by our own benchmark (~128-256 KiB crossover, git-blob workload) instead of their 16 KiB.