docs(research): POC #3 passed — large-blob streaming + sqlite-vs-fs benchmark
- POC #3 (register OQ-BL-06) executed and passed: streaming LFSObject-shaped put/get at the store layer; fanout seam (serve-while-persisting) validated as broadcast above the backends; fs range reads with the verification gap documented (preamble makes slice-vs-whole verification impossible — chunk-tree encoding sharpened to a transfer-layer conditional) - sqlite-vs-fs micro-benchmark: first-party anchor for the small-blob belief (~9-10x sqlite at 1-16 KiB, crossover ~128-256 KiB, 128 KiB default lean); harness re-runnable on real media - Stage-file hygiene tests caught a real leak (one temp file per failed put) — fixed; all failure paths converge on discard - Pack tension resolved in shape from the gix-odb read: packs as large blobs served by range reads; object-level dedup lives in the loose tier - iroh-blobs-eval.md written against the current checkout (e82cbdc / v0.103.0): actor/message-protocol pattern, fs backend layout, verification, do-not-inherit + keep lists - alknet probe re-check demoted to historical cross-check only: alknet's code is not a design input (decomposed; substrate is alkcall); its small-blob perf claim was asserted-never-measured, superseded by A4 - phase-0 register complete: no open POCs; next step is convergence -> Phase 1 Verification: POC crate (standalone /workspace/alkblobs-largeblob-poc) 14 tests pass, clippy -D warnings clean, fmt clean; digests byte-exact vs the real git hash-object CLI on both streaming put paths
This commit is contained in:
glm-5.3-flash committed
1 parent
493516424a
commit
6b278fb490
3 files changed
+653
-67
No files matched your search
@@ -0,0 +1,198 @@
|
||||
---
|
||||
status: draft
|
||||
title: "iroh-blobs store eval — against the current checkout"
|
||||
last_updated: 2026-10-02 (POC #3 reading; checkout e82cbdc, v0.103.0)
|
||||
---
|
||||
|
||||
# iroh-blobs store eval (current checkout)
|
||||
|
||||
Evaluated against `/workspace/iroh-blobs` @ `e82cbdc` ("Release 0.103.0",
|
||||
2026-06-15 — the checkout *is* the 0.103.0 release, so alknet's older
|
||||
probe claims about "published 0.103" verify directly against it).
|
||||
Scope: the store layer (`src/store/**`, `src/api/**`, `src/util/**`).
|
||||
The wire surface (tickets, postcard, provider protocol) is out of scope
|
||||
per convention 7 — we are not adopting it. This eval feeds OQ-BL-02
|
||||
(backend shape), OQ-BL-04 (verification/chunking), and the POC #3
|
||||
large-blob design. GC mechanics were already verified separately
|
||||
(see phase-0 §Prior art; the gc.rs read predates POC #1).
|
||||
|
||||
## The headline: there is no Store trait — and that is instructive
|
||||
|
||||
iroh-blobs 0.103 has **no `Store` trait at all**. The store API is a
|
||||
message protocol: an enum of RPC requests (`src/api/proto.rs:90-139`,
|
||||
`#[rpc_requests]`-derived `Command`), and each backend (MemStore,
|
||||
FsStore, ReadonlyMemStore) is an *actor* consuming that command set
|
||||
over `irpc` channels. The user-facing `Store` struct is just a client
|
||||
handle; `Store::connect/listen` make the same message enum work over
|
||||
the network. The abstraction is "same messages locally and remote."
|
||||
|
||||
Lessons for alkblobs:
|
||||
|
||||
- **The command-enum-as-contract is a working pattern** (it is what
|
||||
makes iroh-blobs' remote/local symmetry possible), but it carries the
|
||||
whole irpc machinery into the store layer and *is* the reason their
|
||||
store is welded to their transport. POC #1's lean `Backend` trait is
|
||||
the smaller hammer for our split: the trait stays dumb (opaque bytes),
|
||||
and any ops surface above it (alkcall) speaks its own protocol.
|
||||
- **Re-borrowed conclusion, not mechanism: per-hash serialized actors.**
|
||||
FsStore runs one serialized state machine *per hash* (entity manager,
|
||||
`src/store/fs/util/entity_manager.rs`) with parallelism across hashes;
|
||||
per-hash state is a `watch::Sender<BaoFileStorage>` (cheap snapshots,
|
||||
observable progress, idle-recycled actor pool). This is a genuinely
|
||||
good concurrency shape for Phase 1's sweep-vs-put story: "one writer
|
||||
per content address, readers snapshot-observe" matches CAS semantics
|
||||
naturally. iroh's DeleteSet was already our named prior art; the
|
||||
per-hash-actor adds a second reusable conclusion.
|
||||
|
||||
## What the fs backend actually looks like on disk (0.103)
|
||||
|
||||
```
|
||||
blobs.db redb — hash → EntryState (+tags, + inline data/outboard)
|
||||
data/<hash>.data complete + partial canonical data
|
||||
data/<hash>.obao4 outboard (bao pre-order, 64-byte pairs)
|
||||
data/<hash>.sizes4 per-chunk-group size slots
|
||||
data/<hash>.bitfield blake3-checksummed postcard bitfield (coverage)
|
||||
temp/<uuid>.temp staging (must be same-device as data/)
|
||||
```
|
||||
|
||||
- **Entry state machine per blob** (`src/store/fs/entry_state.rs`):
|
||||
`Complete { data_location: Inline|Owned|External, outboard_location:
|
||||
Inline|Owned|NotNeeded }` / `Partial { size: Option<u64> }`. The
|
||||
inline/owned split is the small-blob/large-blob dispatch *inside one
|
||||
backend*: ≤16 KiB data lives entirely in the redb `inline-data` row;
|
||||
≥~1 MiB is data file + outboard file; between, data file + inline
|
||||
outboard. Thresholds are `InlineOptions { max_data_inlined: 16 KiB,
|
||||
max_outboard_inlined: 16 KiB }` (`fs/options.rs:71-78`).
|
||||
- **16 KiB is exactly one bao chunk group** (16 × 1024 B chunks), so
|
||||
every blob ≤ 16 KiB has an outboard of size *zero* — a "small blob"
|
||||
is one redb row keyed by plain `blake3(content)`, no tree material at
|
||||
all (`fs/import.rs:160-221`, `DESIGN.md:71`). iroh couples "hash
|
||||
overhead" and "storage tier" at the same boundary; under our git-oid
|
||||
derivation there is no outboard to elide (whole-blob digest only) and
|
||||
the tier boundary is a pure dispatch choice — POC #3's benchmark says
|
||||
~64-128 KiB, not 16 KiB (see `poc-largeblob-findings.md`, B-finding).
|
||||
- **Crash-consistency story, borrowed as a conclusion** (`DESIGN.md:
|
||||
126-142`, "files are hard"): persist data+outboard+sizes *first*, the
|
||||
checksummed bitfield *last*, with truncate-to-zero-on-read marking a
|
||||
bitfile dirty under crash. If the bitfield is missing/corrupt, the
|
||||
entry re-validates by full local hash check. Lazy persistence (on
|
||||
actor idle/shutdown only), never per-chunk fsync. For alkblobs: our
|
||||
put path is *stage-temp → hash → one rename*, which needs no coverage
|
||||
bitfield at all — we only ever need this machinery if we adopt
|
||||
partial/streaming-download entries (range-serve before complete).
|
||||
Recorded as a conditional, like bao.
|
||||
- **Temp-file flow** (<`fs/import.rs`>): byte-stream puts stage to a
|
||||
temp file (after a 16 KiB in-memory peek), path-import reflinks-or-
|
||||
copies (`reflink-copy` crate) to temp, outboard encodes in a second
|
||||
pass, then `fs::rename` atomizes both files before the redb row lands
|
||||
*awaiting* the transaction commit. `ExportMode::TryReference` is the
|
||||
mirror: rename the store's own file into the caller's tree and mark
|
||||
the entry `External`. (TryReference is a neat dedup trick for the
|
||||
appfile/workspace consumer — "move into the store without copying" —
|
||||
Phase 2 note, not Phase 1.)
|
||||
- **Metadata batching**: the redb actor batches read commands (10k msgs
|
||||
/ 1 s window) and write commands (1000 / 500 ms) into single
|
||||
transactions (`meta.rs:785-856`) — "the secret to fast download
|
||||
speeds is to not touch the metadata database at all" (`DESIGN.md:61`).
|
||||
sqlite's WAL gives comparable batching for free at the smaller scale
|
||||
our store keys live at; the *conclusion* (batch metadata writes,
|
||||
never per-blob-fsync) transfers.
|
||||
|
||||
## Verification & chunking (OQ-BL-04 input)
|
||||
|
||||
- Fixed 1024 B bao chunks, grouped 16-wide (iroh `IROH_BLOCK_SIZE`,
|
||||
`store/mod.rs:17-18`); no content-defined chunking exists in-tree.
|
||||
- Outboard encoding is a **custom single-pass encoder over blake3
|
||||
hazmat APIs** (`src/util.rs:224-336`) — post-order chunk iteration,
|
||||
`set_input_offset`/`finalize_non_root` merges, buffered 1 MiB reads.
|
||||
iroh wrote their own rather than using bao-tree's `CreateOutboard`;
|
||||
the *shape* of "encode outboard while streaming through chunks once"
|
||||
is the pattern any chunk-tree encoder follows, bao or CDC.
|
||||
- **Get verification is subtree-local**: `export_bao(hash, ranges)` runs
|
||||
`traverse_ranges_validated` and lazily reads only the parent pairs the
|
||||
requested chunks need. Byte-range `export_ranges` is deliberately
|
||||
*unvalidated* (bitfield-superset check only; `DataReader` is documented
|
||||
"a reader for the unvalidated data file", `fs.rs:337-343`).
|
||||
- **For our canonical digest this exposes a hard fact** (POC #3 finding
|
||||
A2, reproduced independently): the git-blob derivation verifies *the
|
||||
whole content* (preamble includes the length). There is no sub-range
|
||||
relationship between `blob <len>\0+content` and any slice of it, so
|
||||
range reads under our hash cannot be verified against the blob's own
|
||||
digest, period. iroh gets range-verification *from the chunk tree*, not
|
||||
from the hash algorithm. If ranged transfer ever needs per-range
|
||||
verification, the options stay as OQ-BL-04 lists them (caller-carried
|
||||
slice digests = transport-integrity only; conditional chunk-tree
|
||||
encoding; or backend-native). POC #3's verdict: the local-store case
|
||||
doesn't need it (pages fail via EIO, git blobs self-verify on read);
|
||||
the *networked large-blob* case is the only real consumer of a chunk
|
||||
tree — which matches the "conditional, confined to the encoding layer"
|
||||
posture already recorded.
|
||||
|
||||
## Small-blob handling & streaming (POC #3 design input)
|
||||
|
||||
- **Streaming put exists in three shapes**; none hashes on the fly:
|
||||
`add_stream` (bidirectional `ImportByteStream` channel; stages to
|
||||
memory then temp file, computes outboard in a **second pass**,
|
||||
`fs/import.rs:290-352, 372-419`), `import_bao` (pre-verified items,
|
||||
no re-check), and `import_bao_reader` (network decode-verify-push,
|
||||
`blobs.rs:446-488` — the provider's push handler). There is **no
|
||||
put-from-`AsyncRead` API**; callers adapt to `Stream<io::Result<Bytes>>`.
|
||||
- POC #3 found the same constraint from our side *before* reading this
|
||||
in detail: the git preamble needs the length at hash time, so an
|
||||
unknown-length stream *must* stage-then-hash. iroh's architecture
|
||||
arrives at the same stage-then-encode shape (for bao-tree reasons
|
||||
instead). **The one-pass alternative exists only for known-length
|
||||
puts** — which is exactly the git/alkgit dominant case (a staged file
|
||||
has a stat; a protocol offer carries a size). Unknown-length remains
|
||||
the two-pass shape for both crates; POC #3's finding A1 records why
|
||||
(and that the second pass can read from the *staged file*, so it's
|
||||
page-cache-cheap, not network-repetitive).
|
||||
- **Serve-while-persisting is not a bolt-on in iroh — it is the
|
||||
download path**: `get_blob_ranges_impl` opens the import *before*
|
||||
reading from the network and `tokio::try_join!`s the network decode
|
||||
loop with the store import (`src/api/remote.rs:884-944`). POC #3's
|
||||
fanout seam (finding A3) rebuilt this at our layer with a broadcast
|
||||
channel; the mechanism differs (single reader broadcasting vs their
|
||||
per-item verified forwarding) but the conclusion matches: the fetch
|
||||
handler should be "decode-verify → broadcast → (store arm + subscriber
|
||||
arms)", all driven from one receive loop.
|
||||
- **`BlobReader`** implements `AsyncRead + AsyncSeek` built on
|
||||
`export_ranges` per read call (`api/blobs/reader.rs`); seek-from-end
|
||||
is an upstream TODO. Our fs range-read surface covers the same need.
|
||||
|
||||
## Performance posture (no in-tree benches; code choices as evidence)
|
||||
|
||||
No `benches/`, no criterion in the checkout. The perf story is in
|
||||
DESIGN.md + code micro-choices: redb write/read batching; partial
|
||||
entries live entirely outside redb until completion; zero-copy
|
||||
`Bytes` plumbing end to end; reflink-first import/export; 1 MiB copy
|
||||
buffers with `yield_now()` fairness; per-hash channel caps bounding
|
||||
backpressure. The two transferable *conclusions* for us: batch
|
||||
metadata transactions (never per-blob fsync), and zero-copy/`Bytes`
|
||||
at the streaming seams. Our own sqlite-vs-fs numbers are in the POC #3
|
||||
findings (first-party, measured).
|
||||
|
||||
## Do-not-inherit list (re-confirmed against 0.103)
|
||||
|
||||
- The `Command`-actor + irpc store abstraction (welds store to their
|
||||
transport stack).
|
||||
- redb-specific entry-state modeling if we go sqlite (their tables and
|
||||
`EntryState` postcard encoding are theirs; our sqlite schema is
|
||||
`key → (value, size)` and our dispatch is store-layer policy).
|
||||
- Fixed 1024 B chunk trees as the default granularity (principle 7
|
||||
already rejects; nothing new in the eval changes it).
|
||||
- Tickets/postcard/provider — out of scope as always.
|
||||
|
||||
## What we keep (conclusions only)
|
||||
|
||||
1. Per-hash serialized state machines + snapshot-observation for
|
||||
concurrent access (entity-manager pattern).
|
||||
2. Stage-temp → atomic rename as the universal put commit primitive.
|
||||
3. Metadata write batching (amortize sync cost).
|
||||
4. Zero-copy `Bytes` at streaming seams.
|
||||
5. Serve-while-persisting as the fetch-handler shape (POC #3 validated
|
||||
at our layer).
|
||||
6. Crash-consistency ordering (data first, coverage/index last, with
|
||||
dirty-marking) — only if partial entries ever exist.
|
||||
7. The inline threshold concept, re-sized by our own benchmark
|
||||
(~64-128 KiB crossover, git-blob workload) instead of their 16 KiB.
|
||||
Reference in new issue
Block a user