alkblobs/docs/research/poc-largeblob-findings.md

---
status: passed
title: "POC #3 — large-blob streaming path + sqlite-vs-fs micro-benchmark"
last_updated: 2026-10-02
---

# POC: large-blob path (streaming, fanout, range reads) + small-object micro-benchmark — findings

> **POC register #3**, per `docs/research/phase-0.md` OQ-BL-06. Code:
> standalone crate `/workspace/alkblobs-largeblob-poc` (findings land
> here regardless, per the established convention). Companions from the
> same POC's reading work: `iroh-blobs-eval.md` (fresh-store eval
> against checkout e82cbdc / v0.103.0) and the alknet probe re-check
> (below — recorded as **historical cross-check only**).
> Date: 2026-10-02. Status: **passed**, 14 tests, clippy `-D warnings`
> clean, fmt clean. Digests cross-checked against the real `git
> hash-object` CLI (git 2.43), extending POC #1's validation to the
> streaming paths.

## What this POC set out to decide

From the phase-0 register:

1. **The large-blob path** — does an `LFSObject`-shaped `(len, stream)`
   put/get work at the store layer, and what does the git-blob preamble
   do to streaming hashing?
2. **The fanout seam** — "serve the fetch while persisting on receipt"
   (rudolfs' `fanout()`; iroh's `get_blob_ranges_impl`) — does it work
   at our layer, above the backends?
3. **Range reads on the fs backend** — cheap partial reads with *some*
   verification story (OQ-BL-04).
4. **The small-object sqlite-vs-fs micro-benchmark** — anchor the
   "kv is faster than fs for small blobs" belief first-party
   (OQ-BL-02's "dual-belief anchor").
5. **The pack tension** (OQ-BL-05) — analyzed from the gix-odb read,
   no POC of its own.

Note on standing: **alknet's code is not a design input** — it is old,
poorly planned, decomposed into the alk* family; its probe docs are
treated as references to old research only. The substrate this crate
rides is alkcall (like alkhttp/alksocks/alktty/alktunnels).

## Result summary

| Question | Verdict |
|----------|---------|
| Streaming put (`LFSObject` shape) | **Yes** — two paths: known-length one-pass; unknown-length stage-then-hash (forced by the preamble, finding A1) |
| Streaming get without whole-blob load | **Yes** — fs arm streams a file handle; kv arm one bounded read |
| Fanout (serve while persisting) | **Yes** — one reader task, broadcast channel, store arm + N subscriber arms (finding A3) |
| Range reads on fs | **Yes** — pread-window only; **verification has a real gap** (finding A2) |
| sqlite faster than fs for small blobs | **Confirmed and measured** — ~9-10× at 1-16 KiB; crossover ~128-256 KiB (finding A4) |
| Digest = git oid on streaming paths | **Byte-exact vs `git hash-object`** for both put paths |
| Stage-file hygiene | **Verified** — failures (stream error, overflow, early-EOF) leave zero stage files (finding A5) |

## Findings

### Finding A1: the git-blob preamble splits streaming into exactly two paths — and the split is forced, not chosen

The canonical derivation is `H("blob <len>\0" + content)`. The length
is *inside the hashed input*, so:

- **Known length** ⇒ write the preamble into the hasher at construction,
  feed content chunks incrementally, finalize at stream end: **one pass
  over the content**, no staging needed for hashing (staging to a temp
  file still happens for the fs tier, but the hash is done by the time
  the rename commits). This covers the dominant git and protocol cases:
  a pre-staged file has its length from `stat`, a network offer carries
  a size, an HTTP PUT can carry `Content-Length`.
- **Unknown length** ⇒ the preamble can't be written until the stream
  ends, so content must be staged (file, not memory) and hashed in a
  second pass over the staged bytes. **Two passes are forced by the
  derivation**, not by our choice of hash.

iroh-blobs independently arrives at the same stage-then-encode shape
(their `add_stream` buffers then computes outboard in a second pass,
`fs/import.rs:290-419`), for bao-tree reasons instead of
preamble reasons. Their `import_bao_reader` sidesteps it by knowing
both hash and size upfront (a protocol-level offer) — the same
"known-length is the normal case" assumption, enforced by their wire
format.

**Verdict carried to Phase 1:** the store's put seam should be
`(Option<len>, stream)` exactly as built. Known-length puts must be the
encouraged path (documented as such on the API); unknown-length is
supported but slower by one page-cache read. Do *not* invent a
length-prefix-on-the-wire scheme to "fix" unknown-length puts — the
wire carries sizes naturally (it must, for the CAS routing).

### Finding A2: range reads cannot verify against the canonical digest — the preamble includes the length, so slices have no hash relationship to the whole

Confirmed at implementation level (and independently by the
iroh-blobs eval): `blob <len>\0 + content` gives *no* verifiable
relation between `H(whole)` and any `content[a..b]`. iroh gets
per-range verification from the **bao chunk tree**, not from BLAKE3 the
algorithm — the tree is the mechanism, the hash is just its leaf
function.

POC #3's `read_range` therefore returns the slice plus a *slice digest*
(`git-blob-sha-256` over the slice — a transport-integrity check the
caller carries out-of-band, e.g. from the offering side). Consequences:

- **Local-store range reads need no more** — the OS pages honest media;
  EIO surfaces read errors; git objects (the small tier) self-verify
  when git reads them whole.
- **The only genuine consumer of a chunk tree is networked large-blob
  transfer where the caller wants per-chunk integrity.** This sharpens
  OQ-BL-04's "conditional": the bao-like encoding is not about
  *storage* at all, it is a *transfer* encoding. If/when p2p git needs
  verified ranged fetch of LFS-shaped artifacts, the encoding layer
  (BLAKE3-confined per OQ-BL-03 tier 3) registers its whole-content
  root back into the pool as an ordinary git-blob-sha-256 entry.
- Practical corollary: for git-blob-shaped content, byte-range *serving*
  is fine (packfiles will want it; see pack tension below) but per-range
  *verification* is out of scope for the store layer today.

### Finding A3: the fanout seam is a broadcast channel above the store, and late-joiners degrade to get-after-commit

Built `put_stream_fanout(stream, n)`: one reader task consumes the
source stream exactly once and broadcasts each chunk to (a) the store
put arm and (b) N subscriber feeds (`mpsc<io::Result<Bytes>>`);
subscribers read while the put is still landing. Tests show 3
subscribers each receive the full 256 KiB byte-identical while the put
completes, and the put receipt matches the git oid.

Design notes validated:

- **One read, many consumers** — the source stream must never be
  polled twice; anything that wants a second copy subscribes. This
  matches iroh's `get_blob_ranges_impl` shape (network decode loop
  joined with the store import) and rudolfs' `fanout()` (two lock-step
  copies from one stream). All three arrive at "broadcast from a single
  received stream" — treat that as the settled seam shape.
- **Backpressure discipline:** bounded channels (8 items in the POC)
  make a slow subscriber stall the broadcast — which is *correct* for a
  put (the put must land) but means subscriber count is bounded and
  slow-subscriber handling is a policy question for the ops layer
  (drop-and-late-join is the escape hatch; see next note).
- **Late joiners:** a consumer arriving after the live stream has
  passed simply waits for the commit then does a normal verified
  `get_stream` — the CAS guarantees the bytes are identical. The POC
  didn't build this (subscriber set is fixed at start) but the degrade
  path is exactly the normal get; worth documenting in the ops layer.

### Finding A4 (benchmark): sqlite beats fs decisively for small blobs; crossover ~128-256 KiB; fsync discipline dominates both arms

Release build, per-arm timing loops, 2k distinct keys/values random
working set, `synchronous=NORMAL` WAL sqlite vs stage+rename fs puts /
open+read fs gets. Representative run (3 s/arm):

```
backend     size      puts/s    gets/s   put p50µs  put p99µs  get p50µs  get p99µs
sqlite      1024       48319     49508       20.0       37.0       19.0       34.0
fs          1024        5306      3357      205.0      302.0      288.0      384.0
sqlite     16384       14129     14166       70.0      125.0       68.0      107.0
fs         16384        4203      2399      257.0      395.0      433.0      599.0
sqlite     65536        4028      4048      252.0      364.0      243.0      344.0
fs         65536        3212      1852      339.0      507.0      540.0      688.0
sqlite    131072        2067      2042      494.0      698.0      482.0      699.0
fs        131072        1892      1652      593.0      879.0      619.0      809.0
sqlite    262144         981      1003     1033.0     1792.0      994.0     1176.0
fs        262144        1244      1576      822.0     1181.0      741.0      964.0
```

Readings:

- **~9-10× sqlite advantage at 1-16 KiB** (the git small-blob regime —
  iroh's DESIGN.md claim and the old citations hold for our workload).
- **Crossover at ~128-256 KiB**: at 256 KiB fs wins puts (no B-tree
  rewrite of a 256 KiB row) and gets. **The dispatch threshold is a
  tunable**, placed anywhere in 64-256 KiB; POC #1's contract already
  makes it a constructor parameter. Phase 1 default: 128 KiB (midpoint
  of the flat zone) — revisit with the storage-medium reality of real
  consumers (tmpfs vs disk; the POC SSD regime is one data point).
- **fs puts cost ~200 µs floor even at 1 KiB** — that is the
  create+write+rename syscall chain; batching stage files (a shared
  preallocated pool) or O_TMPFILE could shave it, but the sqlite arm
  makes the effort moot below the crossover.
- Caveats recorded honestly: single-node tmpdir on this dev box
  (backend-specific tuning — e.g. iroh's redb batch windows — could
  shift the fs curve), no fsync-in-loop on the fs arm (matching NORMAL
  WAL durability, sqlite arm likewise not full-fsync; both arms are
  "crash-tolerant-ish", so the comparison is fair *as measured*: both
  are the durability tier a store would ship).
- The old alknet claim "<100 KB faster in sqlite" is **confirmed in
  direction, refined in boundary**: the honest crossover is higher
  (~128-256 KiB) and sqlite is never *pathological* below it.

### Finding A5: stage-file hygiene is testable and the tests earned their keep (again)

The `stream_failure_cleans_stage_files` test asserted zero files left in
the data dir after a mid-stream connection failure — and caught a real
bug: the original `chunk?` error path exited *before* discarding the
stage file, leaking one temp file per failed put. Fixed by restructuring
the loops to route all failure paths (stream error, overflow, early-EOF,
write error) through one discard. Two lessons compound with POC #1's:

- **Resource-hygiene invariants need dedicated tests**; code review
  missed what the exact-count assertion caught (echoing POC #1's exact
  deletion counts).
- The store's put path now has the shape Phase 1 should codify: **every
  failure path converges on stage-discard; every success path converges
  on commit-rename** — no early returns that bypass cleanup.

### Finding A6: re-put dedup across the two put paths holds; unknown-length re-put of a small blob lands on fs (a dispatch asymmetry worth one Phase 1 decision)

The same content put via known-length (small) and unknown-length
(staged) paths produces the identical digest and coexists as one entry.
But routing differs: an unknown-length put *always* stages through the
fs arm in the POC (routing must wait for the realized size), so a small
blob arriving via an unknown-length stream occupies an fs file where the
same digest's known-length put would occupy a kv row. If both happen,
the pool holds the content twice (different physical tiers).

Options for Phase 1, decided there, not here (recorded honestly — the
POC as built has the asymmetry): (a) post-commit tier-correction sweep
(move small fs entries into kv during idle GC), (b) require `len` for
the small tier (reject unknown-length small puts — simple, slightly
hostile to streaming producers), (c) accept the duplication (it's
temporary if (a) runs) — or (d) route *unknown-length* puts through a
memory buffer up to the threshold before staging (bounded-memory
reconciliation: small unknown puts never touch fs). Given the
benchmark (small = cheap everywhere), (d) looks natural but should be
weighed against batch-scope pinning from POC #1 finding 3.

## The pack tension (OQ-BL-05 input, from the gix-odb read — analysis only)

gix-odb (v0.84) audit conclusions, in alkblobs terms:

- **Git's own pooling is administratively shared packs**: alternates/
  multi-pack-index give one flat OID→location lookup across many packs
  (slot-map + `ArcSwap` snapshots for lock-free reads), but there is no
  cross-store dedup, no refcounting, no GC, no large-blob handling, no
  streaming reads — every object decompresses whole into memory (writes
  stream; reads do not). gix-lfs is an empty placeholder crate. The
  value-adds alkblobs planned are confirmed unclaimed.
- **Packs are pack-efficient because they delta-chain within
  themselves; a per-object CAS pool fights that in two ways.** First,
  delta chains want objects stored *near their bases*; a flat per-oid
  pool scatters them, so serving a delta-heavy pack read pattern off the
  pool re-inflates from scattered loose entries (loose objects are
  git's *dedup unit and fs bottleneck simultaneously* — gix's own
  discovery docs flag loose-object proliferation as a server-scale
  performance problem). Second, packs are immutable-with-stable-IDs,
  and gitoxide's whole generation/consolidation machinery exists to
  rebind pack IDs when packs churn — a blob pool that *appends per
  object* avoids that machinery entirely (there are no IDs to rebind;
  the address is the content).
- **Resolution shape that the evidence supports** (Phase 1 ADR input):
  git objects enter the pool as **loose-equivalent kv entries** on the
  small tier (the common case — git's median blob is small), and the
  pack question is *deferred to serving*, not storage: if/when serving
  git fetches wants packfile byte-range reads (OQ-BL-05's pack
  tension), the options are (i) store packs as large blobs and serve
  ranges *out of the pack* via the store's range-read (the pack itself
  is the container; per-object addressing rides git's own .idx — the
  store is just a flat byte server for it, which finding A2 says is
  fine locally), or (ii) keep loose-per-object as the only form and
  let consumers synthesize packs above. (i) collapses the tension
  without new store concepts: **the pool stores whole files — including
  packs — and ranged serving is its range-read surface.** Dedup at pack
  granularity is weak, but pack-level dedup across unrelated repos was
  never the goal; object-level dedup lives in the loose tier.
- **Concrete consequence for the store API:** a `metadata(oid)`-shaped
  cheap size/type probe (gix's header-only read is git's cheapest
  primitive) should exist in Phase 1's surface — the POC's
  `LfsObject.len` covers the get side already; make it an explicit
  `stat(digest)` op.

## Alknet probe re-check (historical cross-check only — not design input)

Run for completeness, per the register line that flagged it; per the
session's clarification, **alknet's old code/probe conclusions carry no
decision weight for this crate** (decomposed substrate; alkcall is the
substrate). What the re-check contributes, stripped to the factual:

- The probe's architectural claims about iroh-blobs (no Store trait;
  `Command` enum seam; three-actor pattern; the four pub(crate)
  blockers; the four redb tables; the 16 KiB hybrid) all verified
  against the current checkout — the eval above re-states them
  independently.
- The probe's performance claims ("faster than the filesystem for
  small items") were **asserted-with-citation, never measured** — the
  POC #3 benchmark is the first first-party anchor (finding A4).
- Stale: the version-gap open question (alknet-core is on iroh 1.0
  now); `run_gc` is *not* downstream-importable (private module) —
  irrelevant here, we are not reusing their GC.

## Consequences for the phase-0 OQ register

- **OQ-BL-02 (dispatch):** the small-blob belief is now first-party
  measured (A4). Threshold ~128 KiB default; benchmark harness exists
  (`cargo run --release --bin bench_small`) for re-running on real
  media. Remaining above-trait concerns (migration policy, per-
  namespace config) unchanged — no POC gate.
- **OQ-BL-04 (verification/chunking):** sharpened — range verification
  is impossible under the canonical digest (A2); the chunk-tree
  encoding is a *transfer-layer* conditional whose only consumer is
  networked ranged fetch. Whole-file CAS posture unchanged and
  strengthened. `stat(digest)` surfaces in the Phase 1 API notes.
- **OQ-BL-05 (pooling/GC):** pack tension resolved in shape — packs
  (if stored) are just large blobs served by range reads; object-level
  dedup lives in the loose tier. No new store concepts needed.
- **POC register:** #3 is done; no POCs remain open. Phase 0's next
  step is convergence → Phase 1.

## POC quality notes

- 14 tests (12 integration `large_blob.rs`, 2 CLI cross-check
  `git_cli.rs`); digest validation against real `git hash-object`
  (`--stdin`, repo initialized `--object-format=sha256` — the POC #1
  gotcha holds: the flag is not a hash-object flag).
- clippy `-D warnings` clean (0 warnings), `cargo fmt` clean.
- Deliberate scope limits: single-put-at-a-time (no batch concurrency —
  POC #1's finding 3 covers the pinning story), no persistence-across-
  restart tests (same rationale as POC #1), the fanout seam is
  subscriber-count-fixed (no late-joiner implementation — degrade path
  documented instead), benchmark is one dev box + one working-set
  profile (harness is included for re-running).
- Known micro-detail: `has()` on the kv arm uses a prepared
  `SELECT 1 ... EXISTS`; `get_stream` dispatch does two kv round-trips
  (size, then get) — a schema returning the value directly would avoid
  one; left as-is to keep the size probe identical for both tiers in
  the benchmark comparison.