alkblobs/docs/research/poc-largeblob-findings.md

status, title, last_updated

status	title	last_updated
passed	POC #3 — large-blob streaming path + sqlite-vs-fs micro-benchmark	2026-10-02

POC: large-blob path (streaming, fanout, range reads) + small-object micro-benchmark — findings

POC register #3, per docs/research/phase-0.md OQ-BL-06. Code: standalone crate /workspace/alkblobs-largeblob-poc (findings land here regardless, per the established convention). Companions from the same POC's reading work: iroh-blobs-eval.md (fresh-store eval against checkout e82cbdc / v0.103.0) and the alknet probe re-check (below — recorded as historical cross-check only). Date: 2026-10-02. Status: passed, 14 tests, clippy -D warnings clean, fmt clean. Digests cross-checked against the real git hash-object CLI (git 2.43), extending POC #1's validation to the streaming paths.

What this POC set out to decide

From the phase-0 register:

1. The large-blob path — does an LFSObject-shaped (len, stream) put/get work at the store layer, and what does the git-blob preamble do to streaming hashing?
2. The fanout seam — "serve the fetch while persisting on receipt" (rudolfs' fanout(); iroh's get_blob_ranges_impl) — does it work at our layer, above the backends?
3. Range reads on the fs backend — cheap partial reads with some verification story (OQ-BL-04).
4. The small-object sqlite-vs-fs micro-benchmark — anchor the "kv is faster than fs for small blobs" belief first-party (OQ-BL-02's "dual-belief anchor").
5. The pack tension (OQ-BL-05) — analyzed from the gix-odb read, no POC of its own.

Note on standing: alknet's code is not a design input — it is old, poorly planned, decomposed into the alk* family; its probe docs are treated as references to old research only. The substrate this crate rides is alkcall (like alkhttp/alksocks/alktty/alktunnels).

Result summary

Question	Verdict
Streaming put (LFSObject shape)	Yes — two paths: known-length one-pass; unknown-length stage-then-hash (forced by the preamble, finding A1)
Streaming get without whole-blob load	Yes — fs arm streams a file handle; kv arm one bounded read
Fanout (serve while persisting)	Yes — one reader task, broadcast channel, store arm + N subscriber arms (finding A3)
Range reads on fs	Yes — pread-window only; verification has a real gap (finding A2)
sqlite faster than fs for small blobs	Confirmed and measured — ~9-10× at 1-16 KiB; crossover ~128-256 KiB (finding A4)
Digest = git oid on streaming paths	Byte-exact vs git hash-object for both put paths
Stage-file hygiene	Verified — failures (stream error, overflow, early-EOF) leave zero stage files (finding A5)

Findings

Finding A1: the git-blob preamble splits streaming into exactly two paths — and the split is forced, not chosen

The canonical derivation is H("blob <len>\0" + content). The length is inside the hashed input, so:

• Known length ⇒ write the preamble into the hasher at construction, feed content chunks incrementally, finalize at stream end: one pass over the content, no staging needed for hashing (staging to a temp file still happens for the fs tier, but the hash is done by the time the rename commits). This covers the dominant git and protocol cases: a pre-staged file has its length from stat, a network offer carries a size, an HTTP PUT can carry Content-Length.
• Unknown length ⇒ the preamble can't be written until the stream ends, so content must be staged (file, not memory) and hashed in a second pass over the staged bytes. Two passes are forced by the derivation, not by our choice of hash.

iroh-blobs independently arrives at the same stage-then-encode shape (their add_stream buffers then computes outboard in a second pass, fs/import.rs:290-419), for bao-tree reasons instead of preamble reasons. Their import_bao_reader sidesteps it by knowing both hash and size upfront (a protocol-level offer) — the same "known-length is the normal case" assumption, enforced by their wire format.

Verdict carried to Phase 1: the store's put seam should be (Option<len>, stream) exactly as built. Known-length puts must be the encouraged path (documented as such on the API); unknown-length is supported but slower by one page-cache read. Do not invent a length-prefix-on-the-wire scheme to "fix" unknown-length puts — the wire carries sizes naturally (it must, for the CAS routing).

Finding A2: range reads cannot verify against the canonical digest — the preamble includes the length, so slices have no hash relationship to the whole

Confirmed at implementation level (and independently by the iroh-blobs eval): blob <len>\0 + content gives no verifiable relation between H(whole) and any content[a..b]. iroh gets per-range verification from the bao chunk tree, not from BLAKE3 the algorithm — the tree is the mechanism, the hash is just its leaf function.

POC #3's read_range therefore returns the slice plus a slice digest (git-blob-sha-256 over the slice — a transport-integrity check the caller carries out-of-band, e.g. from the offering side). Consequences:

• Local-store range reads need no more — the OS pages honest media; EIO surfaces read errors; git objects (the small tier) self-verify when git reads them whole.
• The only genuine consumer of a chunk tree is networked large-blob transfer where the caller wants per-chunk integrity. This sharpens OQ-BL-04's "conditional": the bao-like encoding is not about storage at all, it is a transfer encoding. If/when p2p git needs verified ranged fetch of LFS-shaped artifacts, the encoding layer (BLAKE3-confined per OQ-BL-03 tier 3) registers its whole-content root back into the pool as an ordinary git-blob-sha-256 entry.
• Practical corollary: for git-blob-shaped content, byte-range serving is fine (packfiles will want it; see pack tension below) but per-range verification is out of scope for the store layer today.

Finding A3: the fanout seam is a broadcast channel above the store, and late-joiners degrade to get-after-commit

Built put_stream_fanout(stream, n): one reader task consumes the source stream exactly once and broadcasts each chunk to (a) the store put arm and (b) N subscriber feeds (mpsc<io::Result<Bytes>>); subscribers read while the put is still landing. Tests show 3 subscribers each receive the full 256 KiB byte-identical while the put completes, and the put receipt matches the git oid.

Design notes validated:

• One read, many consumers — the source stream must never be polled twice; anything that wants a second copy subscribes. This matches iroh's get_blob_ranges_impl shape (network decode loop joined with the store import) and rudolfs' fanout() (two lock-step copies from one stream). All three arrive at "broadcast from a single received stream" — treat that as the settled seam shape.
• Backpressure discipline: bounded channels (8 items in the POC) make a slow subscriber stall the broadcast — which is correct for a put (the put must land) but means subscriber count is bounded and slow-subscriber handling is a policy question for the ops layer (drop-and-late-join is the escape hatch; see next note).
• Late joiners: a consumer arriving after the live stream has passed simply waits for the commit then does a normal verified get_stream — the CAS guarantees the bytes are identical. The POC didn't build this (subscriber set is fixed at start) but the degrade path is exactly the normal get; worth documenting in the ops layer.

Finding A4 (benchmark): sqlite beats fs decisively for small blobs; crossover ~128-256 KiB; fsync discipline dominates both arms

Release build, per-arm timing loops, 2k distinct keys/values random working set, synchronous=NORMAL WAL sqlite vs stage+rename fs puts / open+read fs gets. Representative run (3 s/arm):

backend     size      puts/s    gets/s   put p50µs  put p99µs  get p50µs  get p99µs
sqlite      1024       48319     49508       20.0       37.0       19.0       34.0
fs          1024        5306      3357      205.0      302.0      288.0      384.0
sqlite     16384       14129     14166       70.0      125.0       68.0      107.0
fs         16384        4203      2399      257.0      395.0      433.0      599.0
sqlite     65536        4028      4048      252.0      364.0      243.0      344.0
fs         65536        3212      1852      339.0      507.0      540.0      688.0
sqlite    131072        2067      2042      494.0      698.0      482.0      699.0
fs        131072        1892      1652      593.0      879.0      619.0      809.0
sqlite    262144         981      1003     1033.0     1792.0      994.0     1176.0
fs        262144        1244      1576      822.0     1181.0      741.0      964.0

Readings:

• ~9-10× sqlite advantage at 1-16 KiB (the git small-blob regime — iroh's DESIGN.md claim and the old citations hold for our workload).
• Crossover at ~128-256 KiB: at 256 KiB fs wins puts (no B-tree rewrite of a 256 KiB row) and gets. The dispatch threshold is a tunable, placed anywhere in 64-256 KiB; POC #1's contract already makes it a constructor parameter. Phase 1 default: 128 KiB (midpoint of the flat zone) — revisit with the storage-medium reality of real consumers (tmpfs vs disk; the POC SSD regime is one data point).
• fs puts cost ~200 µs floor even at 1 KiB — that is the create+write+rename syscall chain; batching stage files (a shared preallocated pool) or O_TMPFILE could shave it, but the sqlite arm makes the effort moot below the crossover.
• Caveats recorded honestly: single-node tmpdir on this dev box (backend-specific tuning — e.g. iroh's redb batch windows — could shift the fs curve), no fsync-in-loop on the fs arm (matching NORMAL WAL durability, sqlite arm likewise not full-fsync; both arms are "crash-tolerant-ish", so the comparison is fair as measured: both are the durability tier a store would ship).
• The old alknet claim "<100 KB faster in sqlite" is confirmed in direction, refined in boundary: the honest crossover is higher (~128-256 KiB) and sqlite is never pathological below it.

Finding A5: stage-file hygiene is testable and the tests earned their keep (again)

The stream_failure_cleans_stage_files test asserted zero files left in the data dir after a mid-stream connection failure — and caught a real bug: the original chunk? error path exited before discarding the stage file, leaking one temp file per failed put. Fixed by restructuring the loops to route all failure paths (stream error, overflow, early-EOF, write error) through one discard. Two lessons compound with POC #1's:

• Resource-hygiene invariants need dedicated tests; code review missed what the exact-count assertion caught (echoing POC #1's exact deletion counts).
• The store's put path now has the shape Phase 1 should codify: every failure path converges on stage-discard; every success path converges on commit-rename — no early returns that bypass cleanup.

Finding A6: re-put dedup across the two put paths holds; unknown-length re-put of a small blob lands on fs (a dispatch asymmetry worth one Phase 1 decision)

The same content put via known-length (small) and unknown-length (staged) paths produces the identical digest and coexists as one entry. But routing differs: an unknown-length put always stages through the fs arm in the POC (routing must wait for the realized size), so a small blob arriving via an unknown-length stream occupies an fs file where the same digest's known-length put would occupy a kv row. If both happen, the pool holds the content twice (different physical tiers).

Options for Phase 1, decided there, not here (recorded honestly — the POC as built has the asymmetry): (a) post-commit tier-correction sweep (move small fs entries into kv during idle GC), (b) require len for the small tier (reject unknown-length small puts — simple, slightly hostile to streaming producers), (c) accept the duplication (it's temporary if (a) runs) — or (d) route unknown-length puts through a memory buffer up to the threshold before staging (bounded-memory reconciliation: small unknown puts never touch fs). Given the benchmark (small = cheap everywhere), (d) looks natural but should be weighed against batch-scope pinning from POC #1 finding 3.

The pack tension (OQ-BL-05 input, from the gix-odb read — analysis only)

gix-odb (v0.84) audit conclusions, in alkblobs terms:

• Git's own pooling is administratively shared packs: alternates/ multi-pack-index give one flat OID→location lookup across many packs (slot-map + ArcSwap snapshots for lock-free reads), but there is no cross-store dedup, no refcounting, no GC, no large-blob handling, no streaming reads — every object decompresses whole into memory (writes stream; reads do not). gix-lfs is an empty placeholder crate. The value-adds alkblobs planned are confirmed unclaimed.
• Packs are pack-efficient because they delta-chain within themselves; a per-object CAS pool fights that in two ways. First, delta chains want objects stored near their bases; a flat per-oid pool scatters them, so serving a delta-heavy pack read pattern off the pool re-inflates from scattered loose entries (loose objects are git's dedup unit and fs bottleneck simultaneously — gix's own discovery docs flag loose-object proliferation as a server-scale performance problem). Second, packs are immutable-with-stable-IDs, and gitoxide's whole generation/consolidation machinery exists to rebind pack IDs when packs churn — a blob pool that appends per object avoids that machinery entirely (there are no IDs to rebind; the address is the content).
• Resolution shape that the evidence supports (Phase 1 ADR input): git objects enter the pool as loose-equivalent kv entries on the small tier (the common case — git's median blob is small), and the pack question is deferred to serving, not storage: if/when serving git fetches wants packfile byte-range reads (OQ-BL-05's pack tension), the options are (i) store packs as large blobs and serve ranges out of the pack via the store's range-read (the pack itself is the container; per-object addressing rides git's own .idx — the store is just a flat byte server for it, which finding A2 says is fine locally), or (ii) keep loose-per-object as the only form and let consumers synthesize packs above. (i) collapses the tension without new store concepts: the pool stores whole files — including packs — and ranged serving is its range-read surface. Dedup at pack granularity is weak, but pack-level dedup across unrelated repos was never the goal; object-level dedup lives in the loose tier.
• Concrete consequence for the store API: a metadata(oid)-shaped cheap size/type probe (gix's header-only read is git's cheapest primitive) should exist in Phase 1's surface — the POC's LfsObject.len covers the get side already; make it an explicit stat(digest) op.

Alknet probe re-check (historical cross-check only — not design input)

Run for completeness, per the register line that flagged it; per the session's clarification, alknet's old code/probe conclusions carry no decision weight for this crate (decomposed substrate; alkcall is the substrate). What the re-check contributes, stripped to the factual:

• The probe's architectural claims about iroh-blobs (no Store trait; Command enum seam; three-actor pattern; the four pub(crate) blockers; the four redb tables; the 16 KiB hybrid) all verified against the current checkout — the eval above re-states them independently.
• The probe's performance claims ("faster than the filesystem for small items") were asserted-with-citation, never measured — the POC #3 benchmark is the first first-party anchor (finding A4).
• Stale: the version-gap open question (alknet-core is on iroh 1.0 now); run_gc is not downstream-importable (private module) — irrelevant here, we are not reusing their GC.

Consequences for the phase-0 OQ register

• OQ-BL-02 (dispatch): the small-blob belief is now first-party measured (A4). Threshold ~128 KiB default; benchmark harness exists (cargo run --release --bin bench_small) for re-running on real media. Remaining above-trait concerns (migration policy, per- namespace config) unchanged — no POC gate.
• OQ-BL-04 (verification/chunking): sharpened — range verification is impossible under the canonical digest (A2); the chunk-tree encoding is a transfer-layer conditional whose only consumer is networked ranged fetch. Whole-file CAS posture unchanged and strengthened. stat(digest) surfaces in the Phase 1 API notes.
• OQ-BL-05 (pooling/GC): pack tension resolved in shape — packs (if stored) are just large blobs served by range reads; object-level dedup lives in the loose tier. No new store concepts needed.
• POC register: #3 is done; no POCs remain open. Phase 0's next step is convergence → Phase 1.

POC quality notes

• 14 tests (12 integration large_blob.rs, 2 CLI cross-check git_cli.rs); digest validation against real git hash-object (--stdin, repo initialized --object-format=sha256 — the POC #1 gotcha holds: the flag is not a hash-object flag).
• clippy -D warnings clean (0 warnings), cargo fmt clean.
• Deliberate scope limits: single-put-at-a-time (no batch concurrency — POC #1's finding 3 covers the pinning story), no persistence-across- restart tests (same rationale as POC #1), the fanout seam is subscriber-count-fixed (no late-joiner implementation — degrade path documented instead), benchmark is one dev box + one working-set profile (harness is included for re-running).
• Known micro-detail: has() on the kv arm uses a prepared SELECT 1 ... EXISTS; get_stream dispatch does two kv round-trips (size, then get) — a schema returning the value directly would avoid one; left as-is to keep the size probe identical for both tiers in the benchmark comparison.