docs(research): iroh/irpc version update findings + alknet-blobs external-store probe
Traces the version gap deferred by the alknet-blobs probe: verifies latest stable iroh 1.0.2 / iroh-blobs 0.103 / irpc 0.17 against crates.io, rules out the 'iroh 0.35 is a stale pin' hypothesis (it's load-bearing — 3 APIs broke), identifies the real cheap win (irpc 0.16 is a dead dep in alknet-call, never imported), and traces the iroh 1.0.x Preset trait so the migration is a concrete 3-edit diff rather than a TODO. Includes the sibling probe doc (alknet-blobs-external-store-probe.md) which the findings cross-references.
This commit is contained in:
glm-5.2 committed
1 parent
1ce11717d8
commit
e024b50b1e
2 files changed
+606
No files matched your search
@@ -0,0 +1,228 @@
|
||||
# alknet-blobs: External SQLite Store — Command-Surface Probe
|
||||
|
||||
**Status:** Probe complete. The external-crate path (no fork) is feasible. The required upstream change is a ~4-line visibility-widening PR with zero logic changes.
|
||||
**Date:** 2026-07-08
|
||||
**Scope:** Resolves open question #1 from `poc-summary.md` ("FsStore (redb + filesystem) vs SQLite — the two-database question"). Determines whether a sibling `alknet-blobs` crate can implement a SQLite-backed blob store that speaks iroh-blobs' `Command` actor protocol *without forking iroh-blobs*, by checking whether the full public API surface is sufficient.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
The POC summary framed this as "fork iroh-blobs to use SQLite instead of redb — a big maintenance commitment." That framing is a **false dichotomy**. There is a third path the summary didn't consider: implement a `SqliteStore` actor in an external crate that speaks iroh-blobs' public `Command` protocol, with **zero changes to iroh-blobs core** plus a trivial upstream visibility PR.
|
||||
|
||||
The probe traced every `Command` variant against the published `iroh-blobs 0.103.0` source. Findings:
|
||||
|
||||
1. **There is no `Store` trait.** `Store` is a concrete struct wrapping an `irpc::Client<proto::Request>` (an mpsc sender). `MemStore` and `FsStore` are two actor implementations that both call `Store::from_sender(sender)`. A third store is a *third actor*, not a trait impl.
|
||||
|
||||
2. **The `Command` enum is `pub`** (generated by the `irpc` `#[rpc_requests]` macro, which propagates the source `pub enum Request`'s visibility). Every request struct, every channel payload type, and every helper type that crosses the actor boundary (`Hash`, `HashAndFormat`, `Tag`, `TagInfo`, `TempTag`, `Bitfield`, `BlobStatus`, `AddProgressItem`, `ExportProgressItem`, `EncodedItem`, `BaoContentItem`, `Leaf`, `Scope`) is `pub`. A downstream actor can match on `Command` and handle every variant using only pub surface.
|
||||
|
||||
3. **The only blockers are four `pub(crate)` items** that need widening to `pub`: `Store::from_sender`, `Store::ref_from_sender`, the `ApiClient` type alias, and the `Scope` field. This is a ~4-line, zero-logic upstream PR. Optionally, exposing `TempTags`/`TempTagScope` would enable iroh-native temp-tag tracking, but a self-managed-GC fallback makes this unnecessary.
|
||||
|
||||
4. **No `Command` variant is BLOCKED.** The redb-specific types (`EntryState`, `DataLocation`, `OutboardLocation`, the redb `tables` module) never cross the pub API boundary — they're confined to the `pub(crate)` `store/fs/meta` module. A SQLite store has its own schema and never touches them.
|
||||
|
||||
5. **GC is reusable, not re-implementable.** `run_gc` (`store/gc.rs:222`) is `pub` and walks the store via the pub `Store` methods (`tags().list()`, `tags().list_temp_tags()`, `blobs().list()`, `blobs().delete()`, `export_bao()`, `sync_db()`). A downstream store that correctly implements those `Command` variants gets iroh's GC for free.
|
||||
|
||||
6. **~100 lines of bao-tree orchestration must be re-implemented** (the `ImportBao`/`ExportBao`/`ExportRanges`/`Observe` handlers). This is re-implementation, not reuse — but all of it sits on pub `bao_tree` APIs (`traverse_ranges_validated`, `PreOrderMemOutboard::create`, `BaoTree`, `ChunkRanges`). `MemStore` already demonstrates this pattern: it has its own `BaoFileStorage`/`PartialMemStorage`; `FsStore` has its own; `alknet-blobs` would have its own. One store impl = one copy of the bao glue.
|
||||
|
||||
**Verdict:** Build `alknet-blobs` as an external crate. Do not fork. Push the 4-line visibility PR upstream. If upstream rejects it, fork — but the fork is ~4 lines different from the external crate, not the "big maintenance commitment" the summary feared.
|
||||
|
||||
---
|
||||
|
||||
## Background: Why the POC summary's framing was wrong
|
||||
|
||||
The POC summary (`poc-summary.md` §"Open Unknowns #1") said:
|
||||
|
||||
> The alternative — forking iroh-blobs to use SQLite instead of redb — is a big maintenance commitment. The POC's write_chunks table proves SQLite can handle chunk-sized inline BLOBs at iroh's granularity, so the swap is mechanically possible. But it should only be done if the two-database coexistence proves problematic, not for aesthetics.
|
||||
|
||||
This assumed the choice was binary: (a) run redb + SQLite side-by-side, or (b) fork iroh-blobs to swap redb for SQLite. Reading the actual code reveals a third option: **an external crate that implements the `Command` actor protocol against SQLite, with iroh-blobs untouched.**
|
||||
|
||||
The key architectural fact the summary missed: **iroh-blobs has no `Store` trait.** `Store` (`src/api.rs:213`) is a concrete struct holding an `irpc::Client<proto::Request>` — a typed mpsc sender. `MemStore` (`src/store/mem.rs`) and `FsStore` (`src/store/fs.rs`) are not trait implementations; they are actor tasks that hold an `mpsc::Receiver<Command>`, handle commands, and produce a `Store` via `Store::from_sender(sender)`. The seam is the `Command` enum, not a trait. A third store is a third actor.
|
||||
|
||||
This re-frames the risk calculus the summary was conservative about. The "two-database" concern (two WALs, two fsync paths, two crash-recovery stories, silent-corruption risk on the seam) is avoided not by "a big maintenance commitment" but by writing a sibling crate that owns its own single SQLite file — exactly the architecture the POC's path-tree + write-chunks + honker work already validated.
|
||||
|
||||
---
|
||||
|
||||
## The Command protocol is pub
|
||||
|
||||
`iroh-blobs 0.103.0`, `src/api/proto.rs`:
|
||||
|
||||
- `pub enum Request` (line 92), decorated with `#[rpc_requests(message = Command, alias = "Msg", rpc_feature = "rpc")]` (line 90).
|
||||
- The `irpc-derive` macro (`/workspace/irpc/irpc-derive/src/lib.rs:146`) emits `#vis enum #message_enum_name { ... }`, propagating the source enum's `pub`. So **`pub enum Command`** is generated, with one variant per `Request` variant, each wrapping `WithChannels<RequestStruct, Request>`.
|
||||
- `WithChannels` fields (`inner`, `tx`, `rx`) are `pub` (`irpc/src/lib.rs:1162-1168`).
|
||||
- Per-variant `*Msg` type aliases (e.g. `ImportBaoMsg`, `ExportBaoMsg`) are generated as `pub type` (`irpc-derive/src/lib.rs:328-335`).
|
||||
- `From<WithChannels<T, Request>> for Command` impls are generated and pub (`irpc-derive/src/lib.rs:279-281`).
|
||||
|
||||
Every request struct (`ImportBaoRequest`, `ExportBaoRequest`, `BlobStatusRequest`, `SetTagRequest`, `ListTagsRequest`, …) is `pub` with `pub` fields. Every channel payload type (`Hash`, `HashAndFormat`, `Tag`, `TagInfo`, `TempTag`, `Bitfield`, `BlobStatus`, `AddProgressItem`, `ExportProgressItem`, `EncodedItem`, `BaoContentItem`, `Leaf`, `Scope`, `BatchResponse`, `ImportByteStreamUpdate`, `ExportRangesItem`) is `pub`.
|
||||
|
||||
The `Command` enum is the actor's inbox type. A downstream crate that spawns a tokio task holding `mpsc::Receiver<Command>` can `match` on every variant and produce a `Store` via `Store::from_sender(sender)` — once that constructor is `pub`.
|
||||
|
||||
---
|
||||
|
||||
## The four items that must be made `pub`
|
||||
|
||||
### Mandatory (the ~4-line PR)
|
||||
|
||||
| Item | File:line | Current | Needed for |
|
||||
|---|---|---|---|
|
||||
| `ApiClient` type alias | `src/api.rs:33` | `pub(crate) type ApiClient = irpc::Client<proto::Request>;` | The external store wrapper holds an `ApiClient` field (as `MemStore`/`FsStore` do). Without it the `from_sender` signature can't be expressed. **Alternative:** make `from_sender` take `irpc::Client<proto::Request>` directly and keep `ApiClient` private — fewer surface changes. |
|
||||
| `Store::from_sender` | `src/api.rs:299` | `pub(crate) fn from_sender(client: ApiClient) -> Self` | External actor constructs a `Store` from its `mpsc::Sender`. This is the single load-bearing change. |
|
||||
| `Store::ref_from_sender` | `src/api.rs:303` | `pub(crate) fn ref_from_sender(client: &ApiClient) -> &Self` | `Deref`/`AsRef<Store>` for the external store wrapper (matches `MemStore`/`FsStore` pattern). |
|
||||
| `Scope` field | `src/api/proto.rs:693` | `pub struct Scope(pub(crate) u64);` | `Batch` must *construct* a `Scope` to send back on its `oneshot::Sender<Scope>`; import variants receive a `Scope` they must *destructure*. Only `Scope::GLOBAL` is constructible today. **Minimum:** make the field `pub` (it's a `u64`), or add `Scope::new(u64)`/`Scope::into_inner()`. |
|
||||
|
||||
### Optional (enables iroh-native temp-tag tracking, NOT required)
|
||||
|
||||
| Item | File:line | Current | Why |
|
||||
|---|---|---|---|
|
||||
| `util::temp_tag` module | `src/util.rs:4` | `pub(crate) mod temp_tag;` | `TempTags`, `TempTagScope`, `TagDrop`, `TagCounter` are sealed. Without this, imports can only emit *untracked* temp tags via `TempTag::new(haf, None)` (pub). |
|
||||
| `TempTags` struct + `create`/`create_scope`/`end_scope` | `src/util/temp_tag.rs:173` | `pub(crate) struct TempTags` (methods individually `pub` but in sealed module) | Refcounted, GC-aware temp tags for imports and `CreateTempTag`. |
|
||||
| `TempTagScope` + `TagDrop` | `src/util/temp_tag.rs:211,60` | sealed | `Batch` scope bookkeeping (`scope.on_drop(&msg)`). |
|
||||
| `Bitfield::new_unchecked` | `src/api/proto/bitfield.rs:74` | `pub(crate)` | `ImportBao` fs handler uses it. `Bitfield::new` (pub) is a fine substitute. **Not required.** |
|
||||
|
||||
**The fallback if `TempTags` stays sealed:** use `TempTag::new(haf, None)` (pub today) for imports. The tag is valid on the wire and protects the blob from GC for the tag's lifetime, but `on_drop` is `None` so the refcount never decrements — effectively a leak until process exit or explicit `DeleteBlobs`. This is acceptable because:
|
||||
|
||||
1. alknet-blobs owns its SQLite schema; the path tree (from the POC) already tracks which hashes are live via path rows + tags. Self-managed GC against SQLite is the natural design.
|
||||
2. `MemStore` and `FsStore` each own their own `TempTags` instance already — there is no shared global temp-tag registry. A third store having its own is the existing pattern, not a deviation.
|
||||
3. iroh's `run_gc` (see below) walks the store via pub `Store` methods, so it works regardless of how temp tags are tracked internally — as long as `ListTempTags` returns the right set.
|
||||
|
||||
Under this fallback, the **only strictly-required upstream change is the 4 lines above.**
|
||||
|
||||
---
|
||||
|
||||
## GC is reusable (a correction to the probe's first pass)
|
||||
|
||||
The first pass of this probe incorrectly concluded that `run_gc` was `pub(crate)` and that a downstream store would have to write its own GC loop. That is wrong:
|
||||
|
||||
- `pub async fn run_gc(store: Store, config: GcConfig)` — `src/store/gc.rs:222`. It is `pub`.
|
||||
- `gc_mark_task` (`gc.rs:34`) walks live hashes via: `store.tags().list()`, `store.tags().list_temp_tags()`, `store.export_bao(hash, ChunkRanges::all()).hashes()`.
|
||||
- `gc_sweep_task` (`gc.rs:83`) deletes dead blobs via: `store.blobs().list().stream()`, `store.blobs().delete(batch)`, `store.sync_db()`.
|
||||
|
||||
Every one of those is a `pub` method on `Store` backed by a `Command` variant (`ListTags`, `ListTempTags`, `ExportBao`, `ListBlobs`, `DeleteBlobs`, `SyncDb`). A downstream `alknet-blobs` store that correctly implements those six variants gets iroh's mark-sweep GC **for free** — it just calls `run_gc(store.clone(), gc_config)` in a spawned task, exactly as `MemStore::new_with_opts` (`mem.rs:143`) and `FsStore::load_with_opts` (`fs.rs:1427`) do.
|
||||
|
||||
The implication: even under the "self-managed temp tags" fallback, GC is not a thing alknet-blobs has to build. It's a thing it gets by implementing the `Command` variants it would implement anyway.
|
||||
|
||||
---
|
||||
|
||||
## Per-variant verdict
|
||||
|
||||
All ~24 `Command` variants traced. Channel payload types are `pub` for every variant. `EntryState`/`DataLocation`/`OutboardLocation` (redb value types) **never appear in pub channel types** — confirmed by reading every `#[rpc(tx=..., rx=...)]` attribute in `proto.rs`. They are confined to the `pub(crate)` redb meta actor.
|
||||
|
||||
| Variant | Verdict | Notes |
|
||||
|---|---|---|
|
||||
| `ListBlobs` | **PUB-OK** | Stream `Hash` from your DB. |
|
||||
| `Batch` | **NEEDS-PUB** | Must construct `Scope` for the `oneshot` reply — needs `Scope` field pub. The `TempTagScope::on_drop` bookkeeping is sealed; implement your own ~30-line scope tracker or skip batch-scoped cleanup (acceptable for a SQLite store that owns its own GC). |
|
||||
| `DeleteBlobs` | **PUB-OK** | Delete in your DB; protected-set is your own. |
|
||||
| `ImportBao` | **PUB-OK (logic-heavy)** | Re-implement ~60-100 lines of bao-tree chunk assembly (`BaoContentItem::{Parent,Leaf}` → outboard + data + `Bitfield`). Uses pub `bao_tree` APIs. See "Re-implementation cost" below. |
|
||||
| `ExportBao` | **PUB-OK** | Call pub `traverse_ranges_validated` with a 6-line `Sender` impl you write yourself (the iroh `BaoTreeSender` is sealed but trivial). |
|
||||
| `ExportRanges` | **PUB-OK** | ~40 lines of range iteration + `Bitfield` superset checks + `Leaf` emission. Pub `Bitfield`/`Leaf`/`RangeSet2`. |
|
||||
| `Observe` | **PUB-OK** | Implement your own `watch::Sender<Bitfield>` → `mpsc::Sender<Bitfield>` forwarder (~15 lines). Pub `Bitfield`. |
|
||||
| `BlobStatus` | **PUB-OK** | Return `NotFound`/`Partial`/`Complete` from your DB. |
|
||||
| `ImportBytes` | **NEEDS-PUB** | `Scope` field needed (request carries `scope`). Emit `AddProgressItem::Done(TempTag)` via `TempTag::new(haf, None)` (pub, untracked) or `TempTags::create` (sealed). |
|
||||
| `ImportByteStream` | **NEEDS-PUB** | Same as `ImportBytes`. |
|
||||
| `ImportPath` | **NEEDS-PUB** | Same as `ImportBytes`. |
|
||||
| `ExportPath` | **PUB-OK** | Write your data to target path; emit `ExportProgressItem`. |
|
||||
| `ListTags` | **PUB-OK** | Query your tag table. |
|
||||
| `SetTag` | **PUB-OK** | |
|
||||
| `DeleteTags` | **PUB-OK** | |
|
||||
| `RenameTag` | **PUB-OK** | |
|
||||
| `CreateTag` | **PUB-OK** | `Tag::auto` is pub (`store/util.rs:72`). |
|
||||
| `ListTempTags` | **PUB-OK** | Return your own temp-tag list. **Required for `run_gc` to work** — it calls this. |
|
||||
| `CreateTempTag` | **NEEDS-PUB** | `Scope` field. `TempTag::new(value, None)` (pub) for untracked, or `TempTags::create` (sealed) for tracked. |
|
||||
| `SyncDb` | **PUB-OK** | Flush your SQLite. |
|
||||
| `WaitIdle` | **PUB-OK** | Track your own task set. |
|
||||
| `Shutdown` | **PUB-OK** | |
|
||||
| `ClearProtected` | **PUB-OK** | Your own protected set. |
|
||||
|
||||
**NEEDS-PUB count:** 6 variants (`Batch`, `ImportBytes`, `ImportByteStream`, `ImportPath`, `CreateTempTag`) — all blocked only by the `Scope` field. One change (`Scope(pub u64)`) unblocks all six.
|
||||
|
||||
**BLOCKED count:** 0.
|
||||
|
||||
---
|
||||
|
||||
## Re-implementation cost (the real maintenance burden)
|
||||
|
||||
The visibility PR is trivial. The actual work — and the honest maintenance cost — is re-implementing the bao-tree orchestration that the `ImportBao`/`ExportBao`/`ExportRanges`/`Observe` handlers do. These handlers don't wire pub pieces together; they *orchestrate* pub `bao_tree` APIs. A downstream store writes its own copy.
|
||||
|
||||
| Handler | Lines to re-implement | Uses (all pub) |
|
||||
|---|---|---|
|
||||
| `ImportBao` | ~60-100 | `BaoContentItem`, `BaoTree`, `ChunkRanges`, `Bitfield::new`, `PreOrderMemOutboard` (for compute-on-import path), `Leaf`. Re-derive the "size-1 leaf validates size" check (`is_validated`, sealed in `pub(crate)` bitfield module — ~3 lines). |
|
||||
| `ExportBao` | ~6 | `traverse_ranges_validated` + a `Sender for &mut mpsc::Sender<EncodedItem>` impl (the iroh `BaoTreeSender` is a 6-line `ref_cast` wrapper at `store/util.rs:400-415`). |
|
||||
| `ExportRanges` | ~40 | `Bitfield` superset check, `RangeSet2` iteration, `Leaf` emission. |
|
||||
| `Observe` | ~15 | `tokio::sync::watch` → `mpsc::Sender<Bitfield>` forwarder. |
|
||||
|
||||
**Total: ~120-160 lines of bao-tree glue** to re-implement in alknet-blobs. This is the same glue `MemStore` has its own copy of (`src/store/mem.rs`, the `BaoFileStorage`/`PartialMemStorage`/`import_bao`/`export_bao` functions) and `FsStore` has its own copy of (`src/store/fs/bao_file.rs`, 749 lines). The pattern is "one store impl = one copy of the bao glue." alknet-blobs is a third store impl; it gets a third copy. The copy is smaller than FsStore's because a SQLite store doesn't need the partial-blob-on-filesystem lifecycle (the POC's `write_chunks` table handles partial state in SQLite).
|
||||
|
||||
This is re-implementation, not reuse — but it is re-implementation against *stable, pub `bao_tree` APIs*, not against iroh internals. The maintenance cost is: track `bao_tree` version bumps. That is a much smaller surface than "track all of iroh-blobs."
|
||||
|
||||
---
|
||||
|
||||
## alknet-crate reuse for the write/read path
|
||||
|
||||
The four existing alknet crates (post-POC) reduce some of the work the POC flagged as unknown:
|
||||
|
||||
- **`alknet-tty`** (`crates/alknet-tty/`) — `TtyHandle` already exposes `tokio::io::AsyncWrite` stdin and `Stream<Item=Bytes>` stdout (`src/backend.rs:139-182`). The `MockStdinSink` (`src/backend.rs:366-403`) is a working `AsyncWrite`-over-`mpsc::Sender<Bytes>` adapter — directly reusable as a pattern for the SFTP write path's chunk bridge into `WriteSession::write_chunk`. The kill-on-Drop contract pattern (`exit_code` future, ADR-056) is the same shape as write-session abort-on-drop.
|
||||
- **`alknet-call`** (`crates/alknet-call/`) — already uses `irpc` (workspace dep `irpc = "0.16"`) for structured RPC. The `EventEnvelope`/length-prefixed JSON framing (`src/protocol/wire.rs`) is a different protocol than iroh-blobs' `Command` actor protocol, but the irpc usage pattern (matching on request variants, channel-based replies) is the same shape alknet-blobs' actor will use.
|
||||
- **`alknet-core`** (`crates/alknet-core/`) — optional `iroh = "0.35"` dep. **Version mismatch:** alknet-core is on `iroh 0.35`/`irpc 0.16`; published `iroh-blobs 0.103` requires `iroh 1.0`/`irpc 0.17`. alknet-blobs will pull iroh-blobs 0.103's irpc 0.17 transitively. This needs resolution at spec time — either alknet-core bumps to iroh 1.0, or alknet-blobs pins an iroh-blobs version compatible with irpc 0.16. **Open question for the spec.**
|
||||
|
||||
None of the existing alknet crates implement the blob store's bao-tree orchestration — that is genuinely new work. But the async-IO adapter patterns (AsyncWrite-over-mpsc, Stream-of-Bytes, kill-on-drop futures) are already proven in `alknet-tty` and reduce the write-path wiring risk.
|
||||
|
||||
---
|
||||
|
||||
## Versioning reality
|
||||
|
||||
Three iroh-blobs generations are in play:
|
||||
|
||||
| Source | Version | iroh | irpc | Notes |
|
||||
|---|---|---|---|---|
|
||||
| `/workspace/iroh-blobs` (local checkout) | 0.100.0 | 0.98 | 0.14 | Pre-n0-rebrand. Used for the original architecture reading. |
|
||||
| crates.io published (cached) | 0.103.0 | 1.0 | 0.17 | What the POC used; what alknet-blobs would target. |
|
||||
| alknet-core dep | — | 0.35 | 0.16 | Two majors behind iroh-blobs 0.103. |
|
||||
|
||||
This probe was run against **0.103.0** (the published version). The visibility findings (`Store::from_sender` pub(crate), `Scope(pub(crate) u64)`, `Command` pub via macro, no `Store` trait, `EntryState` confined to `pub(crate)` meta) all hold on 0.103.0. The 4-line upstream PR would target the 0.103+ line.
|
||||
|
||||
The alknet-core `iroh 0.35` vs iroh-blobs `iroh 1.0` gap is a spec-time decision, not a probe blocker. It affects which iroh QUIC API the distributed-sync layer (POC 3) binds against, but not the local `Command`-actor surface this probe examined.
|
||||
|
||||
---
|
||||
|
||||
## Recommended path
|
||||
|
||||
1. **Build `alknet-blobs` as a workspace member** (`crates/alknet-blobs/`), not a fork. It depends on `iroh-blobs = "0.103"` (published) and implements the `Command` actor against a single SQLite file (schema: the POC's path-tree tables + write_chunks + a blob-metadata table replacing redb's 4 tables + honker tables, all in one .db).
|
||||
|
||||
2. **Push the 4-line visibility PR upstream** to n0-computer/iroh-blobs: `Store::from_sender`/`ref_from_sender` + `ApiClient` (or change `from_sender` to take `irpc::Client<proto::Request>` directly) + `Scope` field. Zero logic changes. If merged, alknet-blobs is a pure downstream crate. If rejected, alknet-blobs forks those 4 lines — the fork is 4 lines different from the external crate, which is the opposite of "big maintenance commitment."
|
||||
|
||||
3. **Use the self-managed-GC fallback** (`TempTag::new(haf, None)` + own SQLite-backed liveness tracking) unless upstream agrees to expose `TempTags`. Reuse iroh's `run_gc` by correctly implementing `ListTempTags`/`ListTags`/`ListBlobs`/`DeleteBlobs`/`ExportBao`/`SyncDb`. This is the same pattern `MemStore`/`FsStore` use (each owns its own `TempTags`); alknet-blobs owns its own, just backed by SQLite.
|
||||
|
||||
4. **Re-implement the ~120-160 lines of bao-tree orchestration** (`ImportBao`/`ExportBao`/`ExportRanges`/`Observe`) against pub `bao_tree` APIs. Copy the structure from `MemStore` (the simpler reference; `src/store/mem.rs`), not `FsStore` (which has the partial-blob-on-filesystem complexity alknet-blobs doesn't need).
|
||||
|
||||
5. **Reuse `alknet-tty`'s async-IO adapter patterns** (AsyncWrite-over-mpsc, Stream-of-Bytes, kill-on-drop) for the SFTP write-path bridge into `WriteSession`.
|
||||
|
||||
6. **Resolve the iroh 0.35 vs 1.0 version gap** at spec time. This is an open question for the architecture doc, not a probe finding.
|
||||
|
||||
---
|
||||
|
||||
## What this closes and what remains
|
||||
|
||||
**Closes:** POC open question #1 ("FsStore (redb + filesystem) vs SQLite — the two-database question"). The answer is: neither. Build a third store (`alknet-blobs`) as an external crate with its own single SQLite file. No fork, no two-database coexistence. The upstream cost is a 4-line visibility PR.
|
||||
|
||||
**Does not close (deferred to spec):**
|
||||
- The iroh 0.35 vs iroh 1.0 version gap (alknet-core dependency alignment).
|
||||
- The exact SQLite schema for the blob-metadata tables (the redb 4-table shape is a reference; a SQLite store may unify inline_data/inline_outboard into BLOB columns or split differently — schema design is a spec task).
|
||||
- Whether to expose `TempTags` upstream or commit to the self-managed-GC fallback (a spec decision; the fallback works, exposing `TempTags` is a nicety).
|
||||
- SFTP wiring (POC open question #3 — unchanged; the `alknet-tty` patterns help but don't close it).
|
||||
- The distributed sync layer's binding to iroh 1.0 vs 0.35 (POC 3 used iroh 1.0; alknet-core is on 0.35).
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Published source probed: `~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/iroh-blobs-0.103.0/`
|
||||
- `src/api.rs:33,213,299,303` — `ApiClient`, `Store`, `from_sender`/`ref_from_sender` (pub(crate))
|
||||
- `src/api/proto.rs:693` — `Scope(pub(crate) u64)`
|
||||
- `src/api/proto.rs:90-139` — `pub enum Request` + `#[rpc_requests]` (generates `pub enum Command`)
|
||||
- `src/store/mem.rs` — reference actor (1127 lines, zero redb)
|
||||
- `src/store/fs/meta.rs`, `src/store/fs/meta/tables.rs` — redb confined here (~1100 lines)
|
||||
- `src/store/fs/entry_state.rs` — `EntryState`/`DataLocation`/`OutboardLocation` (pub(crate), never in pub channel types)
|
||||
- `src/store/gc.rs:222` — `pub async fn run_gc` (reusable, walks store via pub methods)
|
||||
- `src/util/temp_tag.rs:85` — `TempTag::new` (pub, untracked fallback); `:173` — `TempTags` (pub(crate))
|
||||
- irpc derive macro: `/workspace/irpc/irpc-derive/src/lib.rs:146,279,328` — visibility propagation, `From` impls, `*Msg` aliases
|
||||
- POC summary: `docs/research/alknet-filesystem/poc-summary.md` (this doc's parent context)
|
||||
- alknet crates surveyed: `crates/alknet-tty/src/backend.rs` (AsyncWrite/Stream/kill-on-drop patterns), `crates/alknet-call/src/protocol/wire.rs` (irpc usage), `crates/alknet-core/Cargo.toml` (iroh 0.35 version gap)
|
||||
@@ -0,0 +1,378 @@
|
||||
# iroh / irpc Version Update — Findings & Migration Plan
|
||||
|
||||
**Status:** Findings complete. Migration plan drafted; not yet applied. Two cheap wins identified (dead `irpc` dep, stale-pin hypothesis ruled out), one real migration traced end-to-end against iroh 1.0.2.
|
||||
**Date:** 2026-07-08
|
||||
**Scope:** Resolves the version-gap open question deferred by `alknet-blobs-external-store-probe.md` §"Versioning reality" and §"What this closes and what remains". Verifies latest stable versions on crates.io (not training-data defaults), determines whether alknet-core's `iroh 0.35` dep is load-bearing or a stale pin, and traces the exact `Preset` trait surface in iroh 1.0.x so the migration is concrete rather than a TODO.
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
1. **Latest stable versions verified against the crates.io API** (2026-07-08): `iroh 1.0.2`, `iroh-blobs 0.103.0`, `irpc 0.17.0`, `irpc-derive 0.17.0`, `bao-tree 0.16.0`. iroh-blobs has **not** moved past 0.103 — no pre-releases, no newer stable. The probe's target version is the current latest; the 4-line/20-line fork premise still holds.
|
||||
|
||||
2. **`iroh 0.35` in alknet-core is load-bearing, not a stale pin.** The "pure Cargo.toml change" hypothesis is wrong. `crates/alknet-core/src/endpoint.rs` uses ~15 distinct iroh APIs under `#[cfg(feature = "iroh")]`. Three of them broke between 0.35 and 1.0 and require code edits. The bump is small (~3 edits in `endpoint.rs`) but real.
|
||||
|
||||
3. **`irpc 0.16` in the workspace is a dead dep — the actual cheap win.** The probe doc claimed `alknet-call` "already uses `irpc` for structured RPC." That is false: zero `.rs` files in the workspace reference `irpc`. The dep is declared in `Cargo.toml` and never imported. Dropping it (or bumping to 0.17) dissolves the `irpc 0.16 vs 0.17` half of the version gap with zero code impact.
|
||||
|
||||
4. **The iroh 1.0.x migration target is stable.** All three flagged APIs are byte-identical between iroh 1.0.0 and 1.0.2 (the actual latest). Pinning `iroh = "1.0"` resolves to 1.0.2 and the migration edits are stable against it. No "do it again when they update" risk on the 1.0.x line.
|
||||
|
||||
5. **The `Preset` trace is complete.** alknet-core should use `iroh::endpoint::presets::Minimal` — it sets only the now-mandatory `crypto_provider` and leaves relay mode, address lookup, ALPNs, and secret key for alknet-core to configure as it already does. This matches the pattern iroh-blobs 0.103's own tests use (`tests.rs:506,525`).
|
||||
|
||||
---
|
||||
|
||||
## 1. Latest stable versions (crates.io API, 2026-07-08)
|
||||
|
||||
Queried `https://crates.io/api/v1/crates/<name>` for each crate. Versions below are `max_stable_version` (== `newest_version` for all of these — no pre-releases in flight).
|
||||
|
||||
| crate | latest stable | updated | alknet-core pin | iroh-blobs 0.103 requires | gap |
|
||||
|---|---|---|---|---|---|
|
||||
| `iroh` | **1.0.2** | 2026-07-06 | `0.35` (optional) | `^1.0.0` (runtime) | 2 majors |
|
||||
| `iroh-base` | **1.0.2** | 2026-07-06 | (transitive 0.35) | `^1.0.0` (runtime) | follows iroh |
|
||||
| `iroh-relay` | **1.0.2** | 2026-07-06 | (transitive 0.35) | — | follows iroh |
|
||||
| `iroh-blobs` | **0.103.0** | 2026-06-15 | — | — | target version, no newer |
|
||||
| `irpc` | **0.17.0** | 2026-06-15 | `0.16` (workspace, **unused**) | `^0.17.0` (runtime) | 1 minor — dead dep |
|
||||
| `irpc-derive` | **0.17.0** | 2026-06-15 | `0.16` (workspace, **unused**) | — | same as irpc |
|
||||
| `bao-tree` | **0.16.0** | 2025-11-04 | (not declared) | `^0.16` (runtime) | alknet-blobs will add |
|
||||
|
||||
### iroh-blobs has not moved past 0.103
|
||||
|
||||
Recent version history (crates.io `/versions` endpoint):
|
||||
|
||||
```
|
||||
0.103.0 2026-06-15
|
||||
0.102.0 2026-05-27
|
||||
0.101.0 2026-05-08
|
||||
0.100.0 2026-04-20
|
||||
0.99.0 2026-03-17
|
||||
0.98.0 2026-01-28
|
||||
```
|
||||
|
||||
`max_stable == newest == 0.103.0`. No yanked releases, no pre-releases. The probe's `Command`/`Store`/`from_sender`/`Scope` surface findings (all verified against `iroh-blobs-0.103.0/src/api.rs:33,299,303` and `api/proto.rs:693`) are current. The fork premise holds.
|
||||
|
||||
### iroh-blobs 0.103's iroh/irpc deps are runtime, not dev
|
||||
|
||||
The crates.io API's dependency `kind` field was misleading (reported everything as "build"). The authoritative source is the raw `Cargo.toml` in the cargo registry cache:
|
||||
|
||||
```
|
||||
[dependencies.iroh]
|
||||
version = "1.0.0"
|
||||
default-features = false
|
||||
|
||||
[dependencies.iroh-base]
|
||||
version = "1.0.0"
|
||||
|
||||
[dependencies.irpc]
|
||||
version = "0.17.0"
|
||||
features = ["spans", "stream", "derive", "varint-util"]
|
||||
default-features = false
|
||||
```
|
||||
|
||||
All three are in `[dependencies]`, not `[dev-dependencies]`. So any crate depending on `iroh-blobs 0.103` pulls `iroh 1.0` and `irpc 0.17` transitively. The version gap is real and load-bearing for alknet-blobs.
|
||||
|
||||
---
|
||||
|
||||
## 2. `iroh 0.35` is load-bearing (not a stale pin)
|
||||
|
||||
The other session's hypothesis: "if nothing in alknet-core's current code exercises iroh 0.35-specific APIs, bumping to 1.0 may be a pure Cargo.toml change." Verified by grep — this is wrong.
|
||||
|
||||
`crates/alknet-core/src/endpoint.rs` uses these iroh APIs under `#[cfg(feature = "iroh")]`:
|
||||
|
||||
- `iroh::Endpoint::builder()` — `endpoint.rs:681`
|
||||
- `iroh::SecretKey::from_bytes` / `SecretKey::generate` — `endpoint.rs:684,688`
|
||||
- `iroh::RelayMap::from` / `RelayMode::Custom` / `RelayMode::Disabled` — `endpoint.rs:694-697`
|
||||
- `iroh::Endpoint::accept()`, `Endpoint::close()` — `endpoint.rs:403,228`
|
||||
- `iroh::endpoint::{Connection, SendStream, RecvStream, ConnectionError, VarInt, ApplicationClose}` — `endpoint.rs:441`; `types.rs:232,240,366,473,504,824-856`
|
||||
- `iroh::RelayUrl` — `config.rs:28`
|
||||
|
||||
Three of these broke between 0.35 and 1.0 (see §4). The pin is load-bearing.
|
||||
|
||||
---
|
||||
|
||||
## 3. `irpc 0.16` is a dead dep (the actual cheap win)
|
||||
|
||||
The probe doc (`alknet-blobs-external-store-probe.md:163`) claims:
|
||||
|
||||
> **`alknet-call`** — already uses `irpc` (workspace dep `irpc = "0.16"`) for structured RPC.
|
||||
|
||||
**This is false.** Grepped every `.rs` file under `crates/alknet-call/src/` and `crates/alknet-call/tests/` for `irpc` — **zero matches**. The only `irpc` reference in alknet-call is `Cargo.toml:18` (`irpc = { workspace = true }`). The actual wire protocol in `src/protocol/wire.rs` is hand-rolled length-prefixed JSON (`EventEnvelope`/`ResponseEnvelope`/`FrameFramedReader`/`FrameFramedWriter`) — no `irpc` types, no `#[rpc_requests]` macro, no `irpc::Client`.
|
||||
|
||||
The workspace `irpc = "0.16"` / `irpc-derive = "0.16"` deps (`Cargo.toml:19-20`) are **declared but unused by any crate in the workspace**.
|
||||
|
||||
### Why this matters for the version gap
|
||||
|
||||
The only thing pinning `irpc 0.16` in the workspace is a dead dep. Two options, both free:
|
||||
|
||||
1. **Drop the dead dep entirely** (cleanest): remove `irpc`/`irpc-derive` from workspace `Cargo.toml:19-20` and from `alknet-call/Cargo.toml:18`. Re-add when `alknet-blobs` actually needs `irpc 0.17` (it will, as a real dep).
|
||||
2. **Bump to 0.17 now**: change workspace `Cargo.toml` to `irpc = "0.17"` / `irpc-derive = "0.17"`. Zero code impact since nothing uses it. Gets the version in place for `alknet-blobs`.
|
||||
|
||||
Either way, the `irpc 0.16 vs 0.17` half of the gap dissolves with no migration work. This is the "stale pin" cheap win — it just wasn't where the probe said it was (the probe pointed at `iroh 0.35`; the stale pin was `irpc 0.16`).
|
||||
|
||||
---
|
||||
|
||||
## 4. The iroh 0.35 → 1.0.2 migration (3 edits)
|
||||
|
||||
Diffed the actual pub surfaces of `iroh-0.35.0` vs `iroh-1.0.0` (cached in cargo registry) and `iroh-1.0.2` (fetched from static.crates.io). Three APIs broke; everything else alknet-core uses is stable.
|
||||
|
||||
### 4.1 Breaking changes (require code edits)
|
||||
|
||||
| location | 0.35 | 1.0.2 | fix |
|
||||
|---|---|---|---|
|
||||
| `endpoint.rs:681` | `iroh::Endpoint::builder()` (no args) | `iroh::Endpoint::builder(preset: impl Preset)` — `endpoint.rs:950` | pass `presets::Minimal` (see §5) |
|
||||
| `endpoint.rs:688` | `iroh::SecretKey::generate(&mut csprng)` — `iroh-base-0.35.0/src/key.rs:273` takes `R: CryptoRngCore` | `iroh::SecretKey::generate()` — `iroh-base-1.0.2/src/key.rs:318` takes no args (uses internal rng) | drop the `&mut csprng` arg + the `let mut csprng = ...` line |
|
||||
| `endpoint.rs:684` | `iroh::SecretKey::from_bytes(&[u8;32]) -> Result<Self, SignatureError>` — `iroh-base-0.35.0/src/key.rs:109` | `iroh::SecretKey::from_bytes(&[u8;32]) -> Result<Self, KeyParsingError>` — `iroh-base-1.0.2/src/key.rs:122` (error type renamed) | add `?` + map `KeyParsingError` to `EndpointError::TlsConfig` |
|
||||
|
||||
All three confirmed **byte-identical between iroh 1.0.0 and iroh 1.0.2** — the migration target is stable. Pinning `iroh = "1.0"` resolves to 1.0.2 and these edits hold.
|
||||
|
||||
### 4.2 Stable APIs (no edits needed)
|
||||
|
||||
Confirmed by diffing `iroh-0.35.0` vs `iroh-1.0.2` cached/fetched sources:
|
||||
|
||||
- `iroh::RelayMap::from<RelayUrl>` — stable (`iroh-relay-1.0.0/src/relay_map.rs:181`)
|
||||
- `iroh::RelayMode::{Custom, Disabled}` — stable (`iroh-1.0.2/src/endpoint.rs:1922`). Note: 1.0 adds `RelayMode::Default` and `RelayMode::Staging` variants, but alknet-core only uses `Custom`/`Disabled`, which are unchanged.
|
||||
- `iroh::Endpoint::accept()` — stable (`iroh-1.0.2/src/endpoint.rs:1162`)
|
||||
- `iroh::Endpoint::close()` — stable (`iroh-1.0.2/src/endpoint.rs:1703`)
|
||||
- `iroh::endpoint::{Connection, SendStream, RecvStream, ConnectionError, VarInt, ApplicationClose}` — stable (module path unchanged)
|
||||
- `Builder::secret_key`, `Builder::alpns`, `Builder::relay_mode`, `Builder::bind` — all stable (signatures unchanged in 1.0.2)
|
||||
|
||||
### 4.3 The 1.0 Builder also changed structurally (context, not a direct edit)
|
||||
|
||||
The 0.35 `Builder::default()` set `relay_mode: default_relay_mode()` (n0 prod relays) and had no `crypto_provider` field. The 1.0 `Builder::empty()` starts with `RelayMode::Disabled` and **requires** `crypto_provider` to be set before `bind()` (returns `BindError::InvalidCryptoProvider` otherwise — `endpoint.rs:230`).
|
||||
|
||||
This is why the `Preset` arg to `builder()` exists: it's the mechanism for setting the mandatory `crypto_provider`. alknet-core's existing explicit `.relay_mode(...)` call (line 695/697) overrides whatever the preset sets, so the preset only needs to supply the crypto provider. That's exactly what `presets::Minimal` does (see §5).
|
||||
|
||||
---
|
||||
|
||||
## 5. The `Preset` trait trace (iroh 1.0.x)
|
||||
|
||||
**Location:** `iroh-1.0.2/src/endpoint/presets.rs` (re-exported as `iroh::endpoint::presets`)
|
||||
|
||||
### The trait
|
||||
|
||||
```rust
|
||||
// presets.rs:21
|
||||
pub trait Preset {
|
||||
fn apply(self, builder: Builder) -> Builder;
|
||||
}
|
||||
```
|
||||
|
||||
Single-method. `Endpoint::builder(preset)` calls `preset.apply(Builder::empty())`.
|
||||
|
||||
### The four built-in presets
|
||||
|
||||
| preset | cfg gate | what `apply` sets | use case |
|
||||
|---|---|---|---|
|
||||
| `presets::Empty` | always | nothing — returns builder as-is | full manual control; `bind()` fails unless you set `crypto_provider` yourself |
|
||||
| `presets::Minimal` | `with_crypto_provider` | **only `crypto_provider`** (ring if `tls-ring`, aws-lc-rs if `tls-aws-lc-rs`; ring wins if both) | you want control over relay/discovery but not crypto boilerplate |
|
||||
| `presets::N0` | `with_crypto_provider` | `Minimal` + n0 DNS address lookup (pkarr publisher + DNS resolver) + `RelayMode::Default` (n0 prod relays) | the "0.35 `builder()` default" equivalent — production with n0 infra |
|
||||
| `presets::N0DisableRelay` | `with_crypto_provider` | `N0` but overrides relay to `RelayMode::Disabled` | n0 DNS discovery, no relay |
|
||||
|
||||
The cfg gate `with_crypto_provider` is set by iroh's `build.rs:8` to `any(feature = "tls-ring", feature = "tls-aws-lc-rs")`. iroh's **default features include `tls-ring`**, so `Minimal`/`N0`/`N0DisableRelay` are available unless you disable default features.
|
||||
|
||||
### Why `presets::Minimal` is the right choice for alknet-core
|
||||
|
||||
alknet-core's `build_iroh_endpoint` (`endpoint.rs:676-704`) does three things after constructing the builder:
|
||||
1. `builder.secret_key(...)` — sets the iroh identity (line 685/688)
|
||||
2. `builder.alpns(alpns.to_vec())` — sets ALPNs (line 691)
|
||||
3. `builder.relay_mode(...)` — **explicitly** sets either `RelayMode::Custom(relay_map)` or `RelayMode::Disabled` based on config (lines 693-698)
|
||||
|
||||
`presets::N0` would be wrong because:
|
||||
- It sets `RelayMode::Default` (n0 prod relays), but alknet-core **immediately overrides** `relay_mode`. Wasteful, and worse:
|
||||
- It adds n0 DNS address-lookup services (`PkarrPublisher::n0_dns()` + `DnsAddressLookup::n0_dns()` at `presets.rs:121,133`). This would publish alknet node identities to `iroh.link` DNS — not desired for a self-hosted alknet deployment that configures its own relay.
|
||||
- alknet-core's 0.35 `builder()` default did **not** set address-lookup (0.35's `Builder::default()` has no `address_lookup` field — that concept didn't exist).
|
||||
|
||||
`presets::Minimal` is the exact match: it sets **only** the `crypto_provider` (mandatory since 1.0), and leaves relay mode, address lookup, ALPNs, and secret key for alknet-core to set as it already does.
|
||||
|
||||
### Why `presets::Empty` is wrong
|
||||
|
||||
`Empty` sets nothing, so `bind()` returns `BindError::InvalidCryptoProvider` (`endpoint.rs:230`) unless alknet-core adds an explicit `.crypto_provider(...)` call. `Minimal` is strictly better: it handles the crypto provider selection via iroh's feature flags, which is the intended mechanism.
|
||||
|
||||
### Consistency with iroh-blobs 0.103
|
||||
|
||||
iroh-blobs 0.103's own tests use the same pattern (`src/tests.rs:506,525`):
|
||||
|
||||
```rust
|
||||
let ep = Endpoint::builder(presets::Minimal)
|
||||
.relay_mode(RelayMode::Default)
|
||||
.address_lookup(sp.clone())
|
||||
.bind()
|
||||
.await?;
|
||||
```
|
||||
|
||||
`Minimal` + explicit relay mode + explicit address lookup. alknet-core should follow the same shape: `Minimal` + explicit relay mode (no address lookup, since alknet doesn't use iroh's DNS discovery). No preset conflict, no duplicate crypto provider when alknet-blobs (which depends on both) is added later.
|
||||
|
||||
---
|
||||
|
||||
## 6. Cargo.toml changes
|
||||
|
||||
### alknet-core (`crates/alknet-core/Cargo.toml:21`)
|
||||
|
||||
```toml
|
||||
# from:
|
||||
iroh = { version = "0.35", optional = true }
|
||||
# to:
|
||||
iroh = { version = "1.0", optional = true, default-features = false, features = ["tls-aws-lc-rs"] }
|
||||
```
|
||||
|
||||
**Why `default-features = false`:** iroh's defaults include `tls-ring`. alknet-core's quinn path already uses `rustls::crypto::aws_lc_rs::default_provider()` (`endpoint.rs:551,631`). To keep iroh on the same crypto provider (avoid pulling in both ring and aws-lc-rs), disable defaults and enable only `tls-aws-lc-rs`. This makes `presets::Minimal` select aws-lc-rs (the `#[cfg(all(feature = "tls-aws-lc-rs", not(feature = "tls-ring")))]` branch at `presets.rs:71`), matching the quinn path's existing provider choice.
|
||||
|
||||
**What `default-features = false` drops:** `tls-ring`, `metrics`, `fast-apple-datapath`, `portmapper`. None are used by alknet-core today. If metrics or portmapper are wanted later, add them explicitly: `features = ["tls-aws-lc-rs", "metrics"]`. The load-bearing choice is `tls-aws-lc-rs` + not-`tls-ring` so `Minimal` picks aws-lc-rs.
|
||||
|
||||
### Workspace (`Cargo.toml:19-20`) — the dead dep
|
||||
|
||||
```toml
|
||||
# from:
|
||||
irpc = "0.16"
|
||||
irpc-derive = "0.16"
|
||||
# to (option A — cleanest, recommended):
|
||||
# (delete both lines; re-add as 0.17 when alknet-blobs needs them)
|
||||
#
|
||||
# or (option B — bump now, zero code impact):
|
||||
irpc = "0.17"
|
||||
irpc-derive = "0.17"
|
||||
```
|
||||
|
||||
### alknet-call (`crates/alknet-call/Cargo.toml:18`) — the dead dep consumer
|
||||
|
||||
```toml
|
||||
# from:
|
||||
irpc = { workspace = true }
|
||||
# to (option A):
|
||||
# (delete the line — nothing in src/ imports irpc)
|
||||
#
|
||||
# or (option B — leave it, it resolves to the bumped workspace 0.17):
|
||||
irpc = { workspace = true }
|
||||
```
|
||||
|
||||
Option A (drop) is recommended since the dep is genuinely unused; re-adding when needed is trivial. Option B keeps the declaration in case someone planned to wire irpc into alknet-call soon, but the current code doesn't use it.
|
||||
|
||||
---
|
||||
|
||||
## 7. The migration diff for `build_iroh_endpoint`
|
||||
|
||||
`crates/alknet-core/src/endpoint.rs:676-704`:
|
||||
|
||||
```rust
|
||||
async fn build_iroh_endpoint(
|
||||
static_config: &StaticConfig,
|
||||
alpns: &[Vec<u8>],
|
||||
) -> Result<iroh::Endpoint, EndpointError> {
|
||||
- let mut builder = iroh::Endpoint::builder();
|
||||
+ let mut builder = iroh::Endpoint::builder(iroh::endpoint::presets::Minimal);
|
||||
|
||||
if let Some(TlsIdentity::RawKey(secret_key)) = static_config.tls_identity.as_ref() {
|
||||
- let iroh_key = iroh::SecretKey::from_bytes(&secret_key.as_bytes());
|
||||
+ let iroh_key = iroh::SecretKey::from_bytes(&secret_key.as_bytes())
|
||||
+ .map_err(|e| EndpointError::TlsConfig(io::Error::other(e)))?;
|
||||
builder = builder.secret_key(iroh_key);
|
||||
} else {
|
||||
- let mut csprng = rand::rngs::OsRng;
|
||||
- builder = builder.secret_key(iroh::SecretKey::generate(&mut csprng));
|
||||
+ builder = builder.secret_key(iroh::SecretKey::generate());
|
||||
}
|
||||
|
||||
builder = builder.alpns(alpns.to_vec());
|
||||
|
||||
if let Some(relay_url) = static_config.iroh_relay.as_ref() {
|
||||
let relay_map = iroh::RelayMap::from(relay_url.clone());
|
||||
builder = builder.relay_mode(iroh::RelayMode::Custom(relay_map));
|
||||
} else {
|
||||
builder = builder.relay_mode(iroh::RelayMode::Disabled);
|
||||
}
|
||||
|
||||
builder
|
||||
.bind()
|
||||
.await
|
||||
.map_err(|e| EndpointError::BindFailed(io::Error::other(e)))
|
||||
}
|
||||
```
|
||||
|
||||
**Three edits:**
|
||||
1. `builder()` → `builder(presets::Minimal)` — supplies the now-mandatory `crypto_provider` (aws-lc-rs via the `tls-aws-lc-rs` feature).
|
||||
2. `from_bytes(...)` now returns `Result<Self, KeyParsingError>` — add `?` + map to `EndpointError::TlsConfig`. The `TlsConfig` variant already exists (quinn path uses it at `endpoint.rs:554`); no new enum variant needed.
|
||||
3. `generate(&mut csprng)` → `generate()` — the rng arg is gone; `OsRng` is used internally. Delete the `let mut csprng = rand::rngs::OsRng;` line.
|
||||
|
||||
### `EndpointError` note
|
||||
|
||||
`from_bytes`'s new `KeyParsingError` needs a conversion. The cleanest path is `.map_err(|e| EndpointError::TlsConfig(io::Error::other(e)))?` (reusing the existing `TlsConfig` variant), which is what the diff above does. If a dedicated variant is preferred (`InvalidKey(KeyParsingError)`), that's a 1-line enum change + `From` impl — but the reuse path is simpler and matches how the quinn path handles crypto errors.
|
||||
|
||||
---
|
||||
|
||||
## 8. Survey: what else may need updating
|
||||
|
||||
This section catalogs other places in the workspace that touch iroh/irpc and may need attention as part of the bump. Each item is a "check this" — not all will require edits, but all should be verified when the migration branch is opened.
|
||||
|
||||
### 8.1 `crates/alknet-core/src/types.rs` — iroh stream/connection types
|
||||
|
||||
Uses `iroh::endpoint::{SendStream, RecvStream, Connection, ConnectionError, VarInt, ApplicationClose}` (lines 232, 240, 366, 473, 504, 824-856). All confirmed stable in 1.0.2 (same module paths, same enum variants). **Likely no edits**, but the `ConnectionError` match arms at `types.rs:824-856` should be re-checked against 1.0.2's `ConnectionError` enum in case any variants were added (new variants don't break exhaustive matches if the match is on `&_` or has a catch-all, but if it's exhaustive without `_`, the compiler will flag it).
|
||||
|
||||
### 8.2 `crates/alknet-core/src/fingerprint.rs` — ed25519 fingerprint parity test
|
||||
|
||||
The test `fingerprint_from_ed25519_spki_matches_iroh_format` (line 220) constructs an `iroh_fp = format!("ed25519:{}", hex::encode(raw_key))` and compares it against the quinn SPKI path. It doesn't import iroh types — it just asserts string-format parity. **No edits expected.**
|
||||
|
||||
### 8.3 `crates/alknet-core/src/config.rs:28` — `iroh::RelayUrl`
|
||||
|
||||
`pub iroh_relay: Option<iroh::RelayUrl>`. `RelayUrl` is stable across 0.35 → 1.0.2 (`iroh-base-1.0.2/src/relay_url.rs:21`, same `pub struct RelayUrl(Arc<Url>)` shape). **No edits expected.**
|
||||
|
||||
### 8.4 `Cargo.lock` — will regenerate on `cargo update -p iroh`
|
||||
|
||||
After bumping `Cargo.toml`, `cargo update -p iroh` (and `-p iroh-base`, `-p iroh-relay`) will pull 1.0.2 and its transitive deps. The lockfile currently pins `iroh 0.35.0`, `irpc 0.16.0`, `irpc-derive 0.16.0` (verified at `Cargo.lock:1998-1999, 2202-2203, 2224-2225`). Expect a sizable lockfile diff (iroh's dep tree changed substantially between 0.35 and 1.0 — `magicsock` → `socket`, `disco` removed, `iroh-dns` added, etc.).
|
||||
|
||||
### 8.5 `rand` dependency — may become unused
|
||||
|
||||
The 0.35 path uses `rand::rngs::OsRng` at `endpoint.rs:687`. After the migration (`generate()` takes no rng), this import may become dead. Check whether `rand` is used elsewhere in alknet-core before removing it from `Cargo.toml`. (`ed25519-dalek` at `Cargo.toml:39` uses `rand_core` features, but that's a separate dep.)
|
||||
|
||||
### 8.6 `crates/alknet-call/` — the dead `irpc` dep
|
||||
|
||||
Covered in §3 and §6. No source changes; just `Cargo.toml` cleanup.
|
||||
|
||||
### 8.7 Tests under `#[cfg(feature = "iroh")]` in `endpoint.rs`
|
||||
|
||||
`endpoint.rs:1107-1150` (`endpoint_constructs_with_iroh_raw_key_identity`, `iroh_endpoint_runs_accept_loop_and_shutdown`) and `endpoint.rs:1326-1364` (`has_iroh_identity_*`) construct test configs with `iroh_relay: None` and exercise `build_iroh_endpoint`. These will hit the same 3 breaking APIs and need the same edits applied (they call into `build_iroh_endpoint`, so if that function is fixed, the tests should pass — but verify the test helpers don't construct `iroh::SecretKey` or `iroh::Endpoint::builder()` directly).
|
||||
|
||||
### 8.8 Future: `alknet-blobs` crate (not yet created)
|
||||
|
||||
When `alknet-blobs` is added as a workspace member, it will depend on `iroh-blobs = "0.103"`, which pulls `iroh 1.0` and `irpc 0.17` transitively. At that point the workspace must have a single consistent iroh version. The migration in this doc (bumping alknet-core to `iroh 1.0`) is the prerequisite for that — without it, `alknet-blobs` and `alknet-core` would depend on different iroh majors and Cargo would fail to resolve (or pick one, breaking the other).
|
||||
|
||||
The `~20-line fork` of iroh-blobs (widening `Store::from_sender`/`ref_from_sender` + `ApiClient` + `Scope` field, per `alknet-blobs-external-store-probe.md` §"The four items that must be made pub") targets the 0.103 line. Since iroh-blobs hasn't moved past 0.103 (§1), the fork surface is stable. When iroh-blobs does eventually bump, the fork's ~20-line diff is the only thing that needs re-validation — the `Command` protocol shape is the thing to watch, and the probe confirmed it's macro-generated from a `pub enum Request`, so a breaking change there would be visible in iroh-blobs' changelog.
|
||||
|
||||
---
|
||||
|
||||
## 9. Recommended execution order
|
||||
|
||||
1. **Drop the dead `irpc` dep** (§3, §6) — zero code impact, dissolves the `irpc` half of the gap. Do this first as a standalone commit so the workspace is clean before the iroh migration touches anything.
|
||||
2. **Bump `alknet-core` iroh to 1.0** (§4, §6) — `Cargo.toml` change + 3 edits in `endpoint.rs`. Run `cargo update -p iroh -p iroh-base -p iroh-relay`.
|
||||
3. **Verify the build** with `--features iroh` and `--all-features`. Check the `ConnectionError` match arms in `types.rs` (§8.1) for exhaustiveness.
|
||||
4. **Run the iroh-feature tests** (`endpoint.rs:1107-1150, 1326-1364`) to confirm the accept loop and identity paths still work.
|
||||
5. **Survey + clean up** `rand` usage (§8.5) and any newly-dead imports.
|
||||
6. **(Later, separate effort)** Add `alknet-blobs` as a workspace member with the `~20-line` iroh-blobs fork. This is unblocked once step 2 lands, since the workspace is then on a single consistent iroh 1.0.
|
||||
|
||||
---
|
||||
|
||||
## 10. What this closes and what remains
|
||||
|
||||
**Closes:**
|
||||
- The version-gap open question deferred by `alknet-blobs-external-store-probe.md` §"Versioning reality" and §"What this closes and what remains". The gap is real (iroh 0.35 → 1.0.2, irpc 0.16 → 0.17), the irpc half is a free fix (dead dep), and the iroh half is a traced 3-edit migration with a stable target.
|
||||
- The probe's false claim that alknet-call uses `irpc` (§3). Corrected: alknet-call declares but never imports `irpc`.
|
||||
- The "is iroh 0.35 a stale pin" question (§2). Answered: no, it's load-bearing; 3 APIs broke.
|
||||
- The `Preset` selection question (§5). Answered: `presets::Minimal`, with `tls-aws-lc-rs` feature to match the quinn path's existing crypto provider.
|
||||
|
||||
**Does not close (deferred to the migration branch):**
|
||||
- Actually applying the diff and confirming the build compiles (§7, §8). The edits are traced but not yet made.
|
||||
- Verifying the `ConnectionError` match exhaustiveness in `types.rs` (§8.1) — needs a compile check.
|
||||
- Whether `rand` can be dropped from alknet-core's deps after the migration (§8.5).
|
||||
- The `~20-line` iroh-blobs fork itself (that's the `alknet-blobs` crate effort, separate from this update).
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
- Probe doc: `docs/research/alknet-filesystem/alknet-blobs-external-store-probe.md` — the version-gap open question this doc resolves is at §"Versioning reality" (line 170) and §"What this closes and what remains" (line 206).
|
||||
- Storage strategy: `docs/research/alknet-storage-strategy/findings.md` — cross-references iroh-blobs as the content-addressed blob layer.
|
||||
- alknet-core iroh usage: `crates/alknet-core/src/endpoint.rs:676-704` (`build_iroh_endpoint`), `types.rs:231-504` (stream/connection types), `config.rs:28` (`iroh_relay`), `fingerprint.rs:220-228` (parity test).
|
||||
- alknet-call dead dep: `crates/alknet-call/Cargo.toml:18` (`irpc = { workspace = true }`), `crates/alknet-call/src/protocol/wire.rs` (hand-rolled JSON framing, no irpc).
|
||||
- Workspace deps: `Cargo.toml:19-20` (`irpc`/`irpc-derive` at 0.16).
|
||||
- Versions verified via crates.io API: `https://crates.io/api/v1/crates/<name>` (queried 2026-07-08).
|
||||
- iroh 1.0.2 source: fetched from `https://static.crates.io/crates/iroh/iroh-1.0.2.crate` (not in cargo cache). `Preset` trait at `src/endpoint/presets.rs:21`; `Endpoint::builder(preset)` at `src/endpoint.rs:950`; `Builder::bind()` crypto check at `src/endpoint.rs:225-230`.
|
||||
- iroh 1.0.0 source: cached at `~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/iroh-1.0.0/`. Used for 0.35 vs 1.0.0 pub-surface diff.
|
||||
- iroh-blobs 0.103.0 source: cached at `~/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/iroh-blobs-0.103.0/`. `from_sender`/`ref_from_sender`/`ApiClient`/`Scope` visibility confirmed at `src/api.rs:33,299,303` and `src/api/proto.rs:693`. Test preset usage at `src/tests.rs:506,525`.
|
||||
- iroh-base versions: cached 0.35.0 and 1.0.0; fetched 1.0.2. `SecretKey::generate` signature change at `src/key.rs:273` (0.35) vs `:318` (1.0.2); `from_bytes` error type at `:109` (0.35, `SignatureError`) vs `:122` (1.0.2, `KeyParsingError`).
|
||||
Reference in new issue
Block a user