feat: initial repo setup (Phase 0 scaffolding)
- AGENTS.md: repo conventions adapted from alkcall/alksocks (store posture, multi-hash requirement, no-iroh-wire rule; no fuzz gate yet) - Cargo.toml + src/lib.rs: bare lean lib skeleton (tokio async, thiserror) - docs/research/phase-0.md: Phase 0 draft — vision, prior art (iroh-blobs store, gix-odb, alknet appfile probe, alkcall), OQ register, POC plan - .gitignore, LICENSE-APACHE/MIT - Re-pointed stale alknet/alkcall references in .opencode/agents/ and docs/sdd_process.md Verification: cargo build/test, cargo clippy --all-targets -- -D warnings, cargo fmt --check, cargo doc --no-deps — all pass
This commit is contained in:
1 parent
357311d27b
commit
7f7f70b220
10 files changed
+660
-12
No files matched your search
@@ -0,0 +1,233 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-09-30 (initial setup draft: vision, prior art, open
|
||||
questions from the setup discussion; un-numbered — renumber
|
||||
OQ-BL-01..NN as the register solidifies)
|
||||
---
|
||||
|
||||
# alkblobs — Phase 0 (Exploration)
|
||||
|
||||
This document captures Phase 0 (Exploration) for the `alkblobs` crate:
|
||||
vision, guiding principles, prior art, and open questions (OQ-BL-01..NN).
|
||||
Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding
|
||||
principles; research options; validate approaches; converge on a
|
||||
recommended approach.* It is the input to Phase 1 (Architecture), where
|
||||
the Architect will produce `docs/architecture/` specs, ADRs, and the
|
||||
open-questions tracker.
|
||||
|
||||
Drafted 2026-09-30, emerging from the initial setup discussion. Nothing
|
||||
is converged yet — this is the working sketch, not a specification.
|
||||
|
||||
## Vision and guiding principles
|
||||
|
||||
**One sentence:** content-addressed blob storage in the alk* family —
|
||||
a multi-hash put/get/verify store with pluggable backends (small blobs
|
||||
in a kv/sqlite-ish store, large blobs on a filesystem fallback), split
|
||||
from any transport/protocol layer and tolerant of hash-algorithm
|
||||
conflicts (the alkgit case), with auth-gated network operations riding
|
||||
the alkcall seam when they exist.
|
||||
|
||||
**Why this crate exists (two converging consumers):**
|
||||
|
||||
1. **alkgit** (planning phase) needs an object backend that is *not*
|
||||
its proposed default file-based backend ("kind of gross") — blobs
|
||||
stored under git's own object hashes, but with iroh-blobs-like
|
||||
handling for large blobs (where git typically delegates to git-lfs;
|
||||
we could instead follow an iroh-blobs-shaped large-blob path). The
|
||||
conflict: iroh-blobs is BLAKE3-only and content-hashes with bao
|
||||
verification, while git's object database is SHA-1/SHA-256 with
|
||||
git's own object format. A store that insists on one hash algorithm
|
||||
cannot serve git objects directly.
|
||||
2. **The alknet rewrite** (the original alk* project, being decomposed
|
||||
and improved) needs the "appfile" external-store shape: small blobs
|
||||
in a kv/sqlite store (faster than the filesystem for small items —
|
||||
true beyond sqlite: it applies to the kv store iroh-blobs uses
|
||||
too), large blobs on a filesystem fallback, with filename↔hash
|
||||
mapping. alknet's own research hit the core awkwardness of
|
||||
dispatching across more than one backend at a time — which is
|
||||
exactly a store-layer problem this crate should own.
|
||||
|
||||
**The scope line (current posture, revisit as evidence arrives):**
|
||||
this crate is the *store*, not the *transport*. iroh-blobs welds store
|
||||
+ provider protocol + tickets + postcard into one crate; the defining
|
||||
posture here is the opposite split — put/get/verify against a hash is
|
||||
its own layer, and any provider/protocol/ops surface lives above it or
|
||||
in feature-gated modules (the alk* inversion-point pattern). The ops
|
||||
surface (a channels-native "give me the blob with this hash" op family,
|
||||
auth-gated via `AccessControl`) is a Phase 0/1 question, not assumed.
|
||||
|
||||
Guiding principles, inherited from the alk* family:
|
||||
|
||||
1. **Substrate-agnostic by construction.** The store must not know
|
||||
whether bytes arrive over the network, from a local writer, or get
|
||||
reassembled anywhere particular. Backends behind an injected seam
|
||||
(alktty `TtyBackend` / alktunnels pump-halves precedent).
|
||||
2. **Hashes are data, not transport identity.** Multiple hash
|
||||
algorithms must coexist (git SHA-1/SHA-256, BLAKE3, maybe bao-shaped
|
||||
chunk trees as one encoding among several). How this is abstracted —
|
||||
trait, enum, per-backend config — is an open question (OQ-BL-03).
|
||||
3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets,
|
||||
postcard serialization, and provider protocol are design-welded
|
||||
choices we do not inherit. Its *store-shape* lessons (kv + flat
|
||||
backends, verification flow, chunking) are fair game per prior-art
|
||||
reading.
|
||||
4. **Producer/consumer vocabulary** for any network-facing surface;
|
||||
authorization via alkcall's `AccessControl`/identity seam, never an
|
||||
in-band invented scheme (convention 6).
|
||||
5. **Verify where it matters.** iroh-blobs' core virtue is verified
|
||||
transfer (bao outboard encoding). The alkgit case already has
|
||||
verified content (git objects are hash-addressed by git itself).
|
||||
Which verification story the crate owns — bao trees, per-blob
|
||||
digests, backend-native — is open (OQ-BL-04).
|
||||
|
||||
## What is already known (settled, thin)
|
||||
|
||||
Almost nothing is pinned — deliberately. The only postures agreed at
|
||||
setup:
|
||||
|
||||
- **Not a fork of iroh-blobs.** A downstream store crate inspired by
|
||||
its `src/store` work, diverging deliberately on hashing and wire
|
||||
surface (see `iroh-blobs-eval` below when written).
|
||||
- **Multi-hash from day one** (hard requirement from alkgit — do not
|
||||
hardcode BLAKE3).
|
||||
- **Backend pluralism assumed, not designed.** kv/sqlite for small
|
||||
blobs, filesystem fallback for large — the appfile shape — but
|
||||
*how* multi-backend dispatch works is exactly what alknet's research
|
||||
ran into, so it earns research and probably POCs (OQ-BL-02).
|
||||
|
||||
## Prior art
|
||||
|
||||
### iroh-blobs — the shape inspiration (evaluated, not the base)
|
||||
|
||||
`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference).
|
||||
Specifically `src/store`: the kv backend (small blobs, in-process) and
|
||||
flat file backend (large blobs) split; bao outboard encoding and
|
||||
verification flow; chunking. Its BLAKE3-only hashing, tickets, postcard
|
||||
serialization, and provider protocol are the design-welded choices we
|
||||
diverge from. **The store eval should be written against the current
|
||||
checkout** — alknet's older research refers to an older iroh-blobs and
|
||||
its conclusions must be re-verified rather than inherited.
|
||||
|
||||
### gix-odb — git's own object database (alkgit's baseline)
|
||||
|
||||
`/workspace/git-oxide/gix-odb` (read-only reference). The backend
|
||||
alkgit currently plans against; the hashing-algorithm baseline git
|
||||
actually uses (SHA-1/SHA-256), git's loose-object and packfile layout,
|
||||
and git's already-hash-addressed object model. The alkgit question is
|
||||
whether this crate can sit *under or beside* gix-odb semantics — git
|
||||
objects are already content-addressed and verified by git's own model,
|
||||
so the crate's value there is the large-blob story (git-lfs-shaped)
|
||||
and a better small-object backend than the proposed default, without
|
||||
fighting git's hash model.
|
||||
|
||||
### alknet's appfile external-store probe — the direct ancestor
|
||||
|
||||
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
|
||||
(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written
|
||||
against an older iroh-blobs; conclusions re-verify in this Phase 0.
|
||||
The load-bearing bits: the appfile shape (small blobs in kv/sqlite,
|
||||
large on fs fallback), the filename↔hash mapping problem, and the
|
||||
multi-backend dispatch pain the probe hit (which is this crate's
|
||||
reason to own that dispatch). Old-data warning: any API or behavior
|
||||
claims there describe an older upstream; re-check against
|
||||
`/workspace/iroh-blobs` as checked out today.
|
||||
|
||||
### alkcall — the substrate
|
||||
|
||||
`/workspace/@alkdev/alkcall` (the call + channels RPC crate). The
|
||||
authorization seam for network-facing blob ops (`AccessControl`,
|
||||
producer/consumer vocabulary, alkcall ADR-022/037) and, if blob
|
||||
transfer rides channels, the established data-path patterns
|
||||
(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even
|
||||
belongs in this crate is itself open (OQ-BL-01); alkcall is the
|
||||
substrate it composes with *when it does*.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Numbering is provisional until the register solidifies; promote the
|
||||
final set into Phase 1's `docs/architecture/open-questions.md`.
|
||||
|
||||
### OQ-BL-01: Crate scope — store-only, or store + ops surface?
|
||||
|
||||
The store (put/get/verify) is clearly in. What about the network ops
|
||||
layer — a channels-native "fetch/provide blob" op family on the
|
||||
alkcall substrate, auth-gated via `AccessControl`? iroh-blobs has it
|
||||
(provider protocol); our posture is to separate it. Options: (a)
|
||||
store-only crate, ops in a sibling crate later; (b) store + optional
|
||||
feature-gated ops module here; (c) undecided pending the first
|
||||
consumer's shape. Leaning (a)/(b) per the inversion-point pattern, but
|
||||
the alkgit + alknet consumer needs should decide — neither has confirmed
|
||||
a *networked* transfer requirement yet.
|
||||
|
||||
### OQ-BL-02: Multi-backend dispatch — the appfile problem
|
||||
|
||||
Small blobs in kv/sqlite, large on fs fallback — the shape is agreed,
|
||||
the mechanics are not: how does put/get pick a backend (size
|
||||
thresholds? per-algorithm routing? per-namespace config?), how does it
|
||||
work when a blob should migrate between backends, and what happens on a
|
||||
get when the "wrong" backend was probed? alknet's probe hit exactly
|
||||
this ("dealing with more than one backend at a time"); its findings
|
||||
need re-verification against the current iroh-blobs before relying on
|
||||
them. Expected shape: a backend trait + a dispatch layer, but the trait
|
||||
shape is a one-way door once written — deserves a POC before
|
||||
committing.
|
||||
|
||||
### OQ-BL-03: Hash abstraction — trait, enum, or per-backend config?
|
||||
|
||||
The hard requirement: git SHA-1/SHA-256 and BLAKE3 must coexist (alkgit
|
||||
conflict). Sub-questions: is the hash algorithm a *store* parameter
|
||||
(one algorithm per store instance, chosen by the consumer) or
|
||||
*per-blob* data (multi-algorithm within one store)? Does verification
|
||||
(bao trees) get per-algorithm treatment or does bao stay BLAKE3-bound
|
||||
(git objects don't need bao verification anyway)? This interacts with
|
||||
OQ-BL-01 and the wire-format question — a wire ADR must follow whatever
|
||||
this settles.
|
||||
|
||||
### OQ-BL-04: Verification and chunking story
|
||||
|
||||
iroh-blobs' verification is bao outboard encoding over BLAKE3 chunk
|
||||
trees. Git objects are self-verifying under git's model. Raw large
|
||||
blobs (git-lfs-shaped, appfile large files) need *some* verification
|
||||
story from us. Options: bao (borrow the *conclusion*, dependency
|
||||
posture per convention 7), our own digest scheme, or pluggable
|
||||
verification per blob-kind. Also: does chunking exist at the store
|
||||
layer at all (git objects are unit blobs; appfile items may be large
|
||||
files wanting range reads)? Range-read support is an open API-shape
|
||||
question.
|
||||
|
||||
### OQ-BL-05: POC register (draft)
|
||||
|
||||
Numbered POCs, opened as research reaches them (findings land in
|
||||
`docs/research/`; worktree placement per the SDD process):
|
||||
|
||||
| # | What | Status | Where |
|
||||
|---|------|--------|-------|
|
||||
| 1 | Backend-trait + dual-dispatch shape (kv small / fs large) | **Pending** — likely first POC | findings file TBD |
|
||||
| 2 | Multi-hash store (SHA-256 + BLAKE3 coexisting) | **Pending** — rides #1's data | findings file TBD |
|
||||
| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads) | **Pending** | findings file TBD |
|
||||
|
||||
Sequencing note: #1 and #2 are probably one worktree (the dispatch POC
|
||||
naturally exercises two hash algorithms); #3 is a reading-and-design
|
||||
POC against the current iroh-blobs checkout plus the alknet probe
|
||||
re-verification.
|
||||
|
||||
POC placement conventions (inherited from alksocks/alktunnels): a POC
|
||||
that needs code from this repo runs in a worktree/branch
|
||||
(`.worktrees/research/<task-id>/` per the SDD process); a
|
||||
self-contained POC runs as a standalone crate in the global workspace
|
||||
with findings written into `docs/research/` here. Findings always land
|
||||
in `docs/research/` regardless of where the code lives.
|
||||
|
||||
## Phase 0 plan (next steps)
|
||||
|
||||
1. **Write `iroh-blobs-eval.md`** against the *current* checkout,
|
||||
focused on `src/store` (kv + flat backends, bao, chunking) — the
|
||||
conclusions inventory we may borrow, and the weld points we won't.
|
||||
2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`,
|
||||
`poc-summary.md`) against the current upstream; mark what carried
|
||||
over.
|
||||
3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit
|
||||
baseline: its storage layout, object model, and where a blob-store
|
||||
crate under/beside it earns its keep.
|
||||
4. **Open POCs** per OQ-BL-05, in the order above.
|
||||
5. Converge: recommended approach + final OQ register → Phase 1.
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the SDD process for the @alkdev/alkcall package. It
|
||||
This document defines the SDD process for the @alkdev/alkblobs package. It
|
||||
leverages:
|
||||
|
||||
- **OpenCode CLI** as the agent execution environment
|
||||
|
||||
Reference in new issue
Block a user