From 7f7f70b220e95b5c03036c259f8d3be66bcc6161 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Wed, 30 Sep 2026 07:45:40 +0000 Subject: [PATCH] feat: initial repo setup (Phase 0 scaffolding) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - AGENTS.md: repo conventions adapted from alkcall/alksocks (store posture, multi-hash requirement, no-iroh-wire rule; no fuzz gate yet) - Cargo.toml + src/lib.rs: bare lean lib skeleton (tokio async, thiserror) - docs/research/phase-0.md: Phase 0 draft — vision, prior art (iroh-blobs store, gix-odb, alknet appfile probe, alkcall), OQ register, POC plan - .gitignore, LICENSE-APACHE/MIT - Re-pointed stale alknet/alkcall references in .opencode/agents/ and docs/sdd_process.md Verification: cargo build/test, cargo clippy --all-targets -- -D warnings, cargo fmt --check, cargo doc --no-deps — all pass --- .gitignore | 4 + .opencode/agents/architect.md | 14 +- .opencode/agents/coordinator.md | 7 +- AGENTS.md | 175 ++++++++++++++++++++++++ Cargo.toml | 23 ++++ LICENSE-APACHE | 192 ++++++++++++++++++++++++++ LICENSE-MIT | 21 +++ docs/research/phase-0.md | 233 ++++++++++++++++++++++++++++++++ docs/sdd_process.md | 2 +- src/lib.rs | 1 + 10 files changed, 660 insertions(+), 12 deletions(-) create mode 100644 .gitignore create mode 100644 AGENTS.md create mode 100644 Cargo.toml create mode 100644 LICENSE-APACHE create mode 100644 LICENSE-MIT create mode 100644 docs/research/phase-0.md create mode 100644 src/lib.rs diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..0f61703 --- /dev/null +++ b/.gitignore @@ -0,0 +1,4 @@ +target/ +node_modules/ +.worktrees/ +Cargo.lock \ No newline at end of file diff --git a/.opencode/agents/architect.md b/.opencode/agents/architect.md index 836546b..01b0d4b 100644 --- a/.opencode/agents/architect.md +++ b/.opencode/agents/architect.md @@ -326,10 +326,10 @@ result, a concrete use case to arrive. A decision should be `deferred(scope)` when: -- The use case isn't concrete (e.g., "we don't know what the agent crate - will need from the call protocol") +- The use case isn't concrete (e.g., "we don't know what the consumer + crate will need from the blob API") - The options depend on something that doesn't exist yet (e.g., - "depends on the alknet-http crate spec") + "depends on the alkgit crate spec") - The trade-off requires data that can only come from implementation (e.g., "need performance benchmarks to choose between X and Y") - The decision is genuinely not needed for the current scope (e.g., "the @@ -373,10 +373,10 @@ A decision should be `deferred(unclear)` when: (implies it's decided). 2. **State the blocking condition** (`deferred(scope)`) or **investigation target** (`deferred(unclear)`) — what specific thing - would unblock this? Be concrete: "blocked on: alknet-agent crate spec - exists" or "investigation: work through 2+ example outbound-dial use - cases (hub→worker, worker→hub) to see how verifier-selection + - provider + connector compose." + would unblock this? Be concrete: "blocked on: alkgit crate spec + exists" or "investigation: work through 2+ example store use + cases (small-blob kv backend, large-blob fs fallback) to see + how backends + hashes + ACL compose." 3. **State the impacts** — what does this block downstream? Be specific: "blocks the first hub deployment because the hub dials workers" not "blocks the hub crate." This is the triage signal that diff --git a/.opencode/agents/coordinator.md b/.opencode/agents/coordinator.md index 2327646..2035066 100644 --- a/.opencode/agents/coordinator.md +++ b/.opencode/agents/coordinator.md @@ -191,7 +191,7 @@ also include: Example prompt template: ``` -You are an implementation specialist for the @alkdev/alknet project. +You are an implementation specialist for the @alkdev/alkblobs project. Your task: {{task}} @@ -204,11 +204,10 @@ Your task: {{task}} 7. Push: git push origin $(git branch --show-current) 8. Notify: worktree({action: "notify", args: {message: "Task completed: {{task}}. ", level: "info"}}) -Key project constraints (@alkdev/alknet): +Key project constraints (@alkdev/alkblobs): - Rust: use cargo build, cargo clippy, cargo fmt, cargo test - No comments in code -- anyhow::Result for application errors, thiserror for library error types -- Feature flags for transports (tls, iroh, acme) +- thiserror for error types - Async via tokio runtime - No panics in library code ``` diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..87ee20c --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,175 @@ +# AGENTS.md + +Operating instructions for opencode agents working in this repo. opencode +auto-loads this file as instructions, overriding the built-in defaults for +this project. Custom agents in `.opencode/agents/` inherit these rules +unless their own prompts say otherwise. + +## Git Workflow + +**Commit and push when reasonable.** When a change is complete and +verified (build + lint + tests pass), commit and push to `origin/main` +without asking. This overrides the built-in default of "only commit when +explicitly asked." + +The workflow: + +1. Make the change +2. Verify: `cargo test`, `cargo clippy --all-targets -- -D warnings`, + `cargo fmt --check`, `cargo doc --no-deps` if docs changed +3. Inspect `git status` and `git diff` before staging — stage only the + intended files, never secrets +4. Write a concise commit message matching the repo style (see `git log + --oneline -10`). For multi-point changes, use a summary line plus a + body with bullet points and a verification block. +5. `git push origin main` +6. Report the commit hash and the verification summary + +Exceptions — **do not** commit or push without asking: + +- The change is exploratory / speculative (you're not sure the user wants + it kept) +- The user is actively reviewing the diff and may ask for changes +- The change touches the wire format, a trait shape backends implement, + or the hashing/verification story (one-way doors once consumers exist + — see convention 7; nothing is wire-stable yet, but the first + wire-format ADR must be written before the first consumer) +- You'd be force-pushing, amending a published commit, creating an empty + commit, or skipping hooks + +Never commit secrets, keys, or credentials. If a commit fails or hooks +reject it, fix the issue and create a new commit — do not amend the +failed one. + +Git identity is preconfigured (`glm-5.3-flash `). Do +not change `git config`, skip hooks, or use `git commit -i`. + +## Project Conventions (Rust / blob storage crate) + +This is the blob storage crate — content-addressed blob storage in the +alk* family, inspired by iroh-blobs' store work but deliberately not a +fork of it. It sits downstream of alkcall (the call + channels substrate +its network operations ride, when they exist) and its first planned +consumer is alkgit (git object storage, where the iroh-blobs hashing +story collides with git's own object hashes). Status: **Phase 0 +(Exploration)** — `docs/research/` is the working state of the repo; +`docs/architecture/` does not exist yet and no wire format, API shape, +or backend trait is decided. The conventions below apply to all work in +`src/` and `tests/`. They mirror +`.opencode/agents/implementation-specialist.md` §Project Conventions and +are repeated here so they apply to every session. + +1. **No comments in code** unless the user explicitly asks. This is a + project-wide convention. Doc comments (`///`, `//!`) are fine and + expected on public API. Inline `//` comments only when the user asks + or when a non-obvious safety/correctness constraint would otherwise + be missed. + +2. **Error handling** — `thiserror` for library error types. No panics + in library code. No `unwrap()` or `expect()` outside tests. If you + reach for `unwrap`, the error path wasn't specified — stop and decide + what should actually happen. For poisoned `RwLock`/`Mutex`, use + `unwrap_or_else(|e| e.into_inner())` so a panic in one operation does + not cascade to other operations. + +3. **`tokio` is the async runtime** — all I/O is async. Use + `tokio::sync` primitives (`oneshot`, `mpsc`) for lifecycle + correlation; `parking_lot` for short-held internal locks. Do not + introduce blocking I/O on the async path. + +4. **Substrate-agnostic by construction** — the store layer must not + know whether bytes arrive over the network, from a local writer, or + get reassembled anywhere particular. iroh-blobs welds its store to + its provider/ticket/postcard stack; this crate's defining posture is + the opposite split: the store (put/get/verify against a hash) is its + own layer, and transport/protocol concerns live above it or in + feature-gated modules. This is the same inversion-point pattern as + the alk* family (alktty `TtyBackend`, alktunnels pump halves). + +5. **Hashes are data, not identity of transport** — the crate must + tolerate multiple hash algorithms (the alkgit conflict: git's SHA-1/ + SHA-256 object hashes vs iroh-blobs' BLAKE3). How the hash algorithm + is abstracted (trait, enum, per-backend configuration) is a Phase 0/ + 1 decision, not yet pinned. Do not hardcode a single algorithm. + +6. **Auth-gated operations ride the alkcall authorization seam** — when + network-facing operations exist, authorization happens via alkcall's + `AccessControl`/identity mechanisms (the producer/consumer model), + not via an in-band invented auth scheme. Producer/consumer + vocabulary, not "server"/"client" (alkcall ADR-022/037). Not yet + pinned in detail — the ops surface is a Phase 0 question. + +7. **Do not adopt iroh-blobs' wire surface** — tickets, the postcard + serialization, and the provider protocol are iroh-blobs' + design-welded choices we have decided not to inherit. Reading + `/workspace/iroh-blobs` (a fresh upstream checkout) is encouraged for + store-shape lessons (`src/store`, the kv + flat file backends); + borrowing its *conclusions* is fine, wiring in its *types*, framing, + or serialization as load-bearing dependencies is not (yet — the + dependency posture is a Phase 0 question if a small pure piece, like + bao outboard encoding, earns its place). + +8. **Feature flags** — substrate backends (kv, sqlite, flat fs) may be + feature-gated if the need arises. The base crate should compile lean. + Verify both `cargo test` (default) and `cargo test --all-features` + pass if features are added. + +9. **Naming** — Rust standard: `snake_case` for functions/variables/ + modules, `PascalCase` for types/traits, `SCREAMING_SNAKE_CASE` for + constants. + +10. **Module structure** — one module per file under `src/`, re-exported + from `src/lib.rs`. Public API surface is `lib.rs` re-exports. The + expected shape (pending Phase 0 convergence) separates the + backend/store layer from any protocol/ops layer. + +## Verification Commands + +Run these before committing. All must pass. + +```bash +cargo test # full suite +cargo clippy --all-targets -- -D warnings +cargo fmt --check +cargo doc --no-deps # if docs changed +cargo publish --dry-run --allow-dirty # before a release +``` + +If feature flags are added, also run `cargo test --all-features` and +`cargo clippy --all-features --all-targets -- -D warnings`. + +No fuzz targets exist yet for this crate. If/when a wire format is +pinned (Phase 1), a fuzz gate on the model of alkcall/alksocks (`fuzz/` +corpus replay on stable) is expected — until then ignore fuzzing. + +## Architecture Context + +- `docs/research/` — the Phase 0 (Exploration) state of this repo: + research findings, POC records, and `phase-0.md` (vision, prior art, + open questions, converged recommendation). Read it before non-trivial + work. The SDD process lives in `docs/sdd_process.md`. +- `docs/architecture/` — does not exist yet (Phase 1 output). When it + lands, the SDD process applies: specs describe WHAT, `decisions/` + ADRs explain WHY, `open-questions.md` tracks what's unresolved. +- Key prior art (read-only, in the global workspace): + - **iroh-blobs** — `/workspace/iroh-blobs` (fresh upstream checkout): + the shape inspiration, specifically its store work (kv + flat file + backends, bao verification, chunking). Its wire-surface choices + (tickets, postcard, BLAKE3-only) are the ones we are deliberately + diverging from. + - **gix-odb** — `/workspace/git-oxide/gix-odb`: git's own object + database; the backend alkgit currently plans against and the + hashing-algorithm baseline git actually uses. + - **alknet's blobs research** — + `/workspace/@alkdev/alknet/docs/research/alknet-filesystem/` + (`alknet-blobs-external-store-probe.md`, `poc-summary.md`): the + appfile external-store probe — small blobs in a kv/sqlite-ish + store, large blobs on the filesystem fallback, filename↔hash + mapping. Written against an older iroh-blobs than the current + checkout; re-verify conclusions before relying on them. + - **alkcall** — `/workspace/@alkdev/alkcall` (the substrate): + call + channels; its `AccessControl`/identity seam is the + authorization story for network-facing blob ops. +- If a TODO references a design direction that a later ADR has decided + against, the TODO is stale — remove it and align with the ADR. Do not + implement the rejected design. \ No newline at end of file diff --git a/Cargo.toml b/Cargo.toml new file mode 100644 index 0000000..9726c24 --- /dev/null +++ b/Cargo.toml @@ -0,0 +1,23 @@ +[package] +name = "alkblobs" +version = "0.1.0" +edition = "2021" +rust-version = "1.88" +license = "MIT OR Apache-2.0" +description = "Content-addressed blob storage: multi-hash put/get/verify store with pluggable backends, transport- and protocol-agnostic" +repository = "https://git.alk.dev/alkdev/alkblobs" +keywords = ["blobs", "content-addressed", "storage", "gix", "bao"] +categories = ["asynchronous", "filesystem", "database"] +exclude = [".opencode/", "AGENTS.md", "docs/reviews/", "docs/research/", "docs/plans/", "tasks/", "docs/sdd_process.md"] + +[workspace] +members = ["."] +exclude = [] + +[lib] +name = "alkblobs" + +[dependencies] +tokio = { version = "1", default-features = false, features = ["rt", "sync", "io-util", "macros"] } +thiserror = "2" +parking_lot = "0.12" \ No newline at end of file diff --git a/LICENSE-APACHE b/LICENSE-APACHE new file mode 100644 index 0000000..56f9da5 --- /dev/null +++ b/LICENSE-APACHE @@ -0,0 +1,192 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name), or refer to, the Work. + (Note: Derivative Works shall not include works that remain separable from, + or merely link (or bind by name) to the interfaces of, the Work and + Derivative Works thereof.) + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to the Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by the Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + Copyright 2025-2026 Alk Development + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file diff --git a/LICENSE-MIT b/LICENSE-MIT new file mode 100644 index 0000000..9ee173b --- /dev/null +++ b/LICENSE-MIT @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2025-2026 Alk Development + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. \ No newline at end of file diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md new file mode 100644 index 0000000..7538cc1 --- /dev/null +++ b/docs/research/phase-0.md @@ -0,0 +1,233 @@ +--- +status: draft +last_updated: 2026-09-30 (initial setup draft: vision, prior art, open +questions from the setup discussion; un-numbered — renumber +OQ-BL-01..NN as the register solidifies) +--- + +# alkblobs — Phase 0 (Exploration) + +This document captures Phase 0 (Exploration) for the `alkblobs` crate: +vision, guiding principles, prior art, and open questions (OQ-BL-01..NN). +Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding +principles; research options; validate approaches; converge on a +recommended approach.* It is the input to Phase 1 (Architecture), where +the Architect will produce `docs/architecture/` specs, ADRs, and the +open-questions tracker. + +Drafted 2026-09-30, emerging from the initial setup discussion. Nothing +is converged yet — this is the working sketch, not a specification. + +## Vision and guiding principles + +**One sentence:** content-addressed blob storage in the alk* family — +a multi-hash put/get/verify store with pluggable backends (small blobs +in a kv/sqlite-ish store, large blobs on a filesystem fallback), split +from any transport/protocol layer and tolerant of hash-algorithm +conflicts (the alkgit case), with auth-gated network operations riding +the alkcall seam when they exist. + +**Why this crate exists (two converging consumers):** + +1. **alkgit** (planning phase) needs an object backend that is *not* + its proposed default file-based backend ("kind of gross") — blobs + stored under git's own object hashes, but with iroh-blobs-like + handling for large blobs (where git typically delegates to git-lfs; + we could instead follow an iroh-blobs-shaped large-blob path). The + conflict: iroh-blobs is BLAKE3-only and content-hashes with bao + verification, while git's object database is SHA-1/SHA-256 with + git's own object format. A store that insists on one hash algorithm + cannot serve git objects directly. +2. **The alknet rewrite** (the original alk* project, being decomposed + and improved) needs the "appfile" external-store shape: small blobs + in a kv/sqlite store (faster than the filesystem for small items — + true beyond sqlite: it applies to the kv store iroh-blobs uses + too), large blobs on a filesystem fallback, with filename↔hash + mapping. alknet's own research hit the core awkwardness of + dispatching across more than one backend at a time — which is + exactly a store-layer problem this crate should own. + +**The scope line (current posture, revisit as evidence arrives):** +this crate is the *store*, not the *transport*. iroh-blobs welds store ++ provider protocol + tickets + postcard into one crate; the defining +posture here is the opposite split — put/get/verify against a hash is +its own layer, and any provider/protocol/ops surface lives above it or +in feature-gated modules (the alk* inversion-point pattern). The ops +surface (a channels-native "give me the blob with this hash" op family, +auth-gated via `AccessControl`) is a Phase 0/1 question, not assumed. + +Guiding principles, inherited from the alk* family: + +1. **Substrate-agnostic by construction.** The store must not know + whether bytes arrive over the network, from a local writer, or get + reassembled anywhere particular. Backends behind an injected seam + (alktty `TtyBackend` / alktunnels pump-halves precedent). +2. **Hashes are data, not transport identity.** Multiple hash + algorithms must coexist (git SHA-1/SHA-256, BLAKE3, maybe bao-shaped + chunk trees as one encoding among several). How this is abstracted — + trait, enum, per-backend config — is an open question (OQ-BL-03). +3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets, + postcard serialization, and provider protocol are design-welded + choices we do not inherit. Its *store-shape* lessons (kv + flat + backends, verification flow, chunking) are fair game per prior-art + reading. +4. **Producer/consumer vocabulary** for any network-facing surface; + authorization via alkcall's `AccessControl`/identity seam, never an + in-band invented scheme (convention 6). +5. **Verify where it matters.** iroh-blobs' core virtue is verified + transfer (bao outboard encoding). The alkgit case already has + verified content (git objects are hash-addressed by git itself). + Which verification story the crate owns — bao trees, per-blob + digests, backend-native — is open (OQ-BL-04). + +## What is already known (settled, thin) + +Almost nothing is pinned — deliberately. The only postures agreed at +setup: + +- **Not a fork of iroh-blobs.** A downstream store crate inspired by + its `src/store` work, diverging deliberately on hashing and wire + surface (see `iroh-blobs-eval` below when written). +- **Multi-hash from day one** (hard requirement from alkgit — do not + hardcode BLAKE3). +- **Backend pluralism assumed, not designed.** kv/sqlite for small + blobs, filesystem fallback for large — the appfile shape — but + *how* multi-backend dispatch works is exactly what alknet's research + ran into, so it earns research and probably POCs (OQ-BL-02). + +## Prior art + +### iroh-blobs — the shape inspiration (evaluated, not the base) + +`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference). +Specifically `src/store`: the kv backend (small blobs, in-process) and +flat file backend (large blobs) split; bao outboard encoding and +verification flow; chunking. Its BLAKE3-only hashing, tickets, postcard +serialization, and provider protocol are the design-welded choices we +diverge from. **The store eval should be written against the current +checkout** — alknet's older research refers to an older iroh-blobs and +its conclusions must be re-verified rather than inherited. + +### gix-odb — git's own object database (alkgit's baseline) + +`/workspace/git-oxide/gix-odb` (read-only reference). The backend +alkgit currently plans against; the hashing-algorithm baseline git +actually uses (SHA-1/SHA-256), git's loose-object and packfile layout, +and git's already-hash-addressed object model. The alkgit question is +whether this crate can sit *under or beside* gix-odb semantics — git +objects are already content-addressed and verified by git's own model, +so the crate's value there is the large-blob story (git-lfs-shaped) +and a better small-object backend than the proposed default, without +fighting git's hash model. + +### alknet's appfile external-store probe — the direct ancestor + +`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/` +(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written +against an older iroh-blobs; conclusions re-verify in this Phase 0. +The load-bearing bits: the appfile shape (small blobs in kv/sqlite, +large on fs fallback), the filename↔hash mapping problem, and the +multi-backend dispatch pain the probe hit (which is this crate's +reason to own that dispatch). Old-data warning: any API or behavior +claims there describe an older upstream; re-check against +`/workspace/iroh-blobs` as checked out today. + +### alkcall — the substrate + +`/workspace/@alkdev/alkcall` (the call + channels RPC crate). The +authorization seam for network-facing blob ops (`AccessControl`, +producer/consumer vocabulary, alkcall ADR-022/037) and, if blob +transfer rides channels, the established data-path patterns +(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even +belongs in this crate is itself open (OQ-BL-01); alkcall is the +substrate it composes with *when it does*. + +## Open Questions + +Numbering is provisional until the register solidifies; promote the +final set into Phase 1's `docs/architecture/open-questions.md`. + +### OQ-BL-01: Crate scope — store-only, or store + ops surface? + +The store (put/get/verify) is clearly in. What about the network ops +layer — a channels-native "fetch/provide blob" op family on the +alkcall substrate, auth-gated via `AccessControl`? iroh-blobs has it +(provider protocol); our posture is to separate it. Options: (a) +store-only crate, ops in a sibling crate later; (b) store + optional +feature-gated ops module here; (c) undecided pending the first +consumer's shape. Leaning (a)/(b) per the inversion-point pattern, but +the alkgit + alknet consumer needs should decide — neither has confirmed +a *networked* transfer requirement yet. + +### OQ-BL-02: Multi-backend dispatch — the appfile problem + +Small blobs in kv/sqlite, large on fs fallback — the shape is agreed, +the mechanics are not: how does put/get pick a backend (size +thresholds? per-algorithm routing? per-namespace config?), how does it +work when a blob should migrate between backends, and what happens on a +get when the "wrong" backend was probed? alknet's probe hit exactly +this ("dealing with more than one backend at a time"); its findings +need re-verification against the current iroh-blobs before relying on +them. Expected shape: a backend trait + a dispatch layer, but the trait +shape is a one-way door once written — deserves a POC before +committing. + +### OQ-BL-03: Hash abstraction — trait, enum, or per-backend config? + +The hard requirement: git SHA-1/SHA-256 and BLAKE3 must coexist (alkgit +conflict). Sub-questions: is the hash algorithm a *store* parameter +(one algorithm per store instance, chosen by the consumer) or +*per-blob* data (multi-algorithm within one store)? Does verification +(bao trees) get per-algorithm treatment or does bao stay BLAKE3-bound +(git objects don't need bao verification anyway)? This interacts with +OQ-BL-01 and the wire-format question — a wire ADR must follow whatever +this settles. + +### OQ-BL-04: Verification and chunking story + +iroh-blobs' verification is bao outboard encoding over BLAKE3 chunk +trees. Git objects are self-verifying under git's model. Raw large +blobs (git-lfs-shaped, appfile large files) need *some* verification +story from us. Options: bao (borrow the *conclusion*, dependency +posture per convention 7), our own digest scheme, or pluggable +verification per blob-kind. Also: does chunking exist at the store +layer at all (git objects are unit blobs; appfile items may be large +files wanting range reads)? Range-read support is an open API-shape +question. + +### OQ-BL-05: POC register (draft) + +Numbered POCs, opened as research reaches them (findings land in +`docs/research/`; worktree placement per the SDD process): + +| # | What | Status | Where | +|---|------|--------|-------| +| 1 | Backend-trait + dual-dispatch shape (kv small / fs large) | **Pending** — likely first POC | findings file TBD | +| 2 | Multi-hash store (SHA-256 + BLAKE3 coexisting) | **Pending** — rides #1's data | findings file TBD | +| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads) | **Pending** | findings file TBD | + +Sequencing note: #1 and #2 are probably one worktree (the dispatch POC +naturally exercises two hash algorithms); #3 is a reading-and-design +POC against the current iroh-blobs checkout plus the alknet probe +re-verification. + +POC placement conventions (inherited from alksocks/alktunnels): a POC +that needs code from this repo runs in a worktree/branch +(`.worktrees/research//` per the SDD process); a +self-contained POC runs as a standalone crate in the global workspace +with findings written into `docs/research/` here. Findings always land +in `docs/research/` regardless of where the code lives. + +## Phase 0 plan (next steps) + +1. **Write `iroh-blobs-eval.md`** against the *current* checkout, + focused on `src/store` (kv + flat backends, bao, chunking) — the + conclusions inventory we may borrow, and the weld points we won't. +2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`, + `poc-summary.md`) against the current upstream; mark what carried + over. +3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit + baseline: its storage layout, object model, and where a blob-store + crate under/beside it earns its keep. +4. **Open POCs** per OQ-BL-05, in the order above. +5. Converge: recommended approach + final OQ register → Phase 1. \ No newline at end of file diff --git a/docs/sdd_process.md b/docs/sdd_process.md index 0ee8a31..0d477a9 100644 --- a/docs/sdd_process.md +++ b/docs/sdd_process.md @@ -2,7 +2,7 @@ ## Overview -This document defines the SDD process for the @alkdev/alkcall package. It +This document defines the SDD process for the @alkdev/alkblobs package. It leverages: - **OpenCode CLI** as the agent execution environment diff --git a/src/lib.rs b/src/lib.rs new file mode 100644 index 0000000..2200a7e --- /dev/null +++ b/src/lib.rs @@ -0,0 +1 @@ +pub fn placeholder() {}