feat: initial repo setup (Phase 0 scaffolding)
- AGENTS.md: repo conventions adapted from alkcall/alksocks (store posture, multi-hash requirement, no-iroh-wire rule; no fuzz gate yet) - Cargo.toml + src/lib.rs: bare lean lib skeleton (tokio async, thiserror) - docs/research/phase-0.md: Phase 0 draft — vision, prior art (iroh-blobs store, gix-odb, alknet appfile probe, alkcall), OQ register, POC plan - .gitignore, LICENSE-APACHE/MIT - Re-pointed stale alknet/alkcall references in .opencode/agents/ and docs/sdd_process.md Verification: cargo build/test, cargo clippy --all-targets -- -D warnings, cargo fmt --check, cargo doc --no-deps — all pass
This commit is contained in:
1 parent
357311d27b
commit
7f7f70b220
10 files changed
+660
-12
No files matched your search
@@ -0,0 +1,4 @@
|
||||
target/
|
||||
node_modules/
|
||||
.worktrees/
|
||||
Cargo.lock
|
||||
@@ -326,10 +326,10 @@ result, a concrete use case to arrive.
|
||||
|
||||
A decision should be `deferred(scope)` when:
|
||||
|
||||
- The use case isn't concrete (e.g., "we don't know what the agent crate
|
||||
will need from the call protocol")
|
||||
- The use case isn't concrete (e.g., "we don't know what the consumer
|
||||
crate will need from the blob API")
|
||||
- The options depend on something that doesn't exist yet (e.g.,
|
||||
"depends on the alknet-http crate spec")
|
||||
"depends on the alkgit crate spec")
|
||||
- The trade-off requires data that can only come from implementation
|
||||
(e.g., "need performance benchmarks to choose between X and Y")
|
||||
- The decision is genuinely not needed for the current scope (e.g., "the
|
||||
@@ -373,10 +373,10 @@ A decision should be `deferred(unclear)` when:
|
||||
(implies it's decided).
|
||||
2. **State the blocking condition** (`deferred(scope)`) or
|
||||
**investigation target** (`deferred(unclear)`) — what specific thing
|
||||
would unblock this? Be concrete: "blocked on: alknet-agent crate spec
|
||||
exists" or "investigation: work through 2+ example outbound-dial use
|
||||
cases (hub→worker, worker→hub) to see how verifier-selection +
|
||||
provider + connector compose."
|
||||
would unblock this? Be concrete: "blocked on: alkgit crate spec
|
||||
exists" or "investigation: work through 2+ example store use
|
||||
cases (small-blob kv backend, large-blob fs fallback) to see
|
||||
how backends + hashes + ACL compose."
|
||||
3. **State the impacts** — what does this block downstream? Be
|
||||
specific: "blocks the first hub deployment because the hub dials
|
||||
workers" not "blocks the hub crate." This is the triage signal that
|
||||
|
||||
@@ -191,7 +191,7 @@ also include:
|
||||
Example prompt template:
|
||||
|
||||
```
|
||||
You are an implementation specialist for the @alkdev/alknet project.
|
||||
You are an implementation specialist for the @alkdev/alkblobs project.
|
||||
|
||||
Your task: {{task}}
|
||||
|
||||
@@ -204,11 +204,10 @@ Your task: {{task}}
|
||||
7. Push: git push origin $(git branch --show-current)
|
||||
8. Notify: worktree({action: "notify", args: {message: "Task completed: {{task}}. <brief summary>", level: "info"}})
|
||||
|
||||
Key project constraints (@alkdev/alknet):
|
||||
Key project constraints (@alkdev/alkblobs):
|
||||
- Rust: use cargo build, cargo clippy, cargo fmt, cargo test
|
||||
- No comments in code
|
||||
- anyhow::Result for application errors, thiserror for library error types
|
||||
- Feature flags for transports (tls, iroh, acme)
|
||||
- thiserror for error types
|
||||
- Async via tokio runtime
|
||||
- No panics in library code
|
||||
```
|
||||
|
||||
@@ -0,0 +1,175 @@
|
||||
# AGENTS.md
|
||||
|
||||
Operating instructions for opencode agents working in this repo. opencode
|
||||
auto-loads this file as instructions, overriding the built-in defaults for
|
||||
this project. Custom agents in `.opencode/agents/` inherit these rules
|
||||
unless their own prompts say otherwise.
|
||||
|
||||
## Git Workflow
|
||||
|
||||
**Commit and push when reasonable.** When a change is complete and
|
||||
verified (build + lint + tests pass), commit and push to `origin/main`
|
||||
without asking. This overrides the built-in default of "only commit when
|
||||
explicitly asked."
|
||||
|
||||
The workflow:
|
||||
|
||||
1. Make the change
|
||||
2. Verify: `cargo test`, `cargo clippy --all-targets -- -D warnings`,
|
||||
`cargo fmt --check`, `cargo doc --no-deps` if docs changed
|
||||
3. Inspect `git status` and `git diff` before staging — stage only the
|
||||
intended files, never secrets
|
||||
4. Write a concise commit message matching the repo style (see `git log
|
||||
--oneline -10`). For multi-point changes, use a summary line plus a
|
||||
body with bullet points and a verification block.
|
||||
5. `git push origin main`
|
||||
6. Report the commit hash and the verification summary
|
||||
|
||||
Exceptions — **do not** commit or push without asking:
|
||||
|
||||
- The change is exploratory / speculative (you're not sure the user wants
|
||||
it kept)
|
||||
- The user is actively reviewing the diff and may ask for changes
|
||||
- The change touches the wire format, a trait shape backends implement,
|
||||
or the hashing/verification story (one-way doors once consumers exist
|
||||
— see convention 7; nothing is wire-stable yet, but the first
|
||||
wire-format ADR must be written before the first consumer)
|
||||
- You'd be force-pushing, amending a published commit, creating an empty
|
||||
commit, or skipping hooks
|
||||
|
||||
Never commit secrets, keys, or credentials. If a commit fails or hooks
|
||||
reject it, fix the issue and create a new commit — do not amend the
|
||||
failed one.
|
||||
|
||||
Git identity is preconfigured (`glm-5.3-flash <glm-5.3-flash@alk.dev>`). Do
|
||||
not change `git config`, skip hooks, or use `git commit -i`.
|
||||
|
||||
## Project Conventions (Rust / blob storage crate)
|
||||
|
||||
This is the blob storage crate — content-addressed blob storage in the
|
||||
alk* family, inspired by iroh-blobs' store work but deliberately not a
|
||||
fork of it. It sits downstream of alkcall (the call + channels substrate
|
||||
its network operations ride, when they exist) and its first planned
|
||||
consumer is alkgit (git object storage, where the iroh-blobs hashing
|
||||
story collides with git's own object hashes). Status: **Phase 0
|
||||
(Exploration)** — `docs/research/` is the working state of the repo;
|
||||
`docs/architecture/` does not exist yet and no wire format, API shape,
|
||||
or backend trait is decided. The conventions below apply to all work in
|
||||
`src/` and `tests/`. They mirror
|
||||
`.opencode/agents/implementation-specialist.md` §Project Conventions and
|
||||
are repeated here so they apply to every session.
|
||||
|
||||
1. **No comments in code** unless the user explicitly asks. This is a
|
||||
project-wide convention. Doc comments (`///`, `//!`) are fine and
|
||||
expected on public API. Inline `//` comments only when the user asks
|
||||
or when a non-obvious safety/correctness constraint would otherwise
|
||||
be missed.
|
||||
|
||||
2. **Error handling** — `thiserror` for library error types. No panics
|
||||
in library code. No `unwrap()` or `expect()` outside tests. If you
|
||||
reach for `unwrap`, the error path wasn't specified — stop and decide
|
||||
what should actually happen. For poisoned `RwLock`/`Mutex`, use
|
||||
`unwrap_or_else(|e| e.into_inner())` so a panic in one operation does
|
||||
not cascade to other operations.
|
||||
|
||||
3. **`tokio` is the async runtime** — all I/O is async. Use
|
||||
`tokio::sync` primitives (`oneshot`, `mpsc`) for lifecycle
|
||||
correlation; `parking_lot` for short-held internal locks. Do not
|
||||
introduce blocking I/O on the async path.
|
||||
|
||||
4. **Substrate-agnostic by construction** — the store layer must not
|
||||
know whether bytes arrive over the network, from a local writer, or
|
||||
get reassembled anywhere particular. iroh-blobs welds its store to
|
||||
its provider/ticket/postcard stack; this crate's defining posture is
|
||||
the opposite split: the store (put/get/verify against a hash) is its
|
||||
own layer, and transport/protocol concerns live above it or in
|
||||
feature-gated modules. This is the same inversion-point pattern as
|
||||
the alk* family (alktty `TtyBackend`, alktunnels pump halves).
|
||||
|
||||
5. **Hashes are data, not identity of transport** — the crate must
|
||||
tolerate multiple hash algorithms (the alkgit conflict: git's SHA-1/
|
||||
SHA-256 object hashes vs iroh-blobs' BLAKE3). How the hash algorithm
|
||||
is abstracted (trait, enum, per-backend configuration) is a Phase 0/
|
||||
1 decision, not yet pinned. Do not hardcode a single algorithm.
|
||||
|
||||
6. **Auth-gated operations ride the alkcall authorization seam** — when
|
||||
network-facing operations exist, authorization happens via alkcall's
|
||||
`AccessControl`/identity mechanisms (the producer/consumer model),
|
||||
not via an in-band invented auth scheme. Producer/consumer
|
||||
vocabulary, not "server"/"client" (alkcall ADR-022/037). Not yet
|
||||
pinned in detail — the ops surface is a Phase 0 question.
|
||||
|
||||
7. **Do not adopt iroh-blobs' wire surface** — tickets, the postcard
|
||||
serialization, and the provider protocol are iroh-blobs'
|
||||
design-welded choices we have decided not to inherit. Reading
|
||||
`/workspace/iroh-blobs` (a fresh upstream checkout) is encouraged for
|
||||
store-shape lessons (`src/store`, the kv + flat file backends);
|
||||
borrowing its *conclusions* is fine, wiring in its *types*, framing,
|
||||
or serialization as load-bearing dependencies is not (yet — the
|
||||
dependency posture is a Phase 0 question if a small pure piece, like
|
||||
bao outboard encoding, earns its place).
|
||||
|
||||
8. **Feature flags** — substrate backends (kv, sqlite, flat fs) may be
|
||||
feature-gated if the need arises. The base crate should compile lean.
|
||||
Verify both `cargo test` (default) and `cargo test --all-features`
|
||||
pass if features are added.
|
||||
|
||||
9. **Naming** — Rust standard: `snake_case` for functions/variables/
|
||||
modules, `PascalCase` for types/traits, `SCREAMING_SNAKE_CASE` for
|
||||
constants.
|
||||
|
||||
10. **Module structure** — one module per file under `src/`, re-exported
|
||||
from `src/lib.rs`. Public API surface is `lib.rs` re-exports. The
|
||||
expected shape (pending Phase 0 convergence) separates the
|
||||
backend/store layer from any protocol/ops layer.
|
||||
|
||||
## Verification Commands
|
||||
|
||||
Run these before committing. All must pass.
|
||||
|
||||
```bash
|
||||
cargo test # full suite
|
||||
cargo clippy --all-targets -- -D warnings
|
||||
cargo fmt --check
|
||||
cargo doc --no-deps # if docs changed
|
||||
cargo publish --dry-run --allow-dirty # before a release
|
||||
```
|
||||
|
||||
If feature flags are added, also run `cargo test --all-features` and
|
||||
`cargo clippy --all-features --all-targets -- -D warnings`.
|
||||
|
||||
No fuzz targets exist yet for this crate. If/when a wire format is
|
||||
pinned (Phase 1), a fuzz gate on the model of alkcall/alksocks (`fuzz/`
|
||||
corpus replay on stable) is expected — until then ignore fuzzing.
|
||||
|
||||
## Architecture Context
|
||||
|
||||
- `docs/research/` — the Phase 0 (Exploration) state of this repo:
|
||||
research findings, POC records, and `phase-0.md` (vision, prior art,
|
||||
open questions, converged recommendation). Read it before non-trivial
|
||||
work. The SDD process lives in `docs/sdd_process.md`.
|
||||
- `docs/architecture/` — does not exist yet (Phase 1 output). When it
|
||||
lands, the SDD process applies: specs describe WHAT, `decisions/`
|
||||
ADRs explain WHY, `open-questions.md` tracks what's unresolved.
|
||||
- Key prior art (read-only, in the global workspace):
|
||||
- **iroh-blobs** — `/workspace/iroh-blobs` (fresh upstream checkout):
|
||||
the shape inspiration, specifically its store work (kv + flat file
|
||||
backends, bao verification, chunking). Its wire-surface choices
|
||||
(tickets, postcard, BLAKE3-only) are the ones we are deliberately
|
||||
diverging from.
|
||||
- **gix-odb** — `/workspace/git-oxide/gix-odb`: git's own object
|
||||
database; the backend alkgit currently plans against and the
|
||||
hashing-algorithm baseline git actually uses.
|
||||
- **alknet's blobs research** —
|
||||
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
|
||||
(`alknet-blobs-external-store-probe.md`, `poc-summary.md`): the
|
||||
appfile external-store probe — small blobs in a kv/sqlite-ish
|
||||
store, large blobs on the filesystem fallback, filename↔hash
|
||||
mapping. Written against an older iroh-blobs than the current
|
||||
checkout; re-verify conclusions before relying on them.
|
||||
- **alkcall** — `/workspace/@alkdev/alkcall` (the substrate):
|
||||
call + channels; its `AccessControl`/identity seam is the
|
||||
authorization story for network-facing blob ops.
|
||||
- If a TODO references a design direction that a later ADR has decided
|
||||
against, the TODO is stale — remove it and align with the ADR. Do not
|
||||
implement the rejected design.
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
[package]
|
||||
name = "alkblobs"
|
||||
version = "0.1.0"
|
||||
edition = "2021"
|
||||
rust-version = "1.88"
|
||||
license = "MIT OR Apache-2.0"
|
||||
description = "Content-addressed blob storage: multi-hash put/get/verify store with pluggable backends, transport- and protocol-agnostic"
|
||||
repository = "https://git.alk.dev/alkdev/alkblobs"
|
||||
keywords = ["blobs", "content-addressed", "storage", "gix", "bao"]
|
||||
categories = ["asynchronous", "filesystem", "database"]
|
||||
exclude = [".opencode/", "AGENTS.md", "docs/reviews/", "docs/research/", "docs/plans/", "tasks/", "docs/sdd_process.md"]
|
||||
|
||||
[workspace]
|
||||
members = ["."]
|
||||
exclude = []
|
||||
|
||||
[lib]
|
||||
name = "alkblobs"
|
||||
|
||||
[dependencies]
|
||||
tokio = { version = "1", default-features = false, features = ["rt", "sync", "io-util", "macros"] }
|
||||
thiserror = "2"
|
||||
parking_lot = "0.12"
|
||||
+192
@@ -0,0 +1,192 @@
|
||||
Apache License
|
||||
Version 2.0, January 2004
|
||||
http://www.apache.org/licenses/
|
||||
|
||||
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
|
||||
|
||||
1. Definitions.
|
||||
|
||||
"License" shall mean the terms and conditions for use, reproduction,
|
||||
and distribution as defined by Sections 1 through 9 of this document.
|
||||
|
||||
"Licensor" shall mean the copyright owner or entity authorized by
|
||||
the copyright owner that is granting the License.
|
||||
|
||||
"Legal Entity" shall mean the union of the acting entity and all
|
||||
other entities that control, are controlled by, or are under common
|
||||
control with that entity. For the purposes of this definition,
|
||||
"control" means (i) the power, direct or indirect, to cause the
|
||||
direction or management of such entity, whether by contract or
|
||||
otherwise, or (ii) ownership of fifty percent (50%) or more of the
|
||||
outstanding shares, or (iii) beneficial ownership of such entity.
|
||||
|
||||
"You" (or "Your") shall mean an individual or Legal Entity
|
||||
exercising permissions granted by this License.
|
||||
|
||||
"Source" form shall mean the preferred form for making modifications,
|
||||
including but not limited to software source code, documentation
|
||||
source, and configuration files.
|
||||
|
||||
"Object" form shall mean any form resulting from mechanical
|
||||
transformation or translation of a Source form, including but
|
||||
not limited to compiled object code, generated documentation,
|
||||
and conversions to other media types.
|
||||
|
||||
"Work" shall mean the work of authorship, whether in Source or
|
||||
Object form, made available under the License, as indicated by a
|
||||
copyright notice that is included in or attached to the work
|
||||
(an example is provided in the Appendix below).
|
||||
|
||||
"Derivative Works" shall mean any work, whether in Source or Object
|
||||
form, that is based on (or derived from) the Work and for which the
|
||||
editorial revisions, annotations, elaborations, or other modifications
|
||||
represent, as a whole, an original work of authorship. For the purposes
|
||||
of this License, Derivative Works shall not include works that remain
|
||||
separable from, or merely link (or bind by name), or refer to, the Work.
|
||||
(Note: Derivative Works shall not include works that remain separable from,
|
||||
or merely link (or bind by name) to the interfaces of, the Work and
|
||||
Derivative Works thereof.)
|
||||
|
||||
"Contribution" shall mean any work of authorship, including
|
||||
the original version of the Work and any modifications or additions
|
||||
to that Work or Derivative Works thereof, that is intentionally
|
||||
submitted to the Licensor for inclusion in the Work by the copyright owner
|
||||
or by an individual or Legal Entity authorized to submit on behalf of
|
||||
the copyright owner. For the purposes of this definition, "submitted"
|
||||
means any form of electronic, verbal, or written communication sent
|
||||
to the Licensor or its representatives, including but not limited to
|
||||
communication on electronic mailing lists, source code control systems,
|
||||
and issue tracking systems that are managed by, or on behalf of, the
|
||||
Licensor for the purpose of discussing and improving the Work, but
|
||||
excluding communication that is conspicuously marked or otherwise
|
||||
designated in writing by the copyright owner as "Not a Contribution."
|
||||
|
||||
"Contributor" shall mean Licensor and any individual or Legal Entity
|
||||
on behalf of whom a Contribution has been received by the Licensor and
|
||||
subsequently incorporated within the Work.
|
||||
|
||||
2. Grant of Copyright License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
copyright license to reproduce, prepare Derivative Works of,
|
||||
publicly display, publicly perform, sublicense, and distribute the
|
||||
Work and such Derivative Works in Source or Object form.
|
||||
|
||||
3. Grant of Patent License. Subject to the terms and conditions of
|
||||
this License, each Contributor hereby grants to You a perpetual,
|
||||
worldwide, non-exclusive, no-charge, royalty-free, irrevocable
|
||||
(except as stated in this section) patent license to make, have made,
|
||||
use, offer to sell, sell, import, and otherwise transfer the Work,
|
||||
where such license applies only to those patent claims licensable
|
||||
by such Contributor that are necessarily infringed by their
|
||||
Contribution(s) alone or by combination of their Contribution(s)
|
||||
with the Work to which such Contribution(s) was submitted. If You
|
||||
institute patent litigation against any entity (including a
|
||||
cross-claim or counterclaim in a lawsuit) alleging that the Work
|
||||
or a Contribution incorporated within the Work constitutes direct
|
||||
or contributory patent infringement, then any patent licenses
|
||||
granted to You under this License for that Work shall terminate
|
||||
as of the date such litigation is filed.
|
||||
|
||||
4. Redistribution. You may reproduce and distribute copies of the
|
||||
Work or Derivative Works thereof in any medium, with or without
|
||||
modifications, and in Source or Object form, provided that You
|
||||
meet the following conditions:
|
||||
|
||||
(a) You must give any other recipients of the Work or
|
||||
Derivative Works a copy of this License; and
|
||||
|
||||
(b) You must cause any modified files to carry prominent notices
|
||||
stating that You changed the files; and
|
||||
|
||||
(c) You must retain, in the Source form of any Derivative Works
|
||||
that You distribute, all copyright, patent, trademark, and
|
||||
attribution notices from the Source form of the Work,
|
||||
excluding those notices that do not pertain to any part of
|
||||
the Derivative Works; and
|
||||
|
||||
(d) If the Work includes a "NOTICE" text file as part of its
|
||||
distribution, then any Derivative Works that You distribute must
|
||||
include a readable copy of the attribution notices contained
|
||||
within such NOTICE file, excluding those notices that do not
|
||||
pertain to any part of the Derivative Works, in at least one
|
||||
of the following places: within a NOTICE text file distributed
|
||||
as part of the Derivative Works; within the Source form or
|
||||
documentation, if provided along with the Derivative Works; or,
|
||||
within a display generated by the Derivative Works, if and
|
||||
wherever such third-party notices normally appear. The contents
|
||||
of the NOTICE file are for informational purposes only and
|
||||
do not modify the License. You may add Your own attribution
|
||||
notices within Derivative Works that You distribute, alongside
|
||||
or as an addendum to the NOTICE text from the Work, provided
|
||||
that such additional attribution notices cannot be construed
|
||||
as modifying the License.
|
||||
|
||||
You may add Your own copyright statement to Your modifications and
|
||||
may provide additional or different license terms and conditions
|
||||
for use, reproduction, or distribution of Your modifications, or
|
||||
for any such Derivative Works as a whole, provided Your use,
|
||||
reproduction, and distribution of the Work otherwise complies with
|
||||
the conditions stated in this License.
|
||||
|
||||
5. Submission of Contributions. Unless You explicitly state otherwise,
|
||||
any Contribution intentionally submitted for inclusion in the Work
|
||||
by You to the Licensor shall be under the terms and conditions of
|
||||
this License, without any additional terms or conditions.
|
||||
Notwithstanding the above, nothing herein shall supersede or modify
|
||||
the terms of any separate license agreement you may have executed
|
||||
with Licensor regarding such Contributions.
|
||||
|
||||
6. Trademarks. This License does not grant permission to use the trade
|
||||
names, trademarks, service marks, or product names of the Licensor,
|
||||
except as required for reasonable and customary use in describing the
|
||||
origin of the Work and reproducing the content of the NOTICE file.
|
||||
|
||||
7. Disclaimer of Warranty. Unless required by applicable law or
|
||||
agreed to in writing, Licensor provides the Work (and each
|
||||
Contributor provides its Contributions) on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
|
||||
implied, including, without limitation, any warranties or conditions
|
||||
of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
|
||||
PARTICULAR PURPOSE. You are solely responsible for determining the
|
||||
appropriateness of using or redistributing the Work and assume any
|
||||
risks associated with Your exercise of permissions under this License.
|
||||
|
||||
8. Limitation of Liability. In no event and under no legal theory,
|
||||
whether in tort (including negligence), contract, or otherwise,
|
||||
unless required by applicable law (such as deliberate and grossly
|
||||
negligent acts) or agreed to in writing, shall any Contributor be
|
||||
liable to You for damages, including any direct, indirect, special,
|
||||
incidental, or consequential damages of any character arising as a
|
||||
result of this License or out of the use or inability to use the
|
||||
Work (including but not limited to damages for loss of goodwill,
|
||||
work stoppage, computer failure or malfunction, or any and all
|
||||
other commercial damages or losses), even if such Contributor
|
||||
has been advised of the possibility of such damages.
|
||||
|
||||
9. Accepting Warranty or Additional Liability. While redistributing
|
||||
the Work or Derivative Works thereof, You may choose to offer,
|
||||
and charge a fee for, acceptance of support, warranty, indemnity,
|
||||
or other liability obligations and/or rights consistent with this
|
||||
License. However, in accepting such obligations, You may act only
|
||||
on Your own behalf and on Your sole responsibility, not on behalf
|
||||
of any other Contributor, and only if You agree to indemnify,
|
||||
defend, and hold each Contributor harmless for any liability
|
||||
incurred by, or claims asserted against, such Contributor by reason
|
||||
of your accepting any such warranty or additional liability.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
Copyright 2025-2026 Alk Development
|
||||
|
||||
Licensed under the Apache License, Version 2.0 (the "License");
|
||||
you may not use this file except in compliance with the License.
|
||||
You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing, software
|
||||
distributed under the License is distributed on an "AS IS" BASIS,
|
||||
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
See the License for the specific language governing permissions and
|
||||
limitations under the License.
|
||||
+21
@@ -0,0 +1,21 @@
|
||||
MIT License
|
||||
|
||||
Copyright (c) 2025-2026 Alk Development
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
@@ -0,0 +1,233 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-09-30 (initial setup draft: vision, prior art, open
|
||||
questions from the setup discussion; un-numbered — renumber
|
||||
OQ-BL-01..NN as the register solidifies)
|
||||
---
|
||||
|
||||
# alkblobs — Phase 0 (Exploration)
|
||||
|
||||
This document captures Phase 0 (Exploration) for the `alkblobs` crate:
|
||||
vision, guiding principles, prior art, and open questions (OQ-BL-01..NN).
|
||||
Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding
|
||||
principles; research options; validate approaches; converge on a
|
||||
recommended approach.* It is the input to Phase 1 (Architecture), where
|
||||
the Architect will produce `docs/architecture/` specs, ADRs, and the
|
||||
open-questions tracker.
|
||||
|
||||
Drafted 2026-09-30, emerging from the initial setup discussion. Nothing
|
||||
is converged yet — this is the working sketch, not a specification.
|
||||
|
||||
## Vision and guiding principles
|
||||
|
||||
**One sentence:** content-addressed blob storage in the alk* family —
|
||||
a multi-hash put/get/verify store with pluggable backends (small blobs
|
||||
in a kv/sqlite-ish store, large blobs on a filesystem fallback), split
|
||||
from any transport/protocol layer and tolerant of hash-algorithm
|
||||
conflicts (the alkgit case), with auth-gated network operations riding
|
||||
the alkcall seam when they exist.
|
||||
|
||||
**Why this crate exists (two converging consumers):**
|
||||
|
||||
1. **alkgit** (planning phase) needs an object backend that is *not*
|
||||
its proposed default file-based backend ("kind of gross") — blobs
|
||||
stored under git's own object hashes, but with iroh-blobs-like
|
||||
handling for large blobs (where git typically delegates to git-lfs;
|
||||
we could instead follow an iroh-blobs-shaped large-blob path). The
|
||||
conflict: iroh-blobs is BLAKE3-only and content-hashes with bao
|
||||
verification, while git's object database is SHA-1/SHA-256 with
|
||||
git's own object format. A store that insists on one hash algorithm
|
||||
cannot serve git objects directly.
|
||||
2. **The alknet rewrite** (the original alk* project, being decomposed
|
||||
and improved) needs the "appfile" external-store shape: small blobs
|
||||
in a kv/sqlite store (faster than the filesystem for small items —
|
||||
true beyond sqlite: it applies to the kv store iroh-blobs uses
|
||||
too), large blobs on a filesystem fallback, with filename↔hash
|
||||
mapping. alknet's own research hit the core awkwardness of
|
||||
dispatching across more than one backend at a time — which is
|
||||
exactly a store-layer problem this crate should own.
|
||||
|
||||
**The scope line (current posture, revisit as evidence arrives):**
|
||||
this crate is the *store*, not the *transport*. iroh-blobs welds store
|
||||
+ provider protocol + tickets + postcard into one crate; the defining
|
||||
posture here is the opposite split — put/get/verify against a hash is
|
||||
its own layer, and any provider/protocol/ops surface lives above it or
|
||||
in feature-gated modules (the alk* inversion-point pattern). The ops
|
||||
surface (a channels-native "give me the blob with this hash" op family,
|
||||
auth-gated via `AccessControl`) is a Phase 0/1 question, not assumed.
|
||||
|
||||
Guiding principles, inherited from the alk* family:
|
||||
|
||||
1. **Substrate-agnostic by construction.** The store must not know
|
||||
whether bytes arrive over the network, from a local writer, or get
|
||||
reassembled anywhere particular. Backends behind an injected seam
|
||||
(alktty `TtyBackend` / alktunnels pump-halves precedent).
|
||||
2. **Hashes are data, not transport identity.** Multiple hash
|
||||
algorithms must coexist (git SHA-1/SHA-256, BLAKE3, maybe bao-shaped
|
||||
chunk trees as one encoding among several). How this is abstracted —
|
||||
trait, enum, per-backend config — is an open question (OQ-BL-03).
|
||||
3. **Borrow conclusions, not wire surface.** iroh-blobs' tickets,
|
||||
postcard serialization, and provider protocol are design-welded
|
||||
choices we do not inherit. Its *store-shape* lessons (kv + flat
|
||||
backends, verification flow, chunking) are fair game per prior-art
|
||||
reading.
|
||||
4. **Producer/consumer vocabulary** for any network-facing surface;
|
||||
authorization via alkcall's `AccessControl`/identity seam, never an
|
||||
in-band invented scheme (convention 6).
|
||||
5. **Verify where it matters.** iroh-blobs' core virtue is verified
|
||||
transfer (bao outboard encoding). The alkgit case already has
|
||||
verified content (git objects are hash-addressed by git itself).
|
||||
Which verification story the crate owns — bao trees, per-blob
|
||||
digests, backend-native — is open (OQ-BL-04).
|
||||
|
||||
## What is already known (settled, thin)
|
||||
|
||||
Almost nothing is pinned — deliberately. The only postures agreed at
|
||||
setup:
|
||||
|
||||
- **Not a fork of iroh-blobs.** A downstream store crate inspired by
|
||||
its `src/store` work, diverging deliberately on hashing and wire
|
||||
surface (see `iroh-blobs-eval` below when written).
|
||||
- **Multi-hash from day one** (hard requirement from alkgit — do not
|
||||
hardcode BLAKE3).
|
||||
- **Backend pluralism assumed, not designed.** kv/sqlite for small
|
||||
blobs, filesystem fallback for large — the appfile shape — but
|
||||
*how* multi-backend dispatch works is exactly what alknet's research
|
||||
ran into, so it earns research and probably POCs (OQ-BL-02).
|
||||
|
||||
## Prior art
|
||||
|
||||
### iroh-blobs — the shape inspiration (evaluated, not the base)
|
||||
|
||||
`/workspace/iroh-blobs` (fresh upstream checkout; read-only reference).
|
||||
Specifically `src/store`: the kv backend (small blobs, in-process) and
|
||||
flat file backend (large blobs) split; bao outboard encoding and
|
||||
verification flow; chunking. Its BLAKE3-only hashing, tickets, postcard
|
||||
serialization, and provider protocol are the design-welded choices we
|
||||
diverge from. **The store eval should be written against the current
|
||||
checkout** — alknet's older research refers to an older iroh-blobs and
|
||||
its conclusions must be re-verified rather than inherited.
|
||||
|
||||
### gix-odb — git's own object database (alkgit's baseline)
|
||||
|
||||
`/workspace/git-oxide/gix-odb` (read-only reference). The backend
|
||||
alkgit currently plans against; the hashing-algorithm baseline git
|
||||
actually uses (SHA-1/SHA-256), git's loose-object and packfile layout,
|
||||
and git's already-hash-addressed object model. The alkgit question is
|
||||
whether this crate can sit *under or beside* gix-odb semantics — git
|
||||
objects are already content-addressed and verified by git's own model,
|
||||
so the crate's value there is the large-blob story (git-lfs-shaped)
|
||||
and a better small-object backend than the proposed default, without
|
||||
fighting git's hash model.
|
||||
|
||||
### alknet's appfile external-store probe — the direct ancestor
|
||||
|
||||
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
|
||||
(`alknet-blobs-external-store-probe.md`, `poc-summary.md`) — written
|
||||
against an older iroh-blobs; conclusions re-verify in this Phase 0.
|
||||
The load-bearing bits: the appfile shape (small blobs in kv/sqlite,
|
||||
large on fs fallback), the filename↔hash mapping problem, and the
|
||||
multi-backend dispatch pain the probe hit (which is this crate's
|
||||
reason to own that dispatch). Old-data warning: any API or behavior
|
||||
claims there describe an older upstream; re-check against
|
||||
`/workspace/iroh-blobs` as checked out today.
|
||||
|
||||
### alkcall — the substrate
|
||||
|
||||
`/workspace/@alkdev/alkcall` (the call + channels RPC crate). The
|
||||
authorization seam for network-facing blob ops (`AccessControl`,
|
||||
producer/consumer vocabulary, alkcall ADR-022/037) and, if blob
|
||||
transfer rides channels, the established data-path patterns
|
||||
(`BiStream`, two-pump `pump_bidi` ADR-050). Whether transport even
|
||||
belongs in this crate is itself open (OQ-BL-01); alkcall is the
|
||||
substrate it composes with *when it does*.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Numbering is provisional until the register solidifies; promote the
|
||||
final set into Phase 1's `docs/architecture/open-questions.md`.
|
||||
|
||||
### OQ-BL-01: Crate scope — store-only, or store + ops surface?
|
||||
|
||||
The store (put/get/verify) is clearly in. What about the network ops
|
||||
layer — a channels-native "fetch/provide blob" op family on the
|
||||
alkcall substrate, auth-gated via `AccessControl`? iroh-blobs has it
|
||||
(provider protocol); our posture is to separate it. Options: (a)
|
||||
store-only crate, ops in a sibling crate later; (b) store + optional
|
||||
feature-gated ops module here; (c) undecided pending the first
|
||||
consumer's shape. Leaning (a)/(b) per the inversion-point pattern, but
|
||||
the alkgit + alknet consumer needs should decide — neither has confirmed
|
||||
a *networked* transfer requirement yet.
|
||||
|
||||
### OQ-BL-02: Multi-backend dispatch — the appfile problem
|
||||
|
||||
Small blobs in kv/sqlite, large on fs fallback — the shape is agreed,
|
||||
the mechanics are not: how does put/get pick a backend (size
|
||||
thresholds? per-algorithm routing? per-namespace config?), how does it
|
||||
work when a blob should migrate between backends, and what happens on a
|
||||
get when the "wrong" backend was probed? alknet's probe hit exactly
|
||||
this ("dealing with more than one backend at a time"); its findings
|
||||
need re-verification against the current iroh-blobs before relying on
|
||||
them. Expected shape: a backend trait + a dispatch layer, but the trait
|
||||
shape is a one-way door once written — deserves a POC before
|
||||
committing.
|
||||
|
||||
### OQ-BL-03: Hash abstraction — trait, enum, or per-backend config?
|
||||
|
||||
The hard requirement: git SHA-1/SHA-256 and BLAKE3 must coexist (alkgit
|
||||
conflict). Sub-questions: is the hash algorithm a *store* parameter
|
||||
(one algorithm per store instance, chosen by the consumer) or
|
||||
*per-blob* data (multi-algorithm within one store)? Does verification
|
||||
(bao trees) get per-algorithm treatment or does bao stay BLAKE3-bound
|
||||
(git objects don't need bao verification anyway)? This interacts with
|
||||
OQ-BL-01 and the wire-format question — a wire ADR must follow whatever
|
||||
this settles.
|
||||
|
||||
### OQ-BL-04: Verification and chunking story
|
||||
|
||||
iroh-blobs' verification is bao outboard encoding over BLAKE3 chunk
|
||||
trees. Git objects are self-verifying under git's model. Raw large
|
||||
blobs (git-lfs-shaped, appfile large files) need *some* verification
|
||||
story from us. Options: bao (borrow the *conclusion*, dependency
|
||||
posture per convention 7), our own digest scheme, or pluggable
|
||||
verification per blob-kind. Also: does chunking exist at the store
|
||||
layer at all (git objects are unit blobs; appfile items may be large
|
||||
files wanting range reads)? Range-read support is an open API-shape
|
||||
question.
|
||||
|
||||
### OQ-BL-05: POC register (draft)
|
||||
|
||||
Numbered POCs, opened as research reaches them (findings land in
|
||||
`docs/research/`; worktree placement per the SDD process):
|
||||
|
||||
| # | What | Status | Where |
|
||||
|---|------|--------|-------|
|
||||
| 1 | Backend-trait + dual-dispatch shape (kv small / fs large) | **Pending** — likely first POC | findings file TBD |
|
||||
| 2 | Multi-hash store (SHA-256 + BLAKE3 coexisting) | **Pending** — rides #1's data | findings file TBD |
|
||||
| 3 | Large-blob path (iroh-blobs store read under current checkout; fs fallback + range reads) | **Pending** | findings file TBD |
|
||||
|
||||
Sequencing note: #1 and #2 are probably one worktree (the dispatch POC
|
||||
naturally exercises two hash algorithms); #3 is a reading-and-design
|
||||
POC against the current iroh-blobs checkout plus the alknet probe
|
||||
re-verification.
|
||||
|
||||
POC placement conventions (inherited from alksocks/alktunnels): a POC
|
||||
that needs code from this repo runs in a worktree/branch
|
||||
(`.worktrees/research/<task-id>/` per the SDD process); a
|
||||
self-contained POC runs as a standalone crate in the global workspace
|
||||
with findings written into `docs/research/` here. Findings always land
|
||||
in `docs/research/` regardless of where the code lives.
|
||||
|
||||
## Phase 0 plan (next steps)
|
||||
|
||||
1. **Write `iroh-blobs-eval.md`** against the *current* checkout,
|
||||
focused on `src/store` (kv + flat backends, bao, chunking) — the
|
||||
conclusions inventory we may borrow, and the weld points we won't.
|
||||
2. **Re-verify alknet's probe** (`alknet-blobs-external-store-probe.md`,
|
||||
`poc-summary.md`) against the current upstream; mark what carried
|
||||
over.
|
||||
3. **Read gix-odb** (`/workspace/git-oxide/gix-odb`) for the alkgit
|
||||
baseline: its storage layout, object model, and where a blob-store
|
||||
crate under/beside it earns its keep.
|
||||
4. **Open POCs** per OQ-BL-05, in the order above.
|
||||
5. Converge: recommended approach + final OQ register → Phase 1.
|
||||
+1
-1
@@ -2,7 +2,7 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This document defines the SDD process for the @alkdev/alkcall package. It
|
||||
This document defines the SDD process for the @alkdev/alkblobs package. It
|
||||
leverages:
|
||||
|
||||
- **OpenCode CLI** as the agent execution environment
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
pub fn placeholder() {}
|
||||
Reference in new issue
Block a user