alkdev/alkfs
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Set up repo scaffolding: gitignore, AGENTS.md, Phase 0 doc

Browse Source 
- .gitignore matching sibling repos (target/, node_modules/, .worktrees/, Cargo.lock)
- AGENTS.md on the alkcall/alksocks template: git workflow, 17 project
  conventions adapted for a virtual-filesystem crate (no forced host-FS
  access, metadata/content split, durability ordering, storage-trait
  one-way doors), verification commands (docs-only posture for Phase 0),
  and architecture context pointing at the ancestor research and family ADRs
- docs/research/phase-0.md: initial Phase 0 draft — vision/scope sketch,
  guiding principles, what's already settled, prior art (alknet-filesystem
  POCs, iroh-blobs + external-store probe, git/git-lfs, alkgit backend
  seam, russh-sftp, SQLite/honker, rudolfs), 15 open questions
  (OQ-FS-01..15) grouped by theme, a 7-entry candidate POC register
  (proposals, none run), survey list, and an unconverged checklist

Verification: docs-only repo — every referenced path, ADR id, and OQ id
checked to exist (workspace paths, alkcall ADRs 034-051, alkgit research
files); OQ numbering OQ-FS-01..15 complete with no gaps
This commit is contained in:
glm-5.3-flash committed 2026-09-23 08:57:02 +00:00
1 parent eb302f3a7a
commit 9e863747fc
3 files changed
+1057
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

.gitignore
+4
View File
@@ -0,0 +1,4 @@
target/
node_modules/
.worktrees/
Cargo.lock
AGENTS.md
+299
View File
@@ -0,0 +1,299 @@
# AGENTS.md
Operating instructions for opencode agents working in this repo. opencode
auto-loads this file as instructions, overriding the built-in defaults for
this project. Custom agents in `.opencode/agents/` inherit these rules
unless their own prompts say otherwise.
## Git Workflow
**Commit and push when reasonable.** When a change is complete and
verified, commit and push to `origin/main` without asking. This overrides
the built-in default of "only commit when explicitly asked."
The workflow:
1. Make the change
2. Verify (once `src/` exists): `cargo test`,
`cargo clippy --all-targets -- -D warnings`, `cargo fmt --check`,
`cargo doc --no-deps` if docs changed. While the repo is docs-only
(Phase 0), verification is a careful re-read of the changed docs and
a check that every referenced path/ADR/OQ id exists
3. Inspect `git status` and `git diff` before staging — stage only the
intended files, never secrets
4. Write a concise commit message matching the repo style (see `git log
--oneline -10`). For multi-point changes, use a summary line plus a
body with bullet points and a verification block.
5. `git push origin main`
6. Report the commit hash and the verification summary
Exceptions — **do not** commit or push without asking:
- The change is exploratory / speculative (you're not sure the user wants
it kept)
- The user is actively reviewing the diff and may ask for changes
- The change touches a wire format or a trait shape that backends or
consumers implement (one-way doors — see "Wire formats are stable" and
"Backend/storage trait shapes are one-way doors" below; once consumers
exist, those signatures are wire-stable contracts)
- You'd be force-pushing, amending a published commit, creating an empty
commit, or skipping hooks
Never commit secrets, keys, or credentials. If a commit fails or hooks
reject it, fix the issue and create a new commit — do not amend the
failed one.
Git identity is preconfigured (`glm-5.3-flash <glm-5.3-flash@alk.dev>`). Do
not change `git config`, skip hooks, or use `git commit -i`.
## Project Conventions (Rust / virtual filesystem crate)
This is the alkfs crate — a content-addressed, branch-aware virtual
filesystem for the alk family. The intended shape (pending Phase 0
convergence): a local storage engine (path-tree metadata + content-addressed
blob store) with a producer/consumer serving protocol on alkcall channels,
so that alkgit's object storage, the coming alksftp, and the alknet
filesystem vision all compose on the same substrate. It sits in the alk*
family: alkcall (call + channels — the substrate), alktty, alktunnels,
alksocks (the protocol-crate siblings), alkgit (the first in-family
storage consumer). The conventions below apply to all work in `src/` and
`tests/` once they exist. They mirror
`.opencode/agents/implementation-specialist.md` §Project Conventions and
are repeated here so they apply to every session, not just spawned
implementation agents.
1. **No comments in code** unless the user explicitly asks. This is a
project-wide convention. Doc comments (`///`, `//!`) are fine and
expected on public API. Inline `//` comments only when the user asks
or when a non-obvious safety/correctness constraint would otherwise
be missed (e.g., durability ordering constraints — "the blob bytes
must be durable before the path-tree row that names them commits, or
a crash orphans the reference").
2. **Error handling** — `thiserror` for library error types. No panics
in library code. No `unwrap()` or `expect()` outside tests. If you
reach for `unwrap`, the error path wasn't specified — stop and decide
what should actually happen. For poisoned `RwLock`/`Mutex`, use
`unwrap_or_else(|e| e.into_inner())` so a panic in one operation does
not cascade to other operations. Filesystem and storage errors get
faithful, typed mapping — never collapse a durability/corruption
error into a generic one (a silent-corruption bug in a filesystem
crate is the worst-case failure mode).
3. **`tokio` is the async runtime** — all I/O is async. Use the
wasm-clean tokio subset (`rt`, `sync`, `io-util`, `macros`, `time`) —
**do NOT use `features = ["full"]`**. CPU-heavy storage work (hashing,
pack/pao assembly, fsync-heavy batches) goes through
`tokio::task::spawn_blocking`, the alkgit POC-2 pattern. Use
`tokio::sync` primitives (`oneshot`, `mpsc`) for lifecycle
correlation; `parking_lot` for short-held internal locks.
4. **WASM target is load-bearing by default, with an honest escape
hatch** — the family default (alktty/alktunnels/alksocks) is a
wasm-clean protocol layer. A filesystem crate has heavier native
gravity (real FS I/O, SQLite); the expected shape is a wasm-clean
protocol/ops layer with the storage engine native-only behind
features — mirroring how alksocks keeps socket I/O behind `local`.
Whether even the protocol layer stays wasm-clean is a Phase 0
question, not a settled convention; decide deliberately and record it.
Run `cargo check --target wasm32-unknown-unknown` once the split
exists.
5. **Wire formats are stable** — no wire format exists yet; when the
first one is specced (the serving ALPN, its channel open-op params,
any chunk/framing on the data path), it is a one-way door: decide via
ADR before the first consumer exists, then additive-only. The ALPN
string (provisional `alk/fs`; final naming per alkcall ADR-004/006)
and the params-object shape follow the alkcall ADR-039 precedent
(`params` is ALPN-specific, interpreted by the open handler).
6. **Producer/consumer, not server/client** — both sides of a channels
connection can initiate. A producer exposes filesystem resources
(registers openable channels via `ChannelCore::register_openable`);
a consumer opens them and speaks the file protocol inside. Both sides
can be both simultaneously — connection direction is independent of
service direction. Avoid "server" and "client" framing in docs and
API names; use "producer" and "consumer," or "accept side" /
"connect side" for the connection-establishment half specifically.
See alkcall ADR-022, ADR-037.
7. **Substrate-agnostic by construction** — the protocol layer must not
know whether the far side is a channels `BiStream`, an in-process
handle, a local loopback, or a door adapter (alksftp, FUSE, a sync
client). The file-protocol state machine is generic over
`T: AsyncRead + AsyncWrite + Unpin` (the fast-socks5/alksocks
genericity precedent); substrate-specific types are confined to
feature-gated modules injected at the assembly layer (alktty
`TtyBackend` inversion-point pattern).
8. **No forced host-FS access** — the defining requirement of this
crate, the analogue of alksocks' "never binds a port." The virtual
filesystem manages its own content store and never reads, writes, or
mounts the host filesystem unless explicitly configured to. Bridges
to the real world are explicit, optional, feature-gated assembly
shapes: a local blob-file store, an SFTP door (alksftp), a mount
adapter, a directory-sync tool. The engine itself is storage-agnostic
about what backs its blob layer (filesystem, SQLite, remote).
9. **Metadata/content split is the load-bearing architecture** — small
structured state (path edges, branches, snapshots, refs, cached
sizes) lives in a transactional store; content bytes live in a
content-addressed blob layer (dedup by hash, branch sharing for
free). This is the alknet-filesystem POC's three-layer conclusion and
the same "shape" as git + gitlfs and iroh-blobs' inline/outboard
split. Path-tree operations are O(path edges), never O(bytes) —
rename is O(1) on edges regardless of file size. Do not let
byte-scale concerns leak into the metadata layer or vice versa. See
`docs/research/phase-0.md`.
10. **Content addressing and durability ordering** — content is
identified by its hash, not its path; identical content is shared
across branches/paths by construction. The durability contract is
the crash-safety story: content must be durable before any metadata
naming it commits; a crash mid-write leaves the old version intact
and visible (the POC's branch-on-write/merge-on-close property).
Exact hashing/chunking choices are Phase 0 OQs; the *invariant*
(no torn versions visible, no orphaned-name commits) is not.
11. **Vendored core types come from alkcall** — `Connection`,
`ProtocolHandler`, `BiStream`, `BidiStreamSource`, `AuthContext`,
`Identity`, `IdentityProvider`, `AccessControl`,
`OwnershipProvider`, `HandlerError`, `StreamError` come from
`alkcall::core`. Do not vendor copies into this crate. alkcall is
ours and co-developed — breaking changes are expected at this
major-zero stage; find and fix issues upstream rather than working
around them. Pin deliberately and bump deliberately. The
establishment surface (alkcall ADR-049 + amendments) and the
identity seam (CF-005/CF-006) are load-bearing for the serving
protocol, same as the alktty/alktunnels/alksocks pattern.
12. **Access control** — a filesystem is arbitrary read/write by nature:
the open gate and path-scope policy are the security boundary, the
same posture as alksocks' arbitrary egress. Scope-gate file opens
(an `alkfs`-shaped scope following alktty's `TTY_OPEN_SCOPE` /
alksocks' `SOCKS5_OPEN_SCOPE` precedent), wire producer openable
channels through `AccessControl` for free via
`ChannelCore::register_openable`, and treat per-path/per-bucket
policy as a Phase 1 design question (OQ). Multi-tenancy isolation
(the POC's `bucket` concept) is a where-clause, not an afterthought.
13. **Backend/storage trait shapes are one-way doors** — the seam the
protocol crate exposes to storage implementers (the alkgit
`GitRefs`/`GitPackGen`/`GitPackIngest` trait-family precedent,
`docs/architecture/backend.md` there) is a contract once consumers
exist: keep traits small, orthogonal, and substrate-blind; never
leak gix/iroh/SQLite types across them. The alknet-filesystem probe
(`alknet-blobs-external-store-probe.md`) is the cautionary example —
the sealed/`pub(crate)` surface is exactly what turns "external
store" into "fork".
14. **Feature flags** — substrate backends and heavy dependencies are
feature-gated if the need arises. The base crate should compile lean
(no SQLite, no gix, no kernel-FS access unless the feature is on).
Verify both `cargo test` (default) and `cargo test --all-features`
pass if features are added.
15. **Naming** — Rust standard: `snake_case` for functions/variables/
modules, `PascalCase` for types/traits, `SCREAMING_SNAKE_CASE` for
constants.
16. **Module structure** — one module per file under `src/`, re-exported
from `src/lib.rs`. Public API surface is `lib.rs` re-exports. The
expected shape (pending Phase 0/1 pinning): storage engine modules
(path tree, blob store, write sessions, GC), backend traits +
feature-gated implementations, and the producer/consumer protocol
modules mirroring the alktty/alktunnels/alksocks structure. Backend
modules are feature-gated and never imported from the
protocol/adapter/client modules.
17. **Upstream posture** — alkcall, alktunnels, and the alk* crates are
ours to shape: file asks early and land them there rather than
working around them locally (the alktunnels E-01/E-02 precedent —
filed from Phase 0, landed within a day). Third-party crates
(iroh-blobs, sqlite/rusqlite, russh-sftp, gix, automerge, honker)
are NOT ours and never will be: wrap, extract, or fork deliberately
per the alksocks precedent (OQ-SK-04 → ADR-013: extraction as owned
code with provenance notices, differential tests against the
reference checkout, no silent absorption) — and only when the carried
changes pay for themselves. `/workspace/iroh-blobs`,
`/workspace/russh-sftp`, etc. are read-only reference checkouts.
## Verification Commands
Run these before committing (once `src/` exists). All must pass.
```bash
cargo test # full suite
cargo clippy --all-targets -- -D warnings
cargo fmt --check
cargo doc --no-deps # if docs changed
cargo test --all-features # if features are added
cargo check --target wasm32-unknown-unknown # if the wasm posture applies (see convention 4)
cargo publish --dry-run --allow-dirty # before a release
```
While the repo is docs-only (Phase 0), there is no build to verify —
verification means re-reading changed docs and checking that every
referenced path, ADR, and OQ id exists.
## Architecture Context
- `docs/research/phase-0.md` — the Phase 0 (Exploration) document and
the current state of this repo: vision, prior art, open questions
(OQ-FS-NN), and the POC register. Read it before non-trivial work.
The SDD process lives in `docs/sdd_process.md` (Phase 0 in progress;
`docs/architecture/` does not exist yet — do not create it; that is
Phase 1).
- The ancestor research:
- **alknet-filesystem POCs** —
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
(`poc-summary.md` — the three-iteration POC: SQLite path tree +
iroh-blobs + honker, branch-on-write/merge-on-close, automerge sync;
`alknet-blobs-external-store-probe.md` — the iroh-blobs
Command-actor probe). This crate is the decomposition-era
continuation of that research on the published alk* substrate.
- **alkgit** — `/workspace/@alkdev/alkgit`: the first in-family
storage consumer (currently paused on this problem). Its backend
trait family (`docs/architecture/backend.md` — `GitRefs`,
`GitPackGen`, `GitPackIngest` over gix-odb) is the shape alkfs must
serve efficiently: streaming pack generation/ingestion, CAS ref
transactions, budgeted resources. Its `docs/research/` holds the
gitoxide and git-protocol surveys.
- The substrate and sibling crates:
- **alkcall** — `/workspace/@alkdev/alkcall` (v0.8.0, crates.io). The
substrate: call protocol + channels multiplexing. This crate will
consume `alkcall::core` types and the channels
`ChannelCore`/`ChannelClient`/`register_openable_with_establisher`
surface, same as alktty/alktunnels/alksocks.
- **alktty** — `/workspace/@alkdev/alktty`: the first producer/consumer
protocol crate; the backend inversion point, wasm-clean default,
scope-gating, and feature-gated backend precedents.
- **alktunnels** — `/workspace/@alkdev/alktunnels`: `-L`/`-R`/`-D`
tunnels on channels; the Phase 0 findings format and POC placement
conventions (`docs/research/phase-0-findings.md`) this repo
inherits.
- **alksocks** — `/workspace/@alkdev/alksocks`: the SOCKS5 sibling;
the closest Phase 0 template (`docs/research/phase-0.md` shape,
OQ numbering, upstream-posture convention).
- **alkvault** — `/workspace/@alkdev/alkvault`: secure secret
handling; the family rule is metadata stores hold vault references,
never plaintext secrets (alkgit vision principle 3).
- Key upstream ADRs that inform this crate's design (alkcall numbers
unless noted):
- ADR-035 — channels pure channel multiplexing (8-byte header); the
file-session rides inside a channel's `BiStream`
- ADR-050 — two-pump shutdown-on-completion (`channels::pump_bidi`);
use it, do not hand-roll
- ADR-049 (amendments) — the establishment phase;
`register_openable_with_establisher`; refused sessions are typed
`channel:open_failed` call errors, never phantom channels
- ADR-039 — `params` is ALPN-specific; ADR-037 — channel lifecycle
ops on channel 0; ADR-034 — the channels wire format (one-way
door); ADR-036 — channel 0 is pre-negotiated `alk/call`
- ADR-042 — hub relay (terminate-and-re-produce); ADR-051 —
`ChannelRelay`/`HubLegTemplate`
- Ledger CF-005/CF-006 — the per-call opener identity seam on the
open-op hooks
- If a TODO references a design direction that a later ADR has decided
against, the TODO is stale — remove it and align with the ADR. Do not
implement the rejected design.
docs/research/phase-0.md
+754
View File
@@ -0,0 +1,754 @@
---
status: draft
last_updated: 2026-09-23 (initial draft from the setup discussion —
unconverged by design: every OQ is open, the POC register holds proposals
not results, and the prior-art pass is partial)
---
# alkfs — Phase 0 (Exploration)
This document captures Phase 0 (Exploration) for the `alkfs` crate: vision,
guiding principles, prior art, open questions (OQ-FS-01..NN), and the
candidate POC register. Phase 0's objective per `docs/sdd_process.md`:
*capture vision and guiding principles; research options; validate
approaches; converge on a recommended approach.* It is the input to Phase 1
(Architecture), where the Architect will produce `docs/architecture/`
specs, ADRs, and the open-questions tracker.
Drafted 2026-09-23 from the initial setup discussion. The crate is the
storage-engine member of the alk* family: alkcall (call + channels — the
substrate), alktty/alktunnels/alksocks (the protocol-crate siblings),
alkgit (the first in-family storage consumer, currently paused on exactly
the problem this crate exists to solve).
**Honest framing:** this Phase 0 is earlier in its lifecycle than
alksocks' was at the equivalent commit. There are more unknowns here than
in any sibling so far — a filesystem is both a storage engine (hashing,
chunking, durability, GC) and a serving protocol (verbs, sessions, wire
framing), and almost none of those decisions have empirical ground yet.
The POC register below is proposals, not results. Expect this document to
be rewritten several times before convergence.
## Vision and guiding principles
**One sentence:** a content-addressed, branch-aware virtual filesystem for
the alk family — a local storage engine (path-tree metadata +
content-addressed blobs) with a producer/consumer serving protocol on
alkcall channels, so that alkgit's object storage, the coming alksftp
door, and the alknet filesystem vision all compose on the same substrate —
and it never touches the host filesystem unless explicitly configured to.
**Why this crate exists.** The alknet mono-repo's filesystem research
(three POC iterations, 24 passing tests) proved a viable three-layer
architecture but lived inside a project with excessive scope. The alk*
family has since been decomposed and published (alkcall, alktty,
alktunnels, alkvault; alksocks and alkgit in flight). alkgit paused
because its storage problem — content-addressed blob storage with
branch/CAS semantics — is not a git-specific problem, and solving it
inside alkgit would leave alksftp and the alknet filesystem rewrite to
solve it again. The same basic "shape" recurs across the family:
git + git-lfs is small-object CAS metadata over big-blob content storage;
iroh-blobs' FsStore is inline-small/outboard-large; the alknet-filesystem
POC is SQLite path edges over a blob store. alkfs is that shape, made a
crate.
**Scope sketch (unconverged — this is the shape Phase 0 must confirm or
cut):**
1. **Storage engine half** — the path tree (paths, branches, snapshots,
tombstones, cached sizes) in a transactional metadata store; content
bytes in a content-addressed blob layer; write sessions
(branch-on-write/merge-on-close); GC. Fully local, no channels
required — a consumer embeds the engine in-process.
2. **Serving protocol half** — the producer/consumer pair on alkcall
channels: a producer registers openable filesystem channels
(`ChannelCore::register_openable`); a consumer opens them and speaks
the file protocol inside. The `alk/fs`-shaped ALPN resource, the same
"ALPN as a service" shape as alktty/alktunnels/alksocks.
3. **Doors stay out** — alksftp (SFTP door), FUSE/mount adapters, sync
tools are separate crates/features that consume alkfs, per the family
rule (alkgit vision: doors are family infrastructure, not crate
contents).
Guiding principles, inherited from the alk* family plus the filesystem
research:
1. **No forced host-FS access — the defining requirement.** The analogue
of alksocks' "never binds a port": the engine manages its own content
store and never reads, writes, or mounts the host filesystem unless
explicitly configured to. Every bridge to the real world (a blob-file
store, an SFTP door, a mount, a sync tool) is an explicit, optional,
feature-gated assembly shape. The engine itself is storage-agnostic
about what backs its blob layer (filesystem files, SQLite, remote).
2. **Metadata/content split is load-bearing.** Small structured state
(path edges, branch pointers, tombstones, sizes) lives in a
transactional store; content bytes live under their hash in the blob
layer. Path operations are O(path edges), never O(bytes) — rename is
O(1) regardless of file size; identical content dedups across
branches and paths by construction. This is the alknet-filesystem
POC's core conclusion and the same shape as git + git-lfs and
iroh-blobs' inline/outboard split.
3. **Content addressing with a durability contract.** Content is
identified by its hash, not its path. The invariant that must survive
every design choice: content is durable before any metadata row
naming it commits; a crash mid-write leaves the old version intact
and visible; no torn version is ever readable; no metadata commit
orphans its content. Exact hashing/chunking is an OQ; the invariant
is not.
4. **Branch-aware from day one.** Fossil-style branches (a branch is a
parent chain + per-branch overrides + tombstones), branch-on-write/
merge-on-close for the write path. This is what gives free
multi-agent/multi-fork content sharing and the POSIX
concurrent-reader-sees-old-version property — and it is the property
git's object model needs (content shared across refs by hash).
5. **"ALPN as a service," not "a server."** The filesystem is a produced,
ACL-scoped resource on a channels connection. The protocol never
opens host files or binds anything; door-shaped conveniences (local
loopback front door, in-process consumer handle) are optional
assembly-layer shapes on either side. Producer/consumer vocabulary,
never server/client.
6. **Substrate-agnostic protocol, native engine.** The file-protocol
state machine is generic over `T: AsyncRead + AsyncWrite + Unpin`
(fast-socks5/alksocks genericity precedent); the storage engine is
native-only behind features (real FS I/O, SQLite). Whether the
protocol layer itself stays wasm-clean is an OQ, not an assumption.
7. **Multi-tenancy is a where-clause, not an afterthought.** The POC's
`bucket` column is the isolation unit; identity → bucket mapping is
an assembly-layer auth question; per-path/per-bucket policy is a
Phase 1 design question. A filesystem is arbitrary read/write by
nature — the open gate and path-scope policy are the security
boundary.
8. **Bounded resources on every serving path.** alkgit's ADR-009 posture
(wall-clock, size, round limits) is the consumer shape this crate
must serve; the serving protocol inherits alkcall's backpressure and
channel-cap invariants rather than re-deriving them.
## What is already settled
The foundation is POC-validated and (upstream) ADR-pinned; this crate
does not start from zero. It inherits:
- **The channels substrate** — demux→Connection→handler→mux, the
establishment story (`register_openable_with_establisher`, alkcall
ADR-049; refused sessions are typed `channel:open_failed` call errors,
never phantom channels), the identity seam (CF-005/CF-006: per-call
opener identity on the open-op hooks), ACL-for-free via
`register_openable`, the two-pump data plane (`channels::pump_bidi`,
ADR-050 — use it, do not hand-roll), hub relay
(`ChannelRelay`/`HubLegTemplate`, ADR-051). All production-validated by
alktty/alktunnels/alksocks.
- **The metadata/content split, POC-proven.** alknet-filesystem POC
iteration 1: SQLite path tree over a blob store, 8 tests — branch
inheritance, tombstones, O(1) rename, bucket isolation, honker
notify-atomic-with-mutation.
- **The write path, POC-proven.** Iteration 2: branch-on-write/
merge-on-close reconciles "BLAKE3 must hash the complete file" with
"writes arrive in chunks, possibly out of order" — concurrent readers
see the old version until close() commits atomically; crash/abort
leaves the old version intact. 7 tests.
- **The distributed-sync shape, POC-proven but scope-uncommitted.**
Iteration 3: path tree as an automerge CRDT synced over QUIC; 9 tests;
metadata and content sync separately; concurrent-root-map
initialization is a real constraint (root structures must be created
by one node and synced before others write). Whether any of this is
alkfs v1 scope is OQ-FS-14.
- **The iroh-blobs store-seam findings (the external-store probe).** The
alknet-blobs probe traced iroh-blobs 0.103's full `Command` actor
surface: there is **no `Store` trait** — `Store` is a concrete struct
over an `irpc::Client`; `MemStore`/`FsStore` are actors; a third store
is a third actor speaking the pub `Command` enum. Blockers are a
~4-line visibility PR (`Store::from_sender`, `ref_from_sender`,
`ApiClient`, the `Scope` field); GC (`run_gc`) is pub and reusable;
~120-160 lines of bao-tree glue must be re-implemented per store. The
probe's cautionary half: `TempTags`/`TempTagScope` are sealed
`pub(crate)` — sealed surface is exactly what turns "external store"
into "fork" (AGENTS.md convention 13's cautionary example).
- **The consumer shapes this crate must serve.** alkgit's backend trait
family (`docs/architecture/backend.md` there: `GitRefs`, `GitPackGen`,
`GitPackIngest` — streaming pack generation/ingestion, CAS ref
transactions, budgeted resources) and the russh-sftp `Handler` trait
(the POC's near-1:1 SFTP↔PathTree mapping table). "Optimize for git
and sftp-like" is the working posture.
- **The SQLite-as-application-file-format insight.** BLOBs < ~100KB are
faster inline in SQLite than as filesystem files; atomic transactions
over path-tree metadata; the schema is the documentation. iroh-blobs'
FsStore discovered the same shape independently (redb inline tables +
filesystem files). The inline-small/outboard-large hybrid is the
metadata/content split at storage granularity.
- **The family conventions** — AGENTS.md in this repo (error handling,
tokio subset, feature-gating, upstream posture, one-way doors) and the
alkcall ADR set listed there.
## Prior art
### alknet-filesystem POCs — the direct ancestor
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/`
(`poc-summary.md`, `alknet-blobs-external-store-probe.md`), POC crates at
`/workspace/alknet-filesystem-poc` and `/workspace/alknet-fs-sync-poc`.
The three-layer conclusion (SQLite path tree + iroh-blobs content +
honker coordination) and the write-path resolution
(branch-on-write/merge-on-close) are this crate's inheritance, not prior
art to re-litigate — but *every dependency choice* in those POCs is open
again:
- **SQLite path tree** (`rusqlite` 0.39, bundled): `buckets`/`branches`/
`paths`/`tombstones` + `write_sessions`/`write_chunks`; recursive-CTE
chain walk resolves `(bucket, branch, path)` → content link in
sub-millisecond time; only deltas are stored per branch.
- **iroh-blobs `MemStore`** as the content layer (deliberately not
`FsStore` — no redb, no fsync rabbit hole in the POC).
- **honker-core** for notify-in-transaction (the transactional-outbox
pattern built in): `SELECT notify(...)` inside the same transaction as
a path-tree mutation; watchers wake on commit, not poll.
- **What did NOT transfer and why** — the POC's remaining unknowns
(FsStore/redb coexistence, SFTP wiring, GC/tags, chain-depth perf,
snapshot semantics) are this Phase 0's OQ list; the probe resolved the
two-database question by reframing it (external actor, own SQLite
file), and the redb question dissolves entirely if alkfs owns its blob
metadata in its own store.
### iroh-blobs — the nearest CAS store (evaluated; not adopted as-is)
`/workspace/iroh-blobs` (v0.100 checkout) + published 0.103. `DESIGN.md`
is the best available treatment of the blob-store trade space (files are
hard: the bitfield/data/outboard write-ordering problem). What it gives:
BLAKE3 content addressing, bao-tree verified streaming (per-chunk
integrity with an outboard), dedup by construction, partial-range reads,
tags + mark-sweep GC, a hybrid store (small inline in redb, large as
filesystem files) — the same metadata/content split as principle 2.
**The known frictions (why the user called it "a pita" for git):**
- **The transfer protocol is welded to iroh/QUIC.** iroh-blobs' network
story is an iroh `ProtocolHandler` speaking its own ALPN over QUIC. For
alkfs this is irrelevant-by-design — content transfer rides alkcall
channels — but it means iroh-blobs is at most a *local store*
dependency, never the transfer layer. (The store side itself is
transport-independent — that is what the Command-actor probe proved.)
- **Whole-file naming vs incremental writes.** BLAKE3/bao names a blob
by the root hash of its *complete* byte sequence; you cannot compute
the address until all bytes exist. Partial data can live under temp
tags (`BlobStatus::Partial`) but cannot be pinned by its final hash.
The POC's branch-on-write/merge-on-close is one reconciliation; chunk
lists / Merkle-DAG naming is the other (OQ-FS-04). git escapes the
problem structurally: objects are small and independently hashed;
packs are the exception, and pack *generation* streams to a consumer
that hashes on arrival.
- **Two-database coexistence** (redb for blob metadata + SQLite for the
path tree): two WALs, two fsync paths, two crash-recovery stories —
the POC summary's unknown #1, reframed by the probe ("neither fork nor
coexist: a third store actor with its own single SQLite file"), and
dissolvable entirely if alkfs owns blob metadata in its own store
(OQ-FS-02/03).
- **Sealed internals** (`TempTags`, the `BaoTreeSender`) — the probe's
fork-vs-PR calculus. Third-party upstream, not ours, never will be
(AGENTS.md convention 17).
### git + git-lfs — the shape to optimize for
The user-level framing this crate must honor: "the same basic shape
exists with git + gitlfs as with how iroh's blobs handled the mix of
small and large blobs." git itself is the small-object CAS: every object
individually hashed, refs are names over hashes, content shared across
refs/branches for free, rename is metadata-only. git-lfs is the large-blob
sidecar: content-addressed pointer rows in the tree, bytes in a separate
content store (rudolfs is the family's reference server for that half —
namespace/bucket isolation, LRU cache decorator, fanout streaming). The
alkfs analogue: the metadata layer *is* git's object/ref shape; the blob
layer *is* the LFS store; small content can inline into the metadata
store exactly as iroh-blobs inlines small blobs into redb. Whether
alkfs's engine API makes alkgit's gix-odb backend an implementation
detail of alkfs (or leaves gix-odb native and serves only the
general-FS shape) is OQ-FS-12 — the highest-stakes scope question in
this document.
### alkgit — the first consumer
`/workspace/@alkdev/alkgit` — paused on the storage problem. Its backend
doc (`docs/architecture/backend.md`) pins the seam: five traits
(`GitRegistry`, `GitRegistryStore`, `GitRefs`, `GitPackGen`,
`GitPackIngest`), kept small/orthogonal/substrate-blind after the
alknet-blobs sealed-surface lesson. The properties alkfs must serve
efficiently: streaming pack generation (O(counts) memory, blocking-thread
friendly), streaming pack ingestion (budgeted, fsck/connectivity report),
CAS ref transactions, registry records with grants. Its research
(`docs/research/gitoxide.md`, `poc2-findings.md`) carries the gix-odb
API contract notes — including the durability shape (pack bytes durable
before the ref transaction that names them commits) that mirrors
principle 3 exactly.
### russh-sftp / alksftp — the door shape
`/workspace/russh-sftp` (read-only reference): the `Handler` trait is a
near-1:1 mirror of POSIX syscalls as SFTP packets, and the POC mapped
every op onto the PathTree + WriteSession API (open→resolve or
WriteSession::open; write→write_chunk with offset as key; close→hash+
merge+notify; readdir/list_dir; stat→indexed row — cheaper than a real
FS fstat; rename→O(1) edge move). The client's `File` already implements
`AsyncRead + AsyncSeek + AsyncWrite` with pipelined writes. alksftp (the
coming family crate) would speak this verb set; whether alkfs's serving
protocol adopts SFTP-shaped verbs natively or defines its own framing
that alksftp translates is OQ-FS-08. The alknet-tty async-IO adapter
patterns (AsyncWrite-over-mpsc, kill-on-drop) are reusable for the write
bridge.
### SQLite (application file format) + honker
https://sqlite.org/appfileformat.html — the insight that anchored the
POC. honker (`/workspace/honker`, `honker-core` 0.2.4) supplies
notify/locks/queues as SQL functions on *your* connection — the
transactional-outbox property. Caveats: honker is third-party (not ours,
convention 17), single-machine by explicit design ("two servers writing
the same .db over NFS is not a Honker deployment strategy"), and its
scope in alkfs is an OQ (notify-on-commit may be the only load-bearing
piece — locks/queues/scheduler may be excess).
### rudolfs — the git-lfs server reference
`/workspace/@alkdev/alknet/docs/research/references/gitlfs/
rudolfs-reference.md`: `StorageKey = (Namespace, Oid)` tenant isolation;
the decorator composition `Verify ↔ Encrypted ↔ Cached ↔ Retrying(Disk →
S3)`; LRU cache in front of permanent storage; `fanout()` streaming to
client and cache simultaneously. The cache/permanent split and the
fanout pattern are the production-shape references for alkfs's blob
layer and its serving read path.
### Anti-prior-art (what NOT to carry over)
- **The alknet mono-repo's scope.** The filesystem research is inherited
as *findings*, not as a code path; the crate stays small per the
decomposition discipline (alksocks' "socks + channels, nothing else"
resolution is the template — alkfs's version is OQ-FS-01).
- **iroh-blobs' transfer protocol as the content path.** Content
transfer rides alkcall channels, full stop; an iroh/QUIC side-protocol
would fork the family's transport story.
- **iroh-blobs' FsStore partial-file lifecycle as a requirement.** Its
749-line `bao_file.rs` partial-blob-on-filesystem machinery exists
because redb/filesystem is its substrate; a SQLite-inline store (POC
write_chunks) or an alkfs-owned blob store has no need for it.
- **gix-odb's layout as alkfs's layout.** Loose-objects + packfile
directory layout is git's on-disk contract; alkfs is a general
engine. If alkgit keeps gix-odb native, alkfs serves the general shape
and git storage stays in alkgit (OQ-FS-12).
- **NFS-style shared-file multi-node assumptions.** honker's own warning
applies to the whole engine: single-machine local state first;
distribution is an explicit layer (OQ-FS-14), not an ambient property.
## Open Questions
These are the design questions Phase 0 must resolve (or explicitly
defer) before the architecture spec. Numbered OQ-FS-01.. so they can be
referenced, tracked, and promoted into `docs/architecture/
open-questions.md` in Phase 1. Status is open unless stated; the point
of this document is to hold the reasoning without forcing premature
decisions. Grouped by theme; numbering is stable across reorganizations
(do not renumber — findings files and commits reference these ids).
### Core architecture
#### OQ-FS-01: Crate scope — what lives in alkfs?
The component split (§Vision) is a sketch, not a decision. Questions to
converge on:
- One crate with engine + serving protocol (feature-gated apart), or
engine and protocol as separate crates (the alkcall registry/protocol
split precedent)? The engine must be embeddable in-process without
channels (alkgit's use case); the protocol must be usable without the
default storage engine (an embedder with its own backend).
- Is there a backend trait seam at all (the alkgit `GitRefs`-family
pattern: traits in-crate, implementations behind features), or is the
storage engine simply *the* implementation with the protocol generic
over a small handle type? The sealed-surface lesson
(`alknet-blobs-external-store-probe.md`) says: decide this *before*
consumers exist; it is a one-way door.
- What is explicitly out: doors (alksftp, FUSE, mount), sync tooling,
UI/registry applications, the alknet rewrite's client stack.
#### OQ-FS-02: Metadata store substrate — and is blob metadata in the same file?
The POC used `rusqlite` (bundled) for the path tree and it worked well
(schema-as-documentation, recursive CTEs, BLOB columns at chunk
granularity). Open sub-questions:
- SQLite/rusqlite vs redb vs something else for the path tree. The POC
evidence favors SQLite (honker integration, transactional outbox,
sub-ms resolves); redb buys a lighter footprint but loses SQL and the
honker seam.
- The **one-file-vs-two question**: if alkfs owns the blob layer too,
blob metadata (hash index, sizes, refcounts/liveness) can live in the
*same* SQLite file as the path tree — one WAL, one commit boundary,
and the durability ordering of principle 3 becomes a single
transaction instead of a cross-store fsync dance. The probe's
external-actor shape (iroh-blobs with its own store) vs an alkfs-owned
store decides whether this is even on the table. Small content can
then inline as BLOBs (the appfileformat sweet spot) — possibly making
the "blob store" a layer *inside* one database with file-backed
spill-over for large content (the iroh-blobs FsStore hybrid shape).
- Concurrency model: single writer connection + reader pool? WAL
settings (`synchronous=NORMAL` vs `FULL`) as a durability-vs-throughput
knob? What is the honest fsync contract per operation class?
#### OQ-FS-03: Blob substrate — reuse iroh-blobs, own the store, or hybrid?
Three live options, shaped by the probe:
- **(a) Consume iroh-blobs as a store** (implement the `Command` actor
externally per the probe; 4-line upstream PR; ~120-160 lines of bao
glue) — gets BLAKE3+bao verified streaming, GC, tags for free; pays
the iroh-blobs/irpc dependency weight, the sealed-`TempTags`
fallback (self-managed liveness), and still needs the metadata-store
coexistence story (OQ-FS-02). Also carries `bao_tree` version-tracking
as the maintenance surface.
- **(b) Own the blob store.** BLAKE3 (`blake3` crate) + bao
(`bao_tree` crate) directly, alkfs-owned storage layout — full control
of durability ordering (principle 3 becomes enforceable in one
transaction with OQ-FS-02's single-file shape), no sealed surfaces, no
iroh-blobs dependency. Cost: re-derive verified-streaming import/
export (the glue the probe measured), GC, partial states — the
pieces iroh-blobs already solved, now ours to keep correct.
- **(c) Hybrid/no-bao start.** Whole-file BLAKE3 + a simple file-per-blob
or SQLite-inline store first, add verified streaming (bao outboards)
when a consumer needs range verification. git's own objects are small;
packs are written-once-then-verified-by-consumer. Honest question:
does any alkfs v1 consumer actually need per-chunk bao verification on
the read path, or is end-to-end TLS + hash-check-on-close sufficient?
The deciding inputs: the alkgit workload (pack-sized blobs, streaming),
the sftp workload (range reads, seek), and how much of iroh-blobs'
surface survives contact with "no iroh/QUIC transfer."
### Write path, branching, GC
#### OQ-FS-04: Hashing and chunking — whole-file addresses vs chunk DAGs
The load-bearing storage-format question (a one-way door once content
exists in the wild):
- **Whole-file BLAKE3 (+ bao outboard for verified streaming).** Simple,
git-shaped, matches the POC and alkgit's objects; but the address
doesn't exist until the write completes (staging required — the
branch-on-write answer), no partial availability under a final name,
no cross-file chunk dedup (two 1GB files sharing a 500MB prefix
dedup zero bytes).
- **Chunk-list / Merkle-DAG addressing** (content-defined or fixed-size
chunks; the file "name" is a root hash over chunk hashes, iroh-blobs
style): dedup at chunk granularity, partial availability, resumable
writes under a stable name, cheap append; but bigger metadata, a
chunker to pin down (CDC boundary-shift sensitivity), and whole-file
identity becomes "same root hash" rather than "same bytes hash" (root
hash equality still implies byte equality with bao-style trees, at
hash-of-chunk-hashes cost).
- **Hybrid** (the git-lfs/iroh shape): small content inline/whole-hash;
large content chunked. Threshold policy is a knob; the *format* split
is the one-way door.
Sub-questions: CDC (rollsum/buzhash) vs fixed-size vs BLAKE3's native
chunking (`bao_tree`'s chunk size); where chunk indices live (metadata
store rows vs outboard files); whether chunk dedup across *files* is a
v1 requirement or a defer-able nicety; interaction with OQ-FS-05 (a
chunk-DAG write can commit incrementally, weakening the branch-on-write
staging argument).
#### OQ-FS-05: Write path semantics beyond the POC
Branch-on-write/merge-on-close is POC-proven for the happy path. What
Phase 0 must still answer:
- **Durability details** — per-chunk transaction commit (POC shape) vs
batched WAL; `synchronous` level; whether fsync of spilled large
content must precede the path-row commit (principle 3) and how that
is ordered when content lives outside the DB file; crash-mid-close
behavior.
- **Concurrency** — the POC left honker named locks unwired; concurrent
writers to the same path need a real policy (last-close-wins is
session-safe but not POSIX; O_EXCL-ish create semantics; advisory
byte-range locks for the sftp door?).
- **Streaming large writes** — pack generation and 1GB+ uploads: the
POC's per-chunk SQLite rows are proven at ~32KB×1MB; the pack-size
regime needs a spill story (OQ-FS-02's file-backed overflow) and a
backpressure story inherited from alkcall channels.
- **Sparse/out-of-order/ftruncate** — SFTP pipelines out-of-order
(proven); sparse files and explicit truncation/extend are
unaddressed.
#### OQ-FS-06: Branch/snapshot/commit model
The POC has Fossil-style branches (name → parent chain + deltas) but no
explicit commit/snapshot op — writes to a branch are immediately visible
on that branch. Open questions:
- Is a snapshot a new branch (Fossil's model — maps naturally to the
chain walk) or a recorded point-in-time within a branch (git's model)?
What do alkgit (refs + CAS) and the sftp door each need?
- Refs as a first-class concept: alkfs stores `refs` rows naming commits
(branch heads), enabling CAS ref transactions (alkgit's
`GitRefs::apply` shape) over the same metadata store. Is the ref
namespace git-shaped from day one, or a later addition?
- Merge semantics across branches: the POC has none (branch-on-write
collapses trivially at close). Real merges (three-way? automerge-CRDT
for the metadata layer per POC 3?) are presumably out of v1 — say so
explicitly rather than leaving it implied.
- Chain-depth performance: the recursive CTE degrades on deep chains
(POC unknown #6); resolve-cache/materialized-view shape if needed —
perf probe before spec, or accept and document.
#### OQ-FS-07: GC and liveness
Content is shared across branches/paths by hash, so naive per-path tags
(Poc unknown #5's iroh-tags shape) break sharing — deleting a path must
not collect content another path/branch still names. The candidate
shape: **liveness = reachability walk** (mark from all live branch heads
+ refs + write sessions; sweep unreachable) — a metadata-store query
over path rows, no separate tag table needed if blob liveness rows are
refcounted or mark-computed at GC time. Sub-questions: incremental vs
stop-the-world GC; tombstone retention (content behind tombstones is
unreachable-but-recent — grace windows); orphaned write-session chunk
reaping (crash cleanup, the POC's known orphans); cross-bucket sharing
(never — buckets are tenants).
### Serving protocol
#### OQ-FS-08: The file protocol over channels — verbs, sessions, framing
The biggest unspecced surface, and the crate's first real wire format
(one-way door — ADR before the first consumer):
- **Verb set.** SFTP-shaped (the russh-sftp Handler mapping is proven
1:1 against the engine) vs a leaner custom set (open/read/write/
close/stat/readdir/rename/remove + extended ops). Working posture:
SFTP-shaped verbs, SFTP-versioned semantics where it matters
(handles, pipelined writes, `SeekFrom::End` cheapness).
- **Session model.** One channel per file session (alksocks' one-channel-
per-CONNECT shape) vs one control channel multiplexing many file
handles (the alktty demux shape). A directory traversal or a git clone
opens *many* files; per-file channels spend channel IDs and opens
(the 256-cap arithmetic, OQ-SK-09's twin); a multiplexed control
channel reintroduces the demux trade alksocks resolved *for SOCKS5*
(whose RFC structure genuinely has one command per connection — a
file protocol has no such constraint). This trade has different
physics here than it had there; no inherited answer.
- **Framing on the data path.** If sessions are multiplexed, requests
need IDs and payloads need framing (length-prefix + request-id — the
one-way door). If per-session, the payload may be near-pass-through.
Chunk sizes, backpressure (bounded buffers are inherited), and
flow-control for pipelined writes all live here.
- **Read-path streaming.** Range reads (offset+len), EOF semantics,
verified-streaming vs trust-the-store; zero-copy ambitions explicitly
out (channels are the substrate, not splice(2)).
- **Establishment.** `register_openable_with_establisher` with a scope
gate (OQ-FS-10); refused opens are typed errors with per-verb error
mapping (SFTP status codes at the door, faithful engine errors in
protocol — convention 2).
#### OQ-FS-09: ALPN, params, and the resource model
Provisional `alk/fs` (final naming per alkcall ADR-004/006 convention).
Open sub-questions (mostly Phase 1 ADRs with Phase 0 ground):
- The params-object shape (ADR-039 precedent: ALPN-specific, interpreted
by the open handler) — what identifies a produced filesystem resource?
(bucket? branch? scope string? limits?)
- In-process and loopback consumers: the engine half must be directly
embeddable (no channels) — the protocol layer is one substrate among
several, generic over `T` (principle 6). The in-process shape is also
the test seam, the alksocks POC pattern.
- Hub relay story: a filesystem resource traverses relays
terminate-and-re-produce like any produced resource; nothing
alkfs-specific expected — verify during POC rather than assume.
#### OQ-FS-10: Doors — alksftp, mounts, sync (out of crate, but shapes the API)
Family rule: doors are separate crates/features. Phase 0's job is to
keep the protocol layer door-ready: the verb set (OQ-FS-08) must map
onto russh-sftp's `Handler` without translation loss; a FUSE/mount door
needs POSIX-ish error mapping and inode-ish handle semantics — do not
design for FUSE in v1, but record what it would demand so the protocol
doesn't preclude it. Directory-sync tooling (alkfs ↔ host dir) is the
explicit, feature-gated host-FS bridge if/when wanted — never ambient.
#### OQ-FS-11: Multi-tenancy, identity, and path-scope policy
Buckets are the isolation unit (POC-proven free). Open questions: the
`ALKFS_OPEN_SCOPE`-shaped scope gate (alktty/alksocks precedent) — does
scope name buckets, path prefixes, or both; identity → bucket mapping
at the assembly layer (vault references, never plaintext secrets —
family rule); per-path/per-bucket policy as a producer-side policy
callback vs declarative ACL (alksocks OQ-SK-05's dialer-refuses
resolution is the evidence-backed template); write-vs-read grant
separation (a read-only bucket mount is the obvious first split).
### Consumer fit
#### OQ-FS-12: alkgit fit — the question that un-pauses alkgit
The first consumer and the reason this crate moved ahead of alkgit. The
decision tree:
- **(a) alkgit keeps gix-odb native; alkfs serves the general FS.**
Lowest integration risk; but then alkfs doesn't solve the problem that
paused alkgit, and the family carries two storage stories.
- **(b) alkgit's backend traits get an alkfs-backed implementation.**
`GitPackGen` streams a pack *out of* alkfs content (pack assembly is
alkgit-side; the blob bytes come from alkfs reads);
`GitPackIngest` streams a pack *into* alkfs (each unpacked object
lands as content; refs land as metadata rows); `GitRefs` becomes
CAS transactions over alkfs refs. gix-odb's on-disk layout disappears;
git semantics ride alkfs's content addressing. This is the "optimize
for git" path and the probable answer — but it makes alkgit's
workload (pack-scale blobs, O(counts) memory, streaming) the
benchmark OQ-FS-13 must meet.
- **(c) gix-odb *on top of* alkfs** (an odb backend where loose
objects/packs are alkfs files) — keeps gix-odb's pack machinery but
treats alkfs as the disk. Probably the worst of both: gix-odb assumes
filesystem semantics alkfs won't natively provide (mmap, file locks).
Sub-questions: does git's object-id namespace (sha1/sha256) coexist with
alkfs's content addresses (path rows mapping git-oids → alkfs hashes,
dedup at the alkfs layer); CAS ref transaction shape; whether pack
ingestion budget enforcement (alkgit ADR-009) maps onto alkfs write
sessions.
#### OQ-FS-13: Performance shape and budgets
No numbers exist yet. What Phase 0 should collect (empirically, via the
POC register) so Phase 1 can pin budgets: path-resolve cost at realistic
branch depths (POC: sub-ms in-memory; chains of 10+?); small-file
throughput (the inline-BLOB sweet spot claim vs reality at 4KB metadata
ops); large-file write throughput through chunked sessions; pack
gen/ingest streaming rates over a channel vs native gix-odb; memory
bounds under the alkcall channel caps. The alkgit ADR-009 limits shape
is the consumer contract to serve.
### Posture and deferred
#### OQ-FS-14: Distributed sync / multi-node — in scope at all?
POC 3 proved automerge-CRDT sync of the path tree (9 tests, LWW
conflicts, the concurrent-root-init constraint) with content fetched
separately. But sync is a product-layer concern that may not belong in
the engine crate: the family pattern is composition (alkcall relay for
transport, automerge as a third-party sync layer, honker queues as the
local outbox). Working posture: **deferred — out of v1 scope**, recorded
so the metadata schema doesn't preclude it (branch/paths rows are
already CRDT-encodable per POC 3's document structure). Revisit when a
consumer needs multi-node.
#### OQ-FS-15: WASM posture
The expected shape (convention 4): wasm-clean protocol/ops layer; native
storage engine behind features (real FS/SQLite never compiles to
wasm32-unknown-unknown). Open: does any consumer actually want the
protocol layer in wasm (a sandboxed alkfs consumer, mirroring alksocks'
sandboxed-SOCKS-consumer story)? If none is in sight, the honest answer
may be alkhttp-style native-only (the "wasm-clean is preferred, not
mandatory" precedent) — but the protocol layer being wasm-clean by
construction costs little if the engine split exists anyway. Decide
deliberately once the split (OQ-FS-01) exists; verify with
`cargo check --target wasm32-unknown-unknown`.
## Candidate POC register (proposals — none run yet)
POC placement conventions (inherited from alktunnels/alksocks): a POC
needing this repo's code runs in a worktree (`.worktrees/research/
<task-id>/` per `docs/sdd_process.md`); a self-contained POC runs as a
standalone crate in the global workspace with findings written into
`docs/research/` here. Findings always land in `docs/research/`.
| # | Candidate | Hypothesis to validate | Feeds |
|---|---|---|---|
| 1 | Engine skeleton: path tree + inline-blob store in one SQLite file | One-file metadata+blob-metadata (OQ-FS-02) holds up under the POC's 15-test suite ported over; single-transaction durability ordering (principle 3) is expressible | OQ-FS-02, OQ-FS-05 |
| 2 | Write path at pack scale | Chunked write sessions handle 100MB+ streams (spill-to-file regime) with correct durability ordering and crash-abort semantics | OQ-FS-05, OQ-FS-13 |
| 3 | Blob substrate bake-off (narrow) | (a) iroh-blobs external actor vs (b) own blake3/bao store vs (c) inline-hybrid: measure integration cost, GC story, sealed-surface fallout against the probe's predictions | OQ-FS-03 |
| 4 | Chunking probe (paper + micro-bench) | Whole-file vs chunk-DAG: dedup gains on realistic workloads (git packs, agent workspaces), write-path impact, metadata size — enough to decide the format one-way door or at least stage it | OQ-FS-04 |
| 5 | Serving protocol spike | SFTP-verb protocol over channels: per-file-channel vs multiplexed-handles session model, open cost at directory-walk and git-clone fan-out, framing sketch (not a wire ADR) | OQ-FS-08, OQ-FS-09 |
| 6 | alkgit seam spike | `GitPackGen`/`GitPackIngest` shaped against an alkfs-backed engine on the POC's API — does the trait family survive contact; where do git oids map | OQ-FS-12 |
| 7 | GC walk prototype | Reachability-mark GC over path rows + refs: correctness on shared content across branches, cost model, orphaned-session reaping | OQ-FS-07 |
Order is indicative, not committed: #1/#2 de-risk the engine core;
#3/#4 resolve the storage-format one-way doors; #5/#6 are only worth
running once #1 exists (the protocol needs an engine to serve). The
register exists so POC work has home ids; expect entries to be split,
merged, or dropped as OQs sharpen.
## Survey / prior-art list
Read/verified (cited above):
- alknet-filesystem POCs —
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/
poc-summary.md`, `alknet-blobs-external-store-probe.md`; POC crates
`/workspace/alknet-filesystem-poc`, `/workspace/alknet-fs-sync-poc`.
- iroh-blobs — `/workspace/iroh-blobs` (v0.100 checkout + `DESIGN.md`),
published 0.103 in cargo cache (probe source); the family's reference
notes at `/workspace/@alkdev/alknet/docs/research/references/iroh/
iroh-blobs/` (overview, key types, transfer protocol, storage).
- alkgit — `docs/architecture/backend.md` (the trait seam),
`docs/research/{vision,gitoxide,git-protocol,poc2-findings}.md`.
- russh-sftp — `/workspace/russh-sftp` (`server/handler.rs`, client
`File` pipelining).
- SQLite appfileformat — https://sqlite.org/appfileformat.html; honker
— `/workspace/honker` (`honker-core` 0.2.4).
- rudolfs (git-lfs server) —
`/workspace/@alkdev/alknet/docs/research/references/gitlfs/
rudolfs-reference.md`.
- alkcall — ADRs 034/035/036/037/039/042/049/050/051, ledger
CF-005/CF-006 (`docs/architecture/decisions/`); alktty/alktunnels/
alksocks Phase 0 docs as the format/template precedents.
To evaluate (research-specialist queue, feeding the OQs named):
- **Chunking/CDC prior art** — FastCDC, rollsum (bup), rsync-style
boundaries; `bao_tree`'s BLAKE3 chunking as the default candidate
(OQ-FS-04).
- **CAS filesystems** — Plan 9 fossil+venti (the branch/CAS ancestry the
POC name-checks), IPFS unixfs/iros DAG shapes, OST/lessfs-style
inline-vs-outboard hybrids (OQ-FS-04, OQ-FS-06).
- **git storage internals** — gix-odb pack/idx layout and
`gix-pack::data::input` streaming (alkgit research has the survey;
re-read for OQ-FS-12's object-id/namespace questions).
- **SQLite durability posture** — WAL + `synchronous=NORMAL` vs `FULL`
under the commit-boundary contract; SQLite-as-blob-store limits
(OQ-FS-02, OQ-FS-05).
- **FUSE/mount doors** — rust bindings shape, POSIX semantics alkfs
would need to fake (OQ-FS-10; defer design, record constraints).
## Convergence checklist (what Phase 0 must produce)
- [ ] Vision + guiding principles captured (this doc, §Vision — the
scope sketch needs confirmation or cutting, OQ-FS-01)
- [ ] Prior-art pass complete: alknet-filesystem POCs re-read and
mapped (done); iroh-blobs surface re-verified against the probe's
0.103 findings; git/git-lfs shape pinned against alkgit's backend
contract; CDC/chunking survey (OQ-FS-04 input); CAS-filesystem
survey
- [ ] OQ-FS-01 (crate scope) — converged: engine/protocol split, trait
seam or not, explicit non-goals
- [ ] OQ-FS-02 (metadata store) — converged: substrate, one-file-vs-two
(blob metadata placement), durability knob
- [ ] OQ-FS-03 (blob substrate) — converged: consume iroh-blobs vs own
the store vs hybrid (POC #3)
- [ ] OQ-FS-04 (hashing/chunking) — converged: the storage-format
one-way door (POC #4)
- [ ] OQ-FS-05 (write path) — durability/concurrency/spill semantics
pinned (POC #2)
- [ ] OQ-FS-06 (branch/snapshot/refs) — model chosen; ref namespace
decided; merge explicitly in-or-out of v1
- [ ] OQ-FS-07 (GC) — liveness shape chosen; retention/reaping policy
pinned (POC #7)
- [ ] OQ-FS-08 (file protocol) — verbs, session model, framing sketch;
the wire ADR itself is Phase 1's (one-way door, first consumer in
sight) (POC #5)
- [ ] OQ-FS-09 (ALPN/params) — naming + params ground collected; ADR
in Phase 1
- [ ] OQ-FS-10 (doors) — protocol kept door-ready; nothing designed in
- [ ] OQ-FS-11 (tenancy/policy) — scope-string + policy shape settled
- [ ] OQ-FS-12 (alkgit fit) — the (a)/(b)/(c) decision made; alkgit
un-pause plan written (POC #6)
- [ ] OQ-FS-13 (budgets) — numbers collected; budget table drafted
- [ ] OQ-FS-14 (sync) — explicitly deferred or scoped with evidence
- [ ] OQ-FS-15 (wasm) — decided once the engine/protocol split exists
- [ ] Targeted POCs run + findings in `docs/research/`
- [ ] Converge: recommended approach written up, ready to hand to the
Architect for Phase 1