Phase 0 second pass: storage identity + two-crate split + anchor workload

- Reframe storage identity: git-compatible sha1/sha256 (algorithm-tagged),
  not BLAKE3 — the ancestor POCs' BLAKE3 was an iroh-blobs artifact, and
  consuming iroh-blobs (OQ-FS-03 option (a)) dissolves with it; findings
  transfer (inline/outboard hybrid, GC shape, write-ordering analysis),
  code doesn't
- Chunk-level dedup adopted as working goal for large content, with the
  honest calibration recorded: git dedups at object granularity (objects
  are effectively single-chunk); chunk manifests are the generalization
  that pays on workspace-scale files
- Two-crate working direction: alkblobs (bucket-style CAS blob store,
  appfile layout) beneath alkfs (path tree + branches + serving protocol);
  durability contract now crosses the put/get seam
- New OQ-FS-16: remote-mount anchor workload — today's sshfs/vanilla-SFTP
  mount of this dev server's workspace, with an honest decomposition of
  why it's slow and what an indexed VFS fixes server-side
- POC register: #3 rewritten as alkblobs sha1/sha256 store skeleton,
  #4 reframed (git objects ride whole-file), new #8 mount-workload
  baseline (sshfs latency measurement, runnable today)
- AGENTS.md: two-crate direction in the crate description + convention 10
  (identity, chunk-granularity note, seam durability contract) + a
  Phase-0 working-direction quick-reference section (leanings, not
  decisions)

Verification: docs-only repo — all 16 OQ ids contiguous, every referenced
workspace path exists, BLAKE3 references re-checked as historical context
This commit is contained in:
glm-5.3-flash committed 2026-09-23 09:19:58 +00:00
1 parent 9e863747fc
commit 23a39cc9c0
2 files changed
+346 -154

No files matched your search

+297 -136
View File
@@ -1,8 +1,12 @@
---
status: draft
last_updated: 2026-09-23 (initial draft from the setup discussion —
unconverged by design: every OQ is open, the POC register holds proposals
not results, and the prior-art pass is partial)
last_updated: 2026-09-23 (second pass, same day — storage identity
reframed to git-compatible sha1/sha256: the ancestor POCs' BLAKE3 was an
iroh-blobs artifact, not a choice; chunk-level dedup adopted as the
working goal; alkblobs/alkfs two-crate split added as the working
direction; the remote-mount SFTP use case recorded as the anchor
workload. Still unconverged — every OQ is open, the POC register holds
proposals not results)
---
# alkfs — Phase 0 (Exploration)
@@ -15,11 +19,12 @@ approaches; converge on a recommended approach.* It is the input to Phase 1
(Architecture), where the Architect will produce `docs/architecture/`
specs, ADRs, and the open-questions tracker.
Drafted 2026-09-23 from the initial setup discussion. The crate is the
storage-engine member of the alk* family: alkcall (call + channels — the
substrate), alktty/alktunnels/alksocks (the protocol-crate siblings),
alkgit (the first in-family storage consumer, currently paused on exactly
the problem this crate exists to solve).
Drafted 2026-09-23 from the initial setup discussion, revised the same
day as the storage-identity reframe landed (§Working direction). The
crate is the storage-engine member of the alk* family: alkcall (call +
channels — the substrate), alktty/alktunnels/alksocks (the
protocol-crate siblings), alkgit (the first in-family storage consumer,
currently paused on exactly the problem this crate exists to solve).
**Honest framing:** this Phase 0 is earlier in its lifecycle than
alksocks' was at the equivalent commit. There are more unknowns here than
@@ -38,6 +43,33 @@ alkcall channels, so that alkgit's object storage, the coming alksftp
door, and the alknet filesystem vision all compose on the same substrate —
and it never touches the host filesystem unless explicitly configured to.
**Working direction (2026-09-23, from the setup discussion — leanings to
converge, not decisions):**
- **Git-compatible identity, not BLAKE3.** The ancestor POCs used
BLAKE3 because iroh-blobs does; flipping the causality, the blob
layer should carry the appfile concept with the hashes git uses
(sha1/sha256). This dissolves the "BLAKE3 must hash the whole file
before the address exists" problem into a *chosen* design (OQ-FS-04),
and makes git object identity native rather than mapped.
- **Chunk-level dedup is the working goal** for large content
(workspace-scale files, datasets, logs). Whole-file hashing dedups
only identical files; git itself dedups at object granularity (packs
delta-compress within a pack, but the address space is per-object);
chunking is the storage-layer generalization (OQ-FS-04 — measure
whether the workloads pay before committing).
- **The two-crate split: `alkblobs` + `alkfs`.** A bucket-style
content-addressed blob store (the S3 shape) beneath the path-tree
mapping (the alknet-filesystem POC's layer 2). alkgit can consume the
blob store without a filesystem; the FS layer keeps its own crate
(OQ-FS-01).
- **The anchor workload is remote mounting.** The concrete use case:
the dev-server workspace mounted on a local Linux machine under
`remote/` — today over vanilla SFTP (sshfs), which is slow in known
ways (a round trip per syscall, stat storms, full inode work per
stat). An appfile-indexed VFS fixes the server side; caching and
pipelining close the rest (OQ-FS-16).
**Why this crate exists.** The alknet mono-repo's filesystem research
(three POC iterations, 24 passing tests) proved a viable three-layer
architecture but lived inside a project with excessive scope. The alk*
@@ -52,19 +84,27 @@ iroh-blobs' FsStore is inline-small/outboard-large; the alknet-filesystem
POC is SQLite path edges over a blob store. alkfs is that shape, made a
crate.
**Scope sketch (unconverged — this is the shape Phase 0 must confirm or
cut):**
**Scope sketch (unconverged; restructured 2026-09-23 around the
two-crate working direction — Phase 0 must confirm or cut, OQ-FS-01):**
1. **Storage engine half** — the path tree (paths, branches, snapshots,
tombstones, cached sizes) in a transactional metadata store; content
bytes in a content-addressed blob layer; write sessions
(branch-on-write/merge-on-close); GC. Fully local, no channels
required — a consumer embeds the engine in-process.
2. **Serving protocol half** — the producer/consumer pair on alkcall
channels: a producer registers openable filesystem channels
1. **alkblobs (candidate companion crate)** — bucket-style
content-addressed blob storage (the S3/rudolfs shape): put/get/has/
delete by hash, tags/pins for liveness, GC, the appfile layout (one
transactional store for blob metadata + inline small content;
file-backed spill-over for large content), git-compatible sha1/
sha256 identity, chunk-level dedup for large content. Fully local,
embeddable in-process, no channels required; engine-only in v1 (no
serving ALPN of its own yet — OQ-FS-09).
2. **alkfs (this crate)** — the path-tree mapping on top (the POC's
layer 2): paths, branches, snapshots, tombstones, cached sizes,
write sessions (branch-on-write/merge-on-close); plus the
producer/consumer serving protocol on alkcall channels — a producer
registers openable filesystem channels
(`ChannelCore::register_openable`); a consumer opens them and speaks
the file protocol inside. The `alk/fs`-shaped ALPN resource, the same
"ALPN as a service" shape as alktty/alktunnels/alksocks.
the file protocol inside. The `alk/fs`-shaped ALPN resource, the
"ALPN as a service" shape of alktty/alktunnels/alksocks. Depends on
alkblobs, or a narrow blob seam if the split puts a trait there
(OQ-FS-01).
3. **Doors stay out** — alksftp (SFTP door), FUSE/mount adapters, sync
tools are separate crates/features that consume alkfs, per the family
rule (alkgit vision: doors are family infrastructure, not crate
@@ -85,15 +125,22 @@ research:
transactional store; content bytes live under their hash in the blob
layer. Path operations are O(path edges), never O(bytes) — rename is
O(1) regardless of file size; identical content dedups across
branches and paths by construction. This is the alknet-filesystem
POC's core conclusion and the same shape as git + git-lfs and
iroh-blobs' inline/outboard split.
branches and paths by construction (and, with chunking, dedups at
chunk granularity). This is the alknet-filesystem POC's core
conclusion and the same shape as git + git-lfs and iroh-blobs'
inline/outboard split — with git-compatible identity instead of
BLAKE3.
3. **Content addressing with a durability contract.** Content is
identified by its hash, not its path. The invariant that must survive
every design choice: content is durable before any metadata row
naming it commits; a crash mid-write leaves the old version intact
and visible; no torn version is ever readable; no metadata commit
orphans its content. Exact hashing/chunking is an OQ; the invariant
identified by its hash, not its path — with the hashes git uses
(sha1/sha256), so git object identity is native to the store, not
mapped. The invariant that must survive every design choice: content
is durable before any metadata row naming it commits; a crash
mid-write leaves the old version intact and visible; no torn version
is ever readable; no metadata commit orphans its content. With the
two-crate direction the contract crosses the put/get seam (alkfs
must be able to trust that an alkblobs `put` returning means
durable, or get an explicit durability level back) — the seam
contract is an OQ. Exact hashing/chunking is an OQ; the invariant
is not.
4. **Branch-aware from day one.** Fossil-style branches (a branch is a
parent chain + per-branch overrides + tombstones), branch-on-write/
@@ -208,17 +255,32 @@ again:
file), and the redb question dissolves entirely if alkfs owns its blob
metadata in its own store.
### iroh-blobs — the nearest CAS store (evaluated; not adopted as-is)
### iroh-blobs — the nearest CAS store (evaluated; NOT adopted, and the BLAKE3 causality)
`/workspace/iroh-blobs` (v0.100 checkout) + published 0.103. `DESIGN.md`
is the best available treatment of the blob-store trade space (files are
hard: the bitfield/data/outboard write-ordering problem). What it gives:
BLAKE3 content addressing, bao-tree verified streaming (per-chunk
integrity with an outboard), dedup by construction, partial-range reads,
tags + mark-sweep GC, a hybrid store (small inline in redb, large as
filesystem files) — the same metadata/content split as principle 2.
content addressing, bao-tree verified streaming (per-chunk integrity with
an outboard), dedup by construction, partial-range reads, tags +
mark-sweep GC, a hybrid store (small inline in redb, large as filesystem
files) — the same metadata/content split as principle 2.
**The known frictions (why the user called it "a pita" for git):**
**The causality correction (2026-09-23):** the ancestor POCs' BLAKE3
choice was an **iroh-blobs artifact** — BLAKE3 is load-bearing across the
whole iroh-blobs surface (`Hash` type, bao outboards, the `Command`
actor's payload types), so consuming iroh-blobs means inheriting BLAKE3
and its whole-file addressing. Flipping the causality — *we choose the
identity* — dissolves the "must have the full file for blake3" problem
into a design question (OQ-FS-04) and dissolves the "consume iroh-blobs
as a store" option (OQ-FS-03's option (a)) entirely: a sha1/sha256
identity layer over iroh-blobs' Command surface would be a mapping
shim on top of a foreign hash, not a store. What remains worth taking
from iroh-blobs is *findings*, not code: the inline/outboard hybrid
layout, the write-ordering analysis, the mark-sweep GC shape, and the
probe's actor-seam map (what a store API needs to expose to stay
consumable).
**The other frictions (record, for the research they hold):**
- **The transfer protocol is welded to iroh/QUIC.** iroh-blobs' network
story is an iroh `ProtocolHandler` speaking its own ALPN over QUIC. For
@@ -250,18 +312,30 @@ filesystem files) — the same metadata/content split as principle 2.
The user-level framing this crate must honor: "the same basic shape
exists with git + gitlfs as with how iroh's blobs handled the mix of
small and large blobs." git itself is the small-object CAS: every object
individually hashed, refs are names over hashes, content shared across
refs/branches for free, rename is metadata-only. git-lfs is the large-blob
sidecar: content-addressed pointer rows in the tree, bytes in a separate
content store (rudolfs is the family's reference server for that half —
namespace/bucket isolation, LRU cache decorator, fanout streaming). The
alkfs analogue: the metadata layer *is* git's object/ref shape; the blob
layer *is* the LFS store; small content can inline into the metadata
store exactly as iroh-blobs inlines small blobs into redb. Whether
alkfs's engine API makes alkgit's gix-odb backend an implementation
detail of alkfs (or leaves gix-odb native and serves only the
general-FS shape) is OQ-FS-12 — the highest-stakes scope question in
this document.
individually hashed (sha1 today, sha256 in the object-format transition),
refs are names over hashes, content shared across refs/branches for free,
rename is metadata-only. git-lfs is the large-blob sidecar: content-
addressed pointer rows in the tree, bytes in a separate content store
(rudolfs is the family's reference server for that half — namespace/
bucket isolation, LRU cache decorator, fanout streaming). The alkfs
analogue: the metadata layer *is* git's object/ref shape; the blob layer
*is* the LFS store; small content can inline into the metadata store
exactly as iroh-blobs inlines small blobs into redb. With sha1/sha256
identity (the 2026-09-23 reframe), **git objects are natively addressable
in the blob layer** — no oids→hash mapping layer, no dedup seam between
git's CAS and the store's CAS.
**One honest calibration: git dedups at object granularity, not chunk
granularity.** Packs delta-compress similar objects *within* a pack, but
the address space is per-object. Chunk-level dedup (the working goal,
§Working direction) is the storage-layer generalization that pays on
workspace-scale files (datasets, logs, agent workspaces with large
append-only artifacts) — not on git objects, which are effectively
single-chunk. If alkblobs chunks large content, git's pack/idx regime
still rides above it as object-granular rows; the two granularities
compose (OQ-FS-04, OQ-FS-12). Whether alkgit keeps gix-odb native or
serves its object storage from alkfs/alkblobs is OQ-FS-12 — the
highest-stakes scope question in this document.
### alkgit — the first consumer
@@ -347,22 +421,29 @@ decisions. Grouped by theme; numbering is stable across reorganizations
### Core architecture
#### OQ-FS-01: Crate scope — what lives in alkfs?
#### OQ-FS-01: Crate scope — one crate or the alkblobs/alkfs split?
The component split (§Vision) is a sketch, not a decision. Questions to
converge on:
**Working direction (2026-09-23): the two-crate split — `alkblobs` (the
bucket-style CAS blob store) + `alkfs` (the path-tree mapping + serving
protocol).** Rationale: a git-like blob store is a complete, useful
thing without a filesystem (alkgit could consume it directly; S3-bucket-
shaped usage exists on its own), the FS layer is a separate concern with
its own consumers, and the POC's layer separation (blob bridge as an
explicit module) already drew the seam. Open sub-questions:
- One crate with engine + serving protocol (feature-gated apart), or
engine and protocol as separate crates (the alkcall registry/protocol
split precedent)? The engine must be embeddable in-process without
channels (alkgit's use case); the protocol must be usable without the
default storage engine (an embedder with its own backend).
- Is there a backend trait seam at all (the alkgit `GitRefs`-family
pattern: traits in-crate, implementations behind features), or is the
storage engine simply *the* implementation with the protocol generic
over a small handle type? The sealed-surface lesson
(`alknet-blobs-external-store-probe.md`) says: decide this *before*
consumers exist; it is a one-way door.
- One repo or two? The family has both shapes (alkcall single-crate;
alknet's `crates/` workspace). Two crates in one repo (a workspace)
keeps the seam cheap to iterate until first publish; two repos match
the family's one-crate-per-repo posture but tax early co-development.
Working lean: one repo, two crates, until the seam stabilizes.
- **The seam shape**: does alkfs consume a *concrete* `alkblobs` (the
POC's blob_bridge pattern), or does a small blob trait live in alkfs
(or a tiny `alkblobs-types` leaf) so an embedder can substitute its
own store? The sealed-surface lesson
(`alknet-blobs-external-store-probe.md`) says decide before consumers
exist; the alkgit trait-family precedent (`docs/architecture/backend.md`
there) is the template for keeping it small. The put/get durability
contract (principle 3) rides this seam regardless.
- What is explicitly out: doors (alksftp, FUSE, mount), sync tooling,
UI/registry applications, the alknet rewrite's client stack.
@@ -390,66 +471,86 @@ granularity). Open sub-questions:
settings (`synchronous=NORMAL` vs `FULL`) as a durability-vs-throughput
knob? What is the honest fsync contract per operation class?
#### OQ-FS-03: Blob substrate — reuse iroh-blobs, own the store, or hybrid?
#### OQ-FS-03: Blob substrate — own the store (the direction) — what does alkblobs take from where?
Three live options, shaped by the probe:
**Reframed 2026-09-23: option (a) "consume iroh-blobs as a store" is
dissolved** — BLAKE3 is load-bearing across its entire surface (`Hash`
type, bao outboards, `Command` payload types), and the identity decision
has flipped to git-compatible sha1/sha256 (§Working direction). A
sha1/sha256 layer over iroh-blobs' Command surface would be a mapping
shim on a foreign hash, paying its dependency weight and its sealed-
surface risks to inherit a hash we don't want. **The working direction
is option (b): own the store (alkblobs).** What that leaves open:
- **(a) Consume iroh-blobs as a store** (implement the `Command` actor
externally per the probe; 4-line upstream PR; ~120-160 lines of bao
glue) — gets BLAKE3+bao verified streaming, GC, tags for free; pays
the iroh-blobs/irpc dependency weight, the sealed-`TempTags`
fallback (self-managed liveness), and still needs the metadata-store
coexistence story (OQ-FS-02). Also carries `bao_tree` version-tracking
as the maintenance surface.
- **(b) Own the blob store.** BLAKE3 (`blake3` crate) + bao
(`bao_tree` crate) directly, alkfs-owned storage layout — full control
of durability ordering (principle 3 becomes enforceable in one
transaction with OQ-FS-02's single-file shape), no sealed surfaces, no
iroh-blobs dependency. Cost: re-derive verified-streaming import/
export (the glue the probe measured), GC, partial states — the
pieces iroh-blobs already solved, now ours to keep correct.
- **(c) Hybrid/no-bao start.** Whole-file BLAKE3 + a simple file-per-blob
or SQLite-inline store first, add verified streaming (bao outboards)
when a consumer needs range verification. git's own objects are small;
packs are written-once-then-verified-by-consumer. Honest question:
does any alkfs v1 consumer actually need per-chunk bao verification on
the read path, or is end-to-end TLS + hash-check-on-close sufficient?
The deciding inputs: the alkgit workload (pack-sized blobs, streaming),
the sftp workload (range reads, seek), and how much of iroh-blobs'
surface survives contact with "no iroh/QUIC transfer."
- **What to take as findings vs build fresh**: iroh-blobs' inline/
outboard hybrid layout, write-ordering analysis, and mark-sweep GC
shape transfer as *findings* (§Prior art); the `bao_tree` crate
(BLAKE3-keyed) does not transfer for verified streaming — if
per-chunk integrity is wanted under sha1/sha256 identity, the chunk
index itself becomes the verification structure (each chunk row
carries its own hash; reads re-verify per chunk), which is a simpler
design than outboard files and falls out of the chunk manifest
(OQ-FS-04) for free.
- **The appfile layout** (OQ-FS-02's one-file question, now scoped to
alkblobs): one transactional store for blob metadata + inline small
content, file-backed spill-over for large content, single commit
boundary for the durability contract.
- **Bucket semantics**: the S3/rudolfs shape — namespace/bucket as the
isolation + naming prefix. How buckets relate to the alkfs layer
(alkfs buckets = alkblobs buckets? a mapping table? one alkblobs
store per alkfs bucket?) is an OQ-FS-11 sub-question; multi-tenancy
at the blob layer is where-clause isolation per the POC.
- **sha1 vs sha256 vs both**: git's dual-object-format reality (sha1
repos today, sha256 transition) says the store should carry
algorithm-tagged identities (a small enum per hash) rather than
hard-coding either — git object ids are natively addressable then
(OQ-FS-12's coexistence sub-question mostly dissolves into this).
- **Versioned-hash hazard worth recording**: sha1/sha256 are *not*
length-extended-safe for streaming concatenation the way tree hashes
are — any chunk-manifest scheme must be its own construction (hash
list / Merkle over chunk hashes, e.g. the git tree-object shape
itself), not a naive linear hash chain. This is a design constraint,
not a blocker.
### Write path, branching, GC
#### OQ-FS-04: Hashing and chunking — whole-file addresses vs chunk DAGs
#### OQ-FS-04: Hashing and chunking — object-granular identity over a chunk manifest
The load-bearing storage-format question (a one-way door once content
exists in the wild):
**Reframed 2026-09-23.** The identity half is decided in direction:
**sha1/sha256, algorithm-tagged** (OQ-FS-03) — git-compatible, native to
alkgit. What remains open is the *dedup granularity* and the manifest
shape — the load-bearing storage-format one-way door once content
exists in the wild:
- **Whole-file BLAKE3 (+ bao outboard for verified streaming).** Simple,
git-shaped, matches the POC and alkgit's objects; but the address
doesn't exist until the write completes (staging required — the
branch-on-write answer), no partial availability under a final name,
no cross-file chunk dedup (two 1GB files sharing a 500MB prefix
dedup zero bytes).
- **Chunk-list / Merkle-DAG addressing** (content-defined or fixed-size
chunks; the file "name" is a root hash over chunk hashes, iroh-blobs
style): dedup at chunk granularity, partial availability, resumable
writes under a stable name, cheap append; but bigger metadata, a
chunker to pin down (CDC boundary-shift sensitivity), and whole-file
identity becomes "same root hash" rather than "same bytes hash" (root
hash equality still implies byte equality with bao-style trees, at
hash-of-chunk-hashes cost).
- **Hybrid** (the git-lfs/iroh shape): small content inline/whole-hash;
large content chunked. Threshold policy is a knob; the *format* split
is the one-way door.
- **Whole-file sha256 (the git shape).** Objects are addressed whole;
identical files dedup, similar-but-not-identical do not. Simplest,
matches git objects exactly, and matches the POC's tested write path
verbatim. For large workspace content the dedup miss is real
(two 1GB files sharing a 500MB prefix dedup zero bytes).
- **Chunk manifest (the working goal for large content).** A file is a
root manifest over chunk hashes (a hash list — the git tree-object
shape generalized, per OQ-FS-03's versioned-hash constraint); chunks
dedup across files, partial availability and resumable writes come
along, and per-chunk re-verification on read falls out of the
manifest (each row carries its own hash — the bao-outboard role,
without bao). Chunker choice (CDC vs fixed-size) and boundary-shift
sensitivity are the design cost.
- **Hybrid (the probable landing):** whole-file identity under a
size threshold (git objects, source files — effectively single-chunk
anyway); chunk manifests for large content (datasets, logs, pack-
scale blobs). The threshold is a knob; the manifest format for the
chunked side is the one-way door.
Sub-questions: CDC (rollsum/buzhash) vs fixed-size vs BLAKE3's native
chunking (`bao_tree`'s chunk size); where chunk indices live (metadata
store rows vs outboard files); whether chunk dedup across *files* is a
v1 requirement or a defer-able nicety; interaction with OQ-FS-05 (a
chunk-DAG write can commit incrementally, weakening the branch-on-write
staging argument).
Sub-questions: CDC (rollsum/buzhash) vs fixed-size vs git-style
delta-encoding as *adjacent* dedup (packs delta-compress similar
objects — a chunk-manifest store could optionally delta-compress chunk
rows, but that is a compression layer, not identity); where chunk
indices live (alkblobs manifest tables); whether cross-*bucket* chunk
dedup is ever allowed (working answer: never — tenants are isolated,
OQ-FS-11); interaction with OQ-FS-05 (a chunk-manifest write can commit
incrementally, weakening the branch-on-write staging argument for
chunked files — the session model may collapse to "extend the manifest"
for large content).
#### OQ-FS-05: Write path semantics beyond the POC
@@ -605,10 +706,15 @@ decision tree:
filesystem semantics alkfs won't natively provide (mmap, file locks).
Sub-questions: does git's object-id namespace (sha1/sha256) coexist with
alkfs's content addresses (path rows mapping git-oids → alkfs hashes,
dedup at the alkfs layer); CAS ref transaction shape; whether pack
ingestion budget enforcement (alkgit ADR-009) maps onto alkfs write
sessions.
alkfs's content addresses — **mostly dissolved by the identity reframe
(2026-09-23)**: git oids *are* the blob layer's addresses now; what
remains is whether git objects ride as whole-file identities (the
natural fit — git objects are small) while alkfs path rows may point at
chunk-manifest roots for large content (OQ-FS-04's hybrid), and how
pack files relate (a pack is just a large blob to alkblobs; unpacked
object rows are the useful representation — the ingestion mapping is
the question). CAS ref transaction shape; whether pack ingestion
budget enforcement (alkgit ADR-009) maps onto alkfs write sessions.
#### OQ-FS-13: Performance shape and budgets
@@ -648,6 +754,45 @@ construction costs little if the engine split exists anyway. Decide
deliberately once the split (OQ-FS-01) exists; verify with
`cargo check --target wasm32-unknown-unknown`.
#### OQ-FS-16: The remote-mount workload — the anchor use case
**Added 2026-09-23.** The concrete deployment this crate exists for (the
user's own daily workflow): a dev-server workspace mounted on a local
Linux machine under `remote/` — today via sshfs/vanilla SFTP, with
known slowness:
- **Why vanilla SFTP is slow (the honest decomposition):** a network
round trip per syscall (stat storms during directory walks; sshfs
even translates `SeekFrom::End` into a real `fstat` round trip);
attribute-invalidation chatter; no server-side index (the SFTP door
shells to real FS syscalls, each paying host-FS costs); attr-cache
and cache-timeout knobs that trade staleness for speed. The POC's
mapping table already noted `stat` → single indexed lookup is
*cheaper* than a real FS fstat — an appfile-indexed VFS fixes the
server-side half structurally.
- **What the alkfs shape adds:** server-side indexed metadata (stat/
readdir from index rows, no syscall per attribute); pipelined writes
and batched reads native to the protocol (the russh-sftp client's
`write_nowait` + ack window rides the chunked write session);
optional client-side caching/prefetch as door-local policy (a mount
door can cache content chunks locally — content-addressed means
cache coherency is a branch-head-invalidation problem, not a
byte-consistency problem); directory-listing snapshots.
- **What this anchors:** the verb set (OQ-FS-08) must make the
readdir/stat/read/write path fast, not merely correct; the session
model (OQ-FS-08's per-file-channel vs multiplexed question) matters
most for directory-walk fan-out; OQ-FS-13's budget table should
measure against sshfs-over-localhost baselines.
- **alksftp vs a mount protocol:** vanilla SFTP clients (sshfs) work
against any SFTP door with zero client install — that compatibility
is worth keeping as the floor. A leaner alkfs-native mount protocol
(OQ-FS-08's framing question) is the speed tier; whether it warrants
its own FUSE-facing door early or later is a sequencing question
deliberately parked until the protocol shape exists.
This OQ is the *consumer pull* behind the whole crate — if the mount
use case stops mattering, OQ-FS-08's urgency drops with it.
## Candidate POC register (proposals — none run yet)
POC placement conventions (inherited from alktunnels/alksocks): a POC
@@ -660,17 +805,20 @@ standalone crate in the global workspace with findings written into
|---|---|---|---|
| 1 | Engine skeleton: path tree + inline-blob store in one SQLite file | One-file metadata+blob-metadata (OQ-FS-02) holds up under the POC's 15-test suite ported over; single-transaction durability ordering (principle 3) is expressible | OQ-FS-02, OQ-FS-05 |
| 2 | Write path at pack scale | Chunked write sessions handle 100MB+ streams (spill-to-file regime) with correct durability ordering and crash-abort semantics | OQ-FS-05, OQ-FS-13 |
| 3 | Blob substrate bake-off (narrow) | (a) iroh-blobs external actor vs (b) own blake3/bao store vs (c) inline-hybrid: measure integration cost, GC story, sealed-surface fallout against the probe's predictions | OQ-FS-03 |
| 4 | Chunking probe (paper + micro-bench) | Whole-file vs chunk-DAG: dedup gains on realistic workloads (git packs, agent workspaces), write-path impact, metadata size — enough to decide the format one-way door or at least stage it | OQ-FS-04 |
| 3 | alkblobs store skeleton (sha1/sha256 identity) | Own-store direction (OQ-FS-03): algorithm-tagged hash ids, bucket isolation, inline-vs-spill layout, manifest-of-chunks for large content — measure against the probe's store-seam map as the API checklist | OQ-FS-03, OQ-FS-04 |
| 4 | Chunking probe (paper + micro-bench) | Whole-file vs chunk manifest: dedup gains on realistic workloads (agent workspaces, datasets, logs — git objects are single-chunk and ride whole-file), write-path impact, metadata size — enough to decide the format one-way door or at least stage it | OQ-FS-04 |
| 5 | Serving protocol spike | SFTP-verb protocol over channels: per-file-channel vs multiplexed-handles session model, open cost at directory-walk and git-clone fan-out, framing sketch (not a wire ADR) | OQ-FS-08, OQ-FS-09 |
| 6 | alkgit seam spike | `GitPackGen`/`GitPackIngest` shaped against an alkfs-backed engine on the POC's API — does the trait family survive contact; where do git oids map | OQ-FS-12 |
| 6 | alkgit seam spike | `GitPackGen`/`GitPackIngest` shaped against an alkblobs-backed engine — does the trait family survive contact; git oids as native addresses | OQ-FS-12 |
| 7 | GC walk prototype | Reachability-mark GC over path rows + refs: correctness on shared content across branches, cost model, orphaned-session reaping | OQ-FS-07 |
| 8 | Mount-workload baseline | Measure the *current* pain (sshfs against this dev server): op latencies for stat/readdir/read/write at realistic workspace scale — the baseline OQ-FS-16 improvements must beat | OQ-FS-16, OQ-FS-13 |
Order is indicative, not committed: #1/#2 de-risk the engine core;
#3/#4 resolve the storage-format one-way doors; #5/#6 are only worth
running once #1 exists (the protocol needs an engine to serve). The
register exists so POC work has home ids; expect entries to be split,
merged, or dropped as OQs sharpen.
Order is indicative, not committed: #1/#3 de-risk the engine cores
(alkfs metadata, alkblobs store); #2 stress-tests the write path; #4
resolves the storage-format one-way door; #5/#6 are only worth running
once #1/#3 exist (the protocol needs an engine to serve); #8 can run
anytime (it measures today's sshfs, no alkfs code needed) and its
baseline ages well if taken early. The register exists so POC work has
home ids; expect entries to be split, merged, or dropped as OQs sharpen.
## Survey / prior-art list
@@ -700,19 +848,24 @@ Read/verified (cited above):
To evaluate (research-specialist queue, feeding the OQs named):
- **Chunking/CDC prior art** — FastCDC, rollsum (bup), rsync-style
boundaries; `bao_tree`'s BLAKE3 chunking as the default candidate
(OQ-FS-04).
boundaries; git tree-object hashing as the manifest shape (the
versioned-hash constraint's precedent, OQ-FS-03/04); `bao_tree`'s
chunker as a chunk-boundary *algorithm* candidate (decoupled from its
BLAKE3 identity).
- **CAS filesystems** — Plan 9 fossil+venti (the branch/CAS ancestry the
POC name-checks), IPFS unixfs/iros DAG shapes, OST/lessfs-style
inline-vs-outboard hybrids (OQ-FS-04, OQ-FS-06).
- **git storage internals** — gix-odb pack/idx layout and
`gix-pack::data::input` streaming (alkgit research has the survey;
re-read for OQ-FS-12's object-id/namespace questions).
re-read for OQ-FS-12's object/pack mapping questions).
- **SQLite durability posture** — WAL + `synchronous=NORMAL` vs `FULL`
under the commit-boundary contract; SQLite-as-blob-store limits
(OQ-FS-02, OQ-FS-05).
- **FUSE/mount doors** — rust bindings shape, POSIX semantics alkfs
would need to fake (OQ-FS-10; defer design, record constraints).
would need to fake; sshfs latency characteristics (the OQ-FS-16
baseline's "why it's slow" decomposition, verified against real
measurements from POC #8 rather than folklore) (OQ-FS-10,
OQ-FS-16; defer design, record constraints).
## Convergence checklist (what Phase 0 must produce)
@@ -723,14 +876,17 @@ To evaluate (research-specialist queue, feeding the OQs named):
0.103 findings; git/git-lfs shape pinned against alkgit's backend
contract; CDC/chunking survey (OQ-FS-04 input); CAS-filesystem
survey
- [ ] OQ-FS-01 (crate scope) — converged: engine/protocol split, trait
seam or not, explicit non-goals
- [ ] OQ-FS-01 (crate scope) — converged: the alkblobs/alkfs split
confirmed or cut; repo shape (workspace vs two repos); trait seam
at the blob boundary or concrete dependency; explicit non-goals
- [ ] OQ-FS-02 (metadata store) — converged: substrate, one-file-vs-two
(blob metadata placement), durability knob
- [ ] OQ-FS-03 (blob substrate) — converged: consume iroh-blobs vs own
the store vs hybrid (POC #3)
- [ ] OQ-FS-04 (hashing/chunking) — converged: the storage-format
one-way door (POC #4)
(blob metadata placement, now scoped to alkblobs), durability knob
- [ ] OQ-FS-03 (blob substrate) — converged: the own-store direction
pinned down (what transfers as findings, what's built fresh)
(POC #3)
- [ ] OQ-FS-04 (hashing/chunking) — converged: identity settled in
direction (sha1/sha256); dedup granularity + manifest format are
the one-way door (POC #4)
- [ ] OQ-FS-05 (write path) — durability/concurrency/spill semantics
pinned (POC #2)
- [ ] OQ-FS-06 (branch/snapshot/refs) — model chosen; ref namespace
@@ -743,12 +899,17 @@ To evaluate (research-specialist queue, feeding the OQs named):
- [ ] OQ-FS-09 (ALPN/params) — naming + params ground collected; ADR
in Phase 1
- [ ] OQ-FS-10 (doors) — protocol kept door-ready; nothing designed in
- [ ] OQ-FS-11 (tenancy/policy) — scope-string + policy shape settled
- [ ] OQ-FS-11 (tenancy/policy) — scope-string + policy shape settled;
bucket semantics across the alkblobs/alkfs seam
- [ ] OQ-FS-12 (alkgit fit) — the (a)/(b)/(c) decision made; alkgit
un-pause plan written (POC #6)
- [ ] OQ-FS-13 (budgets) — numbers collected; budget table drafted
- [ ] OQ-FS-13 (budgets) — numbers collected (incl. the sshfs baseline,
POC #8); budget table drafted
- [ ] OQ-FS-14 (sync) — explicitly deferred or scoped with evidence
- [ ] OQ-FS-15 (wasm) — decided once the engine/protocol split exists
- [ ] OQ-FS-16 (remote mount) — the anchor workload's requirements
written down (verb-set implications, caching shape); sequencing
(SFTP floor first vs native protocol tier) decided
- [ ] Targeted POCs run + findings in `docs/research/`
- [ ] Converge: recommended approach written up, ready to hand to the
Architect for Phase 1