glm-5.3-flash
281c37876e
- requirements.md (new): the three use cases pinned as REQ-1..4 with the node/pool/fleet/engine vocabulary defined once — ends per-session re-derivation of consumer facts. REQ-2 (replicator fleet over one shared pg pool, incl. large-blob serving) is recorded as a planning fact predating all POCs, on the operator's authority. - ADR-008: Backend trait gains size(key) length probe (before any backend ships data — ADR-002 one-way-door discipline); fleet GC mechanism (DB-backed pin rows committed atomically with entries, TTL+renewal semantics, liveness = embedder-owned table, protect callback is single-node-only, advisory-locked single sweeper, staged re-arbitrated delete window on both engines); fs tier becomes engine-selectable (local default; pg-lo named candidate) with the fleet locality contract (shared media or re-routing; mixed tiers are a documented deployment invariant, not a constructor-provable one). - poc-pglo-spec.md (new): POC #7 spec — pg Large Objects as the fs tier's pg-lo engine; instruments, decision gate, registered in phase-0.md OQ-BL-06. - ADR-003/005 status amendments point to ADR-008's extensions; specs ripple (backends/store-api/gc/ops/overview/README). - open-questions.md: deferral-policy header gains the decisions-vs- sequenced-work distinction; pg-lo's why-not-parked audit trail recorded. - research fixes: postgres POC renumbered #4->#5 to the canonical register (phase-0 OQ-BL-06), redb cross-refs fixed, thinking- artifact sentence in B1 replaced with the honest reading. Verification: docs-only change; reference-integrity sweep across the tree (ADR/REQ/POC refs resolve); architecture-reviewer pass on the delta — original 3 criticals addressed, its follow-up (fleet liveness form, pin TTL, staged-delete semantics, enforceability, shared-media caveats) fixed in this commit.
158 lines
9.1 KiB
Markdown
158 lines
9.1 KiB
Markdown
---
|
||
status: draft
|
||
last_updated: 2026-10-03
|
||
---
|
||
|
||
# Overview
|
||
|
||
## What this crate is
|
||
|
||
`alkblobs` is content-addressed blob storage in the alk* family: a
|
||
put/get/verify store keyed under one canonical content hash (the
|
||
git-family derivation; legacy SHA-1-tolerant), with pluggable backends
|
||
(small blobs in a kv store, large blobs on a filesystem fallback),
|
||
split from any wire/ops-protocol layer. Network ops ride the alkcall
|
||
substrate behind a feature-gated `ops` module (ADR-001).
|
||
|
||
Deployment-relevant consumer facts — what each use case forces on this
|
||
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
|
||
with the **node/pool/fleet/engine** vocabulary defined there). This
|
||
tree references REQs by ID; per-session re-derivation from vision prose
|
||
is exactly what that document exists to prevent.
|
||
|
||
## Why it exists — the extracted shared core
|
||
|
||
Two downstream projects were converging on the same storage stack, so it
|
||
is being built once, here:
|
||
|
||
- **alkgit** was paused mid-Phase-1 (its architecture is reviewed,
|
||
`docs/architecture/` in the alkgit repo) to stop duplicated
|
||
storage-layer work. Its object backend needs git's own oid derivation,
|
||
a better small-object backend than its proposed default file layout,
|
||
and a git-lfs-shaped large-blob story. Under the canonical-hash
|
||
resolution (ADR-002) git objects are first-class pool entries with
|
||
zero indirection.
|
||
- **alkfs** (Phase 0 pending) needs the appfile/workspace shape: small
|
||
content in a fast kv store, large content on the filesystem, path→hash
|
||
mapping above. The path-tree/manifest layer is alkgit's and alkfs's,
|
||
not this crate's (ADR-004).
|
||
|
||
Both demanding consumers (p2p git replicators, agent workspaces) reduce
|
||
to the same shape: **a pooled CAS per node**, where repos and workspaces
|
||
are *sets of hash references*, not walled-off object stores — cross-repo
|
||
dedup by construction (ADR-005). That pooling property is why the
|
||
storage layer was pulled out of both consumers rather than built twice
|
||
or three times.
|
||
|
||
## Layer map
|
||
|
||
```
|
||
┌────────────────────────────────────────────────────────────┐
|
||
│ consumers (above this crate) │
|
||
│ alkgit object storage · alkfs path trees/workspaces │
|
||
│ refs, manifests, path→hash maps, git semantics │
|
||
└────────────┬───────────────────────────────────────────────┘
|
||
▼
|
||
┌────────────────────────────────────────────────────────────┐
|
||
│ ops surface (feature "ops"; ADR-001) │
|
||
│ have/need + verified fetch + put; JSON control ops, │
|
||
│ binary data channels; ACL via alkcall AccessControl │
|
||
└────────────┬───────────────────────────────────────────────┘
|
||
▼
|
||
┌────────────────────────────────────────────────────────────┐
|
||
│ store core (always on) │
|
||
│ hashing & keys (ADR-002) · put/get/stat/range/pin seams │
|
||
│ mark-and-sweep GC + liveness registration (ADR-005) │
|
||
│ size-threshold dispatch (ADR-003) │
|
||
└───────┬───────────────────┬───────────────────────────────┘
|
||
▼ ▼
|
||
│ kv backend fs backend (ADR-003: exactly two tiers;
|
||
│ │ ADR-007: kv = 2 engines;
|
||
│ ▼ ADR-008: fs engines too)
|
||
│ engine per node: sqlite (default) | postgres (feature "postgres")
|
||
│ fs engine per node: local (default) | pg-lo (candidate, POC #7)
|
||
|
||
transport: none, anywhere. alkgit/alkfs or the embedder dials;
|
||
an alkcall Connection is handed in at the ops boundary.
|
||
```
|
||
|
||
## Dependency posture
|
||
|
||
| Layer | Dependencies | Notes |
|
||
|---|---|---|
|
||
| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
|
||
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
|
||
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
|
||
|
||
Backends are feature-gated; both `kv` and `fs` ship default-on
|
||
(batteries-included, the alkgit feature-model pattern — default-on
|
||
features, opt-out rather than opt-in); the `kv` tier's postgres engine
|
||
(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only
|
||
kv node (postgres engine without rusqlite) is **not** a supported
|
||
configuration this crate optimizes for — feature-graph work for it is
|
||
deferred until a named requirement exists (the base `kv` feature
|
||
pulling rusqlite is the accepted v1 shape; convention 8's
|
||
lean-base principle applies to *default* builds, which stay lean since
|
||
`kv` is what the feature model controls, not what default-on means
|
||
downstream). Wasm: the backends are not a wasm story; a wasm client is
|
||
a consumer of the `ops` surface (which rides alkcall, itself
|
||
wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
||
|
||
## Non-goals (each enforced by an ADR)
|
||
|
||
- No wire framing, op dispatch, ACL, or transport inside the store layer
|
||
(ADR-001).
|
||
- No adopted iroh-blobs wire surface — no tickets, no postcard, no
|
||
provider protocol (AGENTS convention 7; ADR-001 §Context).
|
||
- No namespaces-as-physical-storage; the pool is flat, namespaces are
|
||
consumer-level reference sets (ADR-005).
|
||
- No manifest/path-tree/vfs tables; consumers never implement a Backend
|
||
to model paths (ADR-004).
|
||
- No chunk-tree/CDC storage encoding; whole-file CAS is the store's only
|
||
granularity (ADR-006).
|
||
- No per-repo or per-namespace walled stores — one pooled CAS
|
||
(ADR-005).
|
||
|
||
## Design Decisions
|
||
|
||
| ADR | Decision | Summary |
|
||
|---|---|---|
|
||
| [001](decisions/001-substrate-posture-and-ops-placement.md) | Substrate posture & ops placement | store stays substrate-free; ops ride alkcall feature-gated |
|
||
| [002](decisions/002-canonical-hash-and-key-encoding.md) | Canonical hash + key encoding | git-blob-sha-256 canonical; algorithm in-key; one-way door |
|
||
| [003](decisions/003-backend-contract-and-dispatch.md) | Backend contract & dispatch | two backends, size-threshold routing, no migration |
|
||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side |
|
||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
|
||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
|
||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
|
||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
|
||
|
||
## Open Questions
|
||
|
||
Tracked in [open-questions.md](open-questions.md). Ones affecting the
|
||
crate level:
|
||
|
||
- **OQ-07**: how alkgit's object-storage seam consumes alkblobs
|
||
(externally-owned: alkgit's architecture process)
|
||
- **OQ-08**: alkfs requirement intake (externally-owned: alkfs Phase 0)
|
||
- **OQ-09**: default namespace visibility in network ops —
|
||
resolved closed-by-default (deny-unlisted; reopen only on a named
|
||
open-read requirement)
|
||
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
|
||
shipped behind one trait; per-node constructor choice)
|
||
- **POC #7** (`docs/research/poc-pglo-spec.md`): pg Large Objects as
|
||
the fs tier's `pg-lo` engine — spec'd **sequenced work** (not a
|
||
parked question; open-questions.md's policy header distinguishes the
|
||
two): admission evidence for ADR-008's named candidate (REQ-2)
|
||
|
||
## References
|
||
|
||
- `docs/research/phase-0.md` — the converged Phase 0 record; inputs to
|
||
every ADR above
|
||
- `docs/research/poc-trait-dispatch-findings.md`,
|
||
`docs/research/poc-largeblob-findings.md`,
|
||
`docs/research/iroh-blobs-eval.md` — the evidence trail
|
||
- `docs/sdd_process.md` — the process governing this tree
|
||
- `/workspace/@alkdev/alkgit/docs/architecture/backend.md` — the
|
||
reviewed consumer-side seam that OQ-07 maps onto alkblobs
|
||
- AGENTS.md — family conventions (conventions 4–8 gate the decisions) |