docs(architecture): ADR-010/011 — I/O seams, GC-state home, vocabulary; pre-decomposition consistency round
ADR-010 (new): the Backend trait's I/O seams pinned before any code — 'get' returns a crate-internal read cursor (kv engines materialize; 'local' pread loop; pg-lo lo_get windows), 'put' has two named forms (whole-value kv / staged-put large); GC state is store-core-owned via a crate-internal contract-tested engine-state seam on the SQL-backed engines (backends never learn liveness — ADR-005 verbatim); the store core holds the joint entry+pin transaction (corrects ADR-009 §2's ownership statement); kv-tier fleet-validity rule added (mirror of ADR-008 §3); ops pin-token TTL race resolved (the token IS the pin; renewal rides have/re-put; pause-past-TTL falls to delete-then-recover); feature graph pinned (pg-lo does not imply postgres). ADR-011 (new): vocabulary pinned — tier (contract, exactly two) vs engine (concrete impl, exactly five); store instance / node / fleet split (fleet = pool-sharing, not node count — fixes requirements.md's self-contradiction with REQ-2); mem demoted from 'backend'/'testing tier' to the kv tier's third engine (contract-reference engine); fs tier renamed 'large' including the feature name (last free moment before code exists); constructor modes pinned (dual-tier default, kv-only, mem-only) plus the composite fleet-validity predicate. Consistency round across all specs and ADR-003/004/007/008/009: stale pg-lo passages resolved per ADR-009's supersession; ADR-008 owner 'node id' → store-instance id; threshold wording corrected (conservative edge, not midpoint, of the 128-256 KiB crossover zone); SweepReport shape specced; mem classification unified; 008/009 ADR files renamed to match the tier rename; README deferral-policy recap aligned (third category = decided-but-sequenced work, not a parking kind); POC crate list completed. Verification: two independent architecture review rounds (the first found 4 criticals — unpinned trait I/O shapes, missing fleet-state seam, node/fleet self-contradiction, pin-token/TTL conflict — all resolved; final round: zero criticals); all markdown links resolve; ADR tables complete (11 ADRs).
This commit is contained in:
glm-5.3-flash committed
1 parent
9bbe6f3ad9
commit
abf8c5ae9c
18 files changed
+1137
-336
No files matched your search
@@ -1,6 +1,6 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-10-03
|
||||
last_updated: 2026-10-03 (ADR-010/011 — I/O seams pinned; vocabulary fs→large, mem joins kv)
|
||||
---
|
||||
|
||||
# Overview
|
||||
@@ -9,10 +9,11 @@ last_updated: 2026-10-03
|
||||
|
||||
`alkblobs` is content-addressed blob storage in the alk* family: a
|
||||
put/get/verify store keyed under one canonical content hash (the
|
||||
git-family derivation; legacy SHA-1-tolerant), with pluggable backends
|
||||
(small blobs in a kv store, large blobs on a filesystem fallback),
|
||||
split from any wire/ops-protocol layer. Network ops ride the alkcall
|
||||
substrate behind a feature-gated `ops` module (ADR-001).
|
||||
git-family derivation; legacy SHA-1-tolerant), with pluggable tier
|
||||
engines (small blobs in a kv store per ADR-007/011, over-threshold
|
||||
blobs on the `large` tier per ADR-008/009 — the tier formerly named
|
||||
`fs`), split from any wire/ops-protocol layer. Network ops ride the
|
||||
alkcall substrate behind a feature-gated `ops` module (ADR-001).
|
||||
|
||||
Deployment-relevant consumer facts — what each use case forces on this
|
||||
crate — are pinned in [requirements.md](requirements.md) (REQ-1..4,
|
||||
@@ -63,15 +64,22 @@ or three times.
|
||||
│ store core (always on) │
|
||||
│ hashing & keys (ADR-002) · put/get/stat/range/pin seams │
|
||||
│ mark-and-sweep GC + liveness registration (ADR-005) │
|
||||
│ size-threshold dispatch (ADR-003) │
|
||||
│ size-threshold dispatch (ADR-003); GC state owned │
|
||||
│ here, hosted by SQL engines via the seam (ADR-010) │
|
||||
└───────┬───────────────────┬───────────────────────────────┘
|
||||
▼ ▼
|
||||
│ kv backend fs backend (ADR-003: exactly two tiers;
|
||||
│ │ ADR-007: kv = 2 engines;
|
||||
│ ▼ ADR-008: fs engines too)
|
||||
│ engine per node: sqlite (default) | postgres (feature "postgres")
|
||||
│ fs engine per node: local (default) | pg-lo (feature "pg-lo",
|
||||
│ default-off; ADR-009)
|
||||
┌──────────────────────┐ ┌─────────────────────────────────┐
|
||||
│ kv tier (ADR-003/ │ │ large tier (ADR-003/008/011; │
|
||||
│ 007/011) │ │ formerly "fs") │
|
||||
│ sqlite (default) │ │ local (default) │
|
||||
│ postgres ("postgres") │ pg-lo ("pg-lo"; ADR-009) │
|
||||
│ mem ("mem"; tests + │ │
|
||||
│ contract reference) │ │
|
||||
└──────────────────────┘ └─────────────────────────────────┘
|
||||
▲ ▲
|
||||
└── one Backend trait (ADR-003/010: read cursor +
|
||||
staged put I/O seams); engine = per-node
|
||||
constructor choice (ADR-011's table)
|
||||
|
||||
transport: none, anywhere. alkgit/alkfs or the embedder dials;
|
||||
an alkcall Connection is handed in at the ops boundary.
|
||||
@@ -81,36 +89,37 @@ or three times.
|
||||
|
||||
| Layer | Dependencies | Notes |
|
||||
|---|---|---|
|
||||
| base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
|
||||
| `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||||
| `pg-lo` feature (ADR-009) | Large Objects as the fs tier's second engine, default-off | rides the same `tokio-postgres`/`deadpool-postgres` stack as the `postgres` kv engine (SQL `lo_*` functions, no new dependency); same admission/test-gate shape |
|
||||
| base crate (store core + tier engines) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `large`/`mem` | no alkcall, no serialization frameworks |
|
||||
| `postgres` engine feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
|
||||
| `pg-lo` engine feature (ADR-009) | Large Objects as the large tier's second engine, default-off | pulls the same `tokio-postgres`/`deadpool-postgres` cluster as the `postgres` kv engine (SQL `lo_*` functions, no additional dependency); **does not imply the `postgres` feature** — engine features pull the driver cluster independently (ADR-010 §1), so a `large`=pg-lo + kv=sqlite node is coherent and supported; same admission/test-gate shape |
|
||||
| `mem` engine feature (ADR-011) | the kv tier's ephemeral engine, default-off | `BTreeMap`-shaped; the contract-reference engine; no dependencies |
|
||||
| `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
|
||||
| consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
|
||||
|
||||
Backends are feature-gated; both `kv` and `fs` ship default-on
|
||||
(batteries-included, the alkgit feature-model pattern — default-on
|
||||
features, opt-out rather than opt-in); the `kv` tier's postgres engine
|
||||
(`postgres`, ADR-007) and the `ops` feature are default-off. A pg-only
|
||||
kv node (postgres engine without rusqlite) is **not** a supported
|
||||
configuration this crate optimizes for — feature-graph work for it is
|
||||
deferred until a named requirement exists (the base `kv` feature
|
||||
pulling rusqlite is the accepted v1 shape; convention 8's
|
||||
lean-base principle applies to *default* builds, which stay lean since
|
||||
`kv` is what the feature model controls, not what default-on means
|
||||
downstream). Wasm: the backends are not a wasm story; a wasm client is
|
||||
a consumer of the `ops` surface (which rides alkcall, itself
|
||||
wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
||||
Tiers are feature-gated; both default-on (batteries-included, the
|
||||
alkgit feature-model pattern — default-on features, opt-out rather
|
||||
than opt-in); the engine features `postgres` and `pg-lo`, and `mem`,
|
||||
are default-off, as are `ops`. A pg-only kv node (postgres engine
|
||||
without rusqlite) is **not** a supported configuration this crate
|
||||
optimizes for — feature-graph work for it is deferred until a named
|
||||
requirement exists (the base `kv` feature pulling rusqlite is the
|
||||
accepted v1 shape; convention 8's lean-base principle applies to
|
||||
*default* builds, which stay lean since `kv` is what the feature model
|
||||
controls, not what default-on means downstream). Wasm: the tier
|
||||
engines are not a wasm story; a wasm client is a consumer of the `ops`
|
||||
surface (which rides alkcall, itself wasm-clean), never an in-process
|
||||
embedder (ADR-001 §Consequences).
|
||||
|
||||
## Non-goals (each enforced by an ADR)
|
||||
|
||||
- No wire framing, op dispatch, ACL, or transport inside the store layer
|
||||
(ADR-001).
|
||||
- No adopted iroh-blobs wire surface — no tickets, no postcard, no
|
||||
provider protocol (AGENTS convention 7; ADR-001 §Context).
|
||||
- **No adopted iroh-blobs wire surface — no tickets, no postcard, no
|
||||
provider protocol (AGENTS convention 7; ADR-001 §Context).**
|
||||
- No namespaces-as-physical-storage; the pool is flat, namespaces are
|
||||
consumer-level reference sets (ADR-005).
|
||||
- No manifest/path-tree/vfs tables; consumers never implement a Backend
|
||||
to model paths (ADR-004).
|
||||
- No manifest/path-tree/vfs tables; consumers never implement a
|
||||
Backend to model paths (ADR-004).
|
||||
- No chunk-tree/CDC storage encoding; whole-file CAS is the store's only
|
||||
granularity (ADR-006).
|
||||
- No per-repo or per-namespace walled stores — one pooled CAS
|
||||
@@ -126,9 +135,11 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences).
|
||||
| [004](decisions/004-two-backends-no-third.md) | Two backends, no third | manifest layers are consumer-side |
|
||||
| [005](decisions/005-pooled-cas-and-gc-mechanism.md) | Pooled CAS & GC | flat pool, registered liveness, delete windows |
|
||||
| [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
|
||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
|
||||
| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo fs engine | postgres Large Objects as the fs tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
|
||||
| [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + large tier per node |
|
||||
| [008](decisions/008-trait-size-probe-fleet-gc-and-large-engines.md) | `size` probe + fleet GC + large-tier engines | trait length probe; fleet pins/sweeps engine-backed; the large tier (now `large`) engine-selectable |
|
||||
| [009](decisions/009-pg-lo-large-tier-engine.md) | pg-lo large engine | postgres Large Objects as the large tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
|
||||
| [010](decisions/010-backend-io-seams-and-fleet-state-home.md) | I/O seams + GC-state home | read cursor / staged put; store-core-owned GC state via the engine-state seam; kv fleet-validity rule; pin-token renewal |
|
||||
| [011](decisions/011-vocabulary-tiers-engines-and-fleet-state.md) | Vocabulary + rename | tier/engine/instance/node/fleet pinned; mem joins the kv tier; large tier renamed `large`; constructor modes |
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -144,7 +155,7 @@ crate level:
|
||||
- **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
|
||||
shipped behind one trait; per-node constructor choice)
|
||||
- **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects
|
||||
as the fs tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
|
||||
as the large tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
|
||||
a shipped engine by ADR-009 (REQ-2's consolidation option)
|
||||
|
||||
## References
|
||||
|
||||
Reference in new issue
Block a user