glm-5.3-flash
4b5009d85a
- overview/hashing/store-api/backends/gc/ops specs + README index, all draft status with YAML frontmatter and cross-referenced ADR/OQ - ADR-001: substrate posture (store layer alkcall-free = family conformance, not deviation) + ops placement decided (feature-gated module, no consumer-waited deferral) - ADR-002: canonical git-blob-sha-256 + key encoding — the declared wire-format ADR (one-way door) - ADR-003: backend contract, sqlite/fs/mem tiers, pure-function dispatch, no migration, unknown-length buffering semantics pinned - ADR-004: exactly two production backends; manifest/path-tree layers are consumer-side (the corrected two-vs-three-backends framing) - ADR-005: pooled CAS, three liveness sources, delete windows with pin-before-publish + delete-time arbitration (race closed), no ambient scheduling - ADR-006: whole-blob verification; chunk-tree encodings excluded by scoping, not deferred on a paused consumer - open-questions.md: OQ-BL-01..06 resolved with ADR cross-refs; OQ-07 /08 externally-owned, OQ-09 deferred(scope) with SDD tracker task - AGENTS.md: status updated to Phase 1, gix-odb path corrected Verification: cargo test / clippy -D warnings / fmt --check clean
106 lines
5.0 KiB
Markdown
106 lines
5.0 KiB
Markdown
# ADR-006: Verification posture — whole-blob checks; transfer encodings excluded by scoping
|
|
|
|
## Status
|
|
|
|
Accepted
|
|
|
|
## Context
|
|
|
|
OQ-BL-04's settled Phase 0 posture: whole-file CAS as the default
|
|
granularity; per-range verification impossible under the canonical
|
|
digest (the preamble hashes the length — POC #3 finding A2,
|
|
independently reproduced in the iroh-blobs eval; no slice has any hash
|
|
relationship to the whole); the chunk-tree encoding framed as "a
|
|
transfer-layer conditional — deferred until a consumer exists".
|
|
|
|
That last framing is a deferral this crate cannot make under its own
|
|
deferral policy: "the consumer that would want networked verified
|
|
ranged fetch" (p2p git's replicator) is paused *because* this crate is
|
|
being built. A condition gated on it is Schrödinger's code — it never
|
|
collapses. The decision must stand on the evidence in hand, which is
|
|
complete for the *scoping* question even though nothing measures a
|
|
future consumer:
|
|
|
|
- What the consumers need today is verifiable **whole-blob** transfer:
|
|
fetch bytes, hash-check against the canonical digest the caller
|
|
carries. That is exactly verified-fetch (the `Sub` op with the digest
|
|
in the offer), works at any size, and needs no encoding — POC #3's
|
|
streaming paths are validated byte-exact against git.
|
|
- What would need a chunk tree is *per-chunk integrity during ranged
|
|
transfer of one large blob* — a property nothing in the current
|
|
consumer set requires, and which cannot be retrofitted onto the
|
|
canonical digest anyway (impossible under the preamble, finding A2;
|
|
iroh gets it from the bao *tree*, not BLAKE3 the algorithm).
|
|
- Storing chunk trees would add a second stored artifact per blob
|
|
(outboard encoding), a second write path, new GC surface, and a
|
|
second hash domain — speculative cost before any consumer names the
|
|
need.
|
|
|
|
Also settled here: local range reads need only transport-integrity
|
|
slice digests (out-of-band, carried by the caller — the store returns
|
|
one from `read_range`); git objects self-verify under git's own model
|
|
when read whole; small-tier range reads slice whole values in memory.
|
|
|
|
## Decision
|
|
|
|
1. **Verification is whole-blob.** Put and get paths hash-check against
|
|
the canonical derivation (ADR-002). `stat` probes length without
|
|
reading content.
|
|
2. **Range reads return `(slice, slice_digest)`** — the slice digest is
|
|
a transport-integrity check carried out-of-band by the caller (e.g.
|
|
the offering side of a fetch). The store performs no per-range
|
|
verification *because none is derivable* under the canonical digest;
|
|
local serving (packfiles, large-file reads) is sound on honest media
|
|
(finding A2).
|
|
3. **No chunk-tree, CDC, or otherwise partial-verification encoding is
|
|
adopted in this crate — by scoping, not by deferral.** The crate's
|
|
domain is the store; a transfer encoding is a network-layer artifact
|
|
and has no owner here. If a future networked consumer (a p2p
|
|
replicator above an un-paused alkgit, alkfs sync, a future ops
|
|
extension) names a requirement for verified ranged fetch of one
|
|
large blob, that requirement is met by a *new decision at a new
|
|
layer*: an optional encoding module (BLAKE3-confined per ADR-002's
|
|
tier 3 if a tree encoding is chosen; content-defined chunking if
|
|
edit-resistance is the goal) whose verified whole-contents register
|
|
back into the pool as ordinary canonical entries — dedup between
|
|
chunked and whole-file paths is preserved by registration, not by
|
|
shared structure.
|
|
4. **The verified-fetch op (ADR-001) carries whole-blob verification**
|
|
as its integrity story: digest in the offer, hash-check on receipt,
|
|
fanout above the store for concurrent subscribers (POC #3 finding
|
|
A3); late joiners degrade to post-commit verified `get`.
|
|
|
|
## Consequences
|
|
|
|
**Positive**
|
|
|
|
- The store stays single-granularity, single-hash-domain, single
|
|
write-path family — the simplest shape consistent with all
|
|
consumers' *current* needs; no speculative outboard/GC surface.
|
|
- Verified transfer exists at every blob size with zero extra machinery
|
|
(whole-blob digest check), which is what the confirmed ops family
|
|
serves.
|
|
|
|
**Negative**
|
|
|
|
- A future large-blob verified-ranged-fetch consumer pays a new module
|
|
and a second decision — deliberately deferred *cost*, not unmade
|
|
decision (the distinction this ADR records).
|
|
- Callers wanting mid-transfer corruption localization on very large
|
|
blobs get error-at-EOF, not error-at-chunk, without an encoding layer
|
|
(acceptable; named above).
|
|
|
|
**Neutral**
|
|
|
|
- BLAKE3's "large-blob encoding consideration" stays conditional and
|
|
out-of-crate; nothing here allocates an API slot for it.
|
|
|
|
## References
|
|
|
|
- `docs/research/phase-0.md` OQ-BL-04 (posture + surface addendum
|
|
promoted here and to ADR-003/store-api)
|
|
- `docs/research/poc-largeblob-findings.md` findings A2/A3
|
|
- `docs/research/iroh-blobs-eval.md` §Verification
|
|
- ADR-001 (ops surface; verified-fetch op), ADR-002 (digest + preamble
|
|
consequence), ADR-005 (range reads in packfile serving)
|
|
- [store-api.md](../store-api.md); [ops-surface.md](../ops-surface.md) |