alkblobs/docs/architecture/decisions/002-canonical-hash-and-key-encoding.md

ADR-002: Canonical hash and key encoding

Status

Accepted

Context

The store keys every entry under a content hash. Phase 0 (OQ-BL-03) resolved the hash policy — canonical git-blob-sha-256, tolerated git-blob-sha-1, BLAKE3 demoted — after the inherited premise "iroh uses BLAKE3, so we must" was rejected (convention 7: borrow conclusions, not welds; nothing in the consumer set forces BLAKE3: alkgit forces git's derivation by protocol definition, and every other consumer sees addresses, not algorithms).

The residual was the key encoding mechanics — the byte layout of keys inside backends. This is the crate's declared one-way door (AGENTS convention 5): key bytes persist in kv rows and fs filenames, with no schema-migration story; and this is the wire-format ADR that must exist before the first consumer (convention 7's gate on publishing wire-relevant decisions). POC #1 validated the mechanics empirically: byte-exact git hash-object interop (finding 4, both algorithms, external-oid round-trip into the pool), and tagged-key coexistence of both algorithms in one flat pool (finding 5).

Decision

Hash policy (three tiers)

1. Canonical — git-blob-sha-256: H("blob <len>\0" + content), SHA-256. The store's canonical algorithm; git objects are first-class entries with zero indirection; one algorithm domain = one address space, so non-git consumers dedup into the same pool by construction (ADR-005's premise).
2. Tolerated legacy — git-blob-sha-1: same derivation, SHA-1 — accepted for existing-repo interop (a protocol necessity; git's own collision-hardened SHA-1 threat model inherited verbatim).
3. Excluded — BLAKE3: not present in this crate. It was an iroh-blobs inheritance, not a requirement (its virtues — merkle-native trees, raw throughput — are irrelevant here under the canonical digest at our scales). If a chunk-tree transfer encoding is ever adopted it is a new decision at a new layer (ADR-006 excludes it from this crate's scope by scoping, not by hedge).

Security footnote (carried from OQ-BL-03): one algorithm across consumers is safe here — SHA-256 has no practical cross-protocol ambiguity at these input shapes; the preamble domain-separates the derivation. SHA-1's caveat is git's accepted position, inherited.

Key encoding (the one-way door's bytes)

Physical key layout inside all backends:

key := <algorithm byte> <digest bytes>
algorithm byte: 0x01 = git-blob-sha-256 (32-byte digest)
                0x02 = git-blob-sha-1   (20-byte digest)

• Algorithm identity is inside the key. A per-store single-algorithm configuration would not have survived POC #1's external-oid interop test — external git oids and pooled workspace content must be addressable from the same table.
• The tag byte is dropped (POC finding 5: exactly one key kind exists; the algorithm byte is the load-bearing part). Honesty note: the POC validated the tagged layout [tag][algorithm][digest]; dropping the single tag byte is the decided simplification of that layout (the finding's own reasoning — one key kind — is the evidence the simplification rests on; the algorithm byte, the load-bearing half, is the tested part).
• Fixed length per algorithm (32 or 20 bytes); total key lengths 33/21. Fixed-length enum-tagged keys sort cleanly in total order — required for the sweep's live-set diffs (BTreeMap/range-scan walks).
• Unknown-algorithm and truncated keys are rejects, not guesses (POC's length-checking test is the standard).
• Backends are opaque to all of this: they store &[u8] keys; only the store core constructs/interprets typed keys (ADR-003).

Algorithm bytes are assigned from this ADR's registry; new algorithms require a new ADR (another one-way-door byte).

The preamble as abstraction

An algorithm = domain-separated derivation (its own input preprocessing) + digest. The preamble "blob <len>\0" is not a special case — POC #1's domain-separated-premise abstraction held with no special-casing anywhere. Note the structural consequence: the length is inside the hashed input, which is what forces the two put paths (ADR-003) and caps verification at whole-blob (ADR-006) — both accepted rather than worked around.

Consequences

Positive

• Git oids (either format) address pool entries directly from external producers — no mapping layer, byte-exact (tested against the git CLI, not just vectors).
• One address space across all consumers; dedup is structural.
• Key bytes are self-describing; backend data survives digest-layer evolution (opaque at the boundary).

Negative

• The encoding is frozen once a backend ships data (the door closes); algorithm byte 0x00 is reserved and never assigned (a future encoding revision would need a different key universe or a migration ADR).
• SHA-1 acceptance inherits git's threat model — documented, not re-litigated here.

Neutral

• Variable-length length-prefixed keys were considered (POC finding 5) and rejected for now: two algorithms make fixed-length simpler and sort-stable. Extension to a third algorithm re-opens byte layout — new ADR.

References

• docs/research/phase-0.md OQ-BL-03 (resolution + encoding notes)
• docs/research/poc-trait-dispatch-findings.md findings 4/5
• hashing-and-keys.md
• ADR-003 (opaque byte keys; two put paths the preamble forces); ADR-005 (one address space); ADR-006 (whole-blob verification)