alkgit/docs/research/poco-findings.md

POC-O findings: alkblobs as the git object store

Status: complete — proceed with DQ-1a's shape as amended Date: 2026-10-17 Mode: standalone (POC crate at /workspace/alkgit-poco) + a throwaway prototype branch in the sibling (alkblobs branch poc-o-git-objects, commit f0b045c — the DQ-1a registry extension, prototyped there so the POC could test it; the branch is a scratch artifact, its merge to alkblobs main belongs to the ADR conversation). Landing note (2026-10-18): the extension landed on alkblobs main per ADR-018 (docs/architecture/decisions/018-git-object-derivation-family.md — registry bytes 0x03/0x04, the loose object preimage, put_git_object local + fleet, blob-family-only from_git_oid_hex); the prototype branch was deleted after parity confirmation (alkblobs review 002's disposition). The POC suite re-ran green against the landed main surface the same day (alkcall 0.10; all 8 tests, both gates included) — the evidence here holds verbatim with main as the reference. Migration note (alkblobs 0.2.0 — 2026-10-19): ADR-020 folded put/put_git_object into the one derivation-named seam put_as(algorithm, len|None, stream) — the git-object family through Algorithm::GitObjectSha1 at odb.rs/ingest.rs, the generation sink's pack put through put_as(GitBlobSha256, None, …) (the unknown-length arm — facade territory now that the wire put requires a mandatory pre-keyed offer). The DQ-4 composition (substrate-native fetch + unpack-to-loose ingest) is untouched; the suite re-ran green the same day. Depends on: POC-1/POC-2 (the wire path and the generation pipeline, poc-1-findings.md/poc2-findings.md — both reused), substrate-recomposition.md (the POC plan, DQ-1/2/4/7), ADR-018 (pinned signatures), ADR-013 (receive-pack machine).

Hypothesis (restated)

A gix odb fronted entirely by alkblobs (loose git objects under their own oids — registry extension per DQ-1a) can drive the full protocol: fetch of a real repo over the wire from substrate-backed storage, and a push round-trip with CAS refs — including thin-pack bases resolving from an odb fronted partly by substrate blobs, pack generation into a large-tier blob served ranged, and the ref-placement question (rows vs files) with sqlite co-tenancy measured.

Result

Confirmed on every gate, with two course corrections: one protocol fact (the packless delete-only push) and one design simplification inside DQ-1 (one derivation family, not a kind matrix).

The stack under test (git 2.43.0 as the client; alkcall 0.9 bridge over a plain TCP listener at POC time — 0.10 at the 2026-10-18 re-run, the POC-1/2 wire shape):

git clone/push → tcp → alkcall Connection::from_bidi(stream, "alk/git")
  → BiStream → pkt-line codec (fetch: gix-packetline; push: hand-rolled
  reader — lines then raw pack, position control)
  → upload-pack V2 (advertisement, ls-refs, fetch-wait-for-done)
     pack: generated over SubstrateOdb (a custom gix_pack::Find over
     alkblobs) — direct sideband, or generated once → CAS put →
     ranged read_range windows (through-store mode)
  → receive-pack V0 (ADR-013's machine):
     advertisement → commands → pack stream → Bundle index (thin bases
     from the substrate odb) → unpack to loose-form objects in alkblobs
     (pins bridging to the ref apply) → connectivity walk → ref-table
     CAS → band-1 status report

Check	Result
git clone (V2) from substrate-only storage, sqlite + dual-tier (sqlite+local) engines — and the mem engine	completes; git fsck --strict clean; annotated-tag checkout ok — both fetch modes
through-store fetch (generated pack → CAS put → ranged windows on the wire)	clone 0.5 s incl. the CLI spins; fsck --strict clean; per-window slice digests verify mid-flight
cached-pack reuse	the second identical request rides the cache entry (no regeneration); the pack is an ordinary canonical-key blob (0x01 family)
real client push (default thin pack) over the wire	pack ingested with thin bases resolving from the substrate odb, objects verified into the store under their git oids, refs/heads/main CAS-applied; second clone fsck-clean
delete-only push (git push :refs/heads/dev)	works after the packless-push handling (the protocol finding below)
force push (non-FF)	applied per the v1 CAS-only policy (client-displayed [new branch]/forced update)
stale-old push (CAS mismatch)	ng <ref> stale info — row-level, one transaction
atomic push (one bad expected → nothing applied)	rollback verified; per-ref outcomes atomic push failure
GC: cache release → sweep	the pack blobs (and unreachable objects) reclaimed; referenced closure survives (SweepReport.aborted: None)
orphan objects (no ref)	swept; referenced survive
packgen scale over the substrate odb (mem engine, release)	6,000 objects → 49.6 MB pack in 3.3 s; 9,000 objects → 111.8 MB in 7.1 s (no delta synthesis — the recorded backlog; content-heavy fixture inflates bytes)
read amplification (release)	cold: 31 objects' full-db pass = 1 ms (sqlite); memo-warm 2,000 lookups < 1 ms; disk odb baseline 2,000 lookups = 44 ms
DQ-7 sqlite co-tenancy (in-process ingest+CAS latency, release)	no measurable contention: baseline median 5 ms vs co-tenant-loaded median 5 ms (5 pushes × 39..75 objects) — with alkstore's scheduler + queue + 3 watcher pools on the same file

The DQ-1 shape that survived: one derivation family

DQ-1's option (a) as written ("extend the registry with the non-blob git object forms") needs less than a kind×hash matrix: git's own oid function is uniform —

  oid = H("<type> <len>\0" + content)        # the loose-object form

so the registry extension is one derivation family whose stored preimage is the loose-object form itself (the type/length preamble lives inside the stored value):

• key = [tag] [digest], digest = the object's git oid — any kind, blobs included. Two new registry bytes: 0x03 git-object-sha1, 0x04 git-object-sha256 (sha1 is the only one exercised; sha1-only gix builds cannot name sha256 oids).
• put_git_object(algorithm, len, stream) — the known-length put path with a plain (no-preamble) hasher. Known-length only: a git object's length is always known before its content (both the pack-entry and the loose-file forms carry it), which matches the store's preferred put form.
• get verification is automatic: read.rs re-derives with key.algorithm() — the git-object arms derive over the stored preimage byte-identically, so every read verifies against git's own oid function. Verification-on-get, dedup-by-construction, and external-oid addressing (key = [0x03] + oid bytes, no map) all survive unbroken.

What the prototype branch carries (all validated against the git CLI, byte-exact, tests/git_interop.rs): the two Algorithm members, the derivation arms + streaming hasher arm, put_git_object, key-decode of the new tags, and the interop tests (tree/commit/tag against git hash-object -t, round-trip, dedup, and the negative: the same oid is not reachable through the blob tags).

Consequences found while building on it (the upstream ADR's input):

• Value shape is the loose form — <type> <len>\0 + content. The header must live in the value because a lookup by digest cannot carry a kind; the odb layer parses the header back out (cheap, and it also makes stat()'s length the loose-form length — a semantic difference from the blob family worth pinning in the ADR: stat returns the stored preimage length, not the content length).
• The derivation is preimage-identity hashing (no domain separation the store itself adds). This fits ADR-002's structure only as an annotated exception: the separation is git's own header, inside the preimage. The ADR-002 invariants that survive verbatim: digest length fixed per algorithm, rejects-never-guesses key decode, total order (0x03/0x04 sort after the blob members), verification-on-get.
• Namespaces: git oids are global by nature (same content = same oid everywhere — git's own global dedup); the alkblobs global-pool caveat (ADR-017 there) applies unchanged, with the same accepted posture for git objects as POC-L accepted for LFS content.
• Fleet puts not wired in the prototype — the fleet branches' hasher sites still hardcode the canonical blob derivation. The upstream design work threads the derivation through those paths (mechanical, same shape as the non-fleet ones).
• The ops surface (blobs/*, Digest::from_git_oid_hex) needs the same decision: a kind-aware key constructor (40-hex → which family? cannot be answered by length alone once non-blob objects exist). Recorded as part of the same ADR, not prototyped.

DQ-1b (the oid↔key map) was not built here — and doesn't need to be: POC-L's LFS mapping table IS that pattern, already built and measured, and it remains the right shape for its own case (LFS oids are raw hashes, not git derivations — no derivation family can absorb them). For loose git objects, option (a) is strictly better: no per-repo map rows, no map-side liveness, no refcount dance, verification intact, and the upstream cost is one ADR + this measured patch.

DQ-2 (refs): rows won

The ref table is one co-tenant sqlite table ((repo_id, name) → oid, symref_target, HEAD a symref row). Facts:

• The one-transaction CAS home maps directly: apply_updates runs all CAS checks and applies inside one BEGIN IMMEDIATE; the atomic push is a commit/abort, not a flock dance. Demonstrated: two clashing applies (one wins, the loser is stale info), an atomic rollback leaving the table untouched, delete/create/update.
• The serving path holds no file locks at all — the last disk dependency of the object+ref design is gone when the object store moves into the substrate; ref placement follows it.
• gix-ref's file-backed machinery drops out entirely on this shape; its battle-tested transaction work is replaced by the row-CAS (which needs no battle-testing — it is one upsert inside one tx).
• Pinned-type fit: the ADR-018 GitRefs shape (list_refs/apply_updates, RefUpdate/RefOutcome) is exactly what the POC implemented — the seam survived recomposition with zero signature drift.

DQ-4 (pack pipelines): the surviving composition

Fetch side — proven over a custom gix_pack::Find, no disk odb anywhere. gitoxide's generation pipeline (count::objects_unthreaded → entry::iter_from_counts → InOrderIter → bytes::FromEntriesIter) is Find-generic: the same composition POC-2 proved against the disk odb runs against SubstrateOdb (which implements both gix_object::Find + Exists and gix_pack::Find — the only custom-Find notes: location_by_oid / entry_by_location / pack_offsets_and_oid return None (no packed objects), and the pipeline's FindExt blanket impls pick up the rest). The two-stage closure (peel → commit ancestry → TreeContents) is unchanged. Two serving shapes measured, both gate-clean:

1. direct: generation streams through SidebandSink — the POC-2 shape, no storage round trip;
2. through-store (the DQ-4 shape): stream the generation into store.put(None, …) (the store's own two-tier dispatch routes big packs into the large tier), then serve from the stored blob through read_range windows, per-window slice digests verified mid-flight. The pack is an ordinary canonical-key blob; a small request-digest cache makes repeat fetches ride it; cache eviction (or a demo release_all) releases the pins and the next sweep reclaims the pack. GC interop verified end-to-end.

Ingest side — the ADR-013 composition survives substrate-backed objects, with one deviation: thin bases resolve through Bundle::write_to_directory's lookup (the substrate odb — pack deltas against objects the server already has); the indexed pack then unpacks into alkblobs as loose-form objects (the store's verification-on-put IS the oid re-derivation — corruption cannot enter undetected), held under one pin set bridging ingest → CAS apply — the cross-store-atomicity window (DQ-8) closed by pins, confirmed working. The deviation: the pack does not land as a server pack with .keep (ADR-013 §6's shape) — the working area (temporary index + pack) is discarded after unpack; the authoritative object home is the substrate. Connectivity: a full graph walk over the substrate odb from the pushed tips — every reachable object must exist, parents included. (Note for the real implementation: gix-fsck's Connectivity::check_commit checks each commit's tree+subtree+blobs but does not walk parent commits by itself; our owning walk does the complete closure — the same two-stage discipline as generation.)

Recorded (designed, not built): pack mirroring (storing the raw pack blob + its index in the store to give pack-copy serving) — needs in-memory pack handles over stored (idx, pack) pairs; unnecessary for correctness (loose serving is complete), deferrable as the delta-serving optimization, same bucket as the delta-synthesis backlog item from POC-2.

The protocol fact: packless delete-only pushes (an ADR-013 amendment)

ADR-013 §3 says a pack is always expected after the command flush — verified against stdio captures, where EOF == stream end. On duplex transports the real client behaves differently, and the POC captured it live (GIT_TRACE_PACKET against real git-daemon/git-receive-pack on an equivalent repo):

• a delete-only push sends the commands + flush and then sends nothing — no pack bytes at all — and waits for the report;
• real receive-pack answers within ~40 ms (it does not block waiting for a pack): unpack ok + per-ref outcomes over band 1 — the captured report bytes decoded exactly to the POC's framing (000eunpack ok 0041ng refs/heads/main <reason> 0000 … 0000 — inner pkt-frame, inner flush inside the band, outer flush), confirming the report composition byte-for-byte;
• an ordinary push's pack header (50 41 43 4b) arrives within milliseconds of the command flush (verified by peeking on the POC bridge).

Server-side handling: a bounded peek for the pack header (750 ms in the POC; a fast push's header is immediate, a packless client is already waiting) — timeout ⇒ packless path (CAS-only flow). The POC's first no-peek implementation deadlocked exactly as real-stdio-logic would predict; git daemon then proved the packless shape. This amendment goes to ADR-013 §3 as a duplex-transport clarification (two-way door: more packless situations can only be added, the peek recipe generalizes). Also captured: upstream refuses deleting the current branch (HEAD) by default (receive.denyDeleteCurrent) — a policy we don't have; v1 accepts any deletable name per ADR-013 §7, and the HEAD-protection policy belongs in the assembly's config surface if wanted.

Also confirmed live: with atomic negotiated but one ref failing a name check, the whole push aborts with per-ref atomic push failure; the SelectOptions from the client's first command line carried report-status-v2 side-band-64k quiet — and the band-2 line (remote: error: … text) appears exactly as the captures documented. quiet suppresses progress, not errors — the POC sends none of the former and observes the latter.

DQ-7: sqlite co-tenancy measured (no contention observable)

The deployment shape as built: one sqlite file co-hosting (a) alkblobs' kv engine pool, (b) the ref-table connection (its own WAL connection, busy_timeout 5000), (c) two alkstore engine instances (scheduler runner + the machinery's own pool), (d) a listener holding the watcher's data_version polling (~1000 idle polls/s at the default 1 ms cadence). In-process measurements, release build, 5 distinct pushes of 39–75 objects each:

leg	median ingest+CAS (ms)
git assembly alone	5
+ alkstore scheduler (1 s cadence) + queue worker + listener	5 (4–6 range)

Debug-build numbers: 28 vs 26 ms (also indistinguishable). Zero contention events at this scale; the honest caveat is that the sqlite writer serialization is per-file (verified by design, not by saturation — a heavier parallel-push load would surface it, not measured at POC scale). The 1 ms watcher cadence is the assembly-level tuning point DQ-7 named; the POC accepted defaults and did not tune (a heavier deployment should raise it — the alkstore docs' knob).

Also measured: the wire adds a ~50 ms git-CLI process floor per transport op (clone and push alike, both profiles), dwarfing the in-process costs at these object counts — a datum for interpreting any wire-level benchmark at small scale.

The trait posture (DQ-6/ADR-018): the seam held

The whole wire path consumed exactly the ADR-018 family shape — GitRefs (list_refs, apply_updates with RefUpdate/RefOutcome), GitPackGen (generate with a Box<dyn io::Write + Send> sink), GitPackIngest (prepare → PreparedPush.updates, never applying) — with zero signature drift from the pinned shapes. The registry pair (traits 1–2) was out of scope (POC-R's). Two posture facts for the re-baseline:

• The sync Find / async store bridge is Handle::block_on from the blocking pipeline task (gitoxide's Find is sync by design; the substrate is async). Works in spawn_blocking contexts; panics from async contexts ("cannot start a runtime from within a runtime") — the liveness source's walk must bridge with block_in_place instead (POC learned this the live way; the eventual GitPackGen/GitPackIngest impls should own this context rule once, in one place).
• The ADR-018 async-trait posture stayed out of the POC (no traits were declared) — the POC's fns are plain async; the seam fit is measured by shape-compliance, and the DQ-6 macro-posture choice remains open for the re-baseline round.

What this settles / does not

• DQ-1: option (a) with the single-family amendment — validated end-to-end (byte-exactness vs the git CLI, verification, dedup, serving, ingest). The alkblobs ADR + v0.1 ask has its evidence; the throwaway branch is the reference prototype (not to be merged as-is: the ADR round re-implements with the fleet/ops wiring).
• DQ-2: refs in rows — settled for this shape (relational ref table, one-txn CAS); gix-ref file storage drops from the substrate shape.
• DQ-4: fetch = substrate-native generation (custom Find, both serving modes) + stored-pack ranges; ingest = index w/ thin bases from the substrate odb → unpack-to-loose → pins bridge → CAS. ADR-013 survives with the packless-push amendment and the unpack-to-loose deviation recorded.
• DQ-7: co-tenancy measured — no contention observable at POC scale; connection-budget datum stands (worst case four pools + a ref-table connection).
• Not settled: multi-round negotiation (the fetch=wait-for-done shape only — common_haves' seam untested; POC-X territory with replication), delta synthesis (the standing backlog), object-info/partial-clone, sha256 end-to-end (the registry byte exists; the gix side is sha1-pinned per OQ-05), the pack-mirror serving shape (designed, deferred).
• POC-R stands entirely open — this POC consumed one repo id in a in-process table; the reactive composition (wakes, invalidation cache, scheduler-driven sweep + housekeeping) is untested.

Upstream asks (updated)

1. alkblobs hash registry extension: evidence now exists — the single git-object family (0x03/0x04) is the proposed shape (the throwaway branch poc-o-git-objects is the working prototype plus byte-exact CLI-validated tests). This is the concrete v0.1 sneak-in candidate the user held publication open for; its ADR conversation happens at the sibling. Landed 2026-10-18: ADR-018 + the family on alkblobs main, branch deleted.
2. The ADR-013 amendment (packless delete-only pushes on duplex transports) — an alkgit-doc change, not an upstream ask.
3. The busy_timeout ask (#4 in substrate-recomposition) stands unchanged.
4. New datum for the assembly docs: the wire floor (~50 ms/CLI op), the large-tier pack threshold (kv-only deployments reject over-threshold packs — LFS ⇒ large tier present transfers to git packs when through-store serving is wanted), and the sync/async bridge rule.

POC art

• Crate: /workspace/alkgit-poco (standalone; path deps: alkblobs from the prototype branch checkout, alkstore/alkstore-sqlite; alkcall 0.9 published; gitoxide components pinned per gitoxide.md).
• src/odb.rs — SubstrateOdb (the custom Find over alkblobs; loose-form encode/parse; the object graph's children).
• src/refs.rs — the co-tenant relational ref table (CAS/atomic/HEAD symref) + the liveness source (closure from refs, memoized by version).
• src/packgen.rs — generation over any Find + the pack cache (get-or-generate → store → ranged serving → pins).
• src/ingest.rs — receive-pack ingestion (budgeted stream → thin-aware index → unpack-to-loose with pin bridge → full connectivity walk).
• src/serve.rs — the two machines (V2 upload-pack / V0 receive-pack) over the alkcall bridge.
• tests/e2e.rs — the gates (clone/fetch/push/CAS/atomic, through-store + cached pack + GC, large-tier ranged serving, orphan sweep).
• tests/dq7_contention.rs — the co-tenancy measurement + the odb read costs + the scale datum's fixture.
• alkblobs: branch poc-o-git-objects (commit f0b045c) — the DQ-1a prototype; alkblobs main untouched. (Since: landed on main per ADR-018, branch deleted; re-run green against main 2026-10-18.)