alkgit/docs/research/pocx-findings.md

POC-X findings: two-node replication on substrate-backed stores

Status: complete — proceed (both transfer paths converge; the GitSession consumer half has its first working evidence; ADR inputs ready) Date: 2026-10-18 Mode: standalone (POC crate at /workspace/alkgit-pocx; no repo code touched). Deps: published-surface alkblobs (0.1.0, incl. the landed ADR-018 git-object family), alkstore family (path, 0.1), alkcall 0.10, and the 0.89-family gix pins of alkgit's own manifest — the consumer-half work ran against the pins the manifest was moved to (gix-transport 0.61, gix-protocol 0.67, gix-pack 0.76, gix-odb 0.86, gix-hash 0.28, gix-packetline 0.24, the gix-error error model). The POC-O/P parts of the crate (odb/refs/fixture/ ingest/serve, record/registry) were ported across the pin family — their drift is small and recorded below. Migration note (alkblobs 0.2.0 — 2026-10-19): ADR-020 — the family-explicit wire payloads, resolved from review-003 whose finding this POC's ops_gap evidence substantiated — migrated the POC: the facade unification (put_git_object → put_as(algorithm, …)) touched the four call sites (odb.rs/ingest.rs/mirror.rs put handler via Algorithm::GitObjectSha1; packgen.rs's generation sink via put_as(GitBlobSha256, None, …), the unknown-length arm). The replication data plane was already facade/mirror-plane, so nothing wire-shaped changed there; the manifests pin alkblobs = { version = "0.2", path = … }. tests/ops_gap.rs is inverted to pin the fix (see the gap section below); the sibling's ops surface carries the re-homed pinning test (alkblobs commit 7ad2384). Suite re-ran green the same day. Depends on: substrate-recomposition.md (the POC plan, the POC-X row), POC-O findings (the server halves + the receive machine), POC-R findings (the reactive recipe this driver rides), ADR-017 (the consumer half under test).

Hypothesis (restated)

Two nodes, each substrate-backed (alkblobs for objects, rows for refs), mirror each other: a real git push lands on the source's durable change feed; a replication driver (the GitSession-shaped client of ADR-017) carries the diff to the peer, and the peer converges — provable end-to-end with a real git clone fsck-passing from the replica. The open questions: replication completeness under the stream's wake-vs-offset tolerance, and the idempotence recipe. The POC also asked which transfer path survives: git-protocol-native (pack transfer through alk/git) or substrate-native (the blobs/have diff + per-object transfer through a replication plane).

Result

Confirmed on every gate, with one decisive simplification (the diff-driven pass makes replication idempotence nearly free), one protocol-plane finding (the blobs/have wire gap), and one transport-level fact (the RST discipline).

Stack under test: node = alkblobs (kv sqlite / mem) + the row-CAS ref table (POC-O's RefTable) + the substrate odb; source node =

• alkstore-sqlite/mem + the co-tenant registry + the receive-flow announcer; real git 2.43 as the producing/consuming client.

Gate	Result
Two-node mirror converges on a pushed commit	yes, end-to-end: real git push into A → receive-pack (ingest + connectivity + row-CAS) → registry row + event → replicator → GitSession push to B → real git clone from B, fsck --strict clean, the pushed commit checked out — in-process and cross-process (tests/two_process.rs: B in its own binary serving git + mirror plane over real TCP; A's replicator mirrors over TCP; the test only drives git)
The incremental push's pack carries only the delta	the have-aware trim's diff shape measured: initial mirror pack 16 objects / ~1.4 kB, incremental pushes 3 objects / ~0.4 kB (the new commit's commit+tree+blob) — the peer's known closure rides haves
Idempotence recipe	a pass re-run is a noop: the ref-level diff yields zero commands (no pack generation, no push dial); content-addressed puts are CAS-idempotent; per-peer seq bookmarks advance only after a successful pass
Stream wake-vs-offset tolerance	the event leg is latency only: a pass whose event never published (announce suppressed after the row committed — POC-R's crash-between-commits window) is closed by the reconcile tick driving the laggard peer (peer_seq < registry watermark → pass → converge), demonstrated live
Partial-transfer tolerance	the native path's crash window (objects transferred, refs never applied) closes with a strictly smaller need set next pass — demonstrated airtight: N-object need set, 2 pre-landed, next pass transfers exactly N−2
Delete propagation	the mirror policy (source authoritative) deletes peer-only refs on both paths: real git push :refs/heads/dev into A → B's ref goes too (protocol path, packless delete push client-side); the native path's apply CAS deletes
Multi-round negotiation (the seam POC-O left open)	rides now: the pull path's fetch client ran 2 no-done haves rounds + the done round against the wait-for-done server; Arguments::send(false)/Response::from_line_reader machinery handles the loop; each round's acknowledgements parse (NAK-only — the server tracks no common haves, see below)
Substrate-native transfer (the mirror plane)	converges identically: git-object-keyed have diff + per-object verified transfer (put_git_object — the oid re-derivation is the transport's integrity check) + CAS apply + HEAD symref; first pass 16 transfers, delta pass 3
blobs/have as the wire verb for the diff	does not compose for git objects today — see the gap finding below

The consumer half (ADR-017): what the machinery wanted

The ADR-017 seam exists and works, with one simplification worth a line in the eventual implementation ADR:

1. The "custom alkcall transport" is not a new trait impl. gix-transport's client::git::async_io::Connection (the git-daemon transport) is generic over (R, W) — any futures-io read/write pair. The alkcall BiStream split halves, compat-wrapped, are exactly that pair. ADR-017 §2's "one impl, two construction modes" maps onto ConnectMode as-is: Daemon = the request-line preamble (connect_direct's shape), Process = the transport writes nothing (open_via_channels — the preamble rode the alkcall open-op). The GitSession implementation task therefore wraps a construction, not a trait — less code, same story.
2. Fetch rode Arguments + Response directly. The full fetch() (the Negotiate machinery: RefMap, refspecs, the commit-graph) was deliberately NOT exercised — it is the implementation-phase layer on top; the machinery the wire needs (per-round argument assembly, stateless V2 commands, ack parsing, the sideband pack file leg) all works over the custom transport. The negotiation loop structure is exercisable without the RefMap layer (rounds are full stateless commands; the client re-sends wants+haves each round and terminates with done).
3. Push client proven: hand-rolled to ADR-013's grammar (V0 advertisement parse, command lines with caps on the first, the pack raw after the command flush, the status report parsed in the band-wrapped form). The packless delete-only fact (POC-O) holds client-side: commands + flush + nothing, then wait for the report — and the write side must stay open until the report: a client that half-closes after a packless push turns the server's bounded pack-header peek into an EOF error instead of the packless timeout (the server's peek recipe presumes the real-client waiting shape).
4. The 0.89 family's client tree compiles and drives (the error model moved to gix-error; Progress moved to gix_utils::progress; the response reader's sideband decoding activates via set_progress_handler on the ExtendedBufRead reader — the pump pattern: after from_line_reader returns the packfile position, setting the handler and draining fill_buf yields verified band-unwrapped pack bytes).
5. One alkcall connection = one service machine; the V2 session persists across ls-refs → fetch on its stream; push rides its own stream. Matches the ADR-016 preamble-per-stream shape.

The have-aware trim: packgen's haves get their machinery

POC-O's packgen accepted haves but never used them (dead plumbing — the generated pack always carried the full wants closure). The POX implementation:

• the peer's known tips → their full closure walked over the source odb (closure_set — commits via parents, trees via entries, tags via targets);
• commit-ancestry pruning rides gix_traverse::Simple::filtered's predicate (a commit in the peer's closure is not emitted and its ancestry walk stops there);
• the count stage (TreeContents) runs over the trimmed commits, and the peer's tree/blob closure filters the counts before the entry iteration.

The result is correct-by-closure trimming (no thin-pack machinery needed at POC scale); it served BOTH directions — the push client (peer tips via ls_refs) and the server fetch (the request's have lines). Measured: full pack 1.4 kB → incremental 0.4 kB; the substrate odb's memo cache absorbs the re-walk (per-pass closure walk was milliseconds at POC scale).

The blobs/have wire gap (the DQ-1c-class finding)

(Since 2026-10-18: the gap CLOSED — alkblobs review 003 took this finding and ADR-020 (alkblobs 0.2.0) now ships family-explicit payloads: the wire speaks tag-prefixed key hex, put offers carry a mandatory pre-keyed claim. The reproduction below is the historical evidence; tests/ops_gap.rs is inverted to pin the fix — a family key crosses the 0.2 ingress and reaches its entry, while the same digest under the other family's tag still forms a different address.)

The POC plan's row ("blobs/have diff drives object transfer") met a structural wall: the ops surface is blob-family-scoped (alkblobs ADR-018 §Decision 2 — "a family-explicit wire form is a named-consumer + new-ADR door"). The reproduction (tests/ops_gap.rs): a git object stored under its own oid key; the ops-ingress parse (Digest::from_git_oid_hex → Key::from) maps a 40-hex git oid onto the blob-family key — a different tag byte — so have misses the very entry that visibly exists. Consequences for the design:

• wire-expressible replication over blobs/* needs a family-explicit ingress (a new ADR at alkblobs: which family, what wire form — e.g. a type-tagged digest form) — with put/fetch following (blobs/put verifies under the canonical blob derivation, so even offered git loose objects cannot land by git oid);
• OR replication stays protocol-native: the GitSession path needs no blobs ops at all — the have/need diff rides ls_refs + have-closure trimming inside the pack generation, transfer rides the pack, application rides receive-pack's CAS. All machinery measured here.

POC-X's recommendation to carry into the ADR round: protocol-native replication is the v1 posture (less new surface, one tested wire plane, and the git protocol's own have/negotiation semantics already carry the diff); the substrate-native plane remains the optimization bucket (per-object transfer without pack generation — the delta against pack generation cost, the same bucket as pack mirroring and delta synthesis from POC-O/POC-2) — and the mirror plane (the POC's alk/mirror shapes) is the honest sketch of what an alkgit-scoped replication op set would look like if the ADR round wants it: refs/apply(CAS)/head/have/put(verified loose transfer)/fetch. Note its two alkgit-specific verbs beyond git's grammar: the HEAD symref row (outside git's push grammar — a mirror must replicate it or every clone of the replica is checkout-less) and peer-only-ref deletion (the mirror policy).

The idempotence recipe (the named deliverable)

The recipe that survived every tolerance gate:

1. The pass is diff-driven and re-runnable: every pass computes the ref-level diff first (peer refs via ls_refs/mirror-refs vs the source's row table); zero diff ⇒ zero work (no dial, no pack). A re-run is a noop by construction.
2. Content-addressed puts are CAS-idempotent — re-transferred objects are no-op puts; verification (oid re-derivation) is the integrity check on every transfer.
3. Ref application is CAS'd on the peer's current value — a re-apply with the same old/new is either a noop (already applied) or stale info (peer advanced; the next pass re-diffs) — both converge without duplication.
4. Per-peer sequence bookmarks (the registry's monotone seq, applied per peer) decide when runs are needed: the event leg (latency), the reconcile tick over the watermark (the loss closure — exactly POC-R's recipe with the peer as the consumer).
5. No rows_since listing was needed for replication itself — the registry watermark's movement (peers lag ⇒ run) suffices, because the pass is diff-driven and idempotent: the event payload's content is irrelevant to correctness. (This is a real simplification over POC-R's apply-events-by-seq shape: replication state is the peer's own knowledge, not an event log.)

The RST discipline (a transport-level fact, POC-recorded)

The mirror plane chased a ~1/30 flaky ECONNRESET to a crisp cause: a connection closed by drop while its receive side still has unread bytes closes with RST, and the RST destroys the reply bytes still in flight. The mirror route had read only the command line; the client's trailing flush pkt stayed unread; a close-with-unread-data RST'd before the reply could be read. Fix: every command reads its request's trailing flush before replying (the same discipline the git machines already observe implicitly). Recorded as (a) a framing rule any alkgit single-stream session carries (e.g. the receive machine's peek/timeout recipe) and (b) an alkcall-documentation candidate ("drop-with-unread-data closes with RST" — engine behavior, not new machinery; the fix lives in the handler side).

The reactive half, transplanted

The driver's machinery (event leg + watermarks + reconcile) is POC-R's recipe with the peer as the consumer — and the simplification above (watermark movement, not event application). The data_version backstop (sqlite) was not re-probed: the pass runs regardless of the wake's arrival (the wake is a hint; the reconcile tick is the bound), and POC-R's measurements stand.

What this settles / does not

• The POC-X gate: settled — both nodes are substrate-backed, the mirror converges on real pushed commits, with real git on both ends (push produces, clone consumes, fsck is the judge), in-process and cross-process.
• ADR-017's implementation shape: the evidence exists — construction-wrapper transport, Arguments/Response fetch over custom transport, hand-rolled push client, ?Send runtime bound (the replicator lives on a current-thread runtime; the git-server tasks stay on the multi-thread pool — the two coexist over TCP).
• The blobs/have question: answered with the gap finding + the recommendation (protocol-native v1; the plane is the recorded alternative).
• Not settled / deferred: common-haves acknowledgment refinement on the server side (the negotiation's acks stay NAK-only; the trimming is haves-closure-based — sufficient, but a real multi-round negotiator would drive smaller want sets), the Negotiate/RefMap fetch machinery (implementation-phase), divergent two-way replication (v1 is single-direction with the source authoritative; the peer's own pushes are a policy question — out of scope here), sha256 end-to-end, GC interplay across replicas (each replica's own liveness source covers its refs — B's sweep never ate not-yet-referenced objects because the pass applies refs immediately after transfer on the same connection; a scheduled sweep lagging a transfer ride would need the pin bridge — worth carrying into the ADR text for the native path), and a postgres pair leg (the alkstore recipe is engine-generic per POC-R; not re-run here).
• The ported blocks carried small drift: the 0.89 family's gix_error::Result error model (raise-into-error instead of boxed errors), gix_utils::progress types instead of gix_features's, gix_parallel's InOrderIter, count::objects_unthreaded' progress parameter, and one dropped Mode variant (PackCopyAndBaseObjects only). All mechanical; the POC-O shapes survived as-is.

Upstream asks (updated)

None structural from this POC. Two recorded candidates for the ADR round (not asks today):

1. alkblobs family-explicit wire ingress (have/put/fetch for git-object keys) — only if the ADR round picks substrate-native replication at v1; the protocol-native recommendation needs nothing new.
2. An alkcall doc note for the RST discipline (drop-with-unread- data closes with RST — drain before dropping on single-stream handlers); engine behavior is correct, the handler rule is documentation.

POC art

• Crate: /workspace/alkgit-pocx (standalone; README included).
• src/transport.rs / src/session.rs — the GitSession prototype (fetch over gix machinery, hand-rolled push, the pump).
• src/packgen.rs — the have-aware trim + closure_set.
• src/mirror.rs — the substrate-native plane (server + client).
• src/replica.rs — the driver (ref diff, seq bookmarks, event + reconcile legs, both pass bodies, pull-mode).
• src/node.rs — assemblies + serve loops + the registry announcer (with the suppression seam).
• src/{odb,refs,fixture,ingest,record,registry,serve}.rs — POC-O/ POC-R blocks ported to the 0.89 family; serve gains the multi-round loop + the post-CAS announcer hook.
• tests/e2e_mirror.rs — the gate (+ deltas, idempotence, delete propagation, suppressed-announce tolerance).
• tests/pull_mode.rs — the fetch path (full + multi-round trim + idempotent pull).
• tests/native_mirror.rs — the plane (full/delta/exact-remainder crash window/stray-delete).
• tests/ops_gap.rs — the blobs/have wire-gap reproduction (since 2026-10-19: inverted to pin the ADR-020 fix — see the migration note and the gap section's status line).
• tests/mirror_stress.rs — the RST repro probes (kept as the discipline's regression net).
• tests/two_process.rs — the cross-process gate (two demo processes; real git in and out).
• src/bin/demo.rs — node-b / run-a (the demo process pair).