alkgit/docs/research/git-protocol.md

Research: git smart protocol (server side) — what we must implement

Status: initial pass complete Date: 2026-09-19 Sources: gitoxide source reading (client-side parsers as reference), git http-protocol + packfile protocol specs (git-scm.com protocol docs), observation of gitserver's shape as a sanity check that the surface is this small (no code reused — MPL-2.0 incompatible with our MIT/Apache-2.0; see license-note.md).

TL;DR

The server-side surface is well-bounded: capability advertisement (V0/V1 and V2 flavors), ls-refs, fetch negotiation (wants/haves → acks → pack), receive-pack (pack ingestion + ref update CAS + status report), plus the stateless HTTP framing and the SSH exec framing around them. gitoxide gives us every primitive (pkt-line codec, pack parse/write, ref transactions, fsck); what's missing is only the orchestration, which is ours to write — and that's the layer where our auth/ACL model lives, which is exactly where we want to own code.

Protocol surface inventory

V2 (the target; what modern git speaks first)

1. Client sends GIT_PROTOCOL=version=2 (SSH: env line / http: header).
2. Server replies with capability advertisement: version 2, then capabilities as pkt-lines: agent=..., ls-refs=..., fetch=... (shallow, filter, sideband-64k, packfile-uris...), object-format=sha1. Observed (POC-1): the advertisement is sent once per session; commands follow without re-advertisement.
3. command=ls-refs with args (peel, symrefs, ref-prefix) → server streams ref lines then flush.
4. command=fetch with args (want lines, have lines, done, thin-pack, no-progress, include-tag, shallow/deepen...) → server sends acknowledgments section (acked ids or NAK), then either ready + packfile section (pack streamed over sideband) if done, or waits for more haves. Observed (POC-1, git 2.43): on the done path the response is just packfile\n + sideband chunks + flush — no acknowledgments, no ready. The acknowledgments section appears on multi-round negotiation (no done), not on the done path.

git:// framing detail (observed, POC-1)

First pkt-line: git-upload-pack <repo>\0host=<host>\0\0version=2\0 — service + repo, NUL-separated extras, an empty segment, then version=2. The repo name arrives in-band before any ref data; a server must resolve it against the registry (and run ACL) before emitting the advertisement.

V0/V1 (fallback for old clients / http stateless)

• First response line: "<capabilities> <null-octet> <ref> <obj-id>" style advertisement with refs, then fetch loop: wants → haves (NAK/ACK) → pack over sideband. Stateless-http variant requires the client to POST a separate request for each round; ack state must be re-derived or the multi-round negotiation declined (we can require V2 for http and keep V0/V1 for ssh only — OQ candidate).

receive-pack (push; mostly version-independent)

• Client sends update requests (<old> <new> <ref>) + optional shallow lines
  • pack stream (possibly thin).
• Server: validate CAS per ref (old must match current unless zero-id create), fsck/connectivity the pack, apply ref transaction atomically, reply with unpack <ok|ng> + per-ref ok|ng <ref> <reason> lines.

Framing per interface

• SSH: single exec request git-upload-pack '<path>' / git-receive-pack '<path>' / git-upload-archive; pkt-line on stdin/stdout; V2 via env var; sideband on fetch.
• HTTP: GET /info/refs?service=<name> (advertisement in # service=git-upload-pack preamble for smart clients), POST /<name> with pkt-line body; content-type application/x-git-{upload,receive}-pack-*; chunked streaming both ways; Cache-Control: no-cache.

Server-side pack generation — the crux

gitserver (MPL-2.0, read-only reference) demonstrates the pragmatic path: compute the pack from wants/haves and hand it to the response writer — they use gix + hand-rolled protocol_v2 (742 LOC) and receive_pack (549 LOC) for the whole protocol surface. We cannot copy that code, but its size confirms the surface is small enough to own outright with gitoxide primitives:

• Pack generation options to POC: a. gix-pack bundle::write with an in-memory sink (verify it can stream, not just write files). b. gix-pack::data::output::bytes entries-to-bytes writer fed by an object walk over gix-odb (full control over want/have closure; no file intermediates). Resolved (POC-2, poc2-findings.md): (b) wins — it is gitoxide's own generation pipeline (count::objects → entry::iter_from_counts → bytes::FromEntriesIter, as composed by gix pack create) and streams to any io::Write with O(counts) memory. (a) is not a generation tool: it indexes an existing pack stream (receive-pack's job) and always writes a temp file (it mmaps the pack to resolve deltas for the index).
• The closure is two stages: commit ancestry (gix_traverse::commit::Simple, Parents::All, tips peeled to commits) feeding ObjectExpansion::TreeContents (which expands each commit's own tree but does not follow parents). Wants alone through TreeContents produce an incomplete pack that git index-pack --strict rejects.
• odb handle prerequisites for generation: prevent_pack_unload() + ignore_replacements = true (asserts/corruption otherwise). gix_odb::Cache is not Sync (per-thread RefCell caches): share the Arc<Store>, build a handle per session, generate on blocking threads.
• Delta behavior: existing pack deltas are copied through (from_pack_entry); loose objects are written as compressed bases. No delta synthesis exists upstream (gix-delta is apply-only) — an optimization backlog item vs git upload-pack, not a correctness gap.
• Missing objects during generation do not abort by default (they become skipped "invalid" entries); the server must check entry statistics (missing_objects) and abort the fetch with a sideband error band instead of emitting a broken pack.
• Thin packs (delta against client haves) are an optimization — v0 can declare thin-pack unsupported initially; V2 fetch arg parsing must still accept/decline it gracefully.
• filter (partial clone) and packfile-uris can be declined in v1 of alkgit; advertised capabilities must be honest (never advertise what we don't serve — the gitea-class bug pattern is promising something and enforcing elsewhere).
• Fetch-argument inventory, capture-corrected (OQ-19 rider 3, core/w5-oq19-real-client-tokens, real git 2.43.0 against the landed machine — see oq-19-wire-validations.md §3 for the outcomes): the docs-grammar list above ("include-tag" among the common args) is the grammar, not the traffic. Captured live: the capability pre-lines a real client sends on command=fetch are exactly agent= + object-format= (session-id= only when the server advertises the session-id capability, transfer.advertiseSid config; none of shallow/filter/server-option — git's client gates all three on the advertisement's fetch= features), and the argument grammar is thin-pack/no-progress/ofs-delta + wants/haves/done — plus include-tag on configured fetches (transport.c v2.43.0: args.include_tag = data->options.followtags; builtin/fetch.c sets follow-tags for every fetched tracking refspec, and the V2 send path does not gate the token on the advertisement — there is no fetch= feature slot for it, so no honest-retraction option exists in V2's grammar: a server's only non-breaking postures are accept-as-no-op or breaking real clients; alkgit accepts and honors by omission).

Negotiation policy (ours to define)

Minimal correct initial policy:

• V2: respond with full acknowledgments (common ids) each round; send pack on done.
• Enforce max rounds/haves budget per session (DoS bound), max pack size for receive, wall-clock limits for long negotiations.
• Shallow (` deepen/shallow lines) is a v2-later concern; reject with a clear pkt-line error initially (honest capability advertisement again).

Security-relevant protocol notes

• git-upload-archive is a separate command — do not serve it initially.
• Path traversal: repo names on the wire (git-upload-pack '~/x' or ../) must normalize to a registry lookup — never to a filesystem path from the wire. Repo IDs resolve to storage roots configured server-side.
• receive-pack ref names: validate against git ref rules (no .., no control chars, no refs/heads/foo.lock games) — gix-ref name parsing plus our own deny-list for e.g. refs/ reserved namespaces.
• The advertisement phase must run ACL before emitting a single ref line (ref names leak info; private repos must not advertise anything).

Open items → architecture OQs

• OQ: support V0/V1 at all on http (or V2-only http, V0/V1 ssh)?
• OQ: shallow clone support timeline.
• OQ: thin-pack on fetch (requires delta against client have set).
• Resolved: pack streaming composition (POC-2 decided: entries-to-bytes pipeline; see poc2-findings.md).
• OQ: delta synthesis for loose-object serving (optimization backlog).
• OQ: object-format=sha256 support policy (feature flag exists; no real ecosystem need yet — default sha1, keep flag).