alkgit/docs/research/pocl-findings.md

POC-L findings: LFS end-to-end on alkblobs (via an alkhttp-route door)

Status: complete — proceed Date: 2026-10-16 Mode: standalone (POC crate at /workspace/lfs-poc; no repo code touched — alkgit is still an empty lib, and this POC's whole point is the substrate pair alkblobs+alkcall+alkhttp, all consumed as crates). Published deps: alkcall 0.9, alkhttp 0.7 (path dep — the 0.7 registry copy predates the bearer-wraps-extras behavior; the family crate is co-10ant anyway). Unpublished path dep: alkblobs 0.1 (ops, sqlite, postgres, pg-lo, mem features on). Real client: git-lfs/3.7.0 (release tarball, /usr/local/bin/git-lfs). Depends on: substrate-recomposition.md (the POC plan, DQ-5), alkblobs ADR set, alkcall ADR-047/049 (channel openables / establisher), alkhttp ADR-046 §4 (extras auth layering). Landing note (2026-10-18): the two upstream asks landed — alkcall 0.10.0 threads the ownership provider through the gateway dispatch spine and refuses a resource-gated op loudly when none is wired (ADR-048, review 012 resolved; with_static_ownership_fallback() is the declared opt-out), and alkhttp 0.8.0's door_auth_middleware tolerates Basic credentials on door surfaces (ADR-073; ADR-072 threads the ownership wiring onto the HTTP surface, ADR-046 §1 records the static-routes constraint). The POC suite re-ran green against the landed pair the same day; tests/gateway_gap.rs was rewritten from pinning-the-seam to pinning-the-fix (refusal without wiring / the declared static fallback / live revocable grants through a wired provider) — the seam evidence itself is now the fix's regression record. alkblobs itself published 0.1.0 the same day; the door's hand-rolled OperationContext compensation remains valid as described below, and the door's local Basic parsing can be replaced by the shared middleware. Migration note (alkblobs 0.2.0 — 2026-10-19): ADR-020 — the family-explicit wire payloads, born of POC-X's ops_gap finding — reworked the door; the suite re-ran green the same day (alkblobs = { version = "0.2", path = … } pinned). The load-bearing change: the wire blobs/put offer now carries a mandatory pre-keyed claim (key + len before the stream), and an LFS body's canonical key only exists at EOF — so the transfer's PUT leaves the ops plane entirely: both bindings' uploads ride the facade's derivation-named streaming seam (Store::put_as(GitBlobSha256, len|None, stream), the arm that returns the key when the stream finishes — no staging by the door), binding B remains the loopback for blobs/stat|fetch only, and the wire put's departure is ADR-020 §Decision 3's recorded consequence landing in the one consumer that streamed unkeyed bodies. With the wire put gone, the mislabeled-PUT custody-leak reproduction loses its harness (the unconfirmed-token path no longer occurs in this door; the S-5 attack shape and its ADR-019 landing stand on record). The stat/fetch offers speak tag-prefixed key hex now (the gateway_gap.rs probe payload updated to the 0.2 key form — bare 64-hex dies at the schema gate). One design note ADR-020's registry surfaces for the LFS ADR round: the LFS oid is byte-exact the GitObjectSha256 digest of the raw content, so keying LFS media under the identity family would make the key formable pre-EOF (the wire put could carry LFS uploads again) and would dissolve the lfs_oid → key translation — the referenced-oid liveness row stays needed regardless. One transient IncompleteBody failure in small_request_loopback_overhead_probe was observed once mid-migration (the loopback drain hazard class) and never since across five consecutive green runs.

Hypothesis (restated)

A real git lfs client (push/pull) goes end-to-end through an alkhttp-route door onto alkblobs' ops surface (blobs/stat|have for the batch API, blobs/fetch|blobs/put for transfer), with namespace = <repo_id> and liveness = referenced-oid table; the three gate questions (batch+pointer coverage, token hand-off binding, ranged fetch bounds for big media) get measured answers, not guesses.

Result

Confirmed, with one load-bearing course correction (the LFS oid↔key mapping) and one composition finding (the gateway spine's ownership seam).

The wire path under test (identical to the POC-3 shape, with the LFS door as the extras router):

git-lfs → tcp → alkcall Connection::from_bidi(stream, "http/1.1")
  → alkhttp HttpAdapter::handle → accept_bi → BiStream
  → hyper (h1) → axum Router (LFS routes via with_extra_routes)
    POST  /{repo}/objects/batch        (spec batch API)
    PUT   /{repo}/lfs/objects/{oid}   (basic transfer upload)
    GET   /{repo}/lfs/objects/{oid}   (basic transfer download, Range-ful)
  → batch: door builds its own OperationContext (ownership provider wired)
    → OperationRegistry::invoke("blobs/stat", ...)
  → transfer: two bindings measured (below)
  → storage: alkblobs Store (mem / dual_tier_sqlite_local /
    dual_tier_postgres_local engines); liveness: postgres table

Check	Result
git lfs push --object-id origin <oid> (batch upload → PUT)	200s, mapping row live, content under the canonical key — both bindings
git lfs pull (batch download → GET) after wiping the client store	exact bytes restored — both bindings
fresh consumer: GIT_LFS_SKIP_SMUDGE=1 git clone + lfs pull	exact bytes restored
engines: mem, dual_tier_sqlite_local, dual_tier_postgres_local	all pass the same real-client e2e
re-push of a present object	batch answers no actions (idempotent put path)
basic transfer token hand-off (batch → minted href header → PUT/GET)	client applies minted Authorization per action; accepted/refused correctly
unauthenticated batch upload / wrong bearer	401 (authentication required)
PUT with content contradicting the claimed oid	direct: 400, content sweep-eligible (pin dropped); loopback: 400, token left unconfirmed → custody leak (deliberately on record)
Range: bytes=0-99 GET	206 + exact slice + correct Content-Range; bytes=99999- → 416 bytes */len
70 MiB object through the loopback channel PUT	8.9 s (debug) — accepted, verified, attached
45 MiB ranged GET through the loopback fetch	door-split into 2 opens (32 MiB + 13 MiB), slice digests verified per window, byte-exact — 21 s (debug)
70 MiB whole GET through the fetch store arm	byte-exact — 19.5 s (debug)
GC sweep with the referenced-oid table as liveness source	orphaned entry deleted (SweepReport{deleted: 1}), referenced entry survives
GatewayDispatch::invoke with an ungranted identity on blobs/stat	FORBIDDEN via the static Identity.resources fallback; a token-carried {"blob-namespace": ["read"]} resources claim is admitted with no runtime provider consulted — the dynamic grant table is unreachable from the gateway spine

The load-bearing course correction: LFS oid ≠ alkblobs key

The pointer file carries oid sha256:<64hex> — LFS's oid is raw sha256(content). alkblobs' key is the canonical git-blob derivation H("blob <len>\0" + content) (ADR-002), and verification-on-get (the kv tier re-derives and compares on every read) makes storing LFS content under an oid-tagged key impossible: the read re-derives the canonical form and answers Verification. And the canonical key is not derivable in advance from the pointer (the pointer carries the preimage's raw hash and the length, but the blob-derivation is over the content itself), so there is no free oid-addressing shortcut:

Every LFS deployment on alkblobs needs an lfs_oid → canonical key mapping store. The finding (good news): the mapping row wants to be the same table as the referenced-oid liveness source the POC planned — (namespace, lfs_oid) → (key_bytes, size, referenced). One row carries the wire-facing identity, the content address, the size, and the GC liveness. The rejected alternative — store content under a key whose digest bytes happen to be the LFS oid — is the DQ-1b spirit rejection, scoped to LFS: it breaks verification-on-get and dedup-by-construction at the key level.

This is also the DQ-5 pointer-lifecycle answer: pointer-file blobs are ordinary git blobs; LFS content lives in alkblobs; the pointer lifecycle is a per-repo row table owned by the door/adapter layer.

The measured answers to the three gate questions

1. Does the batch adapter + pointer-file flow cover the real client?

Yes. git-lfs 3.7.0 (push, pull, and the skip-smudge fresh-consumer clone) completes against the door in both binding modes across three store engines, with correct auth refusals, idempotent re-push, and Content-Type application/vnd.git-lfs+json responses.

Route-shape facts (empirically pinned):

• With lfs.url = http://host:port/<repo> the client POSTs {url}/objects/batch — the /info/lfs segment is a URL-embedding convention of servers like gitea, not a client rule. The door mounts both static shapes; it never needed a route fallback in the e2e.
• Hrefs are fully door-minted (absolute URLs; per-action Authorization headers accepted by the client — this is exactly the LFS auth hand-off, see finding 4).
• hash_algo arrives as "sha256" (or absent); a server pinned to anything else answers 409. Our door serves sha256 only.
• No verify action was needed; the client treats its absence as fine.

2. Where do transfer/token hand-offs bind best?

Batch (JSON): bind at the ops layer — but through the door's own OperationContext, not the gateway dispatch spine. Direct evidence (tests/gateway_gap.rs): GatewayDispatch::invoke hardwires ownership: None (alkcall gateway/dispatch.rs:245), which makes AccessControl::check fall back to the static Identity.resources model — a token-embedded {"blob-namespace": ["read"]} claim is admitted with no runtime provider consulted, while the dynamic (revocable) grant table is unreachable. So the LFS door builds the OperationContext itself (identity + ownership provider) and calls OperationRegistry::invoke("blobs/stat", …) directly: the namespace ACL (resource_id_path: $.namespace) then runs exactly as it would on the wire, against live grant sets. Upstream consequence recorded below.

Anonymous-public reads: the door synthesizes an anon identity. alkcall's ACL cannot represent "no identity allowed" (an absent identity is Forbidden("authentication required") whenever restrictions exist), so the composition mechanism for public-repo anonymous access is a door-synthesized identity whose grant short-circuits on the public-repo flag — alkgit's ADR-011 posture (public+read anonymous-first-class: the grant decides, the identity exists) drops straight onto the ops ACL this way. With that, all traffic — including anonymous — rides the same ops path; the POC's earlier "anonymous bypasses ops" alternative was dropped once this composed.

Transfer (bytes): bind DirectOps (door → OpsContext) for the embedded door; keep the channel ops for native/fleet consumers. The POC implements and passes both bindings; measurements (debug build, small-object GETs):

binding	10 small GETs	per request
DirectOps	249 ms	~25 ms
LoopbackChannels	795 ms	~80 ms

The loopback bridge (per-request alk/channels half: channel-0 install → registry fork → put/fetch openable registration → dispatcher run — the exact assembly the ops wire path runs, verified in loopback.rs) is correct — its put path exercises the real canonical verification, token mint, and fanout plumbing — but it pays the per-request assembly. The middle option (one persistent door-owned channel server, many channels multiplexed over one client connection) does not fit the door: the open-op ACL runs against the connection-establishment identity, and LFS requests arrive per-request-identity, so the bridge must be per-request (or the binding is DirectOps). That trade-off is the finding: embed → bind at OpsContext; need the ops byte-path (fleet/remote consumers) → per-request loopback or a native client.

3. Is ranged fetch bound set right for big-media objects?

The op's bounds are right; the door owns splitting, and the door owns the whole-arm budget. The ops surface:

• blobs/fetch whole arm (no ranges): unbounded at establish (alkblobs ops/fetch.rs:489 rides store().get directly) — the door must budget it (its own limit, e.g. an object-size ceiling per deployment).
• blobs/fetch ranged arm: max_ranges (default 256) and total <= fetch_max_total_bytes (default 64 MiB) enforced at establish (fetch_ranges_ingress).

Measured: a 45 MiB single range rides as two opens (the POC door windows ≤ 32 MiB), slice digests verified per window (mid-flight verification — the ranged arm's answer), data byte-exact; a 70 MiB whole pull rides the unbounded store arm and verifies canonically at EOF (post-serving residual — recorded; the ranged form is the mid-flight-verified shape for streaming doors).

fetch_max_total_bytes is an OpsConfig knob — an LFS deployment that prefers fewer opens raises it per assembly; no upstream change needed.

The token hand-off, measured in detail

Two token layers compose without conflict:

1. Door href-tokens (lfs:<hex> Bearer): minted per action at batch time, namespace + read/write flagged, carried as action headers. The client applies them verbatim (real client verified). These are door-level capabilities-to-act;
2. alkblobs PinTokens: the store custody layer. The verified flow (real client PUT): body streams → door pins the mapping row (pending) → store put publishes under the canonical key → mint_custodial(namespace, key, pin) → confirm(namespace, token) → the bound PinConverter flips the live-key cache (the LivenessSource) and the row's durable update rides a spawned task (never on the arbitration lock) → custody released, content live.

One structural fact to carry into the design: PinConverter is sync-only, failure-free, and sees only the Key (alkblobs ops/tokens.rs:135 — it acts under the store's arbitration lock). It cannot carry the LFS mapping metadata (oid, size) and cannot fail, so the rich reference row is necessarily door-side; the converter is exactly one in-memory liveness act. The door-side row + confirm-shaped converter binds cleanly; trying to make the converter the row writer does not (no metadata, no failure channel).

Custody behaviors measured (the boundary conditions):

• DirectOps mislabeled PUT: the door drops its pin → the mislabeled content is sweep-eligible immediately (clean, door-owned refusal);
• Loopback mislabeled PUT: the entry is already committed (the put verified its own canonical claim) and the door's non-confirm leaves the token's held pin intact — a custody leak that never expires on a non-fleet store. This is the alkblobs S-5 admission-cap ask (substrate-recomposition upstream-push #2) realized as a concrete attack shape: a hostile LFS upload loop mints held pins indefinitely. The fleet form (pin-row TTL) is the safe posture for LFS deployments; the non-fleet embedded form inherits the embedder's responsibility.

Remaining POC notes (recorded, not blocking)

• alkhttp extras can't carry a route fallback — the merged router already owns one (the decoy) and axum refuses a second merge (verified: panic at construction). Door route tables must be static shapes; multi-segment repo namespaces (user/repo) therefore need enumerated-pattern routes in the eventual alkhttp lfs/git features. Recorded as a door-constraint datum.
• LFS clients authenticate with Basic (URL userinfo) on the batch — the door resolved both Basic and Bearer locally. alkhttp's identity resolution only handles Bearer; a nit-level upstream tolerance would be accepting Basic userinfo in the shared bearer middleware.
• kv-only store shapes reject LFS-sized objects (the over-threshold put refusal) — LFS deployments want a dual-tier engine shape; the deployment matrix (DQ-7) applies unchanged but "LFS ⇒ large tier present" is now a data point for the assembly docs.
• The CAS-global-pool caveat surfaces in liveness bookkeeping: the same content in two repos shares one canonical key; the live-key cache is key-level while the referenced flags are row-level — the mark_orphan path must re-protect the key while any other row references it (implemented crudely with a second live-row scan for the POC; the eventual design should keep a refcount). This is alkblobs ADR-017's caveat landing in alkgit's own table.
• Postgres-bootstrap datum: the deployment needs a root-connection CREATE DATABASE IF MISSING dance (driven connection; the POC's liveness::open) — an assembly-docs datum, not an ask.
• git-lfs quirks for future work: git lfs push --object-id is the remote-less e2e driver (it never consults the git transport); git lfs install --local is required in every fresh consumer (the clone inherits .gitattributes, not the filter config); lfs.locksverify false keeps the client off the (unimplemented here) lock API.

What this settles / does not

• DQ-5 (LFS protocol placement): settled in shape — the door is an alkhttp route-table adapter mapping batch/checking onto blobs/stat|have through the door's own context, and transfer onto the ops byte-path (DirectOps recommended for the embedded case; the alk/lfs-style native op remains the fleet-consumer surface). The lfs_oid ↔ key mapping table + referenced-oid liveness is the pointer-lifecycle store. A formal ADR when the re-baseline lands.
• POC-O's questions (DQ-1, DQ-2, DQ-4): untouched — this POC stores LFS content only (pointer blobs as content is POC-O's world).
• The upstream asks: S-5 (custody admission cap) now has a named attack path; the gateway ownership seam is a new (small) alkhttp candidate; everything else stands as listed.

POC art

• Crate: /workspace/lfs-poc (standalone; see its README).
• tests/e2e_lfs.rs — the real-client round trips (both bindings × mem/sqlite/pg stores + fresh-consumer clone).
• tests/raw_http.rs — ACL denials, transfer-token hand-off, hash-mismatch refusals, range semantics (both bindings).
• tests/sweep.rs — the referenced-oid liveness + sweep demo; mislabeled PUT refusal.
• tests/big_media.rs — 70 MiB through the loopback bridge; 45 MiB ranged split with per-window slice digests; small-request binding overhead probe.
• tests/gateway_gap.rs — the GatewayDispatch ownership-seam evidence.