alkblobs/tasks/ops-e2e-flows.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
ops-e2e-flows	the ops wire-level lifecycle proofs — have/need diffing, the token lifecycle, and the ACL matrix end-to-end	completed	ops-json-ops ops-fetch ops-put	broad	medium	phase	implementation	wave-7 ops e2e tokens acl fleet

Description

The capstone proofs through the harness (ops-call-harness) — the full lifecycle flows the individual handler tests exercise only in pieces. This is wave 7's equivalent of the engine waves' suite columns: the surface proven as a whole, on the wire.

• The have/need flow: a consumer diffs blobs/have over its digest set → fetches the absent digests (the need announcement is the fetch miss — the diff is caller-side; the wire carries only concrete fetch requests). Proven end-to-end: seed a store via the facade, diff from the client side, fetch exactly the absent set, re-diff to all-present.
• The fleet token lifecycle end-to-end (against the harness server — env-driven, skipping cleanly server-less; the pglo-poc convention): put → token → renew via have's renewal form before expiry (present: true; the row's expiry observed moving) → the embedder's registration confirms → the row's protection hands to the registered liveness source (the namespace binding's converter) → the row expires naturally and the entry survives on the registered source's protection. Then the failure half: pause past TTL → the renewal probe answers present: false → the sweeper reaps the row → the re-put backstop (delete-then- recover; CAS makes the re-put idempotent) → the race resolves to a re-put, never to silent loss (ADR-010 §4's expiry semantics verbatim, on the wire).
• The foreign-token refusal: a token minted by node A presented to node B (a second fleet instance's ops module over the same pool) → present: false, not an error — the validity domain is the minting node (ADR-012 §2).
• The non-fleet token lifecycle: put → token (the registry holds the guard) → have-present answers from the custody → the embedder's registration confirms → the guard converts (the namespace binding's converter) → the token retires. Proven on a non-fleet store (sqlite or mem — server-less).
• The ACL matrix end-to-end: an unlisted namespace denies read AND write (closed-by-default — OQ-09); a read-granted identity fetches/stats/haves but cannot put; a write-granted identity puts but a read-only peer cannot fetch its namespace; delete only under the internal authority context (never a namespace grant).
• The re-routing posture's serving half (ADR-008 §3): a large (pg-lo) entry served through blobs/fetch from the storage node's dual-tier fleet store — the shape client nodes in kv-only mode consume; no fleet-specific op exists (the module supports the shape without change).
• The no-side-door audit (structural, ADR-001 §4): the ops module's store access enumerated — the public facade verbs plus the facade's crate-internal pin/fleet access (gc, fleet — the ADR-010 §4 "crosses the seam by construction" access the put handler's guard rides) plus exactly the crate-internal renewal seam (ops_pin_renew, ops_pin_check); no Backend touch, no engine types, no path around the facade. Recorded as the audit the wave-7 review gate re-checks.
• The renewal-cadence boundary honored (ADR-013 §1): the tests drive renewal explicitly (the embedder's cadence role played by the test); no in-crate renewal worker exists.

Acceptance Criteria

• The have/need flow proven end-to-end (diff caller-side; the wire carries only concrete fetches)
• The fleet token lifecycle end-to-end vs the harness server: renew-before-expiry (the row's expiry moves), the registration-handover, pause-past-TTL → reap → the re-put backstop — never silent loss (env-driven; skipping cleanly server-less; acceptance requires green against the server)
• The foreign-token refusal: present: false, not an error
• The non-fleet lifecycle: custody → confirmation → conversion → token retirement (server-less)
• The ACL matrix end-to-end (closed-by-default read+write denial; the read/write split; delete internal-only)
• The re-routing posture's serving half: a large entry fetched through the ops surface from the dual-tier fleet store
• The no-side-door audit recorded (the store-access enumeration)
• Gates green: cargo test (default), cargo test --features ops, cargo test --all-features (the fleet rows live at acceptance), clippy/fmt across the variants, cargo doc --no-deps clean

References

• docs/architecture/ops-surface.md (§Op family — the hand-over contract; the need-half note; §Store-facing relationship — the re-routing posture)
• docs/architecture/decisions/010-backend-io-seams-and-fleet-state-home.md §4 (the expiry semantics; the re-put backstop)
• docs/architecture/decisions/012-pre-decomposition-consistency-rulings.md §2 (the validity domain; the non-fleet generalization)
• docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-large-engines.md §2/§3 (the TTL/3 cadence; the re-routing topology)
• docs/architecture/decisions/013-family-composition-reactive-boundary-and-feature-graph.md §1 (the cadence is embedder composition)
• docs/architecture/open-questions.md OQ-09 (closed-by-default)
• tasks/ops-foundation.md, tasks/ops-call-harness.md, tasks/ops-json-ops.md, tasks/ops-fetch.md, tasks/ops-put.md (the surface this proves as a whole)
• tasks/suite-column-fleet.md (the env-driven harness-server convention this task's fleet rows follow)

Notes

Implementation agent, 2026-10-14. The flows landed as a new section in src/ops.rs's harness test module (the capstone rides the same harness the individual handler tests exercise only in pieces — the serving assembly, the client halves, and the ACL wiring). No shipped module code changed; the task is pure proof. One small test-tree extension: RecordingConverter::captured became pub(crate) so the non-fleet lifecycle flow can read the converter's capture.

The flows, one per acceptance row:

• The have/need flow (the_have_need_flow_diffs_caller_side_..., server-less): a seed via the facade, the caller-side diff over blobs/have → the need set, the miss announcement proven as the fetch refusal (unknown_resource, the digest carried), blobs/need answered NOT_FOUND on the wire (no need op exists — the diff stays caller-side, the wire carries only concrete fetch requests), the putter fills via blobs/put, the putter's cadence rode the renewal form (the test plays it explicitly, ADR-013 §1), the re-diff to all-present, then a fetch of exactly the need set: byte-exact, consumer hash-check passed.
• The fleet lifecycle (env-driven against the harness server, the pglo-poc convention; skipping cleanly server-less; both tests' acceptance green against the live server): the success half (the_fleet_token_lifecycle_hands_over_...) proves the wire put's mint (row owner = the serving instance; guard dropped at response — pin_count 0), two renewals before expiry with the row's expiry observed moving each time (the gc_pins instrument), the confirm verb's hand-over (the converter registered the digest into the embedder's liveness registry), the spent token's probe answering present: false, and the row expiring naturally with the entry surviving two sweeps on the registered source (nothing deleted; the expired row reaped by the maintenance step). The failure half (the_paused_fleet_token_expires_reaps_...) proves pause-past-TTL → present: false (the expiry frozen at the last write — no renewal), the sweep staging exactly the unprotected entry, the maintenance step reaping the row, the execution deleting it exactly once, and the re-put backstop: the digest re-offered through blobs/put recovered the same canonical address (CAS idempotent), a fresh row + a renewing token, and the byte-identical fetch — the race resolves to a re-put, never to silent loss (ADR-010 §4 verbatim, on the wire).
• The foreign-token refusal (the_foreign_token_refusal_is_present_false_not_an_error): a token minted by node A presented to node B's ops module (a second fleet instance over the same pool) answers present: false, not an error; the refusal is symmetric (B's token is foreign to A), and B's own mint over the same pool renews — the validity domain is the minting node (ADR-012 §2).
• The non-fleet lifecycle (server-less): the wire put's custodial mint (the registry holds the guard), have-present answering from the custody, the confirm's conversion (the guard released, the converter captured the digest), the mapping retired, the retired token's probe answering present: false, and the pure probe answering present — the entry stands on the registered source (the hand-over contract's observable).
• The ACL matrix (the_acl_matrix_..., server-less): closed-by- default rows over an unlisted namespace deny read (stat/have/fetch) AND write; the read-granted identity stats/haves/fetches (byte-exact) and cannot put; the write-granted identity puts and cannot fetch (the write grant is not the read grant), and a peer granted on another namespace cannot fetch either (grants are per-namespace); blobs/delete is NOT_FOUND discovered externally and deletes from a composed internal call under the declared composition authority — the internal authority context, never a namespace grant.
• The re-routing posture's serving half (the_rerouting_serving_half_..., env-driven): a 4096-byte entry on the storage node's dual-tier fleet store (dual_tier_postgres_pglo_ fleet — the consolidation shape) landed on the pg-lo tier (kv tier 0 rows, lo_entries 1), and blobs/fetch — unchanged, the standard channels/blobs/sub over alk/blobs, no fleet-specific op — served it byte-exact with the whole-form ack.
• The no-side-door audit (structural): the_no_side_door_audit_... reads the module's non-test source via include_str! and enumerates every store-access method — the observed set equals the public facade verbs (put, get, stat, has, read_range, delete) plus the facade's crate-internal pin/fleet access (fleet, gc — the ADR-010 §4 "crosses the seam by construction" access the put handler's guard rides, pub(crate)) plus exactly the crate-internal renewal seam (ops_pin_check, ops_pin_renew); no Backend, no engine types, no engine crate paths, nothing around the facade. Recorded as the enumeration the wave-7 review gate re-checks.

One shared-harness-hygiene finding worth recording: the re-routing flow's large entry rides the shared postres database's LO space (pg_largeobject is database-level, not schema-level), so its teardown deletes the entry past the pin row's expiry (the facade's one-tx lo_unlink + companion-row delete) and asserts zero lo_entries residue — the pg_lo column's zero-LO-residue gate stays green with the flow in the suite.

Verification: cargo test (default) green; cargo test --features ops green; cargo test --all-features green against the harness server (ALKBLOBS_PG_HOST/PORT/USER/PASSWORD/DB set; both fleet lifecycle rows, the foreign-token refusal, and the re-routing row live at acceptance); clippy -D warnings green on all three shapes; cargo fmt --check green; cargo doc --no-deps clean.

Summary

Ops-e2e-flows landed complete: the wave-7 capstone proofs through the harness — the have/need flow end to end (diff caller-side, NOT_FOUND on "blobs/need", the miss announcement = the fetch refusal, the fill, the byte-exact need-set fetch), the fleet token lifecycle in both halves (renew-before-expiry with the row's expiry observed moving; the registration hand-over to a registered liveness source and survival past natural expiry; pause-past-TTL → present:false → reap → delete-then-recover via the CAS-idempotent wire re-put, never silent loss), the foreign-token refusal (present: false, symmetric, not an error), the non-fleet custody→confirmation→conversion→retirement lifecycle, the ACL matrix end to end (closed-by-default read+write denial; the read/write split; per-namespace grants; delete internal-only), the re-routing serving half (a pg-lo large entry served byte-exact through the unchanged standard fetch op from the dual-tier fleet store), and the no-side-door audit recorded (the enumerated store access = the facade + the crate-internal renewal seam, exactly). All acceptance criteria verified; all gates green on all feature shapes, fleet rows live against the harness server.