alkblobs/tasks/s2-ranged-fetch-bounds.md

id, name, status, depends_on, scope, risk, impact, level, tags

id	name	status	depends_on	scope	risk	impact	level	tags
s2-ranged-fetch-bounds	S-2 — ranged fetch bounded (maxItems + total-bytes) and streamed per-range from the pump	completed	s1-s3-ops-authority-model	moderate	medium	component	implementation	wave-10 review-001 security ops fetch

Description

Review-001 S-2 — the one network-facing unbounded resource. The blobs/fetch ranged form materializes every slice into FetchPlan::Ranges before any byte streams (src/ops/fetch.rs:446-475, fetch_establish_ranges): N ranges x entry-size bytes all resident simultaneously, held for the channel's whole lifetime — a single authenticated (read-granted) request can demand ~TBs. Each range is overflow-safe at the entry boundary (clamped, backend.rs) but the count and the sum are unbounded (the schema has minItems: 1, no maxItems — src/ops/fetch.rs:330-342).

Two fixes, per the review and the plan:

1. Bounds. maxItems on the ranges schema plus a total-bytes bound enforced at the ingress (the schema cannot sum). Design points, judged in-task and recorded in Notes:
   • The bound values and their home. Proposed shape: OpsConfig knobs (the fanout registry's bounds are already constructor state — FanoutRegistry::new(max_subscribers, channel_bound) — and fetch_openables(ctx) builds the schema per-context, so a config value can ride the schema JSON), with named defaults (e.g. maxItems 256; total-bytes 64 MiB summed over the requested lens). Constants are the acceptable fallback if the config shape does not fit cleanly — decide, record.
   • The total-bytes bound sums the requested lens (the ingress sees the wire values; the entry's length is not known there) — clamping stays per-range at the store seam.
   • The over-bound rejection's wire shape: the schema bound rejects at the registry (INVALID_INPUT, the existing malformed-payload sweep's shape); the ingress bound rejects as the establishment HandlerError (the ranges ingress' existing rejection path, src/ops/fetch.rs:480-500) — with a stable domain text (S-4's discipline; coordinate the text shape with the s4 task or land it stable here and let s4 sweep the rest).
2. Streaming. The pump streams per-range instead of pre-materializing: FetchPlan::Ranges carries the key + the ranges (not the slices); the pump calls read_range per range and streams each slice in request order. The slice digests still ride the ack (the establishment reply precedes the pump — the ack contract is unchanged), so establishment still reads per range to compute each digest — bytes transient, one range resident at a time, only the digests retained. The named tradeoff: the ranged form double-reads (the establishment digest pass + the pump's stream pass). Accepted: memory-bounded beats read-count on a network-facing path; the alternative (digests at stream-end) changes the ack contract. Record the tradeoff on the plan/establisher doc. The mid-stream failure posture: a delete racing the pump (or any store failure) ends the pump mid-stream — the consumer's slice-digest check fails, the re-fetch is the degrade (the same posture the whole form's pump already has; the committed entry's immutability makes the honest-case bytes identical).

Tests (the wave-10 gate names them):

• An over-count request rejected at the schema (INVALID_INPUT).
• An over-bytes request rejected at the ingress (the establishment refusal, stable text).
• A bounded ranged request byte-verified end to end through the harness (the existing ranged e2e shape — must stay green, unmodified in its assertions).
• The pump streams per-range: a multi-range fetch's received bytes match the slices and their acked digests (the existing e2e covers this — extend if the plan-shape change needs a closer drive).

Live harness

The ops test tree's fleet column rides the live harness; the ranged fetch tests themselves are server-less (mem/local stores), but run the full matrix:

export ALKBLOBS_PG_HOST=localhost
export ALKBLOBS_PG_PORT=15432
export ALKBLOBS_PG_USER=postgres
export ALKBLOBS_PG_PASSWORD=poc
export ALKBLOBS_PG_DB=blobs

Server-less, the pg tests skip cleanly; acceptance requires green against the live harness.

Acceptance Criteria

• The ranges schema carries maxItems; the ingress enforces a total-bytes bound; both bounds' values and homes recorded in Notes
• FetchPlan::Ranges no longer carries materialized slices; the pump streams per-range via read_range; the double-read tradeoff recorded on the establisher/plan doc
• The slice digests still ride the establishment ack (the ack contract unchanged)
• Tests: over-count rejected at the schema, over-bytes rejected at the ingress (stable text), the bounded ranged e2e byte-verified and green unmodified
• Gates green: cargo test (default), cargo test --all-features server-less, cargo test --all-features + live harness, clippy --all-targets -- -D warnings across the feature variants, cargo fmt --check

References

• docs/reviews/001-storage-surface-general-review.md (S-2 — pre-split coordinates)
• docs/plans/implementation.md (§Wave 10; §Review gates — the security gate's S-2 clause)
• src/ops/fetch.rs (the schema, fetch_ranges_ingress, fetch_establish_ranges, FetchPlan, fetch_pump_slices)
• src/ops/mod.rs (the module doc's ranged-form ack section — update the establishment description if the mechanics moved)
• src/ops/tests.rs (the ranged e2e; the malformed-payload sweep — the over-bound tests' home)
• docs/architecture/decisions/006-verification-posture-and-transfer-encoding.md (the slice-digest convention the ack rides)

Notes

• The bounds' values and homes (decided): OpsConfig knobs, the proposed shape fit cleanly. fetch_max_ranges: usize (default 256) and fetch_max_total_bytes: u64 (default 64 MiB = 67,108,864) on OpsConfig (src/ops/tokens.rs); OpsContext now keeps the OpsConfig it was built with (exposed as OpsContext::config()), fetch_openables(ctx) injects maxItems: fetch_max_ranges into the per-context schema JSON (an over-count open is the registry's INVALID_INPUT), while the total-bytes bound rides fetch_establish → fetch_ranges_ingress (JSON Schema cannot sum). Constants were not needed; the config shape was a natural extension of the fanout's constructor-state discipline. A wire test with a small custom config proves the knobs — not constants — are the home (over-count at 3, over-bytes at 16).
• The over-bytes rejection's wire shape — per the task's design points, both: the schema bound rejects at the registry (INVALID_INPUT); the ingress bound rejects as the establishment HandlerError (wire channel:open_failed, reason handler_error) with stable domain text — the ranges request exceeds the total-bytes bound: {sum} bytes requested, the limit is {bound} bytes (and the checked-add overflow arm's the requested lens sum past the limit of {bound} bytes). The wire numbers are the caller's own u64s and the context's bound — stable shape, no raw payload Value reflection.
• S-4 coordination: the two pre-existing fetch_ranges_ingress texts that interpolated raw payload Values ({values}, {value}) were rewritten to stable texts in this task (the ingress function was rewritten anyway); s4's list already says "verify and skip" for these — they are landed stable here.
• Summing semantics: the total-bytes bound sums the requested lens (the ingress sees the wire values) via u64::checked_add — a sum that overflows is the over-bytes refusal, never a panic; the bound is inclusive (sum == bound establishes, asserted). The requested-vs-served delta stays per-range clamping at the store seam (read_range), exactly as before.
• Streaming mechanics: FetchPlan::Ranges { key, ranges } — the key and the requested ranges, never the slices. Establishment still reads per range (the digest pass: bytes transient, one range resident at a time, only the digests retained — the ack contract is untouched); the pump holds the store handle (fetch_openables captures Arc<Store> alongside the plan type — the plan carries data, not access) and calls read_range per range, streaming each slice in request order (fetch_pump_slices → fetch_pump_ranges).
• The double-read tradeoff: recorded on the [FetchPlan::Ranges] doc (src/ops/fetch.rs) and the module doc's ranged-form ack paragraph (src/ops/mod.rs) — establishment's digest pass plus the pump's stream pass, accepted: memory-bounded beats read-count on a network-facing path, and digest-at-stream-end would change the ack contract.
• The mid-stream failure posture: recorded on fetch_pump_ranges' doc — a delete racing the pump (or any store failure) ends the pump mid-stream; the consumer's slice-digest check fails and the re-fetch is the degrade (the whole form's pump posture; committed entries are immutable, so the honest-case bytes are identical).
• Tests' homes: the over-count case rides the malformed-payload wire sweep at the default config (257 items → INVALID_INPUT); fetch_over_count_rejects_at_the_schema_and_over_bytes_at_the_ingress (src/ops/tests.rs) is the knob-riding wire round (custom small config: count at the schema, single/summed/overflow over-bytes at the ingress, the bound-exact request establishing with its ack digests and byte-verified stream); the establisher-direct sweep gained the default-bound (64 MiB) stable-text assertion. The existing ranged e2e (fetch_serves_ranges_and_the_ack_carries_the_slice_digests) and the pump silent-return round (which drives the ranged plan through the establisher against a torn consumer) stayed green unmodified — the per-range streaming shape exercises through them unchanged.
• API note: OpsContext gained config(); OpsConfig gained the two fields (a breaking field addition to the plain struct literal — the crate is pre-release, and all in-tree constructors use OpsConfig::default() or struct-update syntax).

Summary

S-2 landed. (1) Bounds: the ranges schema carries maxItems from the new OpsConfig::fetch_max_ranges knob (default 256) — an over-count open is the registry's INVALID_INPUT; the establisher's ranges ingress enforces fetch_max_total_bytes (default 64 MiB, summed over the requested lens with checked-add, inclusive at the bound) — the over-bytes establishment HandlerError refusal with stable domain text. (2) Streaming: FetchPlan::Ranges carries the key + ranges only; the pump streams per range via read_range (one range resident at a time), the establishment digest pass keeps the ack contract unchanged, the double-read tradeoff and the mid-stream failure posture recorded on the plan/pump docs. Regression shapes: over-count in the malformed-payload sweep, the knob-riding bound round with the bound-exact byte-verified e2e, the default-bound stable-text assertion in the establisher-direct sweep; the existing ranged e2e and ACL matrix green unmodified.

Gates: cargo test (204 green), cargo test --all-features both server-less and against the live pglo-poc harness (325 lib + 125 conformance + 4 git-interop, green), clippy --all-targets -- -D warnings across both feature variants, cargo fmt --check, cargo doc --no-deps (only the two pre-existing walk_committed intra-doc warnings, unchanged on the clean tree).