# ADR-015: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention

## Status

Accepted

## Context

[OQ-12](../../architecture/open-questions.md) — found by the contract
v1 Phase 1 architecture review: [ADR-008](008-contract-v1-pinning.md)
§1 pinned streams' *method names* (`publish`, `publish_with_key`,
`read_since`, `read_from_consumer`, `save_offset`, `get_offset`,
`subscribe`) but not their *depth*. Four sub-questions were open:

1. **Key semantics** — what `publish_with_key`'s key means, and what
   if any ordering it buys. Honker's realization is a stored nullable
   `key` column (`_honker_stream(topic, key, payload)`) with no
   per-key read path or ordering enforcement anywhere; the doc comment
   says "used for per-key ordering downstream"
   (`packages/honker-rs/src/lib.rs:894`) — per-key FIFO is an
   *emergent* property of a consumer reading `offset ASC` filtered by
   key, not a server guarantee. Three options were on the table: (a)
   pin honker's shape (key = carried metadata, ordering stays global
   FIFO, document the emergent pattern); (b) pin server-enforced
   per-key ordering (real machinery on both engines, gated on a
   consumer-inventory row naming the need); (c) cut `publish_with_key`
   from v1 (no consumer row names keys).

2. **`StreamEvent` shape** — honker's is
   `{ offset: i64, topic: String, key: Option<String>, payload:
   Vec<u8>, created_at: i64 }` with `payload_as<T>` decode
   (lib.rs:1008). The contract needs its own pinned field list.

3. **Ordering guarantee row** — [ADR-006](006-wake-and-delivery-contract.md)
   §2's delivery table pins streams' durability/replay/atomicity but
   its ordering cell content was effectively "readable"; the contract
   suite cannot pin cross-engine stream equivalence without an
   explicit ordering row.

4. **Retention / bounded growth** — the reference read flags the
   stream table as unbounded-growth-unless-swept (honker-machinery
   §8 defect 5); no consumer row names retention
   (replay-deep-history is the *purpose* of the subscription need).

The consumer evidence was completed for this decision by adding the
**alkcall row** to
[consumer-inventory.md](../../research/consumer-inventory.md)
(ecosystem-shape evidence, added per the inventory's own
row-before-assumption convention). Alkcall's pub/sub vocabulary —
`Sub` operations (server→client streaming, ADR-021), `Pub`
(client→server streaming, ADR-046), `channel/resources/subscribe`
(snapshot + change events, ADR-037), and the deferred
broker/fan-out (ADR-046's Gap B, channels OQ-22) — is the shape
downstream crates' reactive ops are expected to take over this store,
and the operator confirmed alkstore is meant as the base storage
layer for that ecosystem. Its ordering posture is per-stream
(ordered, reliable transport streams; correlation by request ID):
**no alkcall mechanism names server-enforced per-key ordering.** The
topic/event-type dimension of its subscriptions routes to *separate
streams* (one stream per topic; e.g. `channel/resources/subscribe` is
its own event source), not to keys within one stream.

## Decision

### 1. Key semantics: option (a) — the key is carried metadata; ordering stays global FIFO

Option (b) is **rejected on the decision rule it named for itself**:
it requires "a consumer row naming the need," and neither the
inventory's original rows nor the alkcall row names server-enforced
per-key ordering anywhere. Real machinery (per-key sequencing,
per-key claim/exhaustion paths, or per-key head-of-line bookkeeping on
both engines) for an unnamed need is exactly the scope discipline
[ADR-002](002-feature-scope.md) exists to enforce.

Option (c) (cut `publish_with_key` from v1) is **rejected**: the
method name is a pinned v1 surface element
([ADR-008](008-contract-v1-pinning.md) §1 — removing it would be a
contract-breaking event for OQ-10 to govern, not a pre-implementation
refinement); the *cost* of carrying the key in the inherited
realizations is one nullable column honker's schema already has; and
the key is genuinely useful as a *grouping token* — the emergent
per-key pattern is real and cheap for consumers (SQL-level filter +
in-order read), and an ecosystem whose ops carry keys (a repo id, an
entity id) deserves a stable way to store that grouping durably.
Cutting the only keyed-write path would push consumers to encode keys
into payloads — an unqueryable convention — the same shape of waste
ADR-010 §7 called out for result storage.

Pinned semantics, exactly honker's:

- `publish_with_key(stream, key, payload)` stores `key` as a nullable
  column value on the event row; `publish` stores `NULL`.
- The key is **carried metadata** — it appears on every `StreamEvent`
  read back and is `Option<String>`-shaped in the event type; the
  engine assigns it no behavioral role (it rides the row the publish
  writes; the wake trigger is the publish itself, key or no key).
- **The ordering guarantee stays global FIFO by offset, per stream** —
  regardless of key. Per-key FIFO-by-offset is the *documented
  emergent pattern*: a consumer that filters a read by key observes
  that stream's events for that key in offset order. The contract
  does not enforce, promise, or price any cross-subscriber per-key
  handoff. (The guarantee row, §3, is written so this remains *true
  under trimming* — offset order, never renumbered.)
- The key is **not** a partitioning instruction, not a dedup key, and
  not an ordering key in any engine-enforced sense. Nothing in the
  contract suite tests keys behaviorally beyond carried-metadata
  round-trip and equivalence across engines.

The alkcall evidence makes this the right ecosystem shape too: its
subscriptions distinguish event *sources* (separate streams per
topic), not positions *within* a source (per-key lanes). When the
channels broker (Gap B) arrives, topic matching selects streams;
within a stream, offset order is the whole story.

### 2. `publish_with_key_tx` — the tx seam carries the key

A gap the key decision exposed in the pinned tx seam:
[TxHandle](../../architecture/core-contract.md) carries
`publish_tx(stream, payload)` but no keyed form — so a keyed event
could not be written commit-atomic with a business write, breaking the
no-ghosts property's uniformity for keyed publishes (the same shape of
hole OQ-13 exposed for outbox enqueue, resolved by
[ADR-014](014-outbox-tx-enqueue.md)). Pinned:
`publish_with_key_tx(stream, key, payload) -> offset` joins the
TxHandle trait — one method, symmetric with `publish_tx`, validating
the key the same way the non-tx form does (`None` or non-empty — an
empty `Some` key is `InvalidName`) with no new error variants
([ADR-008](008-contract-v1-pinning.md) §5's rule). Framed as
completing contract v1 in place, pre-implementation, like ADR-014 —
no versioning event, OQ-10 untouched. A plain-`publish_tx` is
`publish_with_key_tx(stream, None, payload)`, not a separate
engine path. *(Naming pinned here too: the publish family returns the
**assigned offset** — honker's `publish` already returns it — and the
tx-seam sketches' `event_id` placeholder name is corrected to
`offset` so one term covers the value every read/cursor/trim method
speaks.)*

### 3. `StreamEvent` shape — honker's, with one rename

Pinned (inherited near-verbatim, per the fork's API fidelity posture,
[ADR-012](012-forked-substrate-design.md) §3):

```text
struct StreamEvent {
    offset: i64,            // position in the stream's log; immutable
    stream: String,         // the stream name (honker: "topic")
    key: Option<String>,    // §1's carried metadata; None unless
                            // published keyed
    payload: Vec<u8>,       // core value bytes
    created_at: i64,        // unix seconds at publish (the engines'
                            // single clock source)
}
```

- The `topic` field renames to **`stream`** — the contract surface
  names this mechanism streams, not topics (honker-rs's `Stream`
  handle's `topic()` accessor is likewise the stream name); one term
  everywhere, matching the reserved-namespace kinds list
  ([ADR-008](008-contract-v1-pinning.md) §4).
- `payload` is core value bytes crossed as a non-generic value; the
  concrete `StreamEvent` owns the bytes; `payload_as<T>` is the decode
  convenience ([ADR-008](008-contract-v1-pinning.md) §8's rule — the
  deserialization error is `Codec`).
- `created_at` is **unix seconds at publish**, from the engine's
  single clock source (the partial-index/clock-pin rule in
  [queues.md](../../architecture/queues.md)); it is *informational*
  and NOT an ordering candidate — offset is the only ordering field
  (two events can share a second; offset never collides).
- `offset` is the stream-wide position, engine-assigned at publish,
  monotonically increasing per stream on both engines. It is
  **immutable once assigned** and is the value every read/cursor/
  trim method speaks.

### 4. Ordering guarantee row — pinned

ADR-006 §2's table's streams row extends with an explicit ordering
cell (the guarantee cell gains a clause; durability/replay/atomicity
as already pinned):

> **streams** — Guarantee: *global FIFO by offset within each stream
> (`read_since`/`subscribe` yield `offset ASC`); every committed
> event readable by every consumer that hasn't passed its offset;
> offsets immutable and never renumbered.*

This is the row the contract suite pins cross-engine stream
equivalence against: two engines, same publish sequence → same offset
sequence → same `read_since` output order, for direct and subscriber
reads alike. It is deliberately *not* stronger than both engines can
implement by a shared SQL shape (`ORDER BY offset ASC` is already
both engines' claim path) and not weaker than the mechanism's purpose
(replay demands monotone positions).

### 5. Retention: `trim_to` on the stream handle — a consumer-invoked bounded-growth op

The mechanism-shaped answer to defect 5's family (unbounded
`_honker_stream` growth), applying the ADR-010 §6 posture to the one
stream-side table that consumers interact with *as rows* (unlike
notifications, the transport detail whose hygiene stays engine-
internal):

- `store.stream(name)`'s handle gains `trim_to(horizon)` — delete
  events with `offset <= horizon`, emitting no dedicated wake and no
  notify (deletes are not events; consumers whose cursors move
  forward never lose reads). On SQLite the `data_version` watcher may
  still fire on the trim commit — any committed write bumps it; that
  spurious hint is contract-legal (the wake contract's best-effort +
  idempotence posture, ADR-006 §1), not a trim-specific delivery.
  `OffsetGone`-style errors are not needed: reads past the trim
  horizon simply return fewer/no rows (a consumer reading from a
  trimmed region has, by definition, an offset older than the
  horizon — that is the API contract; re-attaching consumers at
  trimmed-away offsets resume at the horizon's first remaining row).
- **No engine-side retention default, no ambient sweeper** (the
  ADR-010 §6 posture; consumer-side, cadenced by the collapse recipe
  like every sweep). Trim is *available*, not automatic: the
  replay-forever default is the *purpose* for the consumer rows, so
  the default is unbounded — with the growth documented squarely,
  this time with an in-contract tool to fix it.
- **Caveat documented in-contract**: offsets are never renumbered and
  never reused; surviving events keep the offsets they were published
  with (gaps after a trim are legal and ordinary). Consumers must
  not assume dense offset sequences (the same discipline the
  scheduler's catch-up/skip-forward already imposes).
- Trim's effect on cursors: a consumer's *saved* offset below the
  trim horizon stays (it remains a valid position marker; the next
  read resumes at the horizon's first remaining row); a consumer may
  re-save after processing. This keeps `get_offset`/`subscribe`
  resume semantics well-defined without engine-side cursor-tracking
  machinery.

Rejected retention alternatives (per the OQ's option set): the
consumer-recipe-without-an-API option was self-admittedly "document
the gap" (no deletion path exists today — this ADR *creates* the
deletion path, which is what the recipe needed); documenting-nothing
repeats the notifications-growth mistake ADR-010 §6 just fixed.

## Consequences

**Positive**

- Streams' depth is fully pinned: names (ADR-008 §1) + semantics (this
  ADR) make the mechanism implementable on both engines from the same
  text, and the contract suite has an ordering row to pin cross-engine
  equivalence against.
- The key decision is honest: a zero-machinery feature (carried
  metadata) documented for exactly what it is, with option (b)'s
  re-entry gate (a consumer-inventory row naming server-enforced
  per-key ordering) recorded — the ecosystem-shape evidence says the
  topic dimension routes to streams, not keys.
- Keyed publishes are tx-seam-native — keyed and unkeyed events carry
  the same commit-atomicity, so an outbox-style keyed event log co-writes
  with business data correctly.
- `trim_to` gives the family a real answer to stream growth in the
  contract, consistent with the no-ambient-sweeper posture (invoked
  deliberately, cadenced from above).

**Negative**

- Per-key *ordering enforcement*, if a consumer ever names it, is a
  later contract extension with real machinery on both engines —
  carrying metadata now does not prejudge or approximate it, and the
  emergent pattern's documented status must not be read as a promise.
- `trim_to` adds one more consumer-invoked maintenance op to the
  surface (backlog rows follow).
- `created_at` is second-precision (the shared clock pin) — consumers
  needing sub-second publish timestamps must encode them in payloads.
- The `StreamEvent.stream` rename diverges from honker's wire-era name
  (`topic`) — the fork keeps the column name `topic` engine-internally
  per the ADR-012 §3 fidelity posture; only the contract type renames.

## Verification backlog additions

- **Cross-engine stream equivalence** (the §3/§4 row): publish
  sequence → identical offset sequence → identical `read_since`
  output order, both engines; keyed and unkeyed interleavings
  preserve global FIFO; `key` round-trips exactly (`None` vs
  `Some`), `StreamEvent.stream`/`created_at` equal on both engines.
- **`publish_with_key_tx` commit-atomicity on both engines** (§2):
  rollback drops the keyed event row with the business write (no
  ghost event); commit makes it visible to `read_since`/`subscribe`;
  empty-`Some`-key `InvalidName` on both engines' tx paths.
- **`trim_to` semantics on both engines** (§5): exact-boundary trim
  (`<=`), surviving rows keep their offsets (gaps legal, never
  renumbered), reads from a trimmed horizon resume correctly,
  `save_offset` below the horizon stays valid, trim emits no
  dedicated wake/notify (on SQLite the `data_version` watcher may
  fire spuriously — pinned as contract-legal), and a concurrent
  subscriber mid-`subscribe` never loses its place
  (offset-anchored reads only move forward).

## References

- [OQ-12](../../architecture/open-questions.md) — this ADR's resolution.
- [core-contract.md](../../architecture/core-contract.md) — the spec
  this ADR's §1–§5 pin into place (streams section, tx seam,
  guarantee table, backlog).
- The alkcall evidence (reference checkout
  `/workspace/@alkdev/alkcall` @ HEAD `main`, 2026-10-05, per the
  AGENTS.md §3 posture):
  decisions/021 (Sub — the Subscription streaming handler),
  decisions/046 (Pub + the `Subscription`→`Sub` rename + Gap B),
  decisions/037 (`channel/resources/subscribe`), open-questions.md
  OQ-22 (fan-out deferral) — the pub/sub vocabulary this contract's
  streams substrate; its no-per-key-ordering posture is the
  inventory row added for this decision.
- [consumer-inventory.md](../../research/consumer-inventory.md) — the
  streams row (operator-authority record) + the alkcall
  ecosystem-shape row; the (b)-gate that rejected server-enforced
  per-key ordering.
- [ADR-006](006-wake-and-delivery-contract.md) §2 — the guarantee
  table the streams ordering cell extends.
- [ADR-008](008-contract-v1-pinning.md) §1/§4/§5/§8 — the pinned
  method names the depth lands on, the namespace kinds, the
  act-differently error rule, the `payload_as`/`Codec` rule.
- [ADR-010](010-queue-semantics-depth.md) §6 — the no-ambient-
  sweeper posture this ADR's retention decision applies to the
  consumer-facing stream table (the notify-table hygiene precedent
  is §6's too).
- [ADR-012](012-forked-substrate-design.md) §3 — the fidelity posture
  the column-name keep rides; [ADR-014](014-outbox-tx-enqueue.md) —
  the amend-in-place framing §2 reuses.
- Honker stream machinery
  (`/workspace/honker` `honker-core/src/honker_ops.rs:1906-1996`, `src/lib.rs:417-429`; `packages/honker-rs/src/lib.rs:876-1010, 1099`)
  — the inherited realization (key column, offset-ASC reads, monotone
  offset saves, event shape); the unbounded-growth defect
  (reference-honker-machinery.md §8 defect 5).