From 8c4ec48f92640d272fc046715e08c9e4801dcaa7 Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Mon, 5 Oct 2026 14:15:28 +0000 Subject: [PATCH] =?UTF-8?q?ADR-015:=20streams=20depth=20=E2=80=94=20carrie?= =?UTF-8?q?d-metadata=20keys,=20global-FIFO=20ordering,=20StreamEvent,=20t?= =?UTF-8?q?rim=5Fto=20(OQ-12=20resolved)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/architecture/README.md | 14 +- docs/architecture/core-contract.md | 101 +++++- .../006-wake-and-delivery-contract.md | 2 +- .../decisions/008-contract-v1-pinning.md | 15 +- .../decisions/014-outbox-tx-enqueue.md | 2 +- .../decisions/015-streams-depth.md | 326 ++++++++++++++++++ docs/architecture/engine-postgres.md | 24 +- docs/architecture/engine-sqlite.md | 17 +- docs/architecture/open-questions.md | 123 ++++--- docs/architecture/overview.md | 2 + docs/architecture/queues.md | 8 +- docs/research/consumer-inventory.md | 38 +- 12 files changed, 562 insertions(+), 110 deletions(-) create mode 100644 docs/architecture/decisions/015-streams-depth.md diff --git a/docs/architecture/README.md b/docs/architecture/README.md index f4306ea..a8e365d 100644 --- a/docs/architecture/README.md +++ b/docs/architecture/README.md @@ -24,9 +24,9 @@ pending architecture review and OQ resolution. | Doc | Status | Purpose | Key OQs | |---|---|---|---| | [overview.md](overview.md) | draft | Crate family, feature surface, non-goals, evidence base | — | -| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-08, OQ-10, OQ-12 | -| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 | -| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-05 (resolved), OQ-08, OQ-12 | +| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-08, OQ-10 | +| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 (resolved), OQ-13 (resolved) | +| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-08, OQ-12 (resolved), OQ-13 (resolved) | | [queues.md](queues.md) | draft | Queue/scheduler/outbox semantics depth (resolved: ADR-009/ADR-010) | OQ-06 (resolved) | | [deployment.md](deployment.md) | draft | Host semantics, connection budgets, knobs, matrix | OQ-08 | | [open-questions.md](open-questions.md) | draft | OQ tracker (promoted from OQ-ST register) | — | @@ -49,6 +49,7 @@ pending architecture review and OQ resolution. | [012](decisions/012-forked-substrate-design.md) | Forked substrate design — contract-blind boundary, fidelity posture, port deltas | Accepted | | [013](decisions/013-fold-substrate-into-sqlite.md) | Fold the forked substrate into `alkstore-sqlite` — no fourth crate | Accepted | | [014](decisions/014-outbox-tx-enqueue.md) | Transactional outbox enqueue — `outbox_enqueue_tx` on the `TxHandle` trait | Accepted | +| [015](decisions/015-streams-depth.md) | Streams depth — carried-metadata keys, global-FIFO ordering row, `StreamEvent` shape, `publish_with_key_tx`, `trim_to` | Accepted | ## Open Questions @@ -57,10 +58,6 @@ Phase 0 register's OQ-ST-01..08 promote one-to-one — OQ-NN mirrors OQ-ST-NN — with new Phase 1 questions appended after). Open, in suggested resolution order: -- **OQ-12** (high): streams depth — key semantics, `StreamEvent` - shape, ordering guarantee row, retention (Phase 1 review find; - evidence gathered in the OQ; check consumer rows for a - per-key-ordering need before choosing among its options). - **OQ-08** (medium): capability-surface shape. - **OQ-10** (medium): contract versioning across engine crates. - **OQ-11** (medium): forked-substrate follow-through (provenance @@ -77,7 +74,8 @@ scheduler guarantee row pinned), ~~OQ-05~~ (queue semantics depth — (honker-core quality read — fork fired, [ADR-011](decisions/011-sqlite-substrate-fork.md)), ~~OQ-13~~ (transactional outbox enqueue shape — -[ADR-014](decisions/014-outbox-tx-enqueue.md)). +[ADR-014](decisions/014-outbox-tx-enqueue.md)), ~~OQ-12~~ (streams +depth — [ADR-015](decisions/015-streams-depth.md)). No deferred OQs: all open questions are actionable Phase 1 work with complete evidence bases. diff --git a/docs/architecture/core-contract.md b/docs/architecture/core-contract.md index 3bde094..df7f1a1 100644 --- a/docs/architecture/core-contract.md +++ b/docs/architecture/core-contract.md @@ -15,7 +15,10 @@ inventory-confirmed features — [ADR-002](decisions/002-feature-scope.md) `TxHandle` representation, wake type, reserved strings, error taxonomy, config split — amended in place, pre-implementation, by [ADR-014](decisions/014-outbox-tx-enqueue.md): `outbox_enqueue_tx` -joins the `TxHandle` trait); ADR-009/ADR-010 add the first *post-v1 +joins the `TxHandle` trait, and by +[ADR-015](decisions/015-streams-depth.md): streams depth — key +semantics, `StreamEvent` shape, the ordering row, `trim_to`, +and `publish_with_key_tx`); ADR-009/ADR-010 add the first *post-v1 contract extensions* (scheduler collapse surface, `QueueOpts` depth — versioning discipline for such extensions is OQ-10's); the *obligations* are this document. @@ -56,7 +59,8 @@ Store::begin_tx() -> Box trait TxHandle { enqueue_tx(name, opts, payload) -> job_id - publish_tx(stream, payload) -> event_id + publish_tx(stream, payload) -> offset + publish_with_key_tx(stream, key, payload) -> offset notify_tx(channel, payload) save_offset_tx(stream, consumer, offset) outbox_enqueue_tx(outbox, opts, payload) -> job_id @@ -68,13 +72,21 @@ trait TxHandle { queue engine-side (the reserved prefix makes the derived name unreachable by `enqueue_tx` by design — [ADR-014](decisions/014-outbox-tx-enqueue.md)). +`publish_with_key_tx` validates the key like the non-tx form (a +present key must be non-empty → `InvalidName`) and is +`publish_tx`'s keyed twin — commit-atomic keyed publishes are the +point ([ADR-015](decisions/015-streams-depth.md) §2). Mechanism handles come off the store (or, for transactional variants, off the handle): ```text store.notify(channel, payload) handle.notify_tx(channel, payload) -store.stream(name) -> Stream handle.publish_tx / save_offset_tx +store.stream(name) -> Stream handle.publish_tx / publish_with_key_tx + handle.save_offset_tx + (the Stream handle also carries + trim_to — stream-side, outside the + tx seam) store.queue(name, opts) -> Queue handle.enqueue_tx store.outbox(name) -> Outbox handle.outbox_enqueue_tx(outbox, opts, payload) store.try_lock(name, owner, ttl) -> Option @@ -138,21 +150,56 @@ cousin of notify: publish is commit-atomic; every committed event is readable by every consumer whose offset hasn't passed it; replay-on-attach is the default; offsets are explicit and transaction-aware (`save_offset_tx` gives exactly-once-within-a- -business-tx shape). +business-tx shape). Depth pinned by +[ADR-015](decisions/015-streams-depth.md). -- `publish` / `publish_tx` / `publish_with_key` — append to the - stream's durable log. *(Depth annotation: the key's semantics, the - `StreamEvent` shape, the ordering row, and log retention are - OQ-12's — method names pinned by - [ADR-008](decisions/008-contract-v1-pinning.md) §1, depth deferred - there; see [open-questions.md](open-questions.md).)* +- `publish` / `publish_tx` / `publish_with_key` / `publish_with_key_tx` + — append to the stream's durable log; return the assigned offset. + **Key semantics ([ADR-015](decisions/015-streams-depth.md) §1): the + key is carried metadata** — stored on the event row, round-tripped + on every read, `Option`-shaped on `StreamEvent` — with no + engine-enforced behavioral role: **the ordering guarantee stays + global FIFO by offset within the stream**; per-key in-order reading + is the documented emergent pattern (filter by key, read in offset + order), not a server promise. A present key must be non-empty + (`InvalidName`). Server-enforced per-key ordering re-enters only + via a consumer-inventory row naming it. +- **`StreamEvent` shape** ([ADR-015](decisions/015-streams-depth.md) + §3): `{ offset: i64, stream: String, key: Option, payload: + Vec, created_at: i64 }` — `stream` renames honker's `topic` + (one term everywhere, the ADR-008 §4 kinds list); `offset` is + engine-assigned, monotone per stream, **immutable, never renumbered** + (gaps after a trim are legal); `created_at` is unix-seconds-at- + publish, informational, **not an ordering field** (offset is the + only one); `payload_as` decodes (the error is `Codec`). +- **Ordering guarantee row** ([ADR-015](decisions/015-streams-depth.md) + §4, extending [ADR-006](decisions/006-wake-and-delivery-contract.md) + §2's streams row): `read_since`/`subscribe` yield `offset ASC` — + global FIFO per stream; committed-event visibility and + replay-on-attach as already pinned; offsets immutable. The + contract suite pins cross-engine equivalence against this row + (same publish sequence → same offset sequence → same read order). - `read_since` / `read_from_consumer(offset)` — cursor-based reads; - `get_offset(consumer)` — checkpoint inspection. + `get_offset(consumer)` — checkpoint inspection. A read from a + trimmed-away offset region (below) resumes at the trim horizon's + first remaining row. - `save_offset` / `save_offset_tx` — consumer checkpoint, explicit; **the contract's save is always explicit** (no auto-checkpoint cadence — honker's per-binding ambiguity is not inherited). The subscription handle exposes no `save_every` / auto-save-on-drop - ([ADR-008](decisions/008-contract-v1-pinning.md) §8). + ([ADR-008](decisions/008-contract-v1-pinning.md) §8). Saves are + monotone; a saved offset below the trim horizon + ([ADR-015](decisions/015-streams-depth.md) §5) stays a valid + position marker. +- `trim_to(horizon)` — delete events with `offset <= horizon` + ([ADR-015](decisions/015-streams-depth.md) §5). The stream-side + bounded-growth op, consumer-invoked, no engine-default + retention and no ambient sweeper (the ADR-010 §6 posture; the + replay-forever default is the mechanism's purpose — growth is + documented with the tool in-contract to bound it). Trim emits no + dedicated wake and no notify — on SQLite the `data_version` + watcher may still fire (any committed write bumps it; a spurious + hint, contract-legal); surviving events keep their offsets. - `subscribe(consumer) -> Box` — durable consumption: attach, read to current tail, resume after restart from the stored offset; explicit `save_offset` on the receiver @@ -302,8 +349,9 @@ inherits them and adds the consumer obligations: consumer's, correctness-of-transition the engine's ([ADR-010](decisions/010-queue-semantics-depth.md) §6); - `*_tx` operations are *only* durable after the caller's commit — - rollback drops job rows, event rows, notifications, and offset saves - together (the no-ghosts property, POC-pinned on both engines). + rollback drops job rows, event rows (keyed and unkeyed alike), + notifications, and offset saves together (the no-ghosts property, + POC-pinned on both engines). ### Errors @@ -412,6 +460,23 @@ before the engine specs are called `stable`: outbox names rejected identically on the tx path (`ReservedName`/`InvalidName`); stamped opts visible via `get_job` equal for both engines. +- **Cross-engine stream equivalence** ([ADR-015](decisions/015-streams-depth.md) + §3/§4) — publish sequence → identical offset sequence → identical + `read_since` output order on both engines, direct and subscriber + reads alike; keyed/unkeyed interleavings preserve global FIFO; + `key` round-trips exactly (`None` vs `Some`), `stream`/`created_at` + fields equal. +- **`publish_with_key_tx` commit-atomicity on both engines** + ([ADR-015](decisions/015-streams-depth.md) §2) — rollback drops the + keyed event row with the business write (no ghost event); commit + makes it visible to `read_since`/`subscribe`; empty-`Some`-key + `InvalidName` on both engines' tx paths. +- **`trim_to` semantics on both engines** ([ADR-015](decisions/015-streams-depth.md) + §5) — exact-boundary trim (`<=`), surviving rows keep their offsets + (gaps legal, never renumbered), reads resume at the trim horizon's + first remaining row, saved offsets below the horizon stay valid, + trim wakes nothing, and a concurrent subscriber never loses its + place. ## Design Decisions @@ -427,6 +492,7 @@ before the engine specs are called `stable`: | [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite substrate owned (`__alkstore_*` naming); queue ops re-derived on contract v1 | | [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; engine-side formula arithmetic pinned equivalent by the contract suite | | [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue (amends 008) | `outbox_enqueue_tx` on `TxHandle`; outbox-name validation; derived backing queue reached only through the outbox surface | +| [015](decisions/015-streams-depth.md) | Streams depth (amends 006/008) | key = carried metadata, global-FIFO ordering row, `StreamEvent` shape, `publish_with_key_tx`, `trim_to` | ## Open Questions @@ -437,11 +503,10 @@ questions affecting this document: - **OQ-10**: contract versioning discipline across engine crates ([open](open-questions.md)) - **OQ-08**: capability-surface shape ([open](open-questions.md)) -- **OQ-12**: streams depth — key semantics, `StreamEvent` shape, - ordering row, retention ([open](open-questions.md)) Resolved on this document's surface: **OQ-09** (scheduler collapse — [ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), -and **OQ-13** (transactional outbox enqueue shape — -[ADR-014](decisions/014-outbox-tx-enqueue.md)), 2026-10-05. \ No newline at end of file +**OQ-13** (transactional outbox enqueue shape — +[ADR-014](decisions/014-outbox-tx-enqueue.md)), and **OQ-12** (streams +depth — [ADR-015](decisions/015-streams-depth.md)), 2026-10-05. \ No newline at end of file diff --git a/docs/architecture/decisions/006-wake-and-delivery-contract.md b/docs/architecture/decisions/006-wake-and-delivery-contract.md index 3f2195d..de7b64e 100644 --- a/docs/architecture/decisions/006-wake-and-delivery-contract.md +++ b/docs/architecture/decisions/006-wake-and-delivery-contract.md @@ -59,7 +59,7 @@ coalesced-but-complete re-read on SQLite). | Mechanism | Durability | Replay | Atomicity | Guarantee | |---|---|---|---|---| | `notify` / listen | none | never | commit-atomic (delivers at commit; rollback drops) | fire-and-forget, at-most-once per listener session | -| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset | +| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset; **global FIFO by offset within each stream** (read paths yield `offset ASC`; offsets immutable, never renumbered — [ADR-015](015-streams-depth.md) §4) | | queues | durable row | claim/ack model | enqueue is commit-atomic | at-least-once *work* with visibility timeouts | - The trait **does not promise replay under `listen()`** — on either diff --git a/docs/architecture/decisions/008-contract-v1-pinning.md b/docs/architecture/decisions/008-contract-v1-pinning.md index 570c91c..22bbbd4 100644 --- a/docs/architecture/decisions/008-contract-v1-pinning.md +++ b/docs/architecture/decisions/008-contract-v1-pinning.md @@ -53,7 +53,11 @@ implement identically): shape, the ordering guarantee row, and log retention are OQ-12's — the method *names* are pinned here, their depth was left un-pinned, mirrored in [core-contract.md](../core-contract.md) - streams.)* + streams. **Resolved 2026-10-05 by [ADR-015](015-streams-depth.md)** — + key = carried metadata, global-FIFO ordering row, event shape + pinned, `trim_to` added; `publish_with_key_tx` also joins the + tx seam (§2 there); the depth was completed in place, + pre-implementation.)* - queues v1 skeleton — `enqueue`, `claim_one`, `claim_batch`, `ack_batch`, `cancel`, `get_job`, `sweep_expired`; job handle `ack / retry / fail / heartbeat`; `EnqueueOpts { delay, run_at, @@ -114,7 +118,8 @@ a **`TxHandle` trait carrying the `*_tx` methods directly**: ```text trait TxHandle { enqueue_tx(name, opts, payload) -> job_id - publish_tx(stream, payload) -> event_id + publish_tx(stream, payload) -> offset + publish_with_key_tx(stream, key, payload) -> offset // ADR-015 notify_tx(channel, payload) save_offset_tx(stream, consumer, offset) outbox_enqueue_tx(outbox, opts, payload) -> job_id // ADR-014 @@ -277,7 +282,10 @@ v1 variants (guaranteed-matchable on every engine): `try_recv` / `recv_timeout` form of the wake close in §3). - `Codec` — payload serialization/deserialization failures (`payload_as` on stream events and job payloads; wakes carry no - payload per §3). + payload per §3). *(Annotated 2026-10-05: a present key on + `publish_with_key(_tx)` being non-empty is validated as + `InvalidName` ([ADR-015](015-streams-depth.md) §2 — the same + entry-point-validation variant, not a new one).)* - `Database` — everything else: driver/connection/SQL errors, opaque to the contract, engine detail preserved via the error source chain. No engine-specific variants are minted for its contents. @@ -339,6 +347,7 @@ differs): | `Lock::heartbeat(ttl)` | `Lock::renew(ttl)` | renew describes the TTL discipline the guarantee row pins; heartbeat is the queue-side renewal term | | `Queue::claim_waker()` | (not surfaced) | engine-internal consumption wake (§1) | | `StreamSubscription` (`save_every`, auto-save on drop) | `subscribe(consumer) -> Box`, explicit saves only | the explicit-offset obligation (core-contract.md streams; the auto-checkpoint ambiguity is not inherited, [ADR-006](006-wake-and-delivery-contract.md)) | +| `StreamEvent.topic` (field) | `StreamEvent.stream` *(added 2026-10-05 by [ADR-015](015-streams-depth.md) §3)* | one term for the mechanism everywhere — the contract names it streams (the §4 kinds list); the substrate's column stays `topic` per the [ADR-012](012-forked-substrate-design.md) §3 fidelity posture | | `update_events()` | (not surfaced) | superseded by the wake subscription — the pinned `WakeReceiver` *is* the reactive-event surface; honker's separate raw update-event stream has no contract role | | `prune_notifications` / `prune_notifications_keep_latest` | (not surfaced) | notifications-table maintenance tooling on the SQLite engine; disposition rides OQ-05's sweep/maintenance design (consumer-visible only if that design surfaces it) | | `Scheduler` surface | (out of v1, OQ-09 decides) | §1 | diff --git a/docs/architecture/decisions/014-outbox-tx-enqueue.md b/docs/architecture/decisions/014-outbox-tx-enqueue.md index f578f1b..b018682 100644 --- a/docs/architecture/decisions/014-outbox-tx-enqueue.md +++ b/docs/architecture/decisions/014-outbox-tx-enqueue.md @@ -53,7 +53,7 @@ One method, added to the `TxHandle` trait: ```text trait TxHandle { enqueue_tx(queue, opts, payload) -> job_id - publish_tx(stream, payload) -> event_id + publish_tx(stream, payload) -> offset notify_tx(channel, payload) save_offset_tx(stream, consumer, offset) outbox_enqueue_tx(outbox, opts, payload) -> job_id // the addition diff --git a/docs/architecture/decisions/015-streams-depth.md b/docs/architecture/decisions/015-streams-depth.md new file mode 100644 index 0000000..9836142 --- /dev/null +++ b/docs/architecture/decisions/015-streams-depth.md @@ -0,0 +1,326 @@ +# ADR-015: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention + +## Status + +Accepted + +## Context + +[OQ-12](../../architecture/open-questions.md) — found by the contract +v1 Phase 1 architecture review: [ADR-008](008-contract-v1-pinning.md) +§1 pinned streams' *method names* (`publish`, `publish_with_key`, +`read_since`, `read_from_consumer`, `save_offset`, `get_offset`, +`subscribe`) but not their *depth*. Four sub-questions were open: + +1. **Key semantics** — what `publish_with_key`'s key means, and what + if any ordering it buys. Honker's realization is a stored nullable + `key` column (`_honker_stream(topic, key, payload)`) with no + per-key read path or ordering enforcement anywhere; the doc comment + says "used for per-key ordering downstream" + (`packages/honker-rs/src/lib.rs:894`) — per-key FIFO is an + *emergent* property of a consumer reading `offset ASC` filtered by + key, not a server guarantee. Three options were on the table: (a) + pin honker's shape (key = carried metadata, ordering stays global + FIFO, document the emergent pattern); (b) pin server-enforced + per-key ordering (real machinery on both engines, gated on a + consumer-inventory row naming the need); (c) cut `publish_with_key` + from v1 (no consumer row names keys). + +2. **`StreamEvent` shape** — honker's is + `{ offset: i64, topic: String, key: Option, payload: + Vec, created_at: i64 }` with `payload_as` decode + (lib.rs:1008). The contract needs its own pinned field list. + +3. **Ordering guarantee row** — [ADR-006](006-wake-and-delivery-contract.md) + §2's delivery table pins streams' durability/replay/atomicity but + its ordering cell content was effectively "readable"; the contract + suite cannot pin cross-engine stream equivalence without an + explicit ordering row. + +4. **Retention / bounded growth** — the reference read flags the + stream table as unbounded-growth-unless-swept (honker-machinery + §8 defect 5); no consumer row names retention + (replay-deep-history is the *purpose* of the subscription need). + +The consumer evidence was completed for this decision by adding the +**alkcall row** to +[consumer-inventory.md](../../research/consumer-inventory.md) +(ecosystem-shape evidence, added per the inventory's own +row-before-assumption convention). Alkcall's pub/sub vocabulary — +`Sub` operations (server→client streaming, ADR-021), `Pub` +(client→server streaming, ADR-046), `channel/resources/subscribe` +(snapshot + change events, ADR-037), and the deferred +broker/fan-out (ADR-046's Gap B, channels OQ-22) — is the shape +downstream crates' reactive ops are expected to take over this store, +and the operator confirmed alkstore is meant as the base storage +layer for that ecosystem. Its ordering posture is per-stream +(ordered, reliable transport streams; correlation by request ID): +**no alkcall mechanism names server-enforced per-key ordering.** The +topic/event-type dimension of its subscriptions routes to *separate +streams* (one stream per topic; e.g. `channel/resources/subscribe` is +its own event source), not to keys within one stream. + +## Decision + +### 1. Key semantics: option (a) — the key is carried metadata; ordering stays global FIFO + +Option (b) is **rejected on the decision rule it named for itself**: +it requires "a consumer row naming the need," and neither the +inventory's original rows nor the alkcall row names server-enforced +per-key ordering anywhere. Real machinery (per-key sequencing, +per-key claim/exhaustion paths, or per-key head-of-line bookkeeping on +both engines) for an unnamed need is exactly the scope discipline +[ADR-002](002-feature-scope.md) exists to enforce. + +Option (c) (cut `publish_with_key` from v1) is **rejected**: the +method name is a pinned v1 surface element +([ADR-008](008-contract-v1-pinning.md) §1 — removing it would be a +contract-breaking event for OQ-10 to govern, not a pre-implementation +refinement); the *cost* of carrying the key in the inherited +realizations is one nullable column honker's schema already has; and +the key is genuinely useful as a *grouping token* — the emergent +per-key pattern is real and cheap for consumers (SQL-level filter + +in-order read), and an ecosystem whose ops carry keys (a repo id, an +entity id) deserves a stable way to store that grouping durably. +Cutting the only keyed-write path would push consumers to encode keys +into payloads — an unqueryable convention — the same shape of waste +ADR-010 §7 called out for result storage. + +Pinned semantics, exactly honker's: + +- `publish_with_key(stream, key, payload)` stores `key` as a nullable + column value on the event row; `publish` stores `NULL`. +- The key is **carried metadata** — it appears on every `StreamEvent` + read back and is `Option`-shaped in the event type; the + engine assigns it no behavioral role (it rides the row the publish + writes; the wake trigger is the publish itself, key or no key). +- **The ordering guarantee stays global FIFO by offset, per stream** — + regardless of key. Per-key FIFO-by-offset is the *documented + emergent pattern*: a consumer that filters a read by key observes + that stream's events for that key in offset order. The contract + does not enforce, promise, or price any cross-subscriber per-key + handoff. (The guarantee row, §3, is written so this remains *true + under trimming* — offset order, never renumbered.) +- The key is **not** a partitioning instruction, not a dedup key, and + not an ordering key in any engine-enforced sense. Nothing in the + contract suite tests keys behaviorally beyond carried-metadata + round-trip and equivalence across engines. + +The alkcall evidence makes this the right ecosystem shape too: its +subscriptions distinguish event *sources* (separate streams per +topic), not positions *within* a source (per-key lanes). When the +channels broker (Gap B) arrives, topic matching selects streams; +within a stream, offset order is the whole story. + +### 2. `publish_with_key_tx` — the tx seam carries the key + +A gap the key decision exposed in the pinned tx seam: +[TxHandle](../../architecture/core-contract.md) carries +`publish_tx(stream, payload)` but no keyed form — so a keyed event +could not be written commit-atomic with a business write, breaking the +no-ghosts property's uniformity for keyed publishes (the same shape of +hole OQ-13 exposed for outbox enqueue, resolved by +[ADR-014](014-outbox-tx-enqueue.md)). Pinned: +`publish_with_key_tx(stream, key, payload) -> offset` joins the +TxHandle trait — one method, symmetric with `publish_tx`, validating +the key the same way the non-tx form does (`None` or non-empty — an +empty `Some` key is `InvalidName`) with no new error variants +([ADR-008](008-contract-v1-pinning.md) §5's rule). Framed as +completing contract v1 in place, pre-implementation, like ADR-014 — +no versioning event, OQ-10 untouched. A plain-`publish_tx` is +`publish_with_key_tx(stream, None, payload)`, not a separate +engine path. *(Naming pinned here too: the publish family returns the +**assigned offset** — honker's `publish` already returns it — and the +tx-seam sketches' `event_id` placeholder name is corrected to +`offset` so one term covers the value every read/cursor/trim method +speaks.)* + +### 3. `StreamEvent` shape — honker's, with one rename + +Pinned (inherited near-verbatim, per the fork's API fidelity posture, +[ADR-012](012-forked-substrate-design.md) §3): + +```text +struct StreamEvent { + offset: i64, // position in the stream's log; immutable + stream: String, // the stream name (honker: "topic") + key: Option, // §1's carried metadata; None unless + // published keyed + payload: Vec, // core value bytes + created_at: i64, // unix seconds at publish (the engines' + // single clock source) +} +``` + +- The `topic` field renames to **`stream`** — the contract surface + names this mechanism streams, not topics (honker-rs's `Stream` + handle's `topic()` accessor is likewise the stream name); one term + everywhere, matching the reserved-namespace kinds list + ([ADR-008](008-contract-v1-pinning.md) §4). +- `payload` is core value bytes crossed as a non-generic value; the + concrete `StreamEvent` owns the bytes; `payload_as` is the decode + convenience ([ADR-008](008-contract-v1-pinning.md) §8's rule — the + deserialization error is `Codec`). +- `created_at` is **unix seconds at publish**, from the engine's + single clock source (the partial-index/clock-pin rule in + [queues.md](../../architecture/queues.md)); it is *informational* + and NOT an ordering candidate — offset is the only ordering field + (two events can share a second; offset never collides). +- `offset` is the stream-wide position, engine-assigned at publish, + monotonically increasing per stream on both engines. It is + **immutable once assigned** and is the value every read/cursor/ + trim method speaks. + +### 4. Ordering guarantee row — pinned + +ADR-006 §2's table's streams row extends with an explicit ordering +cell (the guarantee cell gains a clause; durability/replay/atomicity +as already pinned): + +> **streams** — Guarantee: *global FIFO by offset within each stream +> (`read_since`/`subscribe` yield `offset ASC`); every committed +> event readable by every consumer that hasn't passed its offset; +> offsets immutable and never renumbered.* + +This is the row the contract suite pins cross-engine stream +equivalence against: two engines, same publish sequence → same offset +sequence → same `read_since` output order, for direct and subscriber +reads alike. It is deliberately *not* stronger than both engines can +implement by a shared SQL shape (`ORDER BY offset ASC` is already +both engines' claim path) and not weaker than the mechanism's purpose +(replay demands monotone positions). + +### 5. Retention: `trim_to` on the stream handle — a consumer-invoked bounded-growth op + +The mechanism-shaped answer to defect 5's family (unbounded +`_honker_stream` growth), applying the ADR-010 §6 posture to the one +stream-side table that consumers interact with *as rows* (unlike +notifications, the transport detail whose hygiene stays engine- +internal): + +- `store.stream(name)`'s handle gains `trim_to(horizon)` — delete + events with `offset <= horizon`, emitting no dedicated wake and no + notify (deletes are not events; consumers whose cursors move + forward never lose reads). On SQLite the `data_version` watcher may + still fire on the trim commit — any committed write bumps it; that + spurious hint is contract-legal (the wake contract's best-effort + + idempotence posture, ADR-006 §1), not a trim-specific delivery. + `OffsetGone`-style errors are not needed: reads past the trim + horizon simply return fewer/no rows (a consumer reading from a + trimmed region has, by definition, an offset older than the + horizon — that is the API contract; re-attaching consumers at + trimmed-away offsets resume at the horizon's first remaining row). +- **No engine-side retention default, no ambient sweeper** (the + ADR-010 §6 posture; consumer-side, cadenced by the collapse recipe + like every sweep). Trim is *available*, not automatic: the + replay-forever default is the *purpose* for the consumer rows, so + the default is unbounded — with the growth documented squarely, + this time with an in-contract tool to fix it. +- **Caveat documented in-contract**: offsets are never renumbered and + never reused; surviving events keep the offsets they were published + with (gaps after a trim are legal and ordinary). Consumers must + not assume dense offset sequences (the same discipline the + scheduler's catch-up/skip-forward already imposes). +- Trim's effect on cursors: a consumer's *saved* offset below the + trim horizon stays (it remains a valid position marker; the next + read resumes at the horizon's first remaining row); a consumer may + re-save after processing. This keeps `get_offset`/`subscribe` + resume semantics well-defined without engine-side cursor-tracking + machinery. + +Rejected retention alternatives (per the OQ's option set): the +consumer-recipe-without-an-API option was self-admittedly "document +the gap" (no deletion path exists today — this ADR *creates* the +deletion path, which is what the recipe needed); documenting-nothing +repeats the notifications-growth mistake ADR-010 §6 just fixed. + +## Consequences + +**Positive** + +- Streams' depth is fully pinned: names (ADR-008 §1) + semantics (this + ADR) make the mechanism implementable on both engines from the same + text, and the contract suite has an ordering row to pin cross-engine + equivalence against. +- The key decision is honest: a zero-machinery feature (carried + metadata) documented for exactly what it is, with option (b)'s + re-entry gate (a consumer-inventory row naming server-enforced + per-key ordering) recorded — the ecosystem-shape evidence says the + topic dimension routes to streams, not keys. +- Keyed publishes are tx-seam-native — keyed and unkeyed events carry + the same commit-atomicity, so an outbox-style keyed event log co-writes + with business data correctly. +- `trim_to` gives the family a real answer to stream growth in the + contract, consistent with the no-ambient-sweeper posture (invoked + deliberately, cadenced from above). + +**Negative** + +- Per-key *ordering enforcement*, if a consumer ever names it, is a + later contract extension with real machinery on both engines — + carrying metadata now does not prejudge or approximate it, and the + emergent pattern's documented status must not be read as a promise. +- `trim_to` adds one more consumer-invoked maintenance op to the + surface (backlog rows follow). +- `created_at` is second-precision (the shared clock pin) — consumers + needing sub-second publish timestamps must encode them in payloads. +- The `StreamEvent.stream` rename diverges from honker's wire-era name + (`topic`) — the fork keeps the column name `topic` engine-internally + per the ADR-012 §3 fidelity posture; only the contract type renames. + +## Verification backlog additions + +- **Cross-engine stream equivalence** (the §3/§4 row): publish + sequence → identical offset sequence → identical `read_since` + output order, both engines; keyed and unkeyed interleavings + preserve global FIFO; `key` round-trips exactly (`None` vs + `Some`), `StreamEvent.stream`/`created_at` equal on both engines. +- **`publish_with_key_tx` commit-atomicity on both engines** (§2): + rollback drops the keyed event row with the business write (no + ghost event); commit makes it visible to `read_since`/`subscribe`; + empty-`Some`-key `InvalidName` on both engines' tx paths. +- **`trim_to` semantics on both engines** (§5): exact-boundary trim + (`<=`), surviving rows keep their offsets (gaps legal, never + renumbered), reads from a trimmed horizon resume correctly, + `save_offset` below the horizon stays valid, trim emits no + dedicated wake/notify (on SQLite the `data_version` watcher may + fire spuriously — pinned as contract-legal), and a concurrent + subscriber mid-`subscribe` never loses its place + (offset-anchored reads only move forward). + +## References + +- [OQ-12](../../architecture/open-questions.md) — this ADR's resolution. +- [core-contract.md](../../architecture/core-contract.md) — the spec + this ADR's §1–§5 pin into place (streams section, tx seam, + guarantee table, backlog). +- The alkcall evidence (reference checkout + `/workspace/@alkdev/alkcall` @ HEAD `main`, 2026-10-05, per the + AGENTS.md §3 posture): + decisions/021 (Sub — the Subscription streaming handler), + decisions/046 (Pub + the `Subscription`→`Sub` rename + Gap B), + decisions/037 (`channel/resources/subscribe`), open-questions.md + OQ-22 (fan-out deferral) — the pub/sub vocabulary this contract's + streams substrate; its no-per-key-ordering posture is the + inventory row added for this decision. +- [consumer-inventory.md](../../research/consumer-inventory.md) — the + streams row (operator-authority record) + the alkcall + ecosystem-shape row; the (b)-gate that rejected server-enforced + per-key ordering. +- [ADR-006](006-wake-and-delivery-contract.md) §2 — the guarantee + table the streams ordering cell extends. +- [ADR-008](008-contract-v1-pinning.md) §1/§4/§5/§8 — the pinned + method names the depth lands on, the namespace kinds, the + act-differently error rule, the `payload_as`/`Codec` rule. +- [ADR-010](010-queue-semantics-depth.md) §6 — the no-ambient- + sweeper posture this ADR's retention decision applies to the + consumer-facing stream table (the notify-table hygiene precedent + is §6's too). +- [ADR-012](012-forked-substrate-design.md) §3 — the fidelity posture + the column-name keep rides; [ADR-014](014-outbox-tx-enqueue.md) — + the amend-in-place framing §2 reuses. +- Honker stream machinery + (`/workspace/honker` `honker-core/src/honker_ops.rs:1906-1996`, `src/lib.rs:417-429`; `packages/honker-rs/src/lib.rs:876-1010, 1099`) + — the inherited realization (key column, offset-ASC reads, monotone + offset saves, event shape); the unbounded-growth defect + (reference-honker-machinery.md §8 defect 5). \ No newline at end of file diff --git a/docs/architecture/engine-postgres.md b/docs/architecture/engine-postgres.md index 8107b86..6990481 100644 --- a/docs/architecture/engine-postgres.md +++ b/docs/architecture/engine-postgres.md @@ -64,7 +64,7 @@ obligations live in the core spec. | Contract piece | Engine realization | |---|---| | notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers | -| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) | +| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape). Depth per [ADR-015](decisions/015-streams-depth.md): the event table carries the nullable key column (carried metadata), reads yield `offset ASC` (global FIFO — bigserial offsets, immutable), `publish_with_key_tx` validates + inserts inside the caller's tx, `trim_to` is a `DELETE … WHERE offset <= ?` on a pool connection (wakes nothing) | | queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema; the curve/stamps arithmetic is computed engine-side per [ADR-012](decisions/012-forked-substrate-design.md) §2 (equivalence with the SQLite engine pinned by the contract suite) | | named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; guarantee row pinned by [ADR-008](decisions/008-contract-v1-pinning.md) §7) | | scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule rows in the engine-owned schema, tick re-derived (boundary advance + 64-boundary catch-up cap, honker parity), leadership via the engine's lock machinery on `__alkstore_scheduler`; outbox = helper over queues | @@ -110,6 +110,7 @@ the record in [ADR-003](decisions/003-sqlite-driver.md). | [010](decisions/010-queue-semantics-depth.md) | Queue depth | job-stamped opts, equal-jitter backoff, dead-letter move, no-stranded-rows sweep, one engine-owned schema | | [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate (SQLite side); pg engine owns its own curve/stamps arithmetic, equivalence pinned by the contract suite | | [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` on `TxHandle`; derivation engine-side inside the caller's tx | +| [015](decisions/015-streams-depth.md) | Streams depth | nullable key column (carried metadata); bigserial offsets, global-FIFO reads; keyed tx publish; `trim_to` as a pool-connection delete | ## Open Questions @@ -119,17 +120,16 @@ questions affecting this document: - **OQ-08**: capability surface (shared with [deployment.md](deployment.md)) (open) -- **OQ-12**: streams depth (affects this engine's stream realization - — event-shape/ordering/retention equivalents on the pg side) - ([open](open-questions.md)) -- **OQ-13**: transactional outbox enqueue shape — **resolved** - (2026-10-05, [ADR-014](decisions/014-outbox-tx-enqueue.md)): - `outbox_enqueue_tx(outbox, opts, payload)` on the `TxHandle` trait; - the pg engine validates the outbox name and derives - `__alkstore_outbox:{name}` engine-side inside the caller's - transaction. +- **OQ-12**: streams depth — **resolved** + (2026-10-05, [ADR-015](decisions/015-streams-depth.md)): key = + carried metadata; global-FIFO-by-offset ordering row + (bigserial offsets, immutable); `StreamEvent` shape pinned + (`topic` → `stream` in the contract type); `publish_with_key_tx` + inside the caller's tx; `trim_to` on the stream handle — no + engine-default retention. Resolved: **OQ-09** (scheduler collapse — -[ADR-009](decisions/009-scheduler-collapse.md)) and **OQ-05** (queue -semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), +[ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue +semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), and +**OQ-12** (streams depth — [ADR-015](decisions/015-streams-depth.md)), 2026-10-05. \ No newline at end of file diff --git a/docs/architecture/engine-sqlite.md b/docs/architecture/engine-sqlite.md index e5a953e..e55b975 100644 --- a/docs/architecture/engine-sqlite.md +++ b/docs/architecture/engine-sqlite.md @@ -60,7 +60,7 @@ per-contract obligations live in the core spec and are not restated here. | Contract piece | Engine realization | |---|---| | notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) | -| streams | honker's stream machinery; explicit offset saves through the tx seam | +| streams | the forked substrate's stream machinery (inherited near-verbatim — the `__alkstore_stream` table's field set is already the contract's shape (its `topic` column carries the contract's `stream` field — the one name delta, ADR-012 §3's fidelity posture), the offset-ASC read path, keyed publish's nullable key column, monotone offset saves ([ADR-015](decisions/015-streams-depth.md)); explicit offset saves through the tx seam; `trim_to` is a `DELETE FROM … WHERE offset <= ?` inside the writer-slot lease) | | queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) | | named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) | | scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) | @@ -126,6 +126,7 @@ family is `__alkstore_*` (ADR-010 §8's naming authorization). | [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate, fidelity posture, port deltas, panic-free watcher death | | [013](decisions/013-fold-substrate-into-sqlite.md) | Substrate packaging | folded into `alkstore-sqlite` (`src/substrate/`); no fourth crate | | [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` on `TxHandle`; derivation engine-side through the writer-slot lease | +| [015](decisions/015-streams-depth.md) | Streams depth | key = carried metadata (the inherited nullable column); global-FIFO ordering; `trim_to` as a writer-lease delete; event shape already the substrate's | ## Open Questions @@ -136,8 +137,13 @@ questions affecting this document: - **OQ-06**: honker-core quality read — fork-trigger assessment — **resolved** (2026-10-05, [ADR-011](decisions/011-sqlite-substrate-fork.md); trigger fired). -- **OQ-12**: streams depth (affects this engine's stream realization - — key column, event shape, retention) ([open](open-questions.md)) +- **OQ-12**: streams depth — **resolved** + (2026-10-05, [ADR-015](decisions/015-streams-depth.md)): key = + carried metadata; global-FIFO-by-offset ordering row; + `StreamEvent` shape (honker's, `topic` → `stream` in the contract + type — the substrate's column name stays per the ADR-012 §3 + fidelity posture); `publish_with_key_tx` routes through the + writer-slot lease; `trim_to` is a plain delete in the same lease. - **OQ-13**: transactional outbox enqueue shape — **resolved** (2026-10-05, [ADR-014](decisions/014-outbox-tx-enqueue.md)): `outbox_enqueue_tx(outbox, opts, payload)` on the `TxHandle` trait; @@ -146,6 +152,7 @@ questions affecting this document: substrate's `__alkstore_outbox:{name}` backing queue. Resolved: **OQ-09** (scheduler collapse — -[ADR-009](decisions/009-scheduler-collapse.md)) and **OQ-05** (queue -semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), +[ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue +semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), and +**OQ-12** (streams depth — [ADR-015](decisions/015-streams-depth.md)), 2026-10-05. \ No newline at end of file diff --git a/docs/architecture/open-questions.md b/docs/architecture/open-questions.md index 489867c..8792c6d 100644 --- a/docs/architecture/open-questions.md +++ b/docs/architecture/open-questions.md @@ -23,11 +23,12 @@ is OQ-11); **the fork packaging folded** (2026-10-05, `alkstore-substrate` crate; the forked machinery is a module subtree of `alkstore-sqlite`); **OQ-13 resolved** (2026-10-05, [ADR-014](decisions/014-outbox-tx-enqueue.md) — -`outbox_enqueue_tx` joins the `TxHandle` trait, the hole is closed). -Next: **OQ-12** (streams depth — a Phase 1 review -find: key semantics, `StreamEvent` shape, ordering row, retention; -evidence gathered; check consumer rows for per-key-ordering need -before choosing among its options), OQ-08 (rides the now-pinned trait +`outbox_enqueue_tx` joins the `TxHandle` trait, the hole is closed); +**OQ-12 resolved** (2026-10-05, +[ADR-015](decisions/015-streams-depth.md) — streams depth pinned: +carried-metadata keys, global-FIFO ordering row, `StreamEvent` shape, +`publish_with_key_tx`, `trim_to`). +Next: **OQ-08** (rides the now-pinned trait shape and the ADR-011 substrate fork), OQ-10 (versioning discipline for contract extensions; note ADR-011 changes its substrate-side facts for SQLite — the forked machinery lives in-tree inside the engine @@ -131,7 +132,8 @@ narrowed to the pinning work its own record already scoped.)* resolution. A verification backlog (core-contract.md §Verification backlog) tracks the one-engine-pinned properties. - **Cross-references**: OQ-01, OQ-10, OQ-09, OQ-08; OQ-13 (the one - gap the v1 pinning's own Phase 1 review found in it). + gap the v1 pinning's own Phase 1 review found in it), OQ-12 (the + sibling depth gap — resolved, [ADR-015](decisions/015-streams-depth.md)). ## Theme: Queues and scheduling @@ -308,59 +310,69 @@ narrowed to the pinning work its own record already scoped.)* ## Theme: Core contract (Phase 1 review finds) -### OQ-12: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention +### OQ-12: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention — **RESOLVED** - **Origin**: [core-contract.md](core-contract.md) streams section, [ADR-008](decisions/008-contract-v1-pinning.md) §1 (Phase 1 architecture review, 2026-10-05) -- **Status**: open +- **Status**: resolved (2026-10-05, Phase 1 — + [ADR-015](decisions/015-streams-depth.md)) - **Priority**: high (streams is a first-class [ADR-002](decisions/002-feature-scope.md) mechanism; ADR-008's - "no placeholder semantics remain" positive consequence is false for - `publish_with_key` until this resolves — the contract suite also - cannot pin cross-engine stream equivalence without an ordering row) -- **Resolution**: open. The method *names* are pinned (ADR-008 §1); - the depth is not. Sub-questions, with the evidence already gathered: - - **Key semantics** — honker's realization is a stored `key` column - (`_honker_stream(topic, key, payload)`, NULLable) with no - per-key read path or ordering enforcement anywhere; the honker-rs - doc comment says "used for per-key ordering downstream" - (`packages/honker-rs/src/lib.rs:894`), i.e. per-key global-FIFO - is an *emergent* property (a consumer reading `offset ASC` - filtered by key), not server-enforced. Options: (a) pin exactly - honker's shape — key is carried metadata, ordering stays global - FIFO (document the emergent per-key pattern), cheapest and - honest; (b) pin *server-enforced* per-key ordering (a read or - delivery guarantee keyed on `key`) — real machinery on both - engines, needs a consumer row naming the need; (c) cut - `publish_with_key` from v1 (no consumer row names keys — the - honker-rs doc comment is upstream's intent, not ours). Decision - rule: consumer-inventory row first if (b). - - **`StreamEvent` shape** — honker's: `{ offset: i64, topic: String, - key: Option, payload, created_at: i64 }` with - `payload_as` (upstream lib.rs:1008). The contract needs its own - pinned field list (stream name vs topic token, key in or out per - the key decision, timestamp semantics) — probably inherited - near-verbatim. - - **Ordering guarantee row** — ADR-006's table has no streams - *ordering* cell content beyond "readable"; pin FIFO-by-offset as - the read ordering (`ORDER BY offset ASC` is already both - engines' claim path) and whether any per-key strengthening rides - the key decision. - - **Retention / bounded growth** — the reference read flags - `_honker_stream` as unbounded-growth unless the user sweeps - (honker-machinery §8.5); no consumer row names stream retention - (replay-forever is the *purpose* for the named need — - subscription surfaces). Options: consumer-side trim API - (`trim_to(offset)` — a new contract method), consumer-recipe - only (new stream + advance consumers + unschedule/nothing — - needs a deletion path that doesn't exist yet, so this option is - really "document the gap"), or document nothing (honest but - repeats the notifications-table mistake ADR-010 §6 just fixed). - Rides the no-ambient-sweeper posture either way. + "no placeholder semantics remain" positive consequence was false + for `publish_with_key` until this resolved — the contract suite + also could not pin cross-engine stream equivalence without an + ordering row) +- **Resolution**: Pinned by + [ADR-015](decisions/015-streams-depth.md). (1) **Key semantics — + option (a)**: the key is carried metadata (honker's shape — a + stored nullable column, round-tripped on every read, no + engine-enforced behavioral role); the ordering guarantee stays + **global FIFO by offset per stream**; per-key in-order reading is + the documented emergent pattern, not a promise. Option (b) + (server-enforced per-key ordering) rejected on its own decision + rule — no consumer row names the need; the consumer inventory's + streams row and the alkcall ecosystem-shape row (added for this + decision — alkcall's Sub/Pub/fan-out vocabulary routes the + topic/event-type dimension to *separate streams*, never per-key + lanes) both name none. Re-entry gate: a consumer-inventory row + naming server-enforced per-key ordering. Option (c) (cut + `publish_with_key`) rejected — removing a pinned v1 method name + would be a contract-breaking event for OQ-10, the cost is one + nullable column the inherited realizations already have, and the + key is a legitimately useful grouping token. (2) **A seam gap the + key decision exposed, closed in the same ADR**: + `publish_with_key_tx(stream, key, payload) -> offset` joins the + `TxHandle` trait — keyed publishes were otherwise unreachable + commit-atomically, the same hole shape OQ-13 exposed for the + outbox (`publish_tx` = the `None`-key call, not a separate engine + path; empty-`Some`-key → `InvalidName`, no new variants). + (3) **`StreamEvent` shape** — honker's near-verbatim with one + rename: `{ offset: i64, stream: String, key: Option, + payload: Vec, created_at: i64 }` (`topic` → `stream` — one + term everywhere; the substrate's column name stays per ADR-012 §3 + fidelity); `offset` immutable, never renumbered (gaps after trim + legal); `created_at` unix-seconds-at-publish, informational, not + an ordering field. (4) **Ordering guarantee row** — ADR-006 §2's + streams row extended: global FIFO by offset (read paths yield + `offset ASC`), offsets immutable — the row the contract suite + pins cross-engine equivalence against. (5) **Retention** — + `trim_to(horizon)` added to the stream handle (delete + `offset <= horizon`), consumer-invoked, no engine-default + retention, no ambient sweeper (the ADR-010 §6 posture); the + replay-forever default is documented with the in-contract tool to + bound it; reads from a trimmed-away region resume at the horizon; + saved offsets below the horizon stay valid. Framed as completing + contract v1 in place, pre-implementation — no versioning event, + OQ-10 untouched. Contract-suite rows added (cross-engine stream + equivalence, keyed-tx commit-atomicity, `trim_to` semantics — + both engines). - **Cross-references**: OQ-04 (the pinning that left this depth - open), OQ-01 (scope row), ADR-002, ADR-006 (the guarantee table - the ordering row extends), ADR-008 §1. + open), OQ-13 (the sibling depth gap; the amend-in-place framing), + OQ-01 (scope row), ADR-002, ADR-006 §2 (the guarantee table the + ordering row extends), ADR-008 §1/§4/§5/§8, ADR-010 §6 (the + no-ambient-sweeper posture the trim decision applies), ADR-012 §3 + (the fidelity posture the column-name keep rides). ### OQ-13: Transactional outbox enqueue shape — the `TxHandle` surface cannot express it — **RESOLVED** @@ -396,13 +408,14 @@ narrowed to the pinning work its own record already scoped.)* place, pre-implementation — no versioning event, OQ-10 untouched. Contract-suite row added (commit-atomicity on both engines). - **Cross-references**: OQ-04 (the pinning that missed this), OQ-12 - (the sibling depth gap found by the same review), ADR-002 + (the sibling depth gap found by the same review — resolved the + same day, [ADR-015](decisions/015-streams-depth.md)), ADR-002 (outbox scope row), ADR-007/ADR-008 (the seam design), ADR-010 §3 (the outbox's derived QueueOpts — unchanged by this). ## Deferred / Blocked None currently. Every open OQ above is actionable Phase 1 work -(streams depth, capability-surface shape, versioning discipline, -fork-scaffold follow-through) with its evidence base complete — no +(capability-surface shape, versioning discipline, fork-scaffold +follow-through) with its evidence base complete — no external arrivals are being waited on. diff --git a/docs/architecture/overview.md b/docs/architecture/overview.md index 10116e6..6fa8aac 100644 --- a/docs/architecture/overview.md +++ b/docs/architecture/overview.md @@ -77,6 +77,8 @@ Per [ADR-002](decisions/002-feature-scope.md): | [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted | | [012](decisions/012-forked-substrate-design.md) | Forked substrate design (contract-blind boundary, fidelity, port deltas) | Accepted | | [013](decisions/013-fold-substrate-into-sqlite.md) | Fold the forked substrate into `alkstore-sqlite` (no fourth crate) | Accepted | +| [014](decisions/014-outbox-tx-enqueue.md) | Transactional outbox enqueue (`outbox_enqueue_tx` on `TxHandle`) | Accepted | +| [015](decisions/015-streams-depth.md) | Streams depth (carried-metadata keys, global-FIFO ordering, `StreamEvent`, `trim_to`) | Accepted | ## Non-goals diff --git a/docs/architecture/queues.md b/docs/architecture/queues.md index 4a16156..6d8b338 100644 --- a/docs/architecture/queues.md +++ b/docs/architecture/queues.md @@ -165,7 +165,13 @@ made under; the ADRs carry the WHY. comes from the scheduler machinery above. The SQLite notifications table's hygiene is engine-internal (not a consumer chore; `prune_notifications*` stay out of the contract, resolving ADR-008 - §8's disposition). + §8's disposition). The stream log's bounded-growth answer follows + the same posture but *is* consumer-side, because consumers interact + with stream rows directly ([ADR-015](decisions/015-streams-depth.md) + §5): `stream(name).trim_to(offset)` — invoked deliberately, + cadenced by the collapse recipe like every sweep; no engine-default + retention, the replay-forever default documented with the + in-contract tool to bound it. ## Scheduler (ADR-009) diff --git a/docs/research/consumer-inventory.md b/docs/research/consumer-inventory.md index 2822d13..bbb3bea 100644 --- a/docs/research/consumer-inventory.md +++ b/docs/research/consumer-inventory.md @@ -1,10 +1,11 @@ --- status: draft -last_updated: 2026-10-04 (streams corrected to documented/in-scope — +last_updated: 2026-10-05 (streams corrected to documented/in-scope — operator-authority record: type-filtered event watching from multiple -places, e.g. repo-change subscriptions in a git app. No consumer doc -carries the row yet; applications above the paused crates are the -wanters. rate-limits + result-storage remain the keep-with-flag rows.) +places, e.g. repo-change subscriptions in a git app. alkcall row added +2026-10-05 as ecosystem-shape evidence for streams (no per-key-ordering +need named anywhere); rate-limits + result-storage remain the +keep-with-flag rows.) --- # alkstore — consumer-driven scope inventory @@ -48,6 +49,24 @@ which scope decisions OQ-ST-01 still has to take per feature. ## The consumers and their artifacts +- **alkcall** — `/workspace/@alkdev/alkcall/docs/architecture/` + (Phase 1, 51 ADRs, several reviewed; reference checkout per the + AGENTS.md §3 posture). The call protocol crate + (bidirectional RPC on ALPN `alk/call`: Query/Mutation/Sub/Pub + operation types, subscriptions, channels). Added 2026-10-05 as an + *ecosystem-shape* consumer: it is not (yet) a direct store + consumer with named rows, but its pub/sub vocabulary — + `Subscription` ops (server→client streaming, ADR-021), `Pub` + (client→server streaming, ADR-046), `channel/resources/subscribe` + (snapshot + change events, ADR-037), and the futures + broker/fan-out mechanism (ADR-046's Gap B, deferred to channels, + OQ-22) — is the shape downstream crates' reactive ops will take. + The operator's note (2026-10-05): alkstore is meant as the base + storage layer for this broader ecosystem, and the store's streams + are what backs that vocabulary durably. Its ordering posture is + per-stream/per-channel (ordered, reliable, per-transport-stream) — + **no alkcall document names server-enforced per-key ordering**; + correlation is by request ID and delivery is per-connection. - **alkfs** — `/workspace/@alkdev/alkfs/docs/research/phase-0.md` (Phase 0, 2026-09-23; 17 OQs). The VFS/workspace crate; its ancestor POC (`alknet-filesystem-poc`) used honker directly (honker-core 0.2.4 @@ -136,6 +155,7 @@ feature. | Consumer | Evidence | Need | Confidence | |---|---|---|---| | family-wide (reactivity requirement) | user/planning record, 2026-10-04 (operator's authority — the REQ-2 recording convention) | watching for specific event *types* from several different places at once. Concrete example: a git app at gitea/gitlab scale — users and other apps subscribe to changes on a repo. Many cases along these lines. | **documented** (operator-authority record; no consumer doc has grown the section yet) | +| alkcall (ecosystem shape) | alkcall `docs/architecture/`: decisions/021 (Sub, the Subscription streaming handler), decisions/046 (Pub + the `Subscription`→`Sub` rename + Gap B fan-out), decisions/037 (`channel/resources/subscribe`), open-questions OQ-22 | a durable substrate for the pub/sub vocabulary: subscribe-type ops backed by a stream that survives disconnect/restart (per-consumer offset + replay), and a broker/fan-out (one stream, N matching subscribers) over durable events when the channels session takes Gap B up. Ordering: global per-stream FIFO suffices for every named shape — no alkcall op names per-key ordering; the topic/event-type dimension routes to *separate streams* (one per topic), not keys within one stream. | **operator-authority** (operator record 2026-10-05 — alkstore is the base storage layer for the ecosystem these ops shape; evidence is the alkcall ADRs + that record; added 2026-10-05) | Verdict: **in scope, first-class.** Corrected 2026-10-04 — the initial draft (earlier the same day) graded this **absent** because no consumer @@ -153,7 +173,11 @@ nothing written down yet carries the row — which is expected, not suspicious. Design note for OQ-ST-04: on the Postgres side this is a NOTIFY-triggered durable event table with per-consumer cursors (the pg-boss-family shape); the per-consumer offset contract, not the -storage shape, is the seam to pin. +storage shape, is the seam to pin. **Added 2026-10-05** (the alkcall +row, evidence for OQ-12): the subscription vocabulary's ordering need +is per-stream FIFO — topic/event-type filtering routes to separate +streams, not per-key ordering within one; no current or planned +ecosystem op names server-enforced per-key ordering. ### rate limits @@ -220,7 +244,9 @@ none of them yet. - The inventory covers artifacts existing as of 2026-10-04: alkfs phase-0, alkgit architecture (reviewed), alkblobs architecture - (draft-but-ADR'd), alknet-filesystem POC summary. If a new consumer + (draft-but-ADR'd), alknet-filesystem POC summary. *(Extended + 2026-10-05: alkcall's architecture added as ecosystem-shape + evidence.)* If a new consumer is added to the family (alksftp, the alknet rewrite, the ops platform), a row gets added here *before* it is assumed into a scope decision — the same discipline as alkblobs' requirements.md ("when a