alkdev/alkstore
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

ADR-015: streams depth — carried-metadata keys, global-FIFO ordering, StreamEvent, trim_to (OQ-12 resolved)

Browse Source
This commit is contained in:
glm-5.3-flash committed 2026-10-05 14:15:28 +00:00
1 parent 04a04651dc
commit 8c4ec48f92
12 files changed
+562 -110
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/architecture/README.md
+6 -8
View File
@@ -24,9 +24,9 @@ pending architecture review and OQ resolution.
| Doc | Status | Purpose | Key OQs |
|---|---|---|---|
| [overview.md](overview.md) | draft | Crate family, feature surface, non-goals, evidence base | — |
| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-08, OQ-10, OQ-12 |
| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 |
| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-05 (resolved), OQ-08, OQ-12 |
| [core-contract.md](core-contract.md) | draft | The unified trait surface, delivery guarantees, tx seam | OQ-08, OQ-10 |
| [engine-sqlite.md](engine-sqlite.md) | draft | SQLite engine: forked-substrate/rusqlite mapping | OQ-06 (resolved), OQ-12 (resolved), OQ-13 (resolved) |
| [engine-postgres.md](engine-postgres.md) | draft | Postgres engine: tokio-postgres/LISTEN mapping | OQ-08, OQ-12 (resolved), OQ-13 (resolved) |
| [queues.md](queues.md) | draft | Queue/scheduler/outbox semantics depth (resolved: ADR-009/ADR-010) | OQ-06 (resolved) |
| [deployment.md](deployment.md) | draft | Host semantics, connection budgets, knobs, matrix | OQ-08 |
| [open-questions.md](open-questions.md) | draft | OQ tracker (promoted from OQ-ST register) | — |
@@ -49,6 +49,7 @@ pending architecture review and OQ resolution.
| [012](decisions/012-forked-substrate-design.md) | Forked substrate design — contract-blind boundary, fidelity posture, port deltas | Accepted |
| [013](decisions/013-fold-substrate-into-sqlite.md) | Fold the forked substrate into `alkstore-sqlite` — no fourth crate | Accepted |
| [014](decisions/014-outbox-tx-enqueue.md) | Transactional outbox enqueue — `outbox_enqueue_tx` on the `TxHandle` trait | Accepted |
| [015](decisions/015-streams-depth.md) | Streams depth — carried-metadata keys, global-FIFO ordering row, `StreamEvent` shape, `publish_with_key_tx`, `trim_to` | Accepted |
## Open Questions
@@ -57,10 +58,6 @@ Phase 0 register's OQ-ST-01..08 promote one-to-one — OQ-NN mirrors
OQ-ST-NN — with new Phase 1 questions appended after). Open, in
suggested resolution order:
- **OQ-12** (high): streams depth — key semantics, `StreamEvent`
shape, ordering guarantee row, retention (Phase 1 review find;
evidence gathered in the OQ; check consumer rows for a
per-key-ordering need before choosing among its options).
- **OQ-08** (medium): capability-surface shape.
- **OQ-10** (medium): contract versioning across engine crates.
- **OQ-11** (medium): forked-substrate follow-through (provenance
@@ -77,7 +74,8 @@ scheduler guarantee row pinned), ~~OQ-05~~ (queue semantics depth —
(honker-core quality read — fork fired,
[ADR-011](decisions/011-sqlite-substrate-fork.md)), ~~OQ-13~~
(transactional outbox enqueue shape —
[ADR-014](decisions/014-outbox-tx-enqueue.md)).
[ADR-014](decisions/014-outbox-tx-enqueue.md)), ~~OQ-12~~ (streams
depth — [ADR-015](decisions/015-streams-depth.md)).
No deferred OQs: all open questions are actionable Phase 1 work with
complete evidence bases.
docs/architecture/core-contract.md
+83 -18
View File
@@ -15,7 +15,10 @@ inventory-confirmed features — [ADR-002](decisions/002-feature-scope.md)
`TxHandle` representation, wake type, reserved strings, error
taxonomy, config split — amended in place, pre-implementation, by
[ADR-014](decisions/014-outbox-tx-enqueue.md): `outbox_enqueue_tx`
joins the `TxHandle` trait); ADR-009/ADR-010 add the first *post-v1
joins the `TxHandle` trait, and by
[ADR-015](decisions/015-streams-depth.md): streams depth — key
semantics, `StreamEvent` shape, the ordering row, `trim_to`,
and `publish_with_key_tx`); ADR-009/ADR-010 add the first *post-v1
contract extensions* (scheduler collapse surface, `QueueOpts` depth —
versioning discipline for such extensions is OQ-10's); the
*obligations* are this document.
@@ -56,7 +59,8 @@ Store::begin_tx() -> Box<dyn TxHandle + Send>
trait TxHandle {
enqueue_tx(name, opts, payload) -> job_id
publish_tx(stream, payload) -> event_id
publish_tx(stream, payload) -> offset
publish_with_key_tx(stream, key, payload) -> offset
notify_tx(channel, payload)
save_offset_tx(stream, consumer, offset)
outbox_enqueue_tx(outbox, opts, payload) -> job_id
@@ -68,13 +72,21 @@ trait TxHandle {
queue engine-side (the reserved prefix makes the derived name
unreachable by `enqueue_tx` by design —
[ADR-014](decisions/014-outbox-tx-enqueue.md)).
`publish_with_key_tx` validates the key like the non-tx form (a
present key must be non-empty → `InvalidName`) and is
`publish_tx`'s keyed twin — commit-atomic keyed publishes are the
point ([ADR-015](decisions/015-streams-depth.md) §2).
Mechanism handles come off the store (or, for transactional variants,
off the handle):
```text
store.notify(channel, payload) handle.notify_tx(channel, payload)
store.stream(name) -> Stream handle.publish_tx / save_offset_tx
store.stream(name) -> Stream handle.publish_tx / publish_with_key_tx
handle.save_offset_tx
(the Stream handle also carries
trim_to — stream-side, outside the
tx seam)
store.queue(name, opts) -> Queue handle.enqueue_tx
store.outbox(name) -> Outbox handle.outbox_enqueue_tx(outbox, opts, payload)
store.try_lock(name, owner, ttl) -> Option<Lock>
@@ -138,21 +150,56 @@ cousin of notify: publish is commit-atomic; every committed event is
readable by every consumer whose offset hasn't passed it;
replay-on-attach is the default; offsets are explicit and
transaction-aware (`save_offset_tx` gives exactly-once-within-a-
business-tx shape).
business-tx shape). Depth pinned by
[ADR-015](decisions/015-streams-depth.md).
- `publish` / `publish_tx` / `publish_with_key` — append to the
stream's durable log. *(Depth annotation: the key's semantics, the
`StreamEvent` shape, the ordering row, and log retention are
OQ-12's — method names pinned by
[ADR-008](decisions/008-contract-v1-pinning.md) §1, depth deferred
there; see [open-questions.md](open-questions.md).)*
- `publish` / `publish_tx` / `publish_with_key` / `publish_with_key_tx`
— append to the stream's durable log; return the assigned offset.
**Key semantics ([ADR-015](decisions/015-streams-depth.md) §1): the
key is carried metadata** — stored on the event row, round-tripped
on every read, `Option<String>`-shaped on `StreamEvent` — with no
engine-enforced behavioral role: **the ordering guarantee stays
global FIFO by offset within the stream**; per-key in-order reading
is the documented emergent pattern (filter by key, read in offset
order), not a server promise. A present key must be non-empty
(`InvalidName`). Server-enforced per-key ordering re-enters only
via a consumer-inventory row naming it.
- **`StreamEvent` shape** ([ADR-015](decisions/015-streams-depth.md)
§3): `{ offset: i64, stream: String, key: Option<String>, payload:
Vec<u8>, created_at: i64 }` — `stream` renames honker's `topic`
(one term everywhere, the ADR-008 §4 kinds list); `offset` is
engine-assigned, monotone per stream, **immutable, never renumbered**
(gaps after a trim are legal); `created_at` is unix-seconds-at-
publish, informational, **not an ordering field** (offset is the
only one); `payload_as<T>` decodes (the error is `Codec`).
- **Ordering guarantee row** ([ADR-015](decisions/015-streams-depth.md)
§4, extending [ADR-006](decisions/006-wake-and-delivery-contract.md)
§2's streams row): `read_since`/`subscribe` yield `offset ASC` —
global FIFO per stream; committed-event visibility and
replay-on-attach as already pinned; offsets immutable. The
contract suite pins cross-engine equivalence against this row
(same publish sequence → same offset sequence → same read order).
- `read_since` / `read_from_consumer(offset)` — cursor-based reads;
`get_offset(consumer)` — checkpoint inspection.
`get_offset(consumer)` — checkpoint inspection. A read from a
trimmed-away offset region (below) resumes at the trim horizon's
first remaining row.
- `save_offset` / `save_offset_tx` — consumer checkpoint, explicit;
**the contract's save is always explicit** (no auto-checkpoint
cadence — honker's per-binding ambiguity is not inherited). The
subscription handle exposes no `save_every` / auto-save-on-drop
([ADR-008](decisions/008-contract-v1-pinning.md) §8).
([ADR-008](decisions/008-contract-v1-pinning.md) §8). Saves are
monotone; a saved offset below the trim horizon
([ADR-015](decisions/015-streams-depth.md) §5) stays a valid
position marker.
- `trim_to(horizon)` — delete events with `offset <= horizon`
([ADR-015](decisions/015-streams-depth.md) §5). The stream-side
bounded-growth op, consumer-invoked, no engine-default
retention and no ambient sweeper (the ADR-010 §6 posture; the
replay-forever default is the mechanism's purpose — growth is
documented with the tool in-contract to bound it). Trim emits no
dedicated wake and no notify — on SQLite the `data_version`
watcher may still fire (any committed write bumps it; a spurious
hint, contract-legal); surviving events keep their offsets.
- `subscribe(consumer) -> Box<dyn EventReceiver>` — durable
consumption: attach, read to current tail, resume after restart
from the stored offset; explicit `save_offset` on the receiver
@@ -302,8 +349,9 @@ inherits them and adds the consumer obligations:
consumer's, correctness-of-transition the engine's
([ADR-010](decisions/010-queue-semantics-depth.md) §6);
- `*_tx` operations are *only* durable after the caller's commit —
rollback drops job rows, event rows, notifications, and offset saves
together (the no-ghosts property, POC-pinned on both engines).
rollback drops job rows, event rows (keyed and unkeyed alike),
notifications, and offset saves together (the no-ghosts property,
POC-pinned on both engines).
### Errors
@@ -412,6 +460,23 @@ before the engine specs are called `stable`:
outbox names rejected identically on the tx path
(`ReservedName`/`InvalidName`); stamped opts visible via `get_job`
equal for both engines.
- **Cross-engine stream equivalence** ([ADR-015](decisions/015-streams-depth.md)
§3/§4) — publish sequence → identical offset sequence → identical
`read_since` output order on both engines, direct and subscriber
reads alike; keyed/unkeyed interleavings preserve global FIFO;
`key` round-trips exactly (`None` vs `Some`), `stream`/`created_at`
fields equal.
- **`publish_with_key_tx` commit-atomicity on both engines**
([ADR-015](decisions/015-streams-depth.md) §2) — rollback drops the
keyed event row with the business write (no ghost event); commit
makes it visible to `read_since`/`subscribe`; empty-`Some`-key
`InvalidName` on both engines' tx paths.
- **`trim_to` semantics on both engines** ([ADR-015](decisions/015-streams-depth.md)
§5) — exact-boundary trim (`<=`), surviving rows keep their offsets
(gaps legal, never renumbered), reads resume at the trim horizon's
first remaining row, saved offsets below the horizon stay valid,
trim wakes nothing, and a concurrent subscriber never loses its
place.
## Design Decisions
@@ -427,6 +492,7 @@ before the engine specs are called `stable`:
| [011](decisions/011-sqlite-substrate-fork.md) | Substrate fork | SQLite substrate owned (`__alkstore_*` naming); queue ops re-derived on contract v1 |
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate boundary; engine-side formula arithmetic pinned equivalent by the contract suite |
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue (amends 008) | `outbox_enqueue_tx` on `TxHandle`; outbox-name validation; derived backing queue reached only through the outbox surface |
| [015](decisions/015-streams-depth.md) | Streams depth (amends 006/008) | key = carried metadata, global-FIFO ordering row, `StreamEvent` shape, `publish_with_key_tx`, `trim_to` |
## Open Questions
@@ -437,11 +503,10 @@ questions affecting this document:
- **OQ-10**: contract versioning discipline across engine crates
([open](open-questions.md))
- **OQ-08**: capability-surface shape ([open](open-questions.md))
- **OQ-12**: streams depth — key semantics, `StreamEvent` shape,
ordering row, retention ([open](open-questions.md))
Resolved on this document's surface: **OQ-09** (scheduler collapse —
[ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)),
and **OQ-13** (transactional outbox enqueue shape —
[ADR-014](decisions/014-outbox-tx-enqueue.md)), 2026-10-05.
**OQ-13** (transactional outbox enqueue shape —
[ADR-014](decisions/014-outbox-tx-enqueue.md)), and **OQ-12** (streams
depth — [ADR-015](decisions/015-streams-depth.md)), 2026-10-05.
docs/architecture/decisions/006-wake-and-delivery-contract.md
+1 -1
View File
@@ -59,7 +59,7 @@ coalesced-but-complete re-read on SQLite).
| Mechanism | Durability | Replay | Atomicity | Guarantee |
|---|---|---|---|---|
| `notify` / listen | none | never | commit-atomic (delivers at commit; rollback drops) | fire-and-forget, at-most-once per listener session |
| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset |
| streams | durable table row | yes, per-consumer offset, replay-on-attach default | publish is commit-atomic | every committed event readable by every consumer that hasn't passed its offset; **global FIFO by offset within each stream** (read paths yield `offset ASC`; offsets immutable, never renumbered — [ADR-015](015-streams-depth.md) §4) |
| queues | durable row | claim/ack model | enqueue is commit-atomic | at-least-once *work* with visibility timeouts |
- The trait **does not promise replay under `listen()`** — on either
docs/architecture/decisions/008-contract-v1-pinning.md
+12 -3
View File
@@ -53,7 +53,11 @@ implement identically):
shape, the ordering guarantee row, and log retention are OQ-12's —
the method *names* are pinned here, their depth was left
un-pinned, mirrored in [core-contract.md](../core-contract.md)
streams.)*
streams. **Resolved 2026-10-05 by [ADR-015](015-streams-depth.md)** —
key = carried metadata, global-FIFO ordering row, event shape
pinned, `trim_to` added; `publish_with_key_tx` also joins the
tx seam (§2 there); the depth was completed in place,
pre-implementation.)*
- queues v1 skeleton — `enqueue`, `claim_one`, `claim_batch`,
`ack_batch`, `cancel`, `get_job`, `sweep_expired`; job handle
`ack / retry / fail / heartbeat`; `EnqueueOpts { delay, run_at,
@@ -114,7 +118,8 @@ a **`TxHandle` trait carrying the `*_tx` methods directly**:
```text
trait TxHandle {
enqueue_tx(name, opts, payload) -> job_id
publish_tx(stream, payload) -> event_id
publish_tx(stream, payload) -> offset
publish_with_key_tx(stream, key, payload) -> offset // ADR-015
notify_tx(channel, payload)
save_offset_tx(stream, consumer, offset)
outbox_enqueue_tx(outbox, opts, payload) -> job_id // ADR-014
@@ -277,7 +282,10 @@ v1 variants (guaranteed-matchable on every engine):
`try_recv` / `recv_timeout` form of the wake close in §3).
- `Codec` — payload serialization/deserialization failures
(`payload_as` on stream events and job payloads; wakes carry no
payload per §3).
payload per §3). *(Annotated 2026-10-05: a present key on
`publish_with_key(_tx)` being non-empty is validated as
`InvalidName` ([ADR-015](015-streams-depth.md) §2 — the same
entry-point-validation variant, not a new one).)*
- `Database` — everything else: driver/connection/SQL errors, opaque
to the contract, engine detail preserved via the error source chain.
No engine-specific variants are minted for its contents.
@@ -339,6 +347,7 @@ differs):
| `Lock::heartbeat(ttl)` | `Lock::renew(ttl)` | renew describes the TTL discipline the guarantee row pins; heartbeat is the queue-side renewal term |
| `Queue::claim_waker()` | (not surfaced) | engine-internal consumption wake (§1) |
| `StreamSubscription` (`save_every`, auto-save on drop) | `subscribe(consumer) -> Box<dyn EventReceiver>`, explicit saves only | the explicit-offset obligation (core-contract.md streams; the auto-checkpoint ambiguity is not inherited, [ADR-006](006-wake-and-delivery-contract.md)) |
| `StreamEvent.topic` (field) | `StreamEvent.stream` *(added 2026-10-05 by [ADR-015](015-streams-depth.md) §3)* | one term for the mechanism everywhere — the contract names it streams (the §4 kinds list); the substrate's column stays `topic` per the [ADR-012](012-forked-substrate-design.md) §3 fidelity posture |
| `update_events()` | (not surfaced) | superseded by the wake subscription — the pinned `WakeReceiver` *is* the reactive-event surface; honker's separate raw update-event stream has no contract role |
| `prune_notifications` / `prune_notifications_keep_latest` | (not surfaced) | notifications-table maintenance tooling on the SQLite engine; disposition rides OQ-05's sweep/maintenance design (consumer-visible only if that design surfaces it) |
| `Scheduler` surface | (out of v1, OQ-09 decides) | §1 |
docs/architecture/decisions/014-outbox-tx-enqueue.md
+1 -1
View File
@@ -53,7 +53,7 @@ One method, added to the `TxHandle` trait:
```text
trait TxHandle {
enqueue_tx(queue, opts, payload) -> job_id
publish_tx(stream, payload) -> event_id
publish_tx(stream, payload) -> offset
notify_tx(channel, payload)
save_offset_tx(stream, consumer, offset)
outbox_enqueue_tx(outbox, opts, payload) -> job_id // the addition
docs/architecture/decisions/015-streams-depth.md
+326
View File
@@ -0,0 +1,326 @@
# ADR-015: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention
## Status
Accepted
## Context
[OQ-12](../../architecture/open-questions.md) — found by the contract
v1 Phase 1 architecture review: [ADR-008](008-contract-v1-pinning.md)
§1 pinned streams' *method names* (`publish`, `publish_with_key`,
`read_since`, `read_from_consumer`, `save_offset`, `get_offset`,
`subscribe`) but not their *depth*. Four sub-questions were open:
1. **Key semantics** — what `publish_with_key`'s key means, and what
if any ordering it buys. Honker's realization is a stored nullable
`key` column (`_honker_stream(topic, key, payload)`) with no
per-key read path or ordering enforcement anywhere; the doc comment
says "used for per-key ordering downstream"
(`packages/honker-rs/src/lib.rs:894`) — per-key FIFO is an
*emergent* property of a consumer reading `offset ASC` filtered by
key, not a server guarantee. Three options were on the table: (a)
pin honker's shape (key = carried metadata, ordering stays global
FIFO, document the emergent pattern); (b) pin server-enforced
per-key ordering (real machinery on both engines, gated on a
consumer-inventory row naming the need); (c) cut `publish_with_key`
from v1 (no consumer row names keys).
2. **`StreamEvent` shape** — honker's is
`{ offset: i64, topic: String, key: Option<String>, payload:
Vec<u8>, created_at: i64 }` with `payload_as<T>` decode
(lib.rs:1008). The contract needs its own pinned field list.
3. **Ordering guarantee row** — [ADR-006](006-wake-and-delivery-contract.md)
§2's delivery table pins streams' durability/replay/atomicity but
its ordering cell content was effectively "readable"; the contract
suite cannot pin cross-engine stream equivalence without an
explicit ordering row.
4. **Retention / bounded growth** — the reference read flags the
stream table as unbounded-growth-unless-swept (honker-machinery
§8 defect 5); no consumer row names retention
(replay-deep-history is the *purpose* of the subscription need).
The consumer evidence was completed for this decision by adding the
**alkcall row** to
[consumer-inventory.md](../../research/consumer-inventory.md)
(ecosystem-shape evidence, added per the inventory's own
row-before-assumption convention). Alkcall's pub/sub vocabulary —
`Sub` operations (server→client streaming, ADR-021), `Pub`
(client→server streaming, ADR-046), `channel/resources/subscribe`
(snapshot + change events, ADR-037), and the deferred
broker/fan-out (ADR-046's Gap B, channels OQ-22) — is the shape
downstream crates' reactive ops are expected to take over this store,
and the operator confirmed alkstore is meant as the base storage
layer for that ecosystem. Its ordering posture is per-stream
(ordered, reliable transport streams; correlation by request ID):
**no alkcall mechanism names server-enforced per-key ordering.** The
topic/event-type dimension of its subscriptions routes to *separate
streams* (one stream per topic; e.g. `channel/resources/subscribe` is
its own event source), not to keys within one stream.
## Decision
### 1. Key semantics: option (a) — the key is carried metadata; ordering stays global FIFO
Option (b) is **rejected on the decision rule it named for itself**:
it requires "a consumer row naming the need," and neither the
inventory's original rows nor the alkcall row names server-enforced
per-key ordering anywhere. Real machinery (per-key sequencing,
per-key claim/exhaustion paths, or per-key head-of-line bookkeeping on
both engines) for an unnamed need is exactly the scope discipline
[ADR-002](002-feature-scope.md) exists to enforce.
Option (c) (cut `publish_with_key` from v1) is **rejected**: the
method name is a pinned v1 surface element
([ADR-008](008-contract-v1-pinning.md) §1 — removing it would be a
contract-breaking event for OQ-10 to govern, not a pre-implementation
refinement); the *cost* of carrying the key in the inherited
realizations is one nullable column honker's schema already has; and
the key is genuinely useful as a *grouping token* — the emergent
per-key pattern is real and cheap for consumers (SQL-level filter +
in-order read), and an ecosystem whose ops carry keys (a repo id, an
entity id) deserves a stable way to store that grouping durably.
Cutting the only keyed-write path would push consumers to encode keys
into payloads — an unqueryable convention — the same shape of waste
ADR-010 §7 called out for result storage.
Pinned semantics, exactly honker's:
- `publish_with_key(stream, key, payload)` stores `key` as a nullable
column value on the event row; `publish` stores `NULL`.
- The key is **carried metadata** — it appears on every `StreamEvent`
read back and is `Option<String>`-shaped in the event type; the
engine assigns it no behavioral role (it rides the row the publish
writes; the wake trigger is the publish itself, key or no key).
- **The ordering guarantee stays global FIFO by offset, per stream** —
regardless of key. Per-key FIFO-by-offset is the *documented
emergent pattern*: a consumer that filters a read by key observes
that stream's events for that key in offset order. The contract
does not enforce, promise, or price any cross-subscriber per-key
handoff. (The guarantee row, §3, is written so this remains *true
under trimming* — offset order, never renumbered.)
- The key is **not** a partitioning instruction, not a dedup key, and
not an ordering key in any engine-enforced sense. Nothing in the
contract suite tests keys behaviorally beyond carried-metadata
round-trip and equivalence across engines.
The alkcall evidence makes this the right ecosystem shape too: its
subscriptions distinguish event *sources* (separate streams per
topic), not positions *within* a source (per-key lanes). When the
channels broker (Gap B) arrives, topic matching selects streams;
within a stream, offset order is the whole story.
### 2. `publish_with_key_tx` — the tx seam carries the key
A gap the key decision exposed in the pinned tx seam:
[TxHandle](../../architecture/core-contract.md) carries
`publish_tx(stream, payload)` but no keyed form — so a keyed event
could not be written commit-atomic with a business write, breaking the
no-ghosts property's uniformity for keyed publishes (the same shape of
hole OQ-13 exposed for outbox enqueue, resolved by
[ADR-014](014-outbox-tx-enqueue.md)). Pinned:
`publish_with_key_tx(stream, key, payload) -> offset` joins the
TxHandle trait — one method, symmetric with `publish_tx`, validating
the key the same way the non-tx form does (`None` or non-empty — an
empty `Some` key is `InvalidName`) with no new error variants
([ADR-008](008-contract-v1-pinning.md) §5's rule). Framed as
completing contract v1 in place, pre-implementation, like ADR-014 —
no versioning event, OQ-10 untouched. A plain-`publish_tx` is
`publish_with_key_tx(stream, None, payload)`, not a separate
engine path. *(Naming pinned here too: the publish family returns the
**assigned offset** — honker's `publish` already returns it — and the
tx-seam sketches' `event_id` placeholder name is corrected to
`offset` so one term covers the value every read/cursor/trim method
speaks.)*
### 3. `StreamEvent` shape — honker's, with one rename
Pinned (inherited near-verbatim, per the fork's API fidelity posture,
[ADR-012](012-forked-substrate-design.md) §3):
```text
struct StreamEvent {
offset: i64, // position in the stream's log; immutable
stream: String, // the stream name (honker: "topic")
key: Option<String>, // §1's carried metadata; None unless
// published keyed
payload: Vec<u8>, // core value bytes
created_at: i64, // unix seconds at publish (the engines'
// single clock source)
}
```
- The `topic` field renames to **`stream`** — the contract surface
names this mechanism streams, not topics (honker-rs's `Stream`
handle's `topic()` accessor is likewise the stream name); one term
everywhere, matching the reserved-namespace kinds list
([ADR-008](008-contract-v1-pinning.md) §4).
- `payload` is core value bytes crossed as a non-generic value; the
concrete `StreamEvent` owns the bytes; `payload_as<T>` is the decode
convenience ([ADR-008](008-contract-v1-pinning.md) §8's rule — the
deserialization error is `Codec`).
- `created_at` is **unix seconds at publish**, from the engine's
single clock source (the partial-index/clock-pin rule in
[queues.md](../../architecture/queues.md)); it is *informational*
and NOT an ordering candidate — offset is the only ordering field
(two events can share a second; offset never collides).
- `offset` is the stream-wide position, engine-assigned at publish,
monotonically increasing per stream on both engines. It is
**immutable once assigned** and is the value every read/cursor/
trim method speaks.
### 4. Ordering guarantee row — pinned
ADR-006 §2's table's streams row extends with an explicit ordering
cell (the guarantee cell gains a clause; durability/replay/atomicity
as already pinned):
> **streams** — Guarantee: *global FIFO by offset within each stream
> (`read_since`/`subscribe` yield `offset ASC`); every committed
> event readable by every consumer that hasn't passed its offset;
> offsets immutable and never renumbered.*
This is the row the contract suite pins cross-engine stream
equivalence against: two engines, same publish sequence → same offset
sequence → same `read_since` output order, for direct and subscriber
reads alike. It is deliberately *not* stronger than both engines can
implement by a shared SQL shape (`ORDER BY offset ASC` is already
both engines' claim path) and not weaker than the mechanism's purpose
(replay demands monotone positions).
### 5. Retention: `trim_to` on the stream handle — a consumer-invoked bounded-growth op
The mechanism-shaped answer to defect 5's family (unbounded
`_honker_stream` growth), applying the ADR-010 §6 posture to the one
stream-side table that consumers interact with *as rows* (unlike
notifications, the transport detail whose hygiene stays engine-
internal):
- `store.stream(name)`'s handle gains `trim_to(horizon)` — delete
events with `offset <= horizon`, emitting no dedicated wake and no
notify (deletes are not events; consumers whose cursors move
forward never lose reads). On SQLite the `data_version` watcher may
still fire on the trim commit — any committed write bumps it; that
spurious hint is contract-legal (the wake contract's best-effort +
idempotence posture, ADR-006 §1), not a trim-specific delivery.
`OffsetGone`-style errors are not needed: reads past the trim
horizon simply return fewer/no rows (a consumer reading from a
trimmed region has, by definition, an offset older than the
horizon — that is the API contract; re-attaching consumers at
trimmed-away offsets resume at the horizon's first remaining row).
- **No engine-side retention default, no ambient sweeper** (the
ADR-010 §6 posture; consumer-side, cadenced by the collapse recipe
like every sweep). Trim is *available*, not automatic: the
replay-forever default is the *purpose* for the consumer rows, so
the default is unbounded — with the growth documented squarely,
this time with an in-contract tool to fix it.
- **Caveat documented in-contract**: offsets are never renumbered and
never reused; surviving events keep the offsets they were published
with (gaps after a trim are legal and ordinary). Consumers must
not assume dense offset sequences (the same discipline the
scheduler's catch-up/skip-forward already imposes).
- Trim's effect on cursors: a consumer's *saved* offset below the
trim horizon stays (it remains a valid position marker; the next
read resumes at the horizon's first remaining row); a consumer may
re-save after processing. This keeps `get_offset`/`subscribe`
resume semantics well-defined without engine-side cursor-tracking
machinery.
Rejected retention alternatives (per the OQ's option set): the
consumer-recipe-without-an-API option was self-admittedly "document
the gap" (no deletion path exists today — this ADR *creates* the
deletion path, which is what the recipe needed); documenting-nothing
repeats the notifications-growth mistake ADR-010 §6 just fixed.
## Consequences
**Positive**
- Streams' depth is fully pinned: names (ADR-008 §1) + semantics (this
ADR) make the mechanism implementable on both engines from the same
text, and the contract suite has an ordering row to pin cross-engine
equivalence against.
- The key decision is honest: a zero-machinery feature (carried
metadata) documented for exactly what it is, with option (b)'s
re-entry gate (a consumer-inventory row naming server-enforced
per-key ordering) recorded — the ecosystem-shape evidence says the
topic dimension routes to streams, not keys.
- Keyed publishes are tx-seam-native — keyed and unkeyed events carry
the same commit-atomicity, so an outbox-style keyed event log co-writes
with business data correctly.
- `trim_to` gives the family a real answer to stream growth in the
contract, consistent with the no-ambient-sweeper posture (invoked
deliberately, cadenced from above).
**Negative**
- Per-key *ordering enforcement*, if a consumer ever names it, is a
later contract extension with real machinery on both engines —
carrying metadata now does not prejudge or approximate it, and the
emergent pattern's documented status must not be read as a promise.
- `trim_to` adds one more consumer-invoked maintenance op to the
surface (backlog rows follow).
- `created_at` is second-precision (the shared clock pin) — consumers
needing sub-second publish timestamps must encode them in payloads.
- The `StreamEvent.stream` rename diverges from honker's wire-era name
(`topic`) — the fork keeps the column name `topic` engine-internally
per the ADR-012 §3 fidelity posture; only the contract type renames.
## Verification backlog additions
- **Cross-engine stream equivalence** (the §3/§4 row): publish
sequence → identical offset sequence → identical `read_since`
output order, both engines; keyed and unkeyed interleavings
preserve global FIFO; `key` round-trips exactly (`None` vs
`Some`), `StreamEvent.stream`/`created_at` equal on both engines.
- **`publish_with_key_tx` commit-atomicity on both engines** (§2):
rollback drops the keyed event row with the business write (no
ghost event); commit makes it visible to `read_since`/`subscribe`;
empty-`Some`-key `InvalidName` on both engines' tx paths.
- **`trim_to` semantics on both engines** (§5): exact-boundary trim
(`<=`), surviving rows keep their offsets (gaps legal, never
renumbered), reads from a trimmed horizon resume correctly,
`save_offset` below the horizon stays valid, trim emits no
dedicated wake/notify (on SQLite the `data_version` watcher may
fire spuriously — pinned as contract-legal), and a concurrent
subscriber mid-`subscribe` never loses its place
(offset-anchored reads only move forward).
## References
- [OQ-12](../../architecture/open-questions.md) — this ADR's resolution.
- [core-contract.md](../../architecture/core-contract.md) — the spec
this ADR's §1–§5 pin into place (streams section, tx seam,
guarantee table, backlog).
- The alkcall evidence (reference checkout
`/workspace/@alkdev/alkcall` @ HEAD `main`, 2026-10-05, per the
AGENTS.md §3 posture):
decisions/021 (Sub — the Subscription streaming handler),
decisions/046 (Pub + the `Subscription`→`Sub` rename + Gap B),
decisions/037 (`channel/resources/subscribe`), open-questions.md
OQ-22 (fan-out deferral) — the pub/sub vocabulary this contract's
streams substrate; its no-per-key-ordering posture is the
inventory row added for this decision.
- [consumer-inventory.md](../../research/consumer-inventory.md) — the
streams row (operator-authority record) + the alkcall
ecosystem-shape row; the (b)-gate that rejected server-enforced
per-key ordering.
- [ADR-006](006-wake-and-delivery-contract.md) §2 — the guarantee
table the streams ordering cell extends.
- [ADR-008](008-contract-v1-pinning.md) §1/§4/§5/§8 — the pinned
method names the depth lands on, the namespace kinds, the
act-differently error rule, the `payload_as`/`Codec` rule.
- [ADR-010](010-queue-semantics-depth.md) §6 — the no-ambient-
sweeper posture this ADR's retention decision applies to the
consumer-facing stream table (the notify-table hygiene precedent
is §6's too).
- [ADR-012](012-forked-substrate-design.md) §3 — the fidelity posture
the column-name keep rides; [ADR-014](014-outbox-tx-enqueue.md) —
the amend-in-place framing §2 reuses.
- Honker stream machinery
(`/workspace/honker` `honker-core/src/honker_ops.rs:1906-1996`, `src/lib.rs:417-429`; `packages/honker-rs/src/lib.rs:876-1010, 1099`)
— the inherited realization (key column, offset-ASC reads, monotone
offset saves, event shape); the unbounded-growth defect
(reference-honker-machinery.md §8 defect 5).
docs/architecture/engine-postgres.md
+12 -12
View File
@@ -64,7 +64,7 @@ obligations live in the core spec.
| Contract piece | Engine realization |
|---|---|
| notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers |
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) |
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape). Depth per [ADR-015](decisions/015-streams-depth.md): the event table carries the nullable key column (carried metadata), reads yield `offset ASC` (global FIFO — bigserial offsets, immutable), `publish_with_key_tx` validates + inserts inside the caller's tx, `trim_to` is a `DELETE … WHERE offset <= ?` on a pool connection (wakes nothing) |
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema; the curve/stamps arithmetic is computed engine-side per [ADR-012](decisions/012-forked-substrate-design.md) §2 (equivalence with the SQLite engine pinned by the contract suite) |
| named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; guarantee row pinned by [ADR-008](decisions/008-contract-v1-pinning.md) §7) |
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule rows in the engine-owned schema, tick re-derived (boundary advance + 64-boundary catch-up cap, honker parity), leadership via the engine's lock machinery on `__alkstore_scheduler`; outbox = helper over queues |
@@ -110,6 +110,7 @@ the record in [ADR-003](decisions/003-sqlite-driver.md).
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | job-stamped opts, equal-jitter backoff, dead-letter move, no-stranded-rows sweep, one engine-owned schema |
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate (SQLite side); pg engine owns its own curve/stamps arithmetic, equivalence pinned by the contract suite |
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` on `TxHandle`; derivation engine-side inside the caller's tx |
| [015](decisions/015-streams-depth.md) | Streams depth | nullable key column (carried metadata); bigserial offsets, global-FIFO reads; keyed tx publish; `trim_to` as a pool-connection delete |
## Open Questions
@@ -119,17 +120,16 @@ questions affecting this document:
- **OQ-08**: capability surface (shared with
[deployment.md](deployment.md)) (open)
- **OQ-12**: streams depth (affects this engine's stream realization
— event-shape/ordering/retention equivalents on the pg side)
([open](open-questions.md))
- **OQ-13**: transactional outbox enqueue shape — **resolved**
(2026-10-05, [ADR-014](decisions/014-outbox-tx-enqueue.md)):
`outbox_enqueue_tx(outbox, opts, payload)` on the `TxHandle` trait;
the pg engine validates the outbox name and derives
`__alkstore_outbox:{name}` engine-side inside the caller's
transaction.
- **OQ-12**: streams depth — **resolved**
(2026-10-05, [ADR-015](decisions/015-streams-depth.md)): key =
carried metadata; global-FIFO-by-offset ordering row
(bigserial offsets, immutable); `StreamEvent` shape pinned
(`topic` → `stream` in the contract type); `publish_with_key_tx`
inside the caller's tx; `trim_to` on the stream handle — no
engine-default retention.
Resolved: **OQ-09** (scheduler collapse —
[ADR-009](decisions/009-scheduler-collapse.md)) and **OQ-05** (queue
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)),
[ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), and
**OQ-12** (streams depth — [ADR-015](decisions/015-streams-depth.md)),
2026-10-05.
docs/architecture/engine-sqlite.md
+12 -5
View File
@@ -60,7 +60,7 @@ per-contract obligations live in the core spec and are not restated here.
| Contract piece | Engine realization |
|---|---|
| notify / listen | honker's notify functions inside the caller's tx; `listen()` bridges the watcher's fanout into a tokio receiver (one `spawn_blocking` thread per subscription doing `blocking_send`) |
| streams | honker's stream machinery; explicit offset saves through the tx seam |
| streams | the forked substrate's stream machinery (inherited near-verbatim — the `__alkstore_stream` table's field set is already the contract's shape (its `topic` column carries the contract's `stream` field — the one name delta, ADR-012 §3's fidelity posture), the offset-ASC read path, keyed publish's nullable key column, monotone offset saves ([ADR-015](decisions/015-streams-depth.md)); explicit offset saves through the tx seam; `trim_to` is a `DELETE FROM … WHERE offset <= ?` inside the writer-slot lease) |
| queues | owned queue machinery (forked substrate re-derived on [ADR-010](decisions/010-queue-semantics-depth.md) — stamps, no-stranded-rows sweep, dead-visible `get_job`); ADR-003's ride posture superseded on ownership by [ADR-011](decisions/011-sqlite-substrate-fork.md) |
| named locks | the substrate's lock machinery (`lock_renew` carries the renew semantics the guarantee row pins; re-acquire does not refresh TTL — inherited deliberately, [ADR-012](decisions/012-forked-substrate-design.md) §4) |
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): owned scheduler storage (`__alkstore_scheduler_tasks`, forked substrate) carrying `@every` specs (cron machinery not ported); the leader loop pattern (TTL lock, heartbeat, exit-before-tick-on-loss); the leadership lock is `__alkstore_scheduler`; outbox = helper over queues with the derived backing-queue name ([ADR-008](decisions/008-contract-v1-pinning.md) §4) |
@@ -126,6 +126,7 @@ family is `__alkstore_*` (ADR-010 §8's naming authorization).
| [012](decisions/012-forked-substrate-design.md) | Fork design | contract-blind substrate, fidelity posture, port deltas, panic-free watcher death |
| [013](decisions/013-fold-substrate-into-sqlite.md) | Substrate packaging | folded into `alkstore-sqlite` (`src/substrate/`); no fourth crate |
| [014](decisions/014-outbox-tx-enqueue.md) | Outbox tx enqueue | `outbox_enqueue_tx` on `TxHandle`; derivation engine-side through the writer-slot lease |
| [015](decisions/015-streams-depth.md) | Streams depth | key = carried metadata (the inherited nullable column); global-FIFO ordering; `trim_to` as a writer-lease delete; event shape already the substrate's |
## Open Questions
@@ -136,8 +137,13 @@ questions affecting this document:
- **OQ-06**: honker-core quality read — fork-trigger assessment —
**resolved** (2026-10-05,
[ADR-011](decisions/011-sqlite-substrate-fork.md); trigger fired).
- **OQ-12**: streams depth (affects this engine's stream realization
— key column, event shape, retention) ([open](open-questions.md))
- **OQ-12**: streams depth — **resolved**
(2026-10-05, [ADR-015](decisions/015-streams-depth.md)): key =
carried metadata; global-FIFO-by-offset ordering row;
`StreamEvent` shape (honker's, `topic` → `stream` in the contract
type — the substrate's column name stays per the ADR-012 §3
fidelity posture); `publish_with_key_tx` routes through the
writer-slot lease; `trim_to` is a plain delete in the same lease.
- **OQ-13**: transactional outbox enqueue shape — **resolved**
(2026-10-05, [ADR-014](decisions/014-outbox-tx-enqueue.md)):
`outbox_enqueue_tx(outbox, opts, payload)` on the `TxHandle` trait;
@@ -146,6 +152,7 @@ questions affecting this document:
substrate's `__alkstore_outbox:{name}` backing queue.
Resolved: **OQ-09** (scheduler collapse —
[ADR-009](decisions/009-scheduler-collapse.md)) and **OQ-05** (queue
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)),
[ADR-009](decisions/009-scheduler-collapse.md)), **OQ-05** (queue
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)), and
**OQ-12** (streams depth — [ADR-015](decisions/015-streams-depth.md)),
2026-10-05.
docs/architecture/open-questions.md
+68 -55
View File
@@ -23,11 +23,12 @@ is OQ-11); **the fork packaging folded** (2026-10-05,
`alkstore-substrate` crate; the forked machinery is a module subtree of
`alkstore-sqlite`); **OQ-13 resolved** (2026-10-05,
[ADR-014](decisions/014-outbox-tx-enqueue.md) —
`outbox_enqueue_tx` joins the `TxHandle` trait, the hole is closed).
Next: **OQ-12** (streams depth — a Phase 1 review
find: key semantics, `StreamEvent` shape, ordering row, retention;
evidence gathered; check consumer rows for per-key-ordering need
before choosing among its options), OQ-08 (rides the now-pinned trait
`outbox_enqueue_tx` joins the `TxHandle` trait, the hole is closed);
**OQ-12 resolved** (2026-10-05,
[ADR-015](decisions/015-streams-depth.md) — streams depth pinned:
carried-metadata keys, global-FIFO ordering row, `StreamEvent` shape,
`publish_with_key_tx`, `trim_to`).
Next: **OQ-08** (rides the now-pinned trait
shape and the ADR-011 substrate fork), OQ-10 (versioning discipline
for contract extensions; note ADR-011 changes its substrate-side facts
for SQLite — the forked machinery lives in-tree inside the engine
@@ -131,7 +132,8 @@ narrowed to the pinning work its own record already scoped.)*
resolution. A verification backlog (core-contract.md §Verification
backlog) tracks the one-engine-pinned properties.
- **Cross-references**: OQ-01, OQ-10, OQ-09, OQ-08; OQ-13 (the one
gap the v1 pinning's own Phase 1 review found in it).
gap the v1 pinning's own Phase 1 review found in it), OQ-12 (the
sibling depth gap — resolved, [ADR-015](decisions/015-streams-depth.md)).
## Theme: Queues and scheduling
@@ -308,59 +310,69 @@ narrowed to the pinning work its own record already scoped.)*
## Theme: Core contract (Phase 1 review finds)
### OQ-12: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention
### OQ-12: Streams depth — key semantics, `StreamEvent` shape, ordering row, retention — **RESOLVED**
- **Origin**: [core-contract.md](core-contract.md) streams section,
[ADR-008](decisions/008-contract-v1-pinning.md) §1 (Phase 1
architecture review, 2026-10-05)
- **Status**: open
- **Status**: resolved (2026-10-05, Phase 1 —
[ADR-015](decisions/015-streams-depth.md))
- **Priority**: high (streams is a first-class
[ADR-002](decisions/002-feature-scope.md) mechanism; ADR-008's
"no placeholder semantics remain" positive consequence is false for
`publish_with_key` until this resolves — the contract suite also
cannot pin cross-engine stream equivalence without an ordering row)
- **Resolution**: open. The method *names* are pinned (ADR-008 §1);
the depth is not. Sub-questions, with the evidence already gathered:
- **Key semantics** — honker's realization is a stored `key` column
(`_honker_stream(topic, key, payload)`, NULLable) with no
per-key read path or ordering enforcement anywhere; the honker-rs
doc comment says "used for per-key ordering downstream"
(`packages/honker-rs/src/lib.rs:894`), i.e. per-key global-FIFO
is an *emergent* property (a consumer reading `offset ASC`
filtered by key), not server-enforced. Options: (a) pin exactly
honker's shape — key is carried metadata, ordering stays global
FIFO (document the emergent per-key pattern), cheapest and
honest; (b) pin *server-enforced* per-key ordering (a read or
delivery guarantee keyed on `key`) — real machinery on both
engines, needs a consumer row naming the need; (c) cut
`publish_with_key` from v1 (no consumer row names keys — the
honker-rs doc comment is upstream's intent, not ours). Decision
rule: consumer-inventory row first if (b).
- **`StreamEvent` shape** — honker's: `{ offset: i64, topic: String,
key: Option<String>, payload, created_at: i64 }` with
`payload_as<T>` (upstream lib.rs:1008). The contract needs its own
pinned field list (stream name vs topic token, key in or out per
the key decision, timestamp semantics) — probably inherited
near-verbatim.
- **Ordering guarantee row** — ADR-006's table has no streams
*ordering* cell content beyond "readable"; pin FIFO-by-offset as
the read ordering (`ORDER BY offset ASC` is already both
engines' claim path) and whether any per-key strengthening rides
the key decision.
- **Retention / bounded growth** — the reference read flags
`_honker_stream` as unbounded-growth unless the user sweeps
(honker-machinery §8.5); no consumer row names stream retention
(replay-forever is the *purpose* for the named need —
subscription surfaces). Options: consumer-side trim API
(`trim_to(offset)` — a new contract method), consumer-recipe
only (new stream + advance consumers + unschedule/nothing —
needs a deletion path that doesn't exist yet, so this option is
really "document the gap"), or document nothing (honest but
repeats the notifications-table mistake ADR-010 §6 just fixed).
Rides the no-ambient-sweeper posture either way.
"no placeholder semantics remain" positive consequence was false
for `publish_with_key` until this resolved — the contract suite
also could not pin cross-engine stream equivalence without an
ordering row)
- **Resolution**: Pinned by
[ADR-015](decisions/015-streams-depth.md). (1) **Key semantics —
option (a)**: the key is carried metadata (honker's shape — a
stored nullable column, round-tripped on every read, no
engine-enforced behavioral role); the ordering guarantee stays
**global FIFO by offset per stream**; per-key in-order reading is
the documented emergent pattern, not a promise. Option (b)
(server-enforced per-key ordering) rejected on its own decision
rule — no consumer row names the need; the consumer inventory's
streams row and the alkcall ecosystem-shape row (added for this
decision — alkcall's Sub/Pub/fan-out vocabulary routes the
topic/event-type dimension to *separate streams*, never per-key
lanes) both name none. Re-entry gate: a consumer-inventory row
naming server-enforced per-key ordering. Option (c) (cut
`publish_with_key`) rejected — removing a pinned v1 method name
would be a contract-breaking event for OQ-10, the cost is one
nullable column the inherited realizations already have, and the
key is a legitimately useful grouping token. (2) **A seam gap the
key decision exposed, closed in the same ADR**:
`publish_with_key_tx(stream, key, payload) -> offset` joins the
`TxHandle` trait — keyed publishes were otherwise unreachable
commit-atomically, the same hole shape OQ-13 exposed for the
outbox (`publish_tx` = the `None`-key call, not a separate engine
path; empty-`Some`-key → `InvalidName`, no new variants).
(3) **`StreamEvent` shape** — honker's near-verbatim with one
rename: `{ offset: i64, stream: String, key: Option<String>,
payload: Vec<u8>, created_at: i64 }` (`topic` → `stream` — one
term everywhere; the substrate's column name stays per ADR-012 §3
fidelity); `offset` immutable, never renumbered (gaps after trim
legal); `created_at` unix-seconds-at-publish, informational, not
an ordering field. (4) **Ordering guarantee row** — ADR-006 §2's
streams row extended: global FIFO by offset (read paths yield
`offset ASC`), offsets immutable — the row the contract suite
pins cross-engine equivalence against. (5) **Retention** —
`trim_to(horizon)` added to the stream handle (delete
`offset <= horizon`), consumer-invoked, no engine-default
retention, no ambient sweeper (the ADR-010 §6 posture); the
replay-forever default is documented with the in-contract tool to
bound it; reads from a trimmed-away region resume at the horizon;
saved offsets below the horizon stay valid. Framed as completing
contract v1 in place, pre-implementation — no versioning event,
OQ-10 untouched. Contract-suite rows added (cross-engine stream
equivalence, keyed-tx commit-atomicity, `trim_to` semantics —
both engines).
- **Cross-references**: OQ-04 (the pinning that left this depth
open), OQ-01 (scope row), ADR-002, ADR-006 (the guarantee table
the ordering row extends), ADR-008 §1.
open), OQ-13 (the sibling depth gap; the amend-in-place framing),
OQ-01 (scope row), ADR-002, ADR-006 §2 (the guarantee table the
ordering row extends), ADR-008 §1/§4/§5/§8, ADR-010 §6 (the
no-ambient-sweeper posture the trim decision applies), ADR-012 §3
(the fidelity posture the column-name keep rides).
### OQ-13: Transactional outbox enqueue shape — the `TxHandle` surface cannot express it — **RESOLVED**
@@ -396,13 +408,14 @@ narrowed to the pinning work its own record already scoped.)*
place, pre-implementation — no versioning event, OQ-10 untouched.
Contract-suite row added (commit-atomicity on both engines).
- **Cross-references**: OQ-04 (the pinning that missed this), OQ-12
(the sibling depth gap found by the same review), ADR-002
(the sibling depth gap found by the same review — resolved the
same day, [ADR-015](decisions/015-streams-depth.md)), ADR-002
(outbox scope row), ADR-007/ADR-008 (the seam design), ADR-010 §3
(the outbox's derived QueueOpts — unchanged by this).
## Deferred / Blocked
None currently. Every open OQ above is actionable Phase 1 work
(streams depth, capability-surface shape, versioning discipline,
fork-scaffold follow-through) with its evidence base complete — no
(capability-surface shape, versioning discipline, fork-scaffold
follow-through) with its evidence base complete — no
external arrivals are being waited on.
docs/architecture/overview.md
+2
View File
@@ -77,6 +77,8 @@ Per [ADR-002](decisions/002-feature-scope.md):
| [011](decisions/011-sqlite-substrate-fork.md) | SQLite substrate — fork honker-core into owned code | Accepted |
| [012](decisions/012-forked-substrate-design.md) | Forked substrate design (contract-blind boundary, fidelity, port deltas) | Accepted |
| [013](decisions/013-fold-substrate-into-sqlite.md) | Fold the forked substrate into `alkstore-sqlite` (no fourth crate) | Accepted |
| [014](decisions/014-outbox-tx-enqueue.md) | Transactional outbox enqueue (`outbox_enqueue_tx` on `TxHandle`) | Accepted |
| [015](decisions/015-streams-depth.md) | Streams depth (carried-metadata keys, global-FIFO ordering, `StreamEvent`, `trim_to`) | Accepted |
## Non-goals
docs/architecture/queues.md
+7 -1
View File
@@ -165,7 +165,13 @@ made under; the ADRs carry the WHY.
comes from the scheduler machinery above. The SQLite notifications
table's hygiene is engine-internal (not a consumer chore;
`prune_notifications*` stay out of the contract, resolving ADR-008
§8's disposition).
§8's disposition). The stream log's bounded-growth answer follows
the same posture but *is* consumer-side, because consumers interact
with stream rows directly ([ADR-015](decisions/015-streams-depth.md)
§5): `stream(name).trim_to(offset)` — invoked deliberately,
cadenced by the collapse recipe like every sweep; no engine-default
retention, the replay-forever default documented with the
in-contract tool to bound it.
## Scheduler (ADR-009)
docs/research/consumer-inventory.md
+32 -6
View File
@@ -1,10 +1,11 @@
---
status: draft
last_updated: 2026-10-04 (streams corrected to documented/in-scope —
last_updated: 2026-10-05 (streams corrected to documented/in-scope —
operator-authority record: type-filtered event watching from multiple
places, e.g. repo-change subscriptions in a git app. No consumer doc
carries the row yet; applications above the paused crates are the
wanters. rate-limits + result-storage remain the keep-with-flag rows.)
places, e.g. repo-change subscriptions in a git app. alkcall row added
2026-10-05 as ecosystem-shape evidence for streams (no per-key-ordering
need named anywhere); rate-limits + result-storage remain the
keep-with-flag rows.)
---
# alkstore — consumer-driven scope inventory
@@ -48,6 +49,24 @@ which scope decisions OQ-ST-01 still has to take per feature.
## The consumers and their artifacts
- **alkcall** — `/workspace/@alkdev/alkcall/docs/architecture/`
(Phase 1, 51 ADRs, several reviewed; reference checkout per the
AGENTS.md §3 posture). The call protocol crate
(bidirectional RPC on ALPN `alk/call`: Query/Mutation/Sub/Pub
operation types, subscriptions, channels). Added 2026-10-05 as an
*ecosystem-shape* consumer: it is not (yet) a direct store
consumer with named rows, but its pub/sub vocabulary —
`Subscription` ops (server→client streaming, ADR-021), `Pub`
(client→server streaming, ADR-046), `channel/resources/subscribe`
(snapshot + change events, ADR-037), and the futures
broker/fan-out mechanism (ADR-046's Gap B, deferred to channels,
OQ-22) — is the shape downstream crates' reactive ops will take.
The operator's note (2026-10-05): alkstore is meant as the base
storage layer for this broader ecosystem, and the store's streams
are what backs that vocabulary durably. Its ordering posture is
per-stream/per-channel (ordered, reliable, per-transport-stream) —
**no alkcall document names server-enforced per-key ordering**;
correlation is by request ID and delivery is per-connection.
- **alkfs** — `/workspace/@alkdev/alkfs/docs/research/phase-0.md`
(Phase 0, 2026-09-23; 17 OQs). The VFS/workspace crate; its ancestor
POC (`alknet-filesystem-poc`) used honker directly (honker-core 0.2.4
@@ -136,6 +155,7 @@ feature.
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| family-wide (reactivity requirement) | user/planning record, 2026-10-04 (operator's authority — the REQ-2 recording convention) | watching for specific event *types* from several different places at once. Concrete example: a git app at gitea/gitlab scale — users and other apps subscribe to changes on a repo. Many cases along these lines. | **documented** (operator-authority record; no consumer doc has grown the section yet) |
| alkcall (ecosystem shape) | alkcall `docs/architecture/`: decisions/021 (Sub, the Subscription streaming handler), decisions/046 (Pub + the `Subscription`→`Sub` rename + Gap B fan-out), decisions/037 (`channel/resources/subscribe`), open-questions OQ-22 | a durable substrate for the pub/sub vocabulary: subscribe-type ops backed by a stream that survives disconnect/restart (per-consumer offset + replay), and a broker/fan-out (one stream, N matching subscribers) over durable events when the channels session takes Gap B up. Ordering: global per-stream FIFO suffices for every named shape — no alkcall op names per-key ordering; the topic/event-type dimension routes to *separate streams* (one per topic), not keys within one stream. | **operator-authority** (operator record 2026-10-05 — alkstore is the base storage layer for the ecosystem these ops shape; evidence is the alkcall ADRs + that record; added 2026-10-05) |
Verdict: **in scope, first-class.** Corrected 2026-10-04 — the initial
draft (earlier the same day) graded this **absent** because no consumer
@@ -153,7 +173,11 @@ nothing written down yet carries the row — which is expected, not
suspicious. Design note for OQ-ST-04: on the Postgres side this is a
NOTIFY-triggered durable event table with per-consumer cursors (the
pg-boss-family shape); the per-consumer offset contract, not the
storage shape, is the seam to pin.
storage shape, is the seam to pin. **Added 2026-10-05** (the alkcall
row, evidence for OQ-12): the subscription vocabulary's ordering need
is per-stream FIFO — topic/event-type filtering routes to separate
streams, not per-key ordering within one; no current or planned
ecosystem op names server-enforced per-key ordering.
### rate limits
@@ -220,7 +244,9 @@ none of them yet.
- The inventory covers artifacts existing as of 2026-10-04: alkfs
phase-0, alkgit architecture (reviewed), alkblobs architecture
(draft-but-ADR'd), alknet-filesystem POC summary. If a new consumer
(draft-but-ADR'd), alknet-filesystem POC summary. *(Extended
2026-10-05: alkcall's architecture added as ecosystem-shape
evidence.)* If a new consumer
is added to the family (alksftp, the alknet rewrite, the ops
platform), a row gets added here *before* it is assumed into a scope
decision — the same discipline as alkblobs' requirements.md ("when a