alkcall/docs/research/topic-broker.md

The Topic Broker: Placement, Shape, and the "Op as Topic" Tangle

Status: Phase 0 (exploration — captures the problem and the candidate shapes; no decision taken, no ADR written. Implementation follows only after this converges and Phase 1 produces the ADR + spec). Date: 2026-10-13 Research inputs: operator recollection (2026-10-13, recorded and disentangled in §3), the post-stub-sweep context (reviews 010/011, the v0.9 release-plan discussion), alkcall ADRs 021/024/040/042/046/047/049/052/053 and OQ-22/39/40/42/43, the alkstore consumer inventory + ADR-015 (alkstore /workspace/@alkdev/alkstore), the TypeScript reference implementation (/workspace/@alkdev/pubsub, @alkdev/pubsub — the crates.io-unpublished ancestor of alkcall's EventEnvelope), and the hub-relay composition (ADR-042, ADR-051).

---

1. Why this research exists

Two independent observations converged on the same question in the v0.9 release-planning discussion:

1. The sweep found its own neighbor. The stub sweep removed the last not_implemented surfaces and re-keyed the live docs — but the OQ ledger still carries one deferral whose premise expired and whose escape route is itself unowned. OQ-42 (the broker) is deferred(scope), parked on "a concrete producing crate committing to fan-out," and its home is the composition layer (OQ-43) — whose ownership and shape are also uncommitted. A deferral blocked on a deferral, while two crates downstream (alkgit, alkfs) wait on what it would produce, is the circular-hedging shape in paperwork form: no false advertising on any API surface, no failing handler — just a dependency chain where every link waits on a link that never commits.

2. The deferral was priced under a different dependency graph. ADR-046 §fan-out and ADR-047 Gap B ruled the broker out of alkcall when the alternative was cheap: at that point, no downstream crate built on the pub/sub vocabulary. Today, four published crates consume alkcall, alkblobs' reactive ops surface is the sharpest named need (the alkblobs consumer-remediation input, review 011), and alkgit + alkfs (both paused on alkblobs) grow subscription surfaces on top of it. If the broker is built later, the version bump that delivers it ripples across every crate that grew a subscription surface in the meantime — the operator's stated cost estimate is 6–8 crates, vs riding the v0.9 bump that already touches the 4 published dependents. Deferral here is not free; it compounds.

So this document's job is narrow: disentangle the fused concept that blocked honest specification (§3), inventory what already exists to compose on (§5), assess whether the placement ruling should be amended (§6), and enumerate the open questions that Phase 1 must resolve before any code is written (§8).

2. Prior state of the question (what is decided today)

• The primitives are delivered and load-bearing (ADR-046): the Pub operation type, HandlerKind::Sink + SinkHandler, invoke_sink() dispatch, the call.published wire event (the envelope's sixth event type, wire-stable), and CallConnection::publish().
• The fan-out/broker was ruled out of scope for alkcall (ADR-047 Gap B; ADR-046 §"Fan-out"): topic registry, Pub↔Sub matching by (op_name, params_hash), N-consumer fan-out — placed at the hub composition layer, "the same way the hub relay composes on top of call ops." OQ-22 resolved the primitive half; the broker is OQ-42.
• The named first consumer drifted. OQ-42's blocking condition names "a concrete producing crate committing to fan-out (plausible first candidate: alktrader market-data distribution)." alktrader has since become a quant/backtest library: its market-data path is an analytics input, not a live N-spoke distribution surface. No crate currently matches the blocking condition — which is how OQ-42 ended up uncommittable.
• The "channels" consumer the original rationale named is served differently. ADR-046's deferral reason 2 was "the broker's first consumer is channels" (channel/resources/subscribe, hub-as-broker ADR-042). The discovery-UI half of resources/subscribe shipped as the snapshot-then-changes enumerator surface (ADR-053) — live resource enumeration, not a general topic broker. Hub relay (ADR-042/051) composes call ops without a broker.
• alkstore owns the durable substrate below all of this. Its consumer inventory (streams row) explicitly names alkstore streams as the durable back of alkcall's pub/sub vocabulary — per-stream FIFO by offset, per-consumer cursors, replay-on-attach — and pins the topic dimension to separate streams, never per-key lanes within one stream (alkstore ADR-015 §1).

3. The tangle: two ideas fused in "op as topic"

The operator's recollection (recorded verbatim, then split):

"The 'op as a topic' was a bit hand wavy in hindsight. The general idea was to basically use a hash as a topic and for deduping purposes. The 'op as a topic' was mostly so one could call a subscription operation and receive a stream of events. The reason they were modeled like regular ops is that we can validate the input and provide a schema that is required for that particular subscription (same on the pub side). So it should allow for ACL patterns inside the pubsub implementation."

Two separable design needs got fused into one mechanism:

• Need A — a topic identity for matching and dedup. Something hashable, stable, and comparable: the key used to route a published event to matching subscribers, and (possibly) the dedup key that prevents duplicate delivery. ADR-046 §fan-out's (op_name, params_hash) is one instance of this.
• Need B — a subscription surface with real operation properties. Subscriptions should be validatable (declare the input schema on the subscribe side; declare the publish schema on the publish side), and authorizable (AccessControl::check on both halves). The "model it as a regular op" instinct exists so the registry machinery — schema compile, ACL check, visibility, provenance — is inherited for free rather than hand-rolled inside the broker.

Splitting them exposes a design space the fused formulation hid:

Aspect	Need A (identity)	Need B (op surface)
Decides	which subscribers an event reaches; dedup	who may subscribe/publish; what shapes are accepted
Natural home	topic table / matching engine	registry registration (specs + ACL)
Failure mode if fused	topics inherit op-registry coupling they don't need; hash identity leaks op internals	ad-hoc topic surface re-implements schema + ACL machinery the registry already has

The candidate resolutions for both are the Phase 1 subjects (§8, QP-1/QP-3).

4. Consumer inventory (who waits on this)

Cross-referenced from the alkstore consumer inventory, review 011's downstream input, and the release-plan discussion:

Consumer	Need	Status
alkblobs	its reactive ops surface (Sink ops at their sharpest — REQ-2/ADR-008 lineage); ops-put works against the delivered register_pub_openable path post-fix (reviews 010/011), but cross-put fan-out ("many consumers of one blob stream") wants the broker	published-pending (unpublished crate, pre-first-release)
alkgit	repo-change subscriptions ("users and other apps subscribe to changes on a repo" — the operator's recorded need in alkstore's consumer inventory, streams row); paused on storage → alkblobs	paused
alkfs	the VFS workspace crate; its ancestor POC runs on honker notify; client-side cache invalidation (the "repo + cache invalidation" hot-path pattern the operator cites) wants topic-shaped invalidation signals	not yet decomposed
hub/relay deployments	one producer publishing, N spokes receiving (ADR-042's relay handles bytes-forwarding for channels; the event-shaped fan-out is the broker's)	composition layer, unowned (OQ-43)
alktrader	OQ-42's original named candidate	drifted — analytics, not distribution; do not re-park on it
family-wide	the reactivity requirement row (alkstore consumer inventory, streams): "watching for specific event types from several different places at once," durable, offline-tolerant	the named shape all of the above instantiate

Note the family's shape consensus, recorded independently on both sides: the topic dimension routes to separate channels/streams — one per topic (alkstore ADR-015 §1; ADR-034 §substrate for wire channels), never per-key lanes inside one stream. Whatever identity model Phase 1 picks, it should preserve one-topic-one-delivery-pipe.

5. What exists to compose on (in-crate inventory)

Everything the broker needs as load-bearing inputs is already in this crate and verified:

• Delivery primitives — invoke_sink() (call.published sink-mode), invoke_streaming() (Sub, call.responded), the wire-stable EventEnvelope framing with additive-event rule (ADR-014: new event types may be added; existing shapes frozen).
• Registry machinery for Need B — OperationSpec (schema compile + publish_schema validation), AccessControl + required_scopes, Visibility (ADR-017), provenance (ADR-018), and the registry-layering pattern (ADR-019) — session/connection overlays for dynamic subscription attach/detach.
• The in-crate composition precedents that weaken the placement ruling — ChannelOperations (channel lifecycle ops composed on the call registry, ADR-047), establishers with Establishment plan delivery (ADR-049/052), and channel/resources/subscribe — a live subscription op with per-resource routing, snapshot-then-changes, composed in-crate (ADR-053). None of these needed transport, ALPN, or dial knowledge.
• Peer vocabulary for cross-node fan-out — ADR-024's peer-graph routing model (PeerRef routing, peer-keyed overlays) is the in-crate vocabulary for "deliver to N peers"; OQ-17 keeps multi-hop out of scope.
• Backpressure/lifecycle precedents — bounded buffers, per-identity caps, monotonic IDs (ADR-040/041), the abort cascade with continue-running opt-in for slow-subscriber management (ADR-020).
• The durable substrate downstream — alkstore streams (per-consumer offset, replay, trim), engines mem/sqlite/postgres under one contract.
• The TS reference — @alkdev/pubsub (typed event-target: topic = event type name; subscribe/publish/subscribeOnce vocabulary; the EventEnvelope ancestor). Shape only; the Rust realization composes on the registry instead of CustomEvent.

6. The placement ruling: amend ADR-047 Gap B?

The case for keeping the broker out of alkcall was: (1) point-to-point call protocol, (2) the broker's lifecycle is routing-shaped, (3) designing blind produces half-designs. Each has softened:

1. Point-to-point is true of transport, not of composition. The registry-operations pattern (ADR-047) already puts composition ops in-crate; ADR-053 ships a live subscriber op in-crate. A topic registry is a registry overlay + dispatch bookkeeping, the same class.
2. The lifecycle concern dissolves at the module boundary. A broker module composed on a registry has exactly the lifecycle its registry has; the "topic registry outlives individual calls" objection describes a deployment property, satisfied by any long-lived node, hub or not.
3. The half-design risk is the real surviving constraint — and it now has both halves. The consumer side is no longer hypothetical (alkblobs' ops surface, alkgit's subscription vocabulary recorded in the alkstore inventory), and the substrate side exists (alkstore streams, the TS reference). A Phase 1 spec against these is not a blind design.

Recommendation (non-final until §8 converges): amend the placement — the broker becomes an in-crate, opt-in composition module (the ChannelOperations pattern): it registers topic ops into whatever OperationRegistry the node owner points it at, carries no transport or dial knowledge, and stays unregistered by default so no existing surface silently gains it. What remains out of alkcall: engine wiring, glue, dial — OQ-43's territory. The hub that fans out across connections composes the module with its CallConnections; within one connection, the module composes locally. The durable event log behind a topic's replay is a TopicStore trait seam (§7), not an in-crate engine choice.

This also resolves OQ-42's uncommittable blocking condition: the deferral's "wait for a consumer to commit" gate is replaced by shipped consumers already on record.

7. The durability seam (the "not rocket surgery" constraint)

The operator's requirement: the trait seam must let a downstream assembly add memory/sqlite/postgres adapters with the same ease as the repo + cache-invalidation hot-path pattern — adapters are written once per engine, not per deployment.

Shape to converge on in Phase 1:

• In-memory is the default realization — a TopicBroker with no injected store behaves as the TS reference does: live fan-out, stateless, zero durability.
• The seam is one small trait (working name TopicStore), roughly covering: append a published event (returning the topic's monotonically increasing position), read events at-or-after a position for a topic, and per-subscriber cursor save/load. This maps one-to-one onto the alkstore stream contract (publish_with_key_tx, offset-ordered reads, per-consumer cursor saves — alkstore ADR-015 §2/§4), so the postgres/sqlite adapters are alkstore-streams-backed and the memory adapter is the broker itself.
• Reverse dependency discipline — alkcall gains no alkstore dependency. The seam stays dependency-free (plain trait, core-value bytes crossing it, mirroring the vendored-core "lean, no engine" posture); alkstore-backed adapters live downstream, where engines already live (the OQ-43 glue consumes both crates).
• Honest scope statement for durable mode: durable delivery with per-consumer replay is a deployment capability. The v1 broker contract should specify the seam, not the durability guarantees beyond it (ordering, dedup, retention pricing ride the store behind the seam — for alkstore-backed adapters, those are ADR-015's pinned rows verbatim).

8. Open questions (proposed — to be promoted into Phase 1's ledger)

These are the questions this research identifies as needing decisions before any ADR or code. They're proposed in broker-local numbering (QP-*) until Phase 1 promotes them as OQ-44+.

QP-1: Topic identity — named topics, op-shaped topics, or a split?

Need A's model. Candidates:

• (a) Named topics (strings). topic = stream/event-type name; the TS reference's shape; one-topic-one-pipe; alkstore-stream-aligned. Matching is exact-name; wildcarding is optional and additive.
• (b) Op-shaped ((op_name, params_hash)), ADR-046 §fan-out's literal form. Fan-out targets are Sub-typed ops; identity inherits op internals. Pros: dedup key falls out; wire-consistency with existing ops. Cons: the identity couples routing to op schema shape (params_hash stability depends on the schema's canonicalization — which is a real design burden, see §3's A/B split); one topic per op per registrant, so cross-op co-subscription needs the hash of the matching rule anyway.
• (c) The split: identity is named topics (a); "ops as topics" survives only as Need B — a subscriber declares a topic and the input schema it validates against, so per-subscription input validation is a subscriber-side schema, validated at subscribe time and at delivery time.

Working recommendation: (c) — identity as named topics, schema/ACL inheritance via the registry on the two broker ops, dedup keying re-derived from topic semantics rather than op params.

QP-2: Dedup and delivery semantics

"A hash as a topic ... for deduping purposes" — what does dedup mean here, precisely? Candidate framings to resolve: idempotent-publish keys (producer-supplied event ids, dedup at the store seam), vs subscriber-side dedup (at-least-once delivery + a dedup window), vs no dedup contract in v1. The answer is pinned partly by the store seam: durable adapters can offer producer-supplied-id dedup for free; in-memory mode cannot (and shouldn't claim to). The contract must state delivery semantics honestly per mode (live: at-most-once per attached subscriber; durable-with-replay: at-least-once per cursor) rather than promising a uniform thing two substrates can't both provide.

QP-3: Wire shape — new ops, new event type, or composed handler kinds?

• Broker ops registered into a registry (topic/subscribe, topic/publish, optionally topic/topics listing) — additive, no envelope change required: subscribe rides call.responded (Sub), publish rides call.published (Sink).
• If wire-visible events beyond op names are wanted (e.g. per-topic attach/detach events), they are new envelope event types — legal under ADR-014's additive rule, but each one is a cross-language deliverable to keep in sync; default to zero new event types.

QP-4: ACL and schema on both halves

The operator's requirement, mapped to the registry: the broker ops carry their own AccessControl (who may subscribe/publish at all) — and each topic's authorization model (per-topic publish ACL, per- topic subscribe ACL, maybe scope patterns) is the piece the fused concept gestured at with "ACL patterns inside the pubsub." Decision needed: per-topic ACL tables owned by the broker module vs topic ops as individually registered specs (each topic's publish/subscribe ops registered separately, inheriting the full registry ACL/visibility/ provenance machinery). The latter is the pure Need-B realization but costs a registration per topic; the former is one registration pair with a topic-table lookup. This materially changes the decomposition.

QP-5: Subscription lifecycle

Attach (subscribe with optional position for durable mode), detach, disconnect behavior (live mode: gone on transport drop; durable mode: cursor persists, resubscribe at position — the alkstore streams row's contract), subscriber cap policy (ADR-041 pattern), and interaction with the abort cascade (ADR-020) when a subscriber's sink stream dies.

QP-6: Cross-node fan-out and the hub's margin of work

Within one node: registry dispatch. Across nodes: the hub composes (it holds CallConnections; ADR-024 vocabulary). Decision: does v1 ship any in-crate cross-connection fan-out helper (a FanOut that mirrors call.published streams into peer connections), or does the hub wire that itself from the primitives? ADR-042's relay exists for channels byte-forwarding; its analog for event fan-out is un-specced — it is either a small in-crate helper or hub-side composition, and the placement ruling (§6) should state which explicitly so OQ-43 doesn't inherit an ambiguity.

QP-7: Backpressure across fan-out

One slow subscriber must not back-pressure the topic (the OQ-23/ADR-040 lesson carried into event delivery): per-subscriber bounded queues with an explicit slow-subscriber policy (drop + event, drop + disconnect, block-bounded). Policy default and configurability are to be pinned in Phase 1; the ADR-040 bounded-buffer machinery is the reference.

QP-8: Ordering guarantee

Per-topic FIFO (matches alkstore ADR-015 §1's pinned row and ADR-046's per-stream delivery posture). The open part: does the broker promise ordering across publishers to one topic (single sequencer required) or only per-publisher (interleaving legal, no sequencer needed)? The durable store behind the seam forces single-writer-per-topic-sequencer for (a); in-memory could do either — but the contract has to match the store's row, so this resolves with QP-2 rather than independently.

9. Recommendation and convergence criteria

Direction (to carry into Phase 1, not final): an in-crate, opt-in broker composition module on the call registry — named topics for identity (QP-1c), registry machinery providing schema + ACL on the two broker ops (QP-3, QP-4 resolved as one topic-table vs per-topic registration decision), a dependency-free TopicStore seam with the broker itself as the in-memory adapter (QP-D from §7), per-topic FIFO with an honest per-mode delivery contract (QP-2, QP-8), ADR-040-style per-subscriber backpressure policy (QP-7), and an explicit statement on the cross-node margin (QP-6). ADR-047 Gap B is amended by the Phase 1 ADR (which supersedes the placement ruling for this scope); OQ-42 is resolved by that ADR; OQ-43 remains untouched (dial/engine/glue is still the composition layer's).

Convergence criteria — Phase 0 is done when:

1. Every QP-* above carries a candidate resolution with its rejection cases written down (the ADR-032 one-way-door discipline applies to the wire-visible ones).
2. At least one concrete consumer surface has signed off on the topic model — alkblobs' put-path ops for the publish half, the recorded alkgit subscription vocabulary for the subscribe half (the review 011 downstream-input pattern: the consuming crate drives the contract).
3. The QP-4 decision (topic-table vs per-topic registration) is made — it is the largest decomposition determinant.
4. The placement amendment's blast radius is enumerated: which ADRs/ docs re-key (046 §fan-out, 047 Gap B, open-questions OQ-42, README ADR tables; alkstore consumer-inventory rows are cross-repo notes, updatable but not blocking).

Out of scope for this research: implementation, task decomposition, version assignment, wire-format drafting. Nothing here creates a not_implemented surface — the broker module does not exist and no API names, docs, or changelog entries will advertise it until Phase 1's ADR lands, per the implement-or-stop-advertising ruling (the only permitted deferral statement is "not this session").