From d44dfb5a08b2939d5835ac5d9a446ddbcd987dea Mon Sep 17 00:00:00 2001 From: "glm-5.3-flash" Date: Sun, 4 Oct 2026 05:27:22 +0000 Subject: [PATCH] docs: consumer inventory answers OQ-ST-01; phase-0 consumes it MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit consumer-inventory.md: per-feature scope synthesis over the paused consumers' written artifacts (alkfs phase-0, alkgit architecture, alkblobs ADRs, alknet-filesystem POC) — dissolves the circular 'deferring to consumers who can't run until we exist' framing. notify/locks pinned-or-documented (alkfs invalidation + writer coord, alkblobs fleet sweeper); queues/outbox documented (alkfs sync outbox, alkblobs embedder-owned cadence); scheduler documented-thin; streams/ rate-limits/result-storage have no named consumer — kept per working posture with cut flags, to revisit before implementation. phase-0.md: OQ-ST-01 answered by the inventory; OQ-ST-02 narrowed (uniform feature family across engines favors single-crate shape); OQ-ST-07 sharpened (no consumer needs the loadable-extension surface — cut-only decision); OQ-ST-04 contract-pinning scoped to inventory rows; plan step 1 marked done; references extended. AGENTS.md: architecture context gains the inventory with its add-a-row-before-assuming rule. --- AGENTS.md | 4 + docs/research/consumer-inventory.md | 205 ++++++++++++++++++++++++++++ docs/research/phase-0.md | 109 +++++++++++---- 3 files changed, 288 insertions(+), 30 deletions(-) create mode 100644 docs/research/consumer-inventory.md diff --git a/AGENTS.md b/AGENTS.md index c7b6173..c245792 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -112,6 +112,10 @@ coordinator runs. art, open questions (OQ-ST-01..NN), the POC register, and (eventually) the converged recommendation. Read it before any non-trivial work in this repo. +- `docs/research/consumer-inventory.md` — the per-feature scope + synthesis answering OQ-ST-01 from the paused consumers' documents. + New consumer needs get a row there *before* any scope decision + assumes them. - `docs/sdd_process.md` — the SDD process. Phase 0 in progress; `docs/architecture/` does not exist yet. - The SDD process and agent defs were copied from downstream projects diff --git a/docs/research/consumer-inventory.md b/docs/research/consumer-inventory.md new file mode 100644 index 0000000..101112b --- /dev/null +++ b/docs/research/consumer-inventory.md @@ -0,0 +1,205 @@ +--- +status: draft +last_updated: 2026-10-04 (initial draft — synthesized from the paused +consumers' written artifacts, not from any running consumer. Evidence +quality varies per row and is marked. Rows with weaker evidence are +marked as candidates-for-cut rather than silently included.) +--- + +# alkstore — consumer-driven scope inventory + +This document answers OQ-ST-01 (which honker features are in scope) the +only way that avoids circular hedging: by walking the concrete evidence +the paused consumers already left on disk. alkgit, alkfs, and alkblobs +are paused *processes*, but their *documents* are stable, reviewable +artifacts — requirements they impose on a reactive store, written before +alkstore existed. That is decisively different from deferring to "what +the consumers will say later," which was alkstore's original framing for +OQ-ST-01 and which never collapses: the consumer can't run until the +store exists and the store was waiting on the consumer — the classic +low-level/client deadlock, the exact shape the SDD process's hedging +rules name and the alkblobs open-questions tracker calls +"Schrödinger's code." + +Method: for each feature on honker's surface (the interface finding in +`phase-0.md` §Interface finding), find where a consumer document names a +need for it, quote the need, and grade the evidence. Confidence grades: + +- **pinned** — a consumer ADR or requirement doc names it and some + design leans on it already. +- **documented** — a consumer doc names the need concretely, but no + design has leaned on it yet. +- **recalled** — named in research/summary prose as intended, no + architecture written against it yet. +- **absent** — no consumer document names a need. Included by default + per the working posture, flagged for cut-at-implementation instead of + silently carried. + +The inventory is per-feature need, not per-feature design: how a need is +served (honker-identical, pg-boss-shaped, or re-derived) is OQ-ST-05/06. +This document only decides *which features have a consumer*, and hence +which scope decisions OQ-ST-01 still has to take per feature. + +## The consumers and their artifacts + +- **alkfs** — `/workspace/@alkdev/alkfs/docs/research/phase-0.md` + (Phase 0, 2026-09-23; 17 OQs). The VFS/workspace crate; its ancestor + POC (`alknet-filesystem-poc`) used honker directly (honker-core 0.2.4 + — notify-in-transaction, named locks identified as needed, queues as + the sync outbox). Findings: + `/workspace/@alkdev/alknet/docs/research/alknet-filesystem/poc-summary.md`. +- **alkgit** — `/workspace/@alkdev/alkgit/docs/architecture/` (Phase 1, + reviewed; 18 ADRs). The git protocol crate, paused on storage; its + object backend is being re-based on alkblobs (its OQ-07). +- **alkblobs** — `/workspace/@alkdev/alkblobs/docs/architecture/` + (12 ADRs + requirements REQ-1..4). The pooled CAS store; its `ops` + surface and fleet mode (REQ-2/ADR-008) are the consumers' reactive + needs at their sharpest. +- **alknet-filesystem POC** — the running evidence that the three-layer + shape (SQLite path tree + CAS + honker coordination) works, 8 passing + tests including notify-fires-on-commit inside the mutation txn + (`watch_fires_on_commit`). + +## Per-feature inventory + +### notify / listen (durable pub/sub signal) + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkfs (path tree) | alknet-filesystem `poc-summary.md` (test `watch_fires_on_commit`, Layer 3 notes); alkfs phase-0 §"What is already settled" | `notify()` inside the same transaction as path-tree mutations, atomically; watcher wakes on commit — this is the load-bearing invalidation source for any client-side cache over the tree | **pinned** | +| alkblobs (fleet) | requirements.md REQ-2; ADR-008 (fleet pins, advisory-locked sweeper); decisions/007 §Decision 2 (consumer tables co-tenant the pg instance) | cross-instance visibility of pool/GC state changes among fleet instances over one shared postgres — currently solved by DB rows + polling-free advisory machinery in-ADR; a notify surface is the natural substrate when a fleet node must tell siblings "sweep/manifest changed" | **documented** | +| alkblobs (any deploy) | decisions/007 §Decision 1 (honker ships WAL+NORMAL as the shipped tier defaults — the engine posture already assumes co-tenant reactive tooling on the sqlite file) | notify as the family's invalidation substrate on the sqlite default engine | **documented** | + +Verdict: **in scope, first-class.** This is the feature the crate is +named for — the substrate's answer to "how does a change in the database +become visible to other processes/connections?" (phase-0 §Vision +context). Transactional atomicity (notify inside the caller's +transaction, dropped on rollback) is the property every row leans on — +it is guiding principle 2 in phase-0.md. + +### named locks + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkfs (write path) | alkfs phase-0 OQ-FS-05 ("Concurrent writers to the same path need a real policy"); `poc-summary.md` ("honker named locks for writer coordination — mechanical", unknown #4: `honker_lock_acquire('path:::', writer_id, timeout)` in `WriteSession::open`) | per-path writer coordination: prevent concurrent write sessions stomp-locking each other; last-close-wins is session-safe but not POSIX — the lock is the missing piece that makes write sessions honest | **documented** (mechanically designed, unwired) | +| alkblobs (fleet GC) | gc-and-namespaces.md ("advisory-locked single sweeper"); ADR-008 fleet GC; store-api.md (`SweeperLock` contention as a `GcAbortCause`) | one sweeper at a time across a fleet of store instances sharing one pool — the sweeper advisory lock has TTL semantics and is already ADR-shaped | **pinned** | +| alkgit (ref CAS) | backend.md `GitRefs::apply_updates` (ADR-018: "one CAS transaction per call — the atomic-correctness home") | CAS against expected ref values could be served by pure row-level compare-and-set; a named lock is one admissible mechanism, not a proven need | **recalled** (alternative design exists) | + +Verdict: **in scope.** Two consumers with concrete, one already +ADR-shaped; the third has a competing design that a lock is *allowed* to +replace but needn't. + +### queues + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkfs (sync/replication) | alkfs phase-0 OQ-FS-14 ("the family pattern is composition — honker queues as the local outbox"); `poc-summary.md` ("background sync/replication kicks — honker queues; transactional outbox: enqueue in same txn as mutation"; "The honker queue is the coordination mechanism for background fetching" — the read-miss→async-fetch path) | durable background work whose trigger must be atomic with the mutation that created it (sync kicks, fetch-on-miss workers) — the write-session and deferred-fetch machinery of the FS | **documented** | +| alkblobs (ops/maintenance) | ops-surface.md (pin-token renewal is a TTL/TTL-3 cadence — a renew worker); gc-and-namespaces ("Explicit sweep; embedder owns cadence" — embedders that want interval sweeps add them above, per ADR-005 no-ambient-timers) | periodic maintenance (sweep cadence, pin renewal, orphaned-session reaping per alkfs OQ-FS-07) — every one of these has been *deliberately* left as "an embedder schedules it above"; a durable scheduled-job surface is the substrate answer those embeddings currently have to invent | **documented** | +| family-wide | phase-0 §Vision (queues as part of the one-sentence vision, from honker's default feature set) | the base reactive substrate: consumers shouldn't each hand-roll a poll loop over their own tables | **recalled** | + +Verdict: **in scope.** The strongest need is the transactional-enqueue +half (outbox), which is the same property as notify (principle 2) — +queues extend it to *work* rather than *signal*. + +### scheduler (cron / @every) + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkblobs (sweep cadence) | ADR-005 ("no ambient timers"; embedder owns cadence) | embedding a sweep interval per node is exactly cron/`@every` — with leader election if the deployment is a fleet (honker's scheduler: leader-elected advisory lock + missed-boundary catch-up) | **documented** (the need is ADR-pinned even though the ADR assigns it elsewhere) | +| alkfs (session reaping) | OQ-FS-05/OQ-FS-07 (crash cleanup of orphaned write sessions) | periodic orphan reaping pass | **recalled** | + +Verdict: **in scope, but the thinnest of the confirmed ones.** The need +is real (there is a family-wide "who sweeps/renews/reaps, and when" +problem that every consumer has punted to 'the embedder') but only just +begins to have consumers lean on it. Watch: if the scheduler turns out +to be queues + a tick, it may collapse into queues (OQ-ST-04's job). + +### transactional outbox helper + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkfs | OQ-FS-14, poc-summary (same rows as queues) | the enqueue-inside-business-tx shape, by name | **documented** | + +Verdict: **in scope** (as a *helper over* queues, per honker's shape) — +same evidence as queues; kept separate in the inventory only to note +that the helper is cheap once queues exist, not a separately-needed +feature. + +### streams (durable pub/sub with per-consumer offsets) + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| (none pinned) | — | — | **absent** | + +Verdict: **in scope per the working posture, flagged for +cut-at-implementation.** No consumer document names streams. The known +near-need is the *shape* (durable, replayable event log with explicit +offsets) rather than the feature: if alkfs's sync posture (OQ-FS-14) +grows a real multi-node story, or alkblobs' fleet gossip needs a +durable change-log rather than fire-and-forget notify, streams become +load-bearing — and `pg_notify`-plus-table (the pgboss-family pattern) +gives it on the Postgres side almost for free once queues exist. Keep +it, honest about cost, revisit before implementation. + +### rate limits + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| alkgit (budgeted resources) | ADR-009 budget shape (max pack size, max concurrent blocking pipeline tasks); OQ-FS-16's OQ references it too | alkgit's budgets are enforced in its own wire layer (backend.md: "the wire layer enforces the pipeline-concurrency budget itself") — a store-level rate limit is *not* the proven mechanism; a distributed rate-limit (across processes) is a different thing alkgit has not asked for | **recalled** (alternative design exists, in-crate) | + +Verdict: **weak candidate — first cut candidate.** No document needs +*store-level* rate limiting; alkgit's budgets are enforced above the +storage seam with its own mechanism. Keep listed because honker's +default set includes it and the machinery is trivial next to queues, +but it is the least-needful row after streams. + +### result storage (`save_result` / `get_result` / `sweep_results`) + +| Consumer | Evidence | Need | Confidence | +|---|---|---|---| +| (none pinned) | — | — | **absent** | + +Verdict: **in scope per the working posture, flagged for +cut-at-implementation.** Adjacent-tooling shape (a queue job's outcome +queryable by id), useful to queues+scheduler consumers but named by +none of them yet. + +## What this inventory changes + +- **OQ-ST-01** stops being "blocked on a consumer-driven inventory + pass" — this *is* that pass, run against paused consumers' documents + instead of running consumers. Remaining per-feature scope votes + shrink to: streams, rate limits, result storage (all + keep-with-flag), and the exclusion lines (DAGs/chains/chords, + multi-writer replication, cross-machine locking — no consumer names + these either; they stay out unless a consumer document grows one). +- **OQ-ST-02** gains a data point: the confirmed features (notify, + locks, queues, outbox, scheduler) are the same feature family on + both engines, which argues for the single-crate-with-feature-gated- + engines shape rather than a reactive-core + per-engine crates split + (the split buys something only if the engines' feature sets diverge + sharply, and this inventory says they don't). +- **OQ-ST-07** sharpens: no consumer needs the *loadable extension* + surface (every consumer is in-process Rust attaching to its own + connection, the `honker-core`/`attach_honker_functions` shape per + the POC summary); the embedded-rusqlite-only shape is the whole + consumer story so far. The loadable-extension option is now a + cut-only decision, not an open design fork. +- **OQ-ST-04's** contract-pinning work keeps all of the confirmed + features as contract candidates; the ambiguity found in the + inventory (queues vs scheduler boundary; streams' absent consumer; + rate limits' alternative mechanism) is input to which parts of the + honker-rs surface become contract versus cut. + +## The consumer list, honestly bounded + +- The inventory covers artifacts existing as of 2026-10-04: alkfs + phase-0, alkgit architecture (reviewed), alkblobs architecture + (draft-but-ADR'd), alknet-filesystem POC summary. If a new consumer + is added to the family (alksftp, the alknet rewrite, the ops + platform), a row gets added here *before* it is assumed into a scope + decision — the same discipline as alkblobs' requirements.md ("when a + consumer fact changes, it changes here first"). +- Absence of evidence is marked, not overruled: streams/rate-limits/ + result storage stay on the list with their flags visible, per the + working posture, rather than being cut by this document alone. \ No newline at end of file diff --git a/docs/research/phase-0.md b/docs/research/phase-0.md index 58ec608..5cde4cd 100644 --- a/docs/research/phase-0.md +++ b/docs/research/phase-0.md @@ -1,8 +1,9 @@ --- status: draft -last_updated: 2026-10-03 (initial draft + interface finding: honker-rs surface -read as the unified-API candidate; pgboss-rs LISTEN/NOTIFY-absence folded into -the driver/queue/ownership questions) +last_updated: 2026-10-04 (consumer inventory landed — OQ-ST-01 answered +per-feature from the paused consumers' documents; OQ-ST-02/07 +sharpened; phase-0 plan step 1 done. Interface finding and driver +tension from 2026-10-03 remain trusted-but-unverified working input.) --- # alkstore — Phase 0 (Exploration) @@ -11,8 +12,12 @@ This document captures Phase 0 (Exploration) for the `alkstore` crate: vision, guiding principles, prior art, and the open-question register (OQ-ST-01..NN). Phase 0's objective per `docs/sdd_process.md`: *capture vision and guiding principles; research options; validate approaches; -converge on a recommended approach.* This is the initial draft — none of -the questions below are settled, and there is no POC register yet. +converge on a recommended approach.* The scope question (OQ-ST-01) has +been answered per-feature from the consumers' documents +(`consumer-inventory.md`); the driver and reactive-shape questions +(OQ-ST-03/04) are the open research; there is no POC register yet +(see `consumer-inventory.md` for the first cut candidates that keep its +register small). Context for why this crate starts now: **alkblobs** (`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning) @@ -240,8 +245,8 @@ this crate: finding. Notable honest limitations documented by its own guides — the per-binding processing-guarantees table (auto-checkpoint cadence vs manual offset save; several bindings "may persist an offset on a - cadence... without knowing whether downstream application work - committed"), the Node reverse-order consumer-checkpoint bug, per- + cadence... without knowing whether downstream application work + committed"), the Node reverse-order consumer-checkpoint bug, per- binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++ missing typed pruning) — are exactly the seams a single-crate version designed-for-the-contract from day one can clean up. Its sync-only @@ -303,26 +308,44 @@ only). Promotion target: Phase 1 `docs/architecture/open-questions.md`. ### OQ-ST-01: Scope boundary — which honker features are in-scope? -Honker's feature list (queues, streams, notify, scheduling, locks, rate -limits, outbox helpers) is large; per-feature scope decisions don't -exist yet. Also undetermined: the exclusion lines (honker deliberately -excludes DAGs, task chains/chords, multi-writer replication, -distributed locking). +**Answered by the consumer inventory (2026-10-04) — +`docs/research/consumer-inventory.md`; scope votes now shrink to named +rows, not the whole feature list.** The original framing ("blocked on a +consumer-driven inventory pass") was circular hedging: the consumers +are paused, so the input would never arrive — but their *documents* +are stable evidence, and the inventory walks them per feature. +In-short: notify/listen and named locks are pinned or ADR-shaped +(alkfs path-tree invalidation; alkblobs fleet sweeper lock; alkfs +OQ-FS-05 writer coordination); queues and the outbox helper are +documented needs (alkfs sync/fetch-on-miss outbox; alkblobs +embedder-owned maintenance cadence); the scheduler is documented-thin +(the family-wide "who sweeps/renews/reaps" problem, possibly collapsing +into queues); streams, rate limits, and result storage have **no +named consumer** — carried per the keep-until-implementation posture +with cut-flags visible, cut later rather than silently included. -Not yet decidable without working through the concrete consumers (the -paused alkblobs crates and alkfs planning) — per-feature need hasn't -been articulated. Blocked on a consumer-driven inventory pass. +Per-feature exclusion lines (honker's own: DAGs, task chains/chords, +multi-writer replication, distributed locking) have no consumer either; +they stay out unless a consumer document grows one. New consumers +(alksftp, the alknet rewrite) add a row to the inventory *before* being +assumed into scope. ### OQ-ST-02: Crate scope — one store crate, or reactive-core + engines? Options include: single crate with feature-gated engines (the alk* feature-gate pattern); a core trait crate + per-engine crates; engine crates consuming a thin core. The answer constrains the driver decision -(OQ-ST-04) and the base-crate-lean invariant. +(OQ-ST-03) and the base-crate-lean invariant. -The use case isn't concrete yet (no engine code written, no consumer -wired). Deferred(scope) — concrete use case: the first engine -implementation would force this shape. +Input from the inventory (2026-10-04): the confirmed features +(notify/locks/queues/outbox/scheduler) are the same feature family on +both engines — the feature sets do not diverge sharply, which argues +for the single-crate-with-feature-gated-engines shape; a reactive-core ++ per-engine-crate split buys something only when engines' surfaces +diverge. Not yet decidable without the first engine implementation +forcing the shape; the inventory removed the "no concrete use case" +half of the deferral, the remaining half is real. Deferred(scope), +narrowed. ### OQ-ST-03: Driver story — sqlx, tokio-postgres, or per-engine drivers? @@ -364,9 +387,12 @@ point — the question decomposes into contract-pinning rather than shape-invention: - Which parts of the honker-rs surface become the crate's *contract*: - the `notify`/`listen` pair, the `stream`/offset/consumer model, the - queue claim/ack/visibility model, scheduler, locks/rate-limits, - outbox — all, a subset, or renamed/regrouped? + the `notify`/`listen` pair, the queue claim/ack/visibility model, + locks, outbox, scheduler — all now have named consumers (inventory); + the stream/offset/consumer model does not (streams is keep-with-flag) + and rate-limits have an in-crate alternative mechanism (alkgit's wire + layer) — subset, renamed/regrouped, decided against the inventory + rows rather than against the whole honker menu. - What is the delivery-guarantee contract, per mechanism (honker's own guide table shows how easily per-binding auto-checkpoint vs manual-save ambiguity produces *different* guarantees under one @@ -404,7 +430,8 @@ our tolerance for the alpha-state rc port, and on the verified gap choice, so the queue-machinery reuse value is the honest comparison point, not the whole. -Open; inputs: the OQ-ST-01 inventory + OQ-ST-03 resolution. +Open; inputs: OQ-ST-01's scope vote on queues (inventory: documented +need) + OQ-ST-03 resolution. ### OQ-ST-06: Honker relationship — reference, fork, or vendor? @@ -440,7 +467,13 @@ in-process may be the whole story (the honker-core shape minus the extension/binding packaging). Determines how much of honker is even candidate material. -Open; rides the first consumer-driven scope pass (OQ-ST-01). +**Sharpened by the inventory (2026-10-04):** every identified consumer +is in-process Rust attaching to its own connection (the +`honker-core`/`attach_honker_functions` shape — the alknet-filesystem +POC's actual usage). No consumer needs the loadable-extension surface. +The question is now cut-only: loadable extension is out unless a +consumer appears; the open residue is just how much of honker-core's +machinery survives the extraction. ### OQ-ST-08: Multi-host / deployment posture @@ -455,18 +488,21 @@ capability differences can surface). ## Phase 0 plan -Iteration expected; this register grows as the consumer inventory and -research rounds land. Expected sequence (deliberately rough): +Iteration expected; this register grows as research rounds land. +Expected sequence (deliberately rough): -1. Consumer-driven scope inventory (OQ-ST-01) — what the paused - alkblobs consumers and alkfs planning actually need from the reactive - store; walks back the per-feature scope decisions. +1. ~~Consumer-driven scope inventory (OQ-ST-01)~~ — **done + (2026-10-04)**: `consumer-inventory.md`, run against the paused + consumers' documents (alkfs phase-0, alkgit architecture, alkblobs + architecture + the alknet-filesystem POC). OQ-ST-01 answered down + to named per-feature rows; OQ-ST-02/07 sharpened by it. 2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library capability matrices, then a POC if the unified-trait shape needs validation (likely, given the structural mismatch noted in OQ-ST-04). The honker-rs surface (§Interface finding) is the concrete starting artifact for the OQ-ST-04 work: pinning its contract costs less and - is more honest than inventing a parallel shape. + is more honest than inventing a parallel shape — now scoped against + the inventory's confirmed features rather than the full honker menu. 3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem, after the driver and shape questions narrow the option space. 4. Converge; Phase 1 opens with the ADR backlog this register becomes. @@ -489,5 +525,18 @@ research rounds land. Expected sequence (deliberately rough): paused planning this crate unblocks; its POC findings (`poc-postgres-kv-findings.md`, `poc-pglo-findings.md`) are the tokio-postgres evidence base. +- alkgit — `/workspace/@alkdev/alkgit` (paused mid-Phase-1, + architecture reviewed): its backend.md trait seam and ADR set are + consumer evidence for the inventory (queues/locks rows). +- alkfs — `/workspace/@alkdev/alkfs` (Phase 0 drafted, 2026-09-23): its + phase-0 OQs (OQ-FS-05/07/14/16/17) are consumer evidence for the + inventory (notify/locks/queues rows). +- alknet-filesystem POC — + `/workspace/@alkdev/alknet/docs/research/alknet-filesystem/ + poc-summary.md`: the ran-once evidence that the honker-coordination + layer works (notify-on-commit test; named-locks and outbox usage + identified). +- consumer-inventory.md (`docs/research/consumer-inventory.md`) — the + per-feature synthesis (2026-10-04) answering OQ-ST-01 from the above. - alkcall — `/workspace/@alkdev/alkcall`: the family substrate; referenced for the store-layer-isolation principle only. \ No newline at end of file