alkdev/alkstore
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: consumer inventory answers OQ-ST-01; phase-0 consumes it

Browse Source 
consumer-inventory.md: per-feature scope synthesis over the paused
consumers' written artifacts (alkfs phase-0, alkgit architecture,
alkblobs ADRs, alknet-filesystem POC) — dissolves the circular
'deferring to consumers who can't run until we exist' framing.
notify/locks pinned-or-documented (alkfs invalidation + writer coord,
alkblobs fleet sweeper); queues/outbox documented (alkfs sync outbox,
alkblobs embedder-owned cadence); scheduler documented-thin; streams/
rate-limits/result-storage have no named consumer — kept per working
posture with cut flags, to revisit before implementation.

phase-0.md: OQ-ST-01 answered by the inventory; OQ-ST-02 narrowed
(uniform feature family across engines favors single-crate shape);
OQ-ST-07 sharpened (no consumer needs the loadable-extension surface —
cut-only decision); OQ-ST-04 contract-pinning scoped to inventory rows;
plan step 1 marked done; references extended.

AGENTS.md: architecture context gains the inventory with its
add-a-row-before-assuming rule.
This commit is contained in:
glm-5.3-flash committed 2026-10-04 05:27:22 +00:00
1 parent 1cb007d894
commit d44dfb5a08
3 files changed
+288 -30
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/research/consumer-inventory.md
+205
View File
@@ -0,0 +1,205 @@
---
status: draft
last_updated: 2026-10-04 (initial draft — synthesized from the paused
consumers' written artifacts, not from any running consumer. Evidence
quality varies per row and is marked. Rows with weaker evidence are
marked as candidates-for-cut rather than silently included.)
---
# alkstore — consumer-driven scope inventory
This document answers OQ-ST-01 (which honker features are in scope) the
only way that avoids circular hedging: by walking the concrete evidence
the paused consumers already left on disk. alkgit, alkfs, and alkblobs
are paused *processes*, but their *documents* are stable, reviewable
artifacts — requirements they impose on a reactive store, written before
alkstore existed. That is decisively different from deferring to "what
the consumers will say later," which was alkstore's original framing for
OQ-ST-01 and which never collapses: the consumer can't run until the
store exists and the store was waiting on the consumer — the classic
low-level/client deadlock, the exact shape the SDD process's hedging
rules name and the alkblobs open-questions tracker calls
"Schrödinger's code."
Method: for each feature on honker's surface (the interface finding in
`phase-0.md` §Interface finding), find where a consumer document names a
need for it, quote the need, and grade the evidence. Confidence grades:
- **pinned** — a consumer ADR or requirement doc names it and some
design leans on it already.
- **documented** — a consumer doc names the need concretely, but no
design has leaned on it yet.
- **recalled** — named in research/summary prose as intended, no
architecture written against it yet.
- **absent** — no consumer document names a need. Included by default
per the working posture, flagged for cut-at-implementation instead of
silently carried.
The inventory is per-feature need, not per-feature design: how a need is
served (honker-identical, pg-boss-shaped, or re-derived) is OQ-ST-05/06.
This document only decides *which features have a consumer*, and hence
which scope decisions OQ-ST-01 still has to take per feature.
## The consumers and their artifacts
- **alkfs** — `/workspace/@alkdev/alkfs/docs/research/phase-0.md`
(Phase 0, 2026-09-23; 17 OQs). The VFS/workspace crate; its ancestor
POC (`alknet-filesystem-poc`) used honker directly (honker-core 0.2.4
— notify-in-transaction, named locks identified as needed, queues as
the sync outbox). Findings:
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/poc-summary.md`.
- **alkgit** — `/workspace/@alkdev/alkgit/docs/architecture/` (Phase 1,
reviewed; 18 ADRs). The git protocol crate, paused on storage; its
object backend is being re-based on alkblobs (its OQ-07).
- **alkblobs** — `/workspace/@alkdev/alkblobs/docs/architecture/`
(12 ADRs + requirements REQ-1..4). The pooled CAS store; its `ops`
surface and fleet mode (REQ-2/ADR-008) are the consumers' reactive
needs at their sharpest.
- **alknet-filesystem POC** — the running evidence that the three-layer
shape (SQLite path tree + CAS + honker coordination) works, 8 passing
tests including notify-fires-on-commit inside the mutation txn
(`watch_fires_on_commit`).
## Per-feature inventory
### notify / listen (durable pub/sub signal)
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkfs (path tree) | alknet-filesystem `poc-summary.md` (test `watch_fires_on_commit`, Layer 3 notes); alkfs phase-0 §"What is already settled" | `notify()` inside the same transaction as path-tree mutations, atomically; watcher wakes on commit — this is the load-bearing invalidation source for any client-side cache over the tree | **pinned** |
| alkblobs (fleet) | requirements.md REQ-2; ADR-008 (fleet pins, advisory-locked sweeper); decisions/007 §Decision 2 (consumer tables co-tenant the pg instance) | cross-instance visibility of pool/GC state changes among fleet instances over one shared postgres — currently solved by DB rows + polling-free advisory machinery in-ADR; a notify surface is the natural substrate when a fleet node must tell siblings "sweep/manifest changed" | **documented** |
| alkblobs (any deploy) | decisions/007 §Decision 1 (honker ships WAL+NORMAL as the shipped tier defaults — the engine posture already assumes co-tenant reactive tooling on the sqlite file) | notify as the family's invalidation substrate on the sqlite default engine | **documented** |
Verdict: **in scope, first-class.** This is the feature the crate is
named for — the substrate's answer to "how does a change in the database
become visible to other processes/connections?" (phase-0 §Vision
context). Transactional atomicity (notify inside the caller's
transaction, dropped on rollback) is the property every row leans on —
it is guiding principle 2 in phase-0.md.
### named locks
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkfs (write path) | alkfs phase-0 OQ-FS-05 ("Concurrent writers to the same path need a real policy"); `poc-summary.md` ("honker named locks for writer coordination — mechanical", unknown #4: `honker_lock_acquire('path:<bucket>:<branch>:<path>', writer_id, timeout)` in `WriteSession::open`) | per-path writer coordination: prevent concurrent write sessions stomp-locking each other; last-close-wins is session-safe but not POSIX — the lock is the missing piece that makes write sessions honest | **documented** (mechanically designed, unwired) |
| alkblobs (fleet GC) | gc-and-namespaces.md ("advisory-locked single sweeper"); ADR-008 fleet GC; store-api.md (`SweeperLock` contention as a `GcAbortCause`) | one sweeper at a time across a fleet of store instances sharing one pool — the sweeper advisory lock has TTL semantics and is already ADR-shaped | **pinned** |
| alkgit (ref CAS) | backend.md `GitRefs::apply_updates` (ADR-018: "one CAS transaction per call — the atomic-correctness home") | CAS against expected ref values could be served by pure row-level compare-and-set; a named lock is one admissible mechanism, not a proven need | **recalled** (alternative design exists) |
Verdict: **in scope.** Two consumers with concrete, one already
ADR-shaped; the third has a competing design that a lock is *allowed* to
replace but needn't.
### queues
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkfs (sync/replication) | alkfs phase-0 OQ-FS-14 ("the family pattern is composition — honker queues as the local outbox"); `poc-summary.md` ("background sync/replication kicks — honker queues; transactional outbox: enqueue in same txn as mutation"; "The honker queue is the coordination mechanism for background fetching" — the read-miss→async-fetch path) | durable background work whose trigger must be atomic with the mutation that created it (sync kicks, fetch-on-miss workers) — the write-session and deferred-fetch machinery of the FS | **documented** |
| alkblobs (ops/maintenance) | ops-surface.md (pin-token renewal is a TTL/TTL-3 cadence — a renew worker); gc-and-namespaces ("Explicit sweep; embedder owns cadence" — embedders that want interval sweeps add them above, per ADR-005 no-ambient-timers) | periodic maintenance (sweep cadence, pin renewal, orphaned-session reaping per alkfs OQ-FS-07) — every one of these has been *deliberately* left as "an embedder schedules it above"; a durable scheduled-job surface is the substrate answer those embeddings currently have to invent | **documented** |
| family-wide | phase-0 §Vision (queues as part of the one-sentence vision, from honker's default feature set) | the base reactive substrate: consumers shouldn't each hand-roll a poll loop over their own tables | **recalled** |
Verdict: **in scope.** The strongest need is the transactional-enqueue
half (outbox), which is the same property as notify (principle 2) —
queues extend it to *work* rather than *signal*.
### scheduler (cron / @every)
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkblobs (sweep cadence) | ADR-005 ("no ambient timers"; embedder owns cadence) | embedding a sweep interval per node is exactly cron/`@every` — with leader election if the deployment is a fleet (honker's scheduler: leader-elected advisory lock + missed-boundary catch-up) | **documented** (the need is ADR-pinned even though the ADR assigns it elsewhere) |
| alkfs (session reaping) | OQ-FS-05/OQ-FS-07 (crash cleanup of orphaned write sessions) | periodic orphan reaping pass | **recalled** |
Verdict: **in scope, but the thinnest of the confirmed ones.** The need
is real (there is a family-wide "who sweeps/renews/reaps, and when"
problem that every consumer has punted to 'the embedder') but only just
begins to have consumers lean on it. Watch: if the scheduler turns out
to be queues + a tick, it may collapse into queues (OQ-ST-04's job).
### transactional outbox helper
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkfs | OQ-FS-14, poc-summary (same rows as queues) | the enqueue-inside-business-tx shape, by name | **documented** |
Verdict: **in scope** (as a *helper over* queues, per honker's shape) —
same evidence as queues; kept separate in the inventory only to note
that the helper is cheap once queues exist, not a separately-needed
feature.
### streams (durable pub/sub with per-consumer offsets)
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| (none pinned) | — | — | **absent** |
Verdict: **in scope per the working posture, flagged for
cut-at-implementation.** No consumer document names streams. The known
near-need is the *shape* (durable, replayable event log with explicit
offsets) rather than the feature: if alkfs's sync posture (OQ-FS-14)
grows a real multi-node story, or alkblobs' fleet gossip needs a
durable change-log rather than fire-and-forget notify, streams become
load-bearing — and `pg_notify`-plus-table (the pgboss-family pattern)
gives it on the Postgres side almost for free once queues exist. Keep
it, honest about cost, revisit before implementation.
### rate limits
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| alkgit (budgeted resources) | ADR-009 budget shape (max pack size, max concurrent blocking pipeline tasks); OQ-FS-16's OQ references it too | alkgit's budgets are enforced in its own wire layer (backend.md: "the wire layer enforces the pipeline-concurrency budget itself") — a store-level rate limit is *not* the proven mechanism; a distributed rate-limit (across processes) is a different thing alkgit has not asked for | **recalled** (alternative design exists, in-crate) |
Verdict: **weak candidate — first cut candidate.** No document needs
*store-level* rate limiting; alkgit's budgets are enforced above the
storage seam with its own mechanism. Keep listed because honker's
default set includes it and the machinery is trivial next to queues,
but it is the least-needful row after streams.
### result storage (`save_result` / `get_result` / `sweep_results`)
| Consumer | Evidence | Need | Confidence |
|---|---|---|---|
| (none pinned) | — | — | **absent** |
Verdict: **in scope per the working posture, flagged for
cut-at-implementation.** Adjacent-tooling shape (a queue job's outcome
queryable by id), useful to queues+scheduler consumers but named by
none of them yet.
## What this inventory changes
- **OQ-ST-01** stops being "blocked on a consumer-driven inventory
pass" — this *is* that pass, run against paused consumers' documents
instead of running consumers. Remaining per-feature scope votes
shrink to: streams, rate limits, result storage (all
keep-with-flag), and the exclusion lines (DAGs/chains/chords,
multi-writer replication, cross-machine locking — no consumer names
these either; they stay out unless a consumer document grows one).
- **OQ-ST-02** gains a data point: the confirmed features (notify,
locks, queues, outbox, scheduler) are the same feature family on
both engines, which argues for the single-crate-with-feature-gated-
engines shape rather than a reactive-core + per-engine crates split
(the split buys something only if the engines' feature sets diverge
sharply, and this inventory says they don't).
- **OQ-ST-07** sharpens: no consumer needs the *loadable extension*
surface (every consumer is in-process Rust attaching to its own
connection, the `honker-core`/`attach_honker_functions` shape per
the POC summary); the embedded-rusqlite-only shape is the whole
consumer story so far. The loadable-extension option is now a
cut-only decision, not an open design fork.
- **OQ-ST-04's** contract-pinning work keeps all of the confirmed
features as contract candidates; the ambiguity found in the
inventory (queues vs scheduler boundary; streams' absent consumer;
rate limits' alternative mechanism) is input to which parts of the
honker-rs surface become contract versus cut.
## The consumer list, honestly bounded
- The inventory covers artifacts existing as of 2026-10-04: alkfs
phase-0, alkgit architecture (reviewed), alkblobs architecture
(draft-but-ADR'd), alknet-filesystem POC summary. If a new consumer
is added to the family (alksftp, the alknet rewrite, the ops
platform), a row gets added here *before* it is assumed into a scope
decision — the same discipline as alkblobs' requirements.md ("when a
consumer fact changes, it changes here first").
- Absence of evidence is marked, not overruled: streams/rate-limits/
result storage stay on the list with their flags visible, per the
working posture, rather than being cut by this document alone.
docs/research/phase-0.md
+79 -30
View File
@@ -1,8 +1,9 @@
---
status: draft
last_updated: 2026-10-03 (initial draft + interface finding: honker-rs surface
read as the unified-API candidate; pgboss-rs LISTEN/NOTIFY-absence folded into
the driver/queue/ownership questions)
last_updated: 2026-10-04 (consumer inventory landed — OQ-ST-01 answered
per-feature from the paused consumers' documents; OQ-ST-02/07
sharpened; phase-0 plan step 1 done. Interface finding and driver
tension from 2026-10-03 remain trusted-but-unverified working input.)
---
# alkstore — Phase 0 (Exploration)
@@ -11,8 +12,12 @@ This document captures Phase 0 (Exploration) for the `alkstore` crate:
vision, guiding principles, prior art, and the open-question register
(OQ-ST-01..NN). Phase 0's objective per `docs/sdd_process.md`: *capture
vision and guiding principles; research options; validate approaches;
converge on a recommended approach.* This is the initial draft — none of
the questions below are settled, and there is no POC register yet.
converge on a recommended approach.* The scope question (OQ-ST-01) has
been answered per-feature from the consumers' documents
(`consumer-inventory.md`); the driver and reactive-shape questions
(OQ-ST-03/04) are the open research; there is no POC register yet
(see `consumer-inventory.md` for the first cut candidates that keep its
register small).
Context for why this crate starts now: **alkblobs**
(`/workspace/@alkdev/alkblobs` — spec + POCs only, paused mid-planning)
@@ -240,8 +245,8 @@ this crate:
finding. Notable honest limitations documented by its own guides —
the per-binding processing-guarantees table (auto-checkpoint cadence
vs manual offset save; several bindings "may persist an offset on a
cadence... without knowing whether downstream application work
committed"), the Node reverse-order consumer-checkpoint bug, per-
cadence... without knowing whether downstream application work
committed"), the Node reverse-order consumer-checkpoint bug, per-
binding feature gaps (JVM missing cancel/get_job, Go/Bun/C++
missing typed pruning) — are exactly the seams a single-crate version
designed-for-the-contract from day one can clean up. Its sync-only
@@ -303,26 +308,44 @@ only). Promotion target: Phase 1 `docs/architecture/open-questions.md`.
### OQ-ST-01: Scope boundary — which honker features are in-scope?
Honker's feature list (queues, streams, notify, scheduling, locks, rate
limits, outbox helpers) is large; per-feature scope decisions don't
exist yet. Also undetermined: the exclusion lines (honker deliberately
excludes DAGs, task chains/chords, multi-writer replication,
distributed locking).
**Answered by the consumer inventory (2026-10-04) —
`docs/research/consumer-inventory.md`; scope votes now shrink to named
rows, not the whole feature list.** The original framing ("blocked on a
consumer-driven inventory pass") was circular hedging: the consumers
are paused, so the input would never arrive — but their *documents*
are stable evidence, and the inventory walks them per feature.
In-short: notify/listen and named locks are pinned or ADR-shaped
(alkfs path-tree invalidation; alkblobs fleet sweeper lock; alkfs
OQ-FS-05 writer coordination); queues and the outbox helper are
documented needs (alkfs sync/fetch-on-miss outbox; alkblobs
embedder-owned maintenance cadence); the scheduler is documented-thin
(the family-wide "who sweeps/renews/reaps" problem, possibly collapsing
into queues); streams, rate limits, and result storage have **no
named consumer** — carried per the keep-until-implementation posture
with cut-flags visible, cut later rather than silently included.
Not yet decidable without working through the concrete consumers (the
paused alkblobs crates and alkfs planning) — per-feature need hasn't
been articulated. Blocked on a consumer-driven inventory pass.
Per-feature exclusion lines (honker's own: DAGs, task chains/chords,
multi-writer replication, distributed locking) have no consumer either;
they stay out unless a consumer document grows one. New consumers
(alksftp, the alknet rewrite) add a row to the inventory *before* being
assumed into scope.
### OQ-ST-02: Crate scope — one store crate, or reactive-core + engines?
Options include: single crate with feature-gated engines (the alk*
feature-gate pattern); a core trait crate + per-engine crates; engine
crates consuming a thin core. The answer constrains the driver decision
(OQ-ST-04) and the base-crate-lean invariant.
(OQ-ST-03) and the base-crate-lean invariant.
The use case isn't concrete yet (no engine code written, no consumer
wired). Deferred(scope) — concrete use case: the first engine
implementation would force this shape.
Input from the inventory (2026-10-04): the confirmed features
(notify/locks/queues/outbox/scheduler) are the same feature family on
both engines — the feature sets do not diverge sharply, which argues
for the single-crate-with-feature-gated-engines shape; a reactive-core
+ per-engine-crate split buys something only when engines' surfaces
diverge. Not yet decidable without the first engine implementation
forcing the shape; the inventory removed the "no concrete use case"
half of the deferral, the remaining half is real. Deferred(scope),
narrowed.
### OQ-ST-03: Driver story — sqlx, tokio-postgres, or per-engine drivers?
@@ -364,9 +387,12 @@ point — the question decomposes into contract-pinning rather than
shape-invention:
- Which parts of the honker-rs surface become the crate's *contract*:
the `notify`/`listen` pair, the `stream`/offset/consumer model, the
queue claim/ack/visibility model, scheduler, locks/rate-limits,
outbox — all, a subset, or renamed/regrouped?
the `notify`/`listen` pair, the queue claim/ack/visibility model,
locks, outbox, scheduler — all now have named consumers (inventory);
the stream/offset/consumer model does not (streams is keep-with-flag)
and rate-limits have an in-crate alternative mechanism (alkgit's wire
layer) — subset, renamed/regrouped, decided against the inventory
rows rather than against the whole honker menu.
- What is the delivery-guarantee contract, per mechanism (honker's
own guide table shows how easily per-binding auto-checkpoint vs
manual-save ambiguity produces *different* guarantees under one
@@ -404,7 +430,8 @@ our tolerance for the alpha-state rc port, and on the verified gap
choice, so the queue-machinery reuse value is the honest comparison
point, not the whole.
Open; inputs: the OQ-ST-01 inventory + OQ-ST-03 resolution.
Open; inputs: OQ-ST-01's scope vote on queues (inventory: documented
need) + OQ-ST-03 resolution.
### OQ-ST-06: Honker relationship — reference, fork, or vendor?
@@ -440,7 +467,13 @@ in-process may be the whole story (the honker-core shape minus the
extension/binding packaging). Determines how much of honker is even
candidate material.
Open; rides the first consumer-driven scope pass (OQ-ST-01).
**Sharpened by the inventory (2026-10-04):** every identified consumer
is in-process Rust attaching to its own connection (the
`honker-core`/`attach_honker_functions` shape — the alknet-filesystem
POC's actual usage). No consumer needs the loadable-extension surface.
The question is now cut-only: loadable extension is out unless a
consumer appears; the open residue is just how much of honker-core's
machinery survives the extraction.
### OQ-ST-08: Multi-host / deployment posture
@@ -455,18 +488,21 @@ capability differences can surface).
## Phase 0 plan
Iteration expected; this register grows as the consumer inventory and
research rounds land. Expected sequence (deliberately rough):
Iteration expected; this register grows as research rounds land.
Expected sequence (deliberately rough):
1. Consumer-driven scope inventory (OQ-ST-01) — what the paused
alkblobs consumers and alkfs planning actually need from the reactive
store; walks back the per-feature scope decisions.
1. ~~Consumer-driven scope inventory (OQ-ST-01)~~ — **done
(2026-10-04)**: `consumer-inventory.md`, run against the paused
consumers' documents (alkfs phase-0, alkgit architecture, alkblobs
architecture + the alknet-filesystem POC). OQ-ST-01 answered down
to named per-feature rows; OQ-ST-02/07 sharpened by it.
2. Research rounds on OQ-ST-03/04 (drivers + reactive shape) — library
capability matrices, then a POC if the unified-trait shape needs
validation (likely, given the structural mismatch noted in OQ-ST-04).
The honker-rs surface (§Interface finding) is the concrete starting
artifact for the OQ-ST-04 work: pinning its contract costs less and
is more honest than inventing a parallel shape.
is more honest than inventing a parallel shape — now scoped against
the inventory's confirmed features rather than the full honker menu.
3. Ownership decisions (OQ-ST-05/06) — adopt/fork/derive per subsystem,
after the driver and shape questions narrow the option space.
4. Converge; Phase 1 opens with the ADR backlog this register becomes.
@@ -489,5 +525,18 @@ research rounds land. Expected sequence (deliberately rough):
paused planning this crate unblocks; its POC findings
(`poc-postgres-kv-findings.md`, `poc-pglo-findings.md`) are the
tokio-postgres evidence base.
- alkgit — `/workspace/@alkdev/alkgit` (paused mid-Phase-1,
architecture reviewed): its backend.md trait seam and ADR set are
consumer evidence for the inventory (queues/locks rows).
- alkfs — `/workspace/@alkdev/alkfs` (Phase 0 drafted, 2026-09-23): its
phase-0 OQs (OQ-FS-05/07/14/16/17) are consumer evidence for the
inventory (notify/locks/queues rows).
- alknet-filesystem POC —
`/workspace/@alkdev/alknet/docs/research/alknet-filesystem/
poc-summary.md`: the ran-once evidence that the honker-coordination
layer works (notify-on-commit test; named-locks and outbox usage
identified).
- consumer-inventory.md (`docs/research/consumer-inventory.md`) — the
per-feature synthesis (2026-10-04) answering OQ-ST-01 from the above.
- alkcall — `/workspace/@alkdev/alkcall`: the family substrate;
referenced for the store-layer-isolation principle only.