glm-5.3-flash
79a135c934
ADR-009: scheduler collapses into queues — schedule()/unschedule()/ run_schedules (opt-in, no ambient timers), @every-only v1 grammar (dissolves honker's local-TZ cron brittleness), boundary guarantee row (at-least-once per boundary, fixed 64-cap catch-up with skip-forward, row-locked fire tx as engine-generic no-double-fire floor), __alkstore_scheduler leadership lock, InvalidSpec + LeadershipLost taxonomy additions. ADR-010: queue depth pinned engine-uniformly — three-state machine (pending/processing/dead) with delete-on-ack, get_job sees dead rows, heartbeat = renewal with late-heartbeat refusal, reclaim-consumes- an-attempt stated as contract text, equal-jitter exponential backoff (range definitionally pinned, 1 h cap), QueueOpts stamped onto job rows at enqueue (no per-queue registry), move-to-dead dead-letter with retention-sweep support and no redrive API, sweep_expired carries the no-stranded-rows property (fixes honker's expired- processing zombie hole — SQLite-side realization rides OQ-06 as a concrete fork candidate), one engine-owned pg schema, queues are rows not tables, result-storage cut-flag stands. Also: full honker-machinery and pgboss-rs reference reads persisted (docs/research/reference-*.md — the honker defect list pre-stages the OQ-06 quality read), queues.md rewritten from design-space frame to resolved-depth spec, core-contract/engines/README/overview/deployment/ ADR-002 propagated.
124 lines
7.1 KiB
Markdown
124 lines
7.1 KiB
Markdown
---
|
||
status: draft
|
||
last_updated: 2026-10-05
|
||
---
|
||
|
||
# Postgres engine
|
||
|
||
The `alkstore-postgres` engine implements
|
||
[core-contract.md](core-contract.md) on tokio-postgres +
|
||
deadpool-postgres. This spec records WHAT the engine is internally —
|
||
connection architecture, the LISTEN forwarder, queue machinery
|
||
re-derivation — not code-level HOW. Decisions live in ADRs; contract
|
||
obligations live in the core spec.
|
||
|
||
## Identity and posture
|
||
|
||
- Single driver, natively async: tokio-postgres 0.7.x +
|
||
deadpool-postgres 0.14.x ([ADR-004](decisions/004-postgres-driver.md)).
|
||
No bridge, no `spawn_blocking`; the tx handle holds the pooled
|
||
object directly (client is `Send + Sync` — POC #2 compile-probe
|
||
verified).
|
||
- Multi-host by nature: connections are per-process state; nothing
|
||
assumes a shared host ([ADR-004](decisions/004-postgres-driver.md),
|
||
verified all-through-network in POC #2). See
|
||
[deployment.md](deployment.md).
|
||
- Queue machinery re-derived on this driver with the pg-boss schema
|
||
family as design reference
|
||
([ADR-005](decisions/005-dependency-ownership.md)); semantics depth
|
||
pinned by [ADR-010](decisions/010-queue-semantics-depth.md) — all
|
||
engine-owned tables (job, dead, stream, offsets, schedule) in one
|
||
PostgreSQL schema (default `alkstore`), queues as rows, no
|
||
per-queue tables.
|
||
|
||
## Connection architecture
|
||
|
||
- **Pool** (deadpool) — queries, claims, and all non-transactional
|
||
work. Per-connection statement cache, `RecyclingMethod::Fast`
|
||
(no `DISCARD ALL` recycling; claim SQL re-prepared implicitly with
|
||
zero errors at POC scale).
|
||
- **Listener connection** — one dedicated, **non-pooled** connection
|
||
per process that listens. Pooled connections cannot carry LISTEN
|
||
(deadpool#360 — registration succeeds, delivery is impossible; the
|
||
client-wrapper exposes no notification surface, source-verified and
|
||
test-pinned as `pooled_listen_registers_but_cannot_deliver`). The
|
||
listener is therefore a per-process budget line *outside* the pool:
|
||
`max_size + 1` per LISTEN-ing process
|
||
([deployment.md](deployment.md)).
|
||
- **Forwarder** — the listener's loop: `poll_message`
|
||
fanning out into a bounded broadcast channel (lag surfaced, not
|
||
silent), re-LISTEN from the channel list after every reconnect
|
||
(exponential backoff 50 ms → 2 s cap), and the synthetic
|
||
reconnect-wake on the reserved channel
|
||
(`__alkstore_listener_reconnected__`,
|
||
[ADR-008](decisions/008-contract-v1-pinning.md) §4) — broadcast to
|
||
*every* subscriber's receiver, per the wake contract's
|
||
reconnect-recovery semantics
|
||
([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||
- One listener serves N channels and N subscribers; re-attach is a
|
||
broadcast re-subscribe (no server round-trips); per-channel
|
||
connections are never warranted at this scale (POC-verified).
|
||
|
||
## Mapping the contract
|
||
|
||
| Contract piece | Engine realization |
|
||
|---|---|
|
||
| notify / listen | `pg_notify(...)` inside the caller's tx (delivers at commit — native commit-atomicity, [ADR-007](decisions/007-transactional-seam.md)); `listen()` via LISTEN on the forwarder's connection, fanout to receivers |
|
||
| streams | durable event table + per-consumer offset cursors; `pg_notify` as the wake trigger ([ADR-006](decisions/006-wake-and-delivery-contract.md) mechanism split: durable row, LISTEN wake — the pg-boss-family shape) |
|
||
| queues | re-derived queue table + `FOR UPDATE SKIP LOCKED` claim + LISTEN-driven wake with re-poll safety net (default consumption posture, measured 5–16× vs 50 ms poll; poll-only fallback); states/dead-letter/backoff/visibility per [ADR-010](decisions/010-queue-semantics-depth.md), all in the engine-owned schema |
|
||
| named locks | advisory-lock-semantics TTL locks (pg-boss-family design reference; guarantee row pinned by [ADR-008](decisions/008-contract-v1-pinning.md) §7) |
|
||
| scheduler / outbox | collapse shape ([ADR-009](decisions/009-scheduler-collapse.md)): schedule rows in the engine-owned schema, tick re-derived (boundary advance + 64-boundary catch-up cap, honker parity), leadership via the engine's lock machinery on `__alkstore_scheduler`; outbox = helper over queues |
|
||
| begin_tx | pool checkout + `BEGIN`, returning the caller-held handle ([ADR-007](decisions/007-transactional-seam.md)) |
|
||
| handle ops | straight `.await`s through the held object; commit/rollback returns the object to the pool |
|
||
|
||
## Owned failure modes (all test-pinned in POC #2)
|
||
|
||
Hand-rolling the forwarder means owning its pitfalls — they are
|
||
*learned territory*, pinned as passing tests:
|
||
|
||
1. **Query-vs-poll starvation deadlock** — the poll loop must be
|
||
running before the first client query on the listener connection.
|
||
2. **Client-drop closes the server session** — a long-lived listener
|
||
keeps its `Client` alive for the listener's lifetime.
|
||
3. **The no-replay hole** — commit during a connection gap is never
|
||
re-delivered; recovery = reconnect + synthetic wake + consumer
|
||
re-read ([ADR-006](decisions/006-wake-and-delivery-contract.md)).
|
||
The `!saw_replay` test pins the honesty.
|
||
4. **Payload boundary** — `pg_notify` ≤ 8000 bytes; client-side
|
||
checked, typed error before the round-trip. Large payloads ride a
|
||
table row with the id in the notification (outbox shape).
|
||
5. **Read-your-writes** — `read committed` default verified in both
|
||
directions (in-tx and post-commit).
|
||
|
||
Constraint also carried: mixed rusqlite+sqlx binaries would need a
|
||
vendored patch today — excluded by construction in this engine
|
||
(single driver, [ADR-001](decisions/001-crate-split.md)), noted for
|
||
the record in [ADR-003](decisions/003-sqlite-driver.md).
|
||
|
||
## Design Decisions
|
||
|
||
| ADR | Decision | Summary |
|
||
|---|---|---|
|
||
| [001](decisions/001-crate-split.md) | Crate split | single-driver engine crate |
|
||
| [002](decisions/002-feature-scope.md) | Feature scope | which rows this engine serves |
|
||
| [004](decisions/004-postgres-driver.md) | Driver | tokio-postgres + deadpool; hand-rolled forwarder; re-derived queues |
|
||
| [005](decisions/005-dependency-ownership.md) | Ownership | published libs as-is; `postgres-notify` derive-not-adopt |
|
||
| [006](decisions/006-wake-and-delivery-contract.md) | Wake contract | LISTEN push, no replay, synthetic reconnect-wake |
|
||
| [007](decisions/007-transactional-seam.md) | Tx seam | direct pooled-object handle, no bridging |
|
||
| [008](decisions/008-contract-v1-pinning.md) | Contract v1 | pinned surface; reserved reconnect-wake channel string; `PayloadTooLarge` taxonomy variant |
|
||
| [009](decisions/009-scheduler-collapse.md) | Scheduler collapse | schedule rows in the engine schema, re-derived tick, row-locked fire tx, `__alkstore_scheduler` leadership |
|
||
| [010](decisions/010-queue-semantics-depth.md) | Queue depth | job-stamped opts, equal-jitter backoff, dead-letter move, no-stranded-rows sweep, one engine-owned schema |
|
||
|
||
## Open Questions
|
||
|
||
Open questions are tracked in
|
||
[open-questions.md](open-questions.md). Key
|
||
questions affecting this document:
|
||
|
||
- **OQ-08**: capability surface (shared with
|
||
[deployment.md](deployment.md)) (open)
|
||
|
||
Resolved: **OQ-09** (scheduler collapse —
|
||
[ADR-009](decisions/009-scheduler-collapse.md)) and **OQ-05** (queue
|
||
semantics depth — [ADR-010](decisions/010-queue-semantics-depth.md)),
|
||
2026-10-05. |