alkstore/tasks/sqlite-fix-watcher-death-subscribe.md

---
id: sqlite-fix-watcher-death-subscribe
name: Dead-watcher subscribe hang + initial-stat self-kill (review 004 items 2+3)
status: completed
depends_on: []
scope: moderate
risk: medium
impact: component
level: implementation
tags: [review-004-fixes, sqlite-engine, watcher]
---

## Description

Two bundled watcher-state defects in the SQLite engine, per
`docs/reviews/004-pre-release-general-review.md` §"Findings recorded,
not fixed" items 2 and 3 (the review says bundle them — both are
watcher-state changes).

**(a) Post-watcher-death subscribe hang (item 2).** After the watcher
thread exits (db-file replacement), `subscribe()` still enrolls new
senders with no poller alive (`substrate/watcher.rs:391-396`): a
`listen()` or stream `subscribe()` on the dead store parks forever,
contradicting engine-sqlite.md's "on watcher death every subscriber
closes" posture — which is enforced only for *pre-death* subscribers.
Fix shape: a dead-store/dead-watcher sign that the new subscription
paths consult — a state-machine change across `store.rs`/`notify.rs`/
`stream.rs`. New subscriptions on a dead store must fail closed (a
typed error, the established failure vocabulary) rather than park.

**(b) Initial-stat failure self-kill (item 3).** An identity-`stat`
failure at watcher start seeds `(0, 0)` (`watcher.rs:164-170`); the
first *successful* later stat mismatches it and drives the death path,
spuriously closing every subscriber on recovery from a transient stat
hiccup. Rare, but the fix — an identity-unknown state instead of a
false identity — is the same watcher-state family: the watcher treats
identity as unknown until the first successful stat, and that first
successful stat must not trigger the death path.

**Test shapes**: (a) kill the watcher (the db-file replacement seam or
a `cfg(test)` death injection), then `listen()`/`subscribe()` — assert
the typed failure, not a park; the pre-death subscribers' close-at-
death pins stay green. (b) inject an initial stat failure, let the
first successful stat land, assert the watcher survives and
subscribers stay open.

## Acceptance Criteria

- [x] Subscribing (listen or stream) after watcher death fails closed
      with a typed error instead of parking — pinned by test
- [x] Pre-death subscribers still close at watcher death — the
      existing pins green unchanged
- [x] An initial identity-stat failure no longer seeds a false
      identity: the first successful stat does not trigger the death
      path and subscribers stay open — pinned by injected-failure test
- [x] engine-sqlite.md's watcher-death posture text covers the
      new-subscriber arm (and the identity-unknown state if it changes
      observable behavior)
- [x] `cargo test -p alkstore-sqlite`, clippy `-D warnings`, fmt
      clean; workspace gates green server-less

## References

- docs/reviews/004-pre-release-general-review.md §"Findings recorded,
  not fixed" items 2–3 (the findings, the fix shapes, the bundling)
- docs/architecture/engine-sqlite.md (the watcher-death posture)
- alkstore-sqlite/src/substrate/watcher.rs (both sites),
  store.rs / notify.rs / stream.rs (the subscription paths)

## Notes

Decisions of record the description didn't pin:

- The dead sign lives on `SharedUpdateWatcher` as an
  `Arc<AtomicBool>`, set by `run_poll_loop` itself just before the
  watcher-fatal return — it is set *before* the death-guard's senders
  clear (which happens after the loop returns), so the guard-side
  ordering is fixed by program order.
- `SharedUpdateWatcher::subscribe` became fallible
  (`Result<_, String>`) and checks **two** closed arms atomically with
  the enrollment under the senders lock: the dead sign *and* the
  taken watcher slot (the post-`close` enroll-onto-empty-watcher race
  of the same shape). Failing closed on the taken slot is new
  belt-and-braces beyond the finding's shape; the store's `closed`
  flag already guarded the normal path. Rationale for checking under
  the lock: a check-then-enroll done outside the lock could interleave
  between the death and the guard's clear and enroll a ghost.
- The substrate stays contract-blind: the failure is a `String`
  naming the death, mapped into the taxonomy engine-side —
  `Error::Database` (the opaque fallback with the source chain), the
  same shape the engine's closed-store checks use. `Error::Closed`
  was rejected: the contract docs scope that variant to *receiver*
  closed arms (`try_recv`/`recv_timeout`), not entry points.
- The initial-identity fix is an in-place re-derivation of the same
  code (`Option<(u64, u64)>` baseline vs. the `(0, 0)` seed); the
  death-log message keeps its precise diagnostic (now printing the
  captured baseline).
- The injected-failure test for (b) is the substrate-level drive
  (`run_poll_loop` with the db file absent at start — the initial
  stat genuinely fails; on non-unix `stat_identity` never fails, so
  the scenario cannot arise and the test is unix-gated like the other
  file-replacement drives); no `cfg(test)` injection seam was added to
  `stat_identity` itself. Aliveness after the first successful stat is
  proven by wake delivery relative to a pre-insert snapshot (failing
  identity stats fire `on_change`, so absolute wake counts are not a
  clean liveness signal).
- Also updated: engine-sqlite.md's watcher-failure posture bullet
  (both arms), notify.rs/stream.rs/store.rs module docs, open_tests'
  three direct `watcher.subscribe()` call sites (now `.unwrap()`d),
  and one PROVENANCE.md row (D-39 — the register wants divergence to
  the ported watcher recorded even though the task text didn't name it).

## Summary

Both defects landed with tests:

- (a) Post-watcher-death subscribe hang: the poll loop sets the dead
  sign before exiting through the identity-replacement death path;
  `SharedUpdateWatcher::subscribe` fails closed on it (and on a taken
  watcher slot) atomically with enrollment under the senders lock;
  `notify.rs::listen` and `stream.rs::spawn_bridge` map the failure to
  `Error::Database` with the death named in the source chain. Pinned
  by four tests: substrate-level
  (`subscribe_fails_closed_after_watcher_death`, unix file-replacement
  drive) and engine-level (`listen_after_watcher_death_fails_closed`,
  `subscribe_after_watcher_death_fails_closed` — same drive through
  the store surface, error-shape + no-enrollment assertions). The
  pre-death subscriber close-at-death pins
  (`watcher_death_closes_the_receiver_terminally`,
  `watcher_death_closes_the_subscription_terminally`, and the
  substrate death-signal tests) are green unchanged — the death-signal
  substrate test also now pins the dead sign being set.
- (b) Initial-stat self-kill: the loop tracks the identity baseline as
  `Option<(u64, u64)>`; a failed startup stat leaves it unknown, the
  first successful stat captures it without driving death, and a later
  genuine mismatch still kills. Pinned by
  `initial_stat_failure_does_not_seed_a_false_identity`
  (`run_poll_loop` drive: absent file → unknown baseline → file
  created → no death + wake delivery lands → replacement → death
  fires; unix-gated).
- Verified: `cargo build`, `cargo test` (workspace, all green),
  `cargo test -p alkstore-sqlite` (201 tests), fuzz gate
  (`cargo test --manifest-path fuzz/shared/Cargo.toml`), clippy
  `-D warnings` clean, `cargo fmt --check` clean, rustdoc
  `-D warnings` doc gate clean.