docs(architecture): ADR-009 — pg-lo admitted as the fs tier's second engine on POC #7

- ADR-009 (new): postgres Large Objects ship as the fs tier's second
  engine (feature pg-lo, default-off), admitted on POC #7's passed
  gate (contract 10/10, durable put ~= fs at >=1 MiB, lo_get window
  gets, transactional LO lifecycle = zero crash orphans). Engine
  shape: companion table (lo_entries) is the contract authority; put =
  one tx (lo_create + lowrite + row + pin); get = descriptorless
  lo_get windows (lo_*64 discipline); named operator deltas (catalog
  space never returned / autovacuum inherited / sweep for the legacy
  bypass class only) and named performance deltas (cached-get 20-50x
  behind page-cache fs single-stream, ~700 MB/s aggregate at 16
  readers = the fleet serving picture).
- Ripple: backends-and-dispatch (pg-lo shipped; fleet locality
  contract gains the whole-fleet-pg-lo option), overview (layer map,
  feature table + pg-lo row, OQ -> shipped pointer), requirements
  (engine def + REQ-2 note updated), README (current-state + ADR
  table), store-api (invariant 1/6 engine lists), ADR-008 Status
  amendment (pg-lo posture superseded by ADR-009), open-questions
  (audit trail completed: decided -> sequenced -> completed).

Verification: docs-only; all cross-references resolve (ADR/REQ/POC);
register, findings, spec, and both engine-ADR status lines agree.

docs/architecture/README.md

@@ -1,6 +1,6 @@
 ---
 status: draft
-last_updated: 2026-10-03
+last_updated: 2026-10-03 (ADR-009 — pg-lo admitted on POC #7)
 ---
 
 # alkblobs — Architecture
@@ -47,6 +47,18 @@ there — not inherited as hedges:
   admission-gated on POC #7). It also adds
   [requirements.md](requirements.md) — the REQ anchor that ends
   per-session re-derivation of consumer facts.
+- **ADR-009** (2026-10-03) admits `pg-lo` — postgres Large Objects as
+  the fs tier's second engine (feature `pg-lo`, default-off) — on
+  POC #7's passed gate (`docs/research/poc-pglo-findings.md`):
+  contract 10/10 exact-count tests, durable put ≈ fs durable put at
+  ≥1 MiB, descriptorless `lo_get` window gets, transactional LO
+  lifecycle (zero crash orphans), companion-table contract authority.
+  REQ-2's fleet consolidation option is now shipped: a fleet node can
+  run postgres-only. Named deltas: cached-get 20–50× behind
+  page-cache fs single-stream (aggregate serving ~700 MB/s at 16
+  readers), catalog space reused-but-never-returned, autovacuum
+  inherited. The admission door (ADR-003, exercised by ADR-007 and
+  ADR-009) has now shipped both its tier-extending engines.
 
 ## Architecture Documents
 
@@ -73,6 +85,7 @@ there — not inherited as hedges:
 | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Whole-blob verification; transfer encoding excluded by scoping | Accepted |
 | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines (sqlite + postgres) behind one trait; constructor-selected | Accepted |
 | [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | Trait `size` probe; fleet GC (DB-backed pins, advisory-locked sweeper); fs-tier engines | Accepted |
+| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo — postgres Large Objects as the fs tier's second engine (admitted on POC #7) | Accepted |
 
 ## Open Questions
 
@@ -84,8 +97,10 @@ alkfs intake; carried for visibility, gating nothing here). OQ-09
 (second kv engine) resolved by ADR-007. The pg-lo fs-engine question
 resolved to *sequenced work*, not a parked question: REQ-2/ADR-008
 named it the candidate fs-tier engine and POC #7
-(`docs/research/poc-pglo-spec.md`) is its admission evidence — a task,
-not a wait. **Deferral
+(`docs/research/poc-pglo-spec.md`) was its admission evidence — a
+task, not a wait; that task completed 2026-10-03
+(`docs/research/poc-pglo-findings.md`, passed) and ADR-009 shipped
+the engine. **Deferral
 policy (the "Schrödinger's code" rule):** a *decision this crate needs
 before shipping* may not be deferred on a dependency that is itself
 waiting for this crate to exist. Two parking kinds remain legitimate:

docs/architecture/backends-and-dispatch.md

@@ -128,7 +128,7 @@ sweep-outcome CI gate, run against dockerized postgres under
 near-linearly to ~37k puts/s at 24 conns — the only engine whose
 throughput *increases* under concurrency.
 
-### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (candidate)
+### fs backend (feature `fs`, default-on); engines: `local` (default) + `pg-lo` (feature `pg-lo`, default-off)
 
 Large blobs. Like the kv tier (ADR-007), the fs tier is **one contract,
 multiple engines** — the medium is a per-node constructor choice
@@ -142,24 +142,31 @@ for fleets):
   rename for the two-pass unknown-length path; pread-based range
   reads (POC #3 finding A2 — local range serving is sound, e.g. for
   packfiles).
-- **`pg-lo` engine** (candidate, *not shipped*; ADR-008): postgres
-  Large Objects as the fs tier's storage — the same engine-behind-
-  one-trait move as ADR-007, one tier over. Admission is gated on
-  POC #7's measured evidence (passed 2026-10-03:
-  `docs/research/poc-pglo-findings.md` — durable put ≈ fs's durable
-  put, `lo_get` window gets, companion-table authority, clean
-  crash-orphan behavior) and its own engine ADR (see
-  `docs/research/poc-pglo-spec.md`).
-- **Fleet locality contract (ADR-008):** over one shared pool, the fs
-  tier is either shared media (every node's `local` root on the same
-  fleet-shared media — soundness caveats and the required media
-  guarantees in ADR-008) or re-routed (one storage node serves pool
-  large-blob content via the ops surface; client nodes are
-  kv-tier-only over pool content). Mixed per-node-local fs tiers over
-  one pool is the partitioning failure this contract exists to
-  prevent — a documented deployment invariant the constructor cannot
-  fully prove but must be declared against (ADR-008 names the
-  enforceable seam and the detection symptom).
+- **`pg-lo` engine** (shipped, feature `pg-lo`, default-off; ADR-009):
+  postgres Large Objects as the fs tier's storage — the same
+  engine-behind-one-trait move as ADR-007, one tier over, admitted on
+  POC #7's measured evidence (contract 10/10 exact-count tests;
+  durable put ≈ fs durable put at ≥1 MiB, beats it <1 MiB on the POC
+  disk; gets ride descriptorless `lo_get(oid, off, len)` windows over
+  the companion table as contract authority; LO creation
+  transactional — zero crash orphans). Named deltas: catalog space
+  reused-but-never-returned (monitor rel size; `VACUUM FULL` is the
+  operator's shrink path), autovacuum inherited, cached-get 20–50×
+  behind page-cache fs single-stream (~700 MB/s aggregate at 16
+  readers — the fleet serving picture).
+- **Fleet locality contract (ADR-008, extended by ADR-009):** over one
+  shared pool, the fs tier is one of: shared media (every node's
+  `local` root on the same fleet-shared media — soundness caveats and
+  the required media guarantees in ADR-008), re-routed (one storage
+  node serves pool large-blob content via the ops surface; client
+  nodes are kv-tier-only over pool content), **or `pg-lo` for the
+  whole fleet** (ADR-009 — one engine, no shared media, no re-routing;
+  pool content lives in the same pg instance the kv tier rides).
+  Mixed per-node-local fs tiers over one pool is the partitioning
+  failure this contract exists to prevent — a documented deployment
+  invariant the constructor cannot fully prove but must be declared
+  against (ADR-008 names the enforceable seam and the detection
+  symptom).
 
 ### mem backend (feature `mem`, default-off)
 

docs/architecture/decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md

@@ -2,7 +2,9 @@
 
 ## Status
 
-Accepted
+Accepted (the §3 pg-lo bullet's "candidate, not shipped" posture was
+superseded 2026-10-03 by ADR-009, which admitted pg-lo on POC #7's
+passed gate; every other part of this ADR stands as written)
 
 ## Context
 

docs/architecture/decisions/009-pg-lo-fs-tier-engine.md

@@ -0,0 +1,156 @@
+# ADR-009: pg-lo — postgres Large Objects as the fs tier's second engine
+
+## Status
+
+Accepted (admission evidence: POC #7, passed 2026-10-03 —
+`docs/research/poc-pglo-findings.md`; the ADR-008-named candidate's
+gate)
+
+## Context
+
+ADR-008 named `pg-lo` as the candidate fs-tier engine for REQ-2's
+fleet topology (multiple store instances sharing one pg pool, serving
+large/beyond-threshold pool content) — expressly gated on the ADR-003
+admission door: measured evidence first, in the ADR-007 shape. POC #7
+ran that door (`poc-pglo-spec.md` → `poc-pglo-findings.md`,
+findings C1–C7): a ~400-line `PgLoBackend` over the ADR-003/008 trait
+contract (including `size`), 10/10 exact-count sweep-outcome contract
+tests, on the POC #5 driver stack (tokio-postgres + deadpool-postgres
+via SQL `lo_*` functions — no new dependency, as ADR-008 §Neutral
+projected), against dockerized postgres:16.
+
+The gate's three legs all passed:
+
+- **Contract** — complete `list()` (= companion table == LO catalog
+  oids, after mixed ops), GC-participating delete, virgin-store
+  no-ops, CAS re-put, stage rollback, ranged reads; proven by the
+  exact-count sweep-outcome shape ADR-007's gate uses.
+- **Performance** — durable puts 60–65 MB/s at ≥1 MiB, within 5–10% of
+  durable local fs at 16–128 MiB and *beating* it below 1 MiB on the
+  POC box (fsync-dominated); cached gets 70–180 MB/s single-stream,
+  ~700 MB/s aggregate at 16 readers — 20–50× behind page-cache fs
+  single-stream, named honestly (see Consequences).
+- **Ops posture** — the tx-scoped-descriptor question resolved in
+  pg's favor (descriptorless `lo_get(oid, off, len)` windows: pool-
+  friendly, stateless, ranged reads never pay the tx cost); LO
+  catalog churn page-granular and reusing; LO creation transactional —
+  crash orphans structurally zero.
+
+## Decision
+
+**The fs tier ships `pg-lo` as its second engine — postgres Large
+Objects — behind the same `Backend` trait, feature `pg-lo` (default-
+off), constructor-selected per node.** REQ-2's deployment choice set
+(ADR-008) extends from {shared media, re-routing} to include the
+consolidation option as a shipped engine.
+
+1. **Engine shape (POC #7 C1 is the contract):** LOs hold the bytes;
+   the companion table `lo_entries (key bytea PK, loid oid, size
+   bigint, committed_at)` is the *contract authority* — `list()`,
+   `size()`, `has()`, CAS-existence are all table-level. Never trust
+   or read `pg_largeobject` for addressing (the C6 sweep validates
+   against the table only).
+2. **Put = one transaction**: BEGIN → `lo_create` → chunked `lowrite`
+   (512 KiB statements — the measured knee; statement count, not
+   LOBLKSIZE, is what costs) → companion-row insert → COMMIT.
+   Pin-before-publish (ADR-005/008) rides this shape: entry row + pin
+   row commit together with the content. Unknown-length puts stage
+   into the LO and commit the row on completion; rollback discards
+   both.
+3. **Get = `lo_get(oid, off, len)` windows by default** (POC #7 C4):
+   the fs-tier handle closes over the companion row (oid + length) —
+   immutable, so the row is authoritative — not over a tx-scoped
+   descriptor. Ranged reads are single stateless statements,
+   pool-friendly (acquire p99 1–6 ms ≤ pool size). The held-descriptor
+   tx posture exists as the fallback (measured: no advantage even for
+   sequential full reads). **Reads ride `lo_*` 64-bit variants
+   end-to-end** (`lo_lseek64`/int8 offsets; `lo_lseek`'s int4 cap is a
+   C7 trap the engine must not inherit).
+4. **Delete = one tx** (`lo_unlink` + row delete) under the ADR-008
+   delete-window arbitration (table-level; nothing fs-specific).
+5. **Posture deltas, as operator requirements (the honest cost of the
+   engine, POC #7 C5/C6):**
+   - **LO catalog space is reused but never returned** — the engine's
+     inverted-fs property. The engine ADR requirement: rel-size
+     monitoring (`pg_total_relation_size('pg_largeobject')`) named in
+     ops docs; `VACUUM FULL`/`pg_repack` is the only shrink path —
+     an operator runbook line, never automatic.
+   - **Autovacuum inherited; keep it on.** `pg_largeobject` is an
+     ordinary catalog heap (autovacuum applies, measured active); a
+     deployment disabling autovacuum gets an operator-visible note.
+   - **Orphan-recovery sweep exists for the legacy/bypass class only**
+     (committed LOs without companion rows — mixed/legacy stores). The
+     crash story is structurally clean: LO lifecycle is transactional
+     in modern postgres; kill mid-tx leaves zero pages, zero
+     half-commits (cleaner than fs `local`'s stage-file residue,
+     which needs its own recovery sweep).
+6. **Performance deltas, as named capacity facts (not hidden, not
+   solved):** single-stream cached get runs 20–50× behind page-cache
+   local fs at ≥1 MiB (server-side LO page-walk + copy ceiling,
+   ~65–70 MB/s per in-flight statement); the fleet serving picture
+   aggregates ~700 MB/s at 16 readers (server-side parallelism) —
+   which is REQ-2's relevant number — and exceeds-threshold single
+   clone streams get ~120 MB/s. Deployments whose large-blob serving
+   is dominated by single-stream page-cache-bound reads stay better
+   served by `local`-on-shared-media; pg-lo's case is the
+   consolidated fleet (one engine to operate) whose serving burden is
+   many-user aggregate. A third engine (S3-like) remains the
+   ADR-003 door; nothing here touches it.
+7. **redb-class guardrail maintained:** this is the second engine the
+   admission door ships (ADR-007: kv; ADR-009: fs) — both on the same
+   measured-evidence shape; no third engine is opened by this one.
+
+## Consequences
+
+**Positive**
+
+- REQ-2's consolidation option is real and shipped: a fleet node runs
+  *postgres only* — kv (bytea) tier + fs (LO) tier from one engine,
+  no shared-media requirement, no re-routing topology, no consumer-ops
+  dependency for pool content.
+- The engine adds ~400 lines of impl over an already-fixed contract —
+  the additive shape both ADR-007 and ADR-008 projected, now
+  evidenced.
+- Crash hygiene is *better* than fs `local` (transactional LO
+  lifecycle vs. stage-file residue); the engine's only recovery sweep
+  targets a class (bypassed-LO) that healthy deployments never produce.
+
+**Negative**
+
+- Single-stream cached get is 20–50× behind local fs — a real,
+  named cost. Deployments with page-cache-friendly media and single-
+  dominant readers should stay on `local` engines. (Aggregate serving
+  — the fleet case — scales to ~700 MB/s on the POC box; per-box
+  numbers shift with cores.)
+- LO catalog space is never returned (monitor + occasional
+  operator-run `VACUUM FULL`), the inverse of fs `local`'s
+  immediate-return.
+- Feature/maintenance surface: the fs tier now carries two engines
+  (like the kv tier), each with posture deltas to keep honest
+  (ADR-007's negative, inherited).
+
+**Neutral**
+
+- The `Backend` trait is unchanged (ADR-008's `size` probe is the
+  last trait change — pg-lo fits the contract as it stands).
+- Driver stack: unchanged (tokio-postgres + deadpool; SQL `lo_*`
+  functions, `postgres_large_object` crate stays avoided).
+- The `pg-lo` feature defaults off; CI's `--all-features` gate adds
+  it to the sweep-outcome suite (dockerized postgres:16, same shape
+  as the `postgres` kv engine's gate).
+
+## References
+
+- `docs/research/poc-pglo-findings.md` (POC #7, findings C1–C7 — the
+  evidence base); `poc-pglo-spec.md` (the gate this ADR's legs
+  mirror)
+- [requirements.md](../requirements.md) — REQ-2 (the fleet topology
+  this engine serves); REQ-1/3/4 unchanged postures
+- ADR-003 (the trait contract + admission door exercised twice now),
+  ADR-004 (tier ≠ engine, fs tier edition), ADR-005/008 (the GC
+  invariants this engine participates in — table-level, nothing
+  fs-specific), ADR-007 (the engine-admission precedent and driver
+  stack)
+- [backends-and-dispatch.md](../backends-and-dispatch.md) (the fs
+  engine surface this amends); [overview.md](../overview.md);
+  [store-api.md](../store-api.md)

docs/architecture/open-questions.md

@@ -253,10 +253,14 @@ multi-instance-with-large-blobs topology a standing requirement
 nothing to defer *on*. It resolved to ADR-008 naming pg-lo as the
 candidate fs-tier engine + POC #7 (`docs/research/poc-pglo-spec.md`)
 as its admission evidence — sequenced work, per the deferral-policy
-header's third category. An earlier draft of that OQ parked on
-"alkgit's architecture process names the topology" — rendered moot the
-same day by the REQ-2 fact; recorded here because the audit trail of
-*why not parked* is part of this document's job.
+header's third category. That work completed the same day: POC #7
+passed (`docs/research/poc-pglo-findings.md`, gate verdict passed
+with named deltas) and ADR-009 shipped the engine. An earlier draft
+of that OQ parked on "alkgit's architecture process names the
+topology" — rendered moot by the REQ-2 fact before it was even
+written; recorded here because the audit trail of *why not parked* is
+part of this document's job, and because it is the worked example of
+the header's third category (decided → sequenced → completed).
 
 OQ-07/OQ-08 are owned by other repos' processes and are not alkblobs
 tracker tasks (their outcome arrives *through* their owners, not

docs/architecture/overview.md

@@ -70,7 +70,8 @@ or three times.
 │     │                                     ADR-007: kv = 2 engines;
 │     ▼                                     ADR-008: fs engines too)
 │   engine per node: sqlite (default) | postgres (feature "postgres")
-│   fs engine per node: local (default) | pg-lo (candidate, POC #7)
+│   fs engine per node: local (default) | pg-lo (feature "pg-lo",
+│   default-off; ADR-009)
 
    transport: none, anywhere. alkgit/alkfs or the embedder dials;
    an alkcall Connection is handed in at the ops boundary.
@@ -82,6 +83,7 @@ or three times.
 |---|---|---|
 | base crate (store core + backends) | `tokio`, `thiserror`, `parking_lot`, plus `rusqlite` behind the `kv` feature and nothing behind `fs`/`mem` | no alkcall, no serialization frameworks |
 | `postgres` feature (ADR-007) | adds `tokio-postgres` + `deadpool-postgres` behind the kv tier's second engine, default-off | depends on `kv`'s contract surface (the `Backend` trait definition is base-crate code); the engine's *impl* is additive. Same `Backend` contract, own sweep-safety CI gate under `--all-features` |
+| `pg-lo` feature (ADR-009) | Large Objects as the fs tier's second engine, default-off | rides the same `tokio-postgres`/`deadpool-postgres` stack as the `postgres` kv engine (SQL `lo_*` functions, no new dependency); same admission/test-gate shape |
 | `ops` feature | adds `alkcall` (feature-gated, default-off) | JSON op specs + binary channels + `AccessControl`; ADR-001 |
 | consumer crates | depend on alkblobs (and on alkcall directly when they speak ops) | alkgit, alkfs |
 
@@ -126,6 +128,7 @@ wasm-clean), never an in-process embedder (ADR-001 §Consequences).
 | [006](decisions/006-verification-posture-and-transfer-encoding.md) | Verification posture | whole-blob verification; chunk trees excluded by scoping |
 | [007](decisions/007-two-kv-engines-sqlite-and-postgres.md) | Two kv engines | sqlite default + postgres behind one trait; one relational engine + fs per node |
 | [008](decisions/008-trait-size-probe-fleet-gc-and-fs-engines.md) | `size` probe + fleet GC + fs engines | trait length probe; fleet pins/sweeps engine-backed; fs tier engine-selectable |
+| [009](decisions/009-pg-lo-fs-tier-engine.md) | pg-lo fs engine | postgres Large Objects as the fs tier's second engine, admitted on POC #7; a fleet node can run postgres-only |
 
 ## Open Questions
 
@@ -140,10 +143,9 @@ crate level:
   open-read requirement)
 - **OQ-10**: second kv engine — resolved by ADR-007 (sqlite + postgres
   shipped behind one trait; per-node constructor choice)
-- **POC #7** (`docs/research/poc-pglo-spec.md`): pg Large Objects as
-  the fs tier's `pg-lo` engine — spec'd **sequenced work** (not a
-  parked question; open-questions.md's policy header distinguishes the
-  two): admission evidence for ADR-008's named candidate (REQ-2)
+- **POC #7** (`docs/research/poc-pglo-findings.md`): pg Large Objects
+  as the fs tier's `pg-lo` engine — **passed** 2026-10-03; admitted as
+  a shipped engine by ADR-009 (REQ-2's consolidation option)
 
 ## References
 

docs/architecture/requirements.md

@@ -1,6 +1,6 @@
 ---
 status: draft
-last_updated: 2026-10-03
+last_updated: 2026-10-03 (pg-lo admitted by ADR-009 on POC #7 — REQ-2's consolidation option is shipped)
 ---
 
 # Requirements
@@ -16,8 +16,7 @@ the ADRs re-derive from it.**
 
 - **Engine** — a concrete storage implementation of one tier's
   contract: the kv tier's engines are sqlite and postgres (ADR-007);
-  the fs tier's engines are `local` (default) and `pg-lo` (candidate —
-  see REQ-2, ADR-008).
+  the fs tier's engines are `local` (default) and `pg-lo` (ADR-009).
 - **Pool** — the flat content-addressed namespace of one deployment
   (`hash → bytes` everywhere; ADR-005).
 - **Node** — one running store instance (one process embedder; its fs
@@ -62,7 +61,8 @@ replicators; ACL/ownership above (alkcall layer).
   Consequences are derived in ADR-008: fleet GC (DB-backed pins,
   advisory-locked sweeper, SQL delete arbitration), the fs-tier
   engine concept with its locality contract, and the pg-lo candidate
-  engine (POC #7).
+  engine (POC #7 — **passed** 2026-10-03, admitted as a shipped
+  engine by ADR-009: a fleet node can consolidate on postgres-only).
 - **REQ-3 (traces: ADR-007, POC #5 B3/B4)** — concurrency scale-out:
   the fleet's hub-class nodes need put throughput that *increases*
   under concurrency (pg: near-linear to ~37k puts/s at 24 conns);

docs/architecture/store-api.md

@@ -129,7 +129,8 @@ lesson generalizes to any kv engine).
 
 1. Whole-blob put/get round-trips byte-identical under the canonical
    derivation, both tiers, both put paths, **and every shipped engine
-   of both tiers** (kv: sqlite + postgres; fs: `local`; ADR-007/008)
+   of both tiers** (kv: sqlite + postgres; fs: `local` + pg-lo;
+   ADR-007/008/009)
    (interop with real git remains the source of truth, per POC #1's
    lesson about hardcoded vectors).
 2. Dedup: putting identical content twice (same or different path) is
@@ -147,8 +148,8 @@ lesson generalizes to any kv engine).
    tests assert through sweep outcomes (POC #1 finding 2's lesson
    codified).
 6. `size`/`stat` on any engine returns the entry length without
-   fetching content (asserted per-backend: kv probe, fs stat —
-   ADR-008).
+   fetching content (asserted per-backend: kv probe, fs stat, pg-lo
+   companion-row probe — ADR-008/009).
 
 ## Design Decisions