alkblobs/docs/architecture/decisions/011-vocabulary-tiers-engines-and-fleet-state.md

# ADR-011: Vocabulary — tiers, engines, store instances, fleets; mem joins the kv tier; the fs tier is renamed `large`

## Status

Accepted

## Context

The engine story (ADR-007/008/009) grew the physical layer faster than
its vocabulary grew, and the pre-decomposition review found the drift
is now self-contradictory in the anchor document itself:

1. **Node vs store instance.** requirements.md defines **Node** as
   "one running store instance" — then REQ-2 states "a replicator node
   may run multiple store instances (a fleet)" and **Fleet** as "two
   or more nodes over one shared pool". Under the written definition,
   one process hosting two instances is simultaneously one node (by
   definition) and two nodes (by REQ-2), and the GC arbitration
   domain ("node is the GC arbitration domain") is undefined for the
   exact topology ADR-008 exists for. ADR-010 fixes pin ownership;
   the pin's *owner id* and the node/instance distinction it needs
   are still unpinned vocabulary.
2. **The "fs" tier name is now wrong.** The tier was named when its
   only engine was a filesystem (ADR-003's `local` layout). After
   ADR-009, half its engines are postgres Large Objects — the name
   describes one engine's medium, not the tier. Worse, "fs" collides
   with the dispatch language: "large content goes to fs" never said
   *why* — the tier's actual routing criterion is content length
   (ADR-003's size-threshold dispatch). The docs already speak the
   true name informally: store-api.md says "small tier"/"large tier".
3. **mem's classification drifts.** backends-and-dispatch.md calls it
   a "backend" in one place and a "testing/utility tier" in another;
   ADR-003 lists it under "shipped backends … plus a testing tier";
   ADR-004 calls it an ephemeral tier and carves it out of the
   two-backend count. In truth it is neither a tier nor a separate
   backend: it implements exactly the kv tier's contract (including
   `size`, trivially), it is the contract-reference engine every
   other engine's conformance tests mirror, and "engine" names what
   it is (ADR-007 established that an engine is a concrete
   implementation of one tier's contract — requirements.md already
   defines the word this way).
4. **Tier and Backend were never formally defined** even though
   ADR-004's scope note ("the 'two' here counts tiers, not engines")
   leans on both words. Also undecided: whether the feature names
   follow the tier rename now or after code exists — the last free
   moment is now (zero code; after implementation the rename breaks
   crates.io/feature ergonomics for every consumer).

Deciding facts are in hand: REQ-2's fleet fact, the engine set's
final shape (ADR-007/009), the dispatch mechanism (ADR-003), and
ADR-010's GC-state-home table. This is a vocabulary-and-boundary
decision, not an architecture change: ADR-004's tier≠engine note
already established the shape; this ADR gives it final, consistent
names.

Candidates weighed for the tier rename:

- **`lo`/`los`** — names the *postgres* engine (Large Objects), so
  "the lo tier's `local` engine" would be backwards. Rejected.
- **`lfs`** — workable, echoes git-lfs specifically; but the tier
  also serves packfiles and any oversized workspace artifact, and the
  echo binds the crate's naming to one consumer's protocol shape.
  Rejected.
- **`large`** — the dispatch-true name (size-threshold routing), the
  word the specs already use informally ("large tier"), medium-agnostic.
  **Chosen.** For symmetry in prose, the kv tier stays `kv` (its
  routing criterion — "small" — lives in the prose as "small blobs"
  and needs no rename: `kv` names the mechanism class, and `small`
  as a feature/tier name would read as an adjective; the pair's
  asymmetry is cosmetic, the *load-bearing* correction is that
  neither tier is named after a medium).

Feature-name mirroring: `fs` → `large`; `pg-lo` keeps its name (it
names the engine's mechanism, accurate); `postgres` keeps its name
(it names the kv engine); `mem` keeps its name; `kv` and `ops` keep
their names.

## Decision

### 1. Vocabulary (the pinned definitions; requirements.md is the canonical copy)

- **Tier** — one *contract* (a `Backend` trait implementation
  boundary) serving one routing class of the pool. The crate ships
  exactly two tiers: **kv** (small content) and **large** (over
  the size-threshold content). Tier count (2) is ADR-004's count.
- **Engine** — a concrete storage implementation of one tier's
  contract (the existing definition, unchanged). Engine count (5) is
  this ADR's count.
- **Backend** — *the* contract (the `Backend` trait) plus its shipped
  engine set, i.e. the tier as shipped. "Backend" and "tier" name the
  same thing at different altitudes; the crate says "tier" for
  counting, "the `Backend` trait" for the contract. "Two backends"
  reads forever as "two tiers".
- **Store instance** — one running instantiation of the store facade
  (one engine set — one or two engines per the constructor mode, §4 —
  plus dispatch and GC state). **The GC
  arbitration domain** for single-instance topologies; the pin-row
  owner id under fleet engines (ADR-010's `owner`; this corrects
  ADR-008's "node id" wording, which predates the instance/node
  split — a store instance is what owns pins).
- **Node** — one running *process* (an embedder's binary). A node
  hosts **one or more store instances**. Node is the process/ops unit:
  alkcall connections, ACL grants, sweep cadence, and deployment
  identity attach to nodes; GC arbitration and pin ownership attach
  to store instances.
- **Fleet** — **two or more store instances over one shared pool**
  (via a shared SQL engine). Deliberately not defined by node count:
  two instances in one process over one postgres pool are a fleet
  under this definition (and REQ-2's phrasing "a replicator node may
  run multiple store instances (a fleet)" reads true as written);
  two processes on two sqlite files are two pools, never a fleet.
- **Deployment** — unchanged (one operator's whole system).

### 2. mem joins the kv tier as its third engine

`mem` is the **kv tier's ephemeral engine** (feature `mem`,
default-off): a `BTreeMap`-shaped implementation of the kv contract,
including `size` (trivially) and list; never fleet-valid (ADR-010 §3's
rule and this ADR §4's constructor table); the **contract-reference
engine** — every other engine's
conformance tests mirror its suite. It is not a tier, not a third
backend, and the ADR-004 count ("two") survives untouched because
tier and engine were never the same count — this ADR is that
distinction's final form.

### 3. The fs tier is renamed `large`

In every document, status line, feature name, and constructor
parameter: tier `fs` → tier `large`; feature `fs` → feature `large`
(default-on, as `fs` was); dispatch prose says "large tier"/"small
tier" as store-api.md already does; the `local` engine's name is
kept (its meaning — this node's own filesystem — stays the
constructor's meaning even under the shared-media fleet posture,
where it is explicitly *not* only-node-local; the constructor's
explicit shared-media assertion, ADR-008 §3, is what disambiguates).

### 4. The constructor table (the load-bearing copy of the fleet-validity summary; ADR-010 §3 points here)

| Tier | Engine | Feature | Default | Fleet-valid |
|---|---|---|---|---|
| kv | sqlite | — (tier default) | yes (tier default-on) | no |
| kv | postgres | `postgres` | no | **yes** |
| kv | mem | `mem` | no | no |
| large | local | — (tier default) | yes (tier default-on) | shared media only, explicit assertion |
| large | pg-lo | `pg-lo` | no | **yes** |

Constructor modes (dispatch shapes) — pinned here (implied by
ADR-008 §3's re-routing bullet and the mem tests, but never named):
- **dual-tier** (the default): kv + large, size-threshold dispatch
  between them (ADR-003).
- **kv-only** (ADR-008's "kv-tier-only dispatch", now named): a
  constructor mode with no large tier; over-threshold puts are a
  constructor-time error (the client-node re-routing posture handles
  them via the ops surface instead — its composition, not a store
  mode).
- **mem-only** (test/embedder-ephemeral mode): the mem engine alone,
  no dispatch threshold in effect (everything is local and small);
  a testing posture, explicitly not a production story.
- Single-tier SQL modes (postgres-only, pg-lo-only) are **not**
  constructor modes this crate ships: the two-tier shape is
  ADR-004's requirement (both tiers are load-bearing by measured
  economics); "postgres-only" as in *one SQL instance serving both
  tiers* already exists (kv=postgres + large=pg-lo — the ADR-009
  consolidation), which is the dual-tier mode over one engine.

### 5. Deferral-policy wording (README alignment)

The deferral policy's categories are: `deferred(scope)`,
externally-owned questions, and decided-but-sequenced work — three
names, defined in open-questions.md's header. README.md's recap says
"two parking kinds" and is corrected to name all three (the third is
not a *parking* kind — it is work, not a parked question — hence
this ADR's wording here rather than a silent edit).

## Consequences

**Positive**

- The anchor document's self-contradiction (node/fleet) becomes a
  three-level vocabulary (store instance / node / deployment) whose
  levels map one-to-one onto the mechanisms (GC domain / process-ops
  unit / operator whole).
- The fs→large rename removes the last medium-named tier; feature
  names, tier names, and dispatch prose now agree, before any code
  exists to break.
- mem's classification is stated once, as the contract-reference kv
  engine; the "plus a testing tier" carve-outs dissolve.
- The constructor table gives task decomposition a closed set: 5
  engine implementations, 3 constructor modes, 1 dispatch policy.

**Negative**

- Vocabulary churn in every doc that says "fs" (the rename is
  mechanical but touches all specs and ADR-003/004/007/008/009 status
  lines and reference sections). Zero code exists — this is the last
  free moment; after Phase 1 starts, feature renames break consumers.
- "Backend"/"tier" as interchangeable words risks reader confusion
  with ADR-003's "Backend trait" — mitigated by the rule that the
  *trait* is always named with "trait" ("the `Backend` trait"), never
  bare "the backend".

**Neutral**

- No engine's contract, feature flag semantics, or constructor
  parameter changes; `mem` gains the word "engine" and loses the
  word "backend".
- REQ-2's meaning is unchanged (the fleet definition is re-derived
  from the same fact, not re-fact'd).

## References

- ADR-003 (the trait contract; its "shipped backends" list is now
  read through this ADR's vocabulary), ADR-004 (tier≠engine — the
  scope note this ADR gives final names), ADR-007 (kv engines;
  constructor selection), ADR-008 (the second tier's engine candidate
  and the fleet topology; the vocabulary it assumed is now pinned),
  ADR-009 (pg-lo admission; the consolidation option whose
  constructor mode this ADR names dual-tier), [ADR-010](010-backend-io-seams-and-fleet-state-home.md)
  (the I/O seams and GC-state home this vocabulary must fit)
- [requirements.md](../requirements.md) (the canonical vocabulary
  copy, REQ-1..4); [backends-and-dispatch.md](../backends-and-dispatch.md)
  (the tier/engine table's spec home)
- `docs/research/poc-trait-dispatch-findings.md` finding 1 (the
  reference-impl role mem already played in the POCs);
  `poc-pglo-findings.md` (the gate the engine suite mirrors)