hub/docs/architecture/hub-config.md

---
status: draft
last_updated: 2026-05-18
---

# Hub Configuration System

## Overview

The hub and spoke share a base configuration schema for common concerns (logging, MCP servers, operation directories). Hub config extends this base with infrastructure settings (Postgres, Redis, HTTP server) and encryption keys. Sensitive values in the config file are AES-256-GCM encrypted; a master key provisioned via Docker secret decrypts them at startup.

**Hard rule**: No important keys or configuration options in environment variables. The `/proc/PID/environ` leak is real. Non-sensitive convenience vars (e.g., `ALKHUB_CONFIG_PATH`) are acceptable. Everything that would be damaging if read by another process on the host must come from Docker secrets or encrypted config fields.

**Why this spec exists**: Previous implementations fell back to env vars because the config system didn't provide a clear path for every subsystem. This spec enumerates every subsystem's config needs and the precise mechanism for satisfying them, eliminating any ambiguity that could lead to env-var shortcuts.

## Architecture

### Two-Layer Key Model

```
┌──────────────────────────────────────────────┐
│  Docker Secret (master key)                   │
│  /run/secrets/hub_master_key                  │
│  Provisioned once. Rarely rotated.            │
│  tmpfs-backed, never on container filesystem. │
└──────────────┬───────────────────────────────┘
                │
                ▼
┌──────────────────────────────────────────────┐
│  Config File (JSON)                           │
│  /etc/alkhub/config.json                      │
│  Encrypted fields are EncryptedData objects.  │
│  Can be version-controlled (ciphertext safe). │
└──────────────┬───────────────────────────────┘
                │
                ▼
┌──────────────────────────────────────────────┐
│  Fully-Resolved HubConfig (in memory)         │
│  All encrypted fields decrypted, validated.   │
│  Data encryption keys (v1, v2, ...) available │
│  for client_secrets encrypt/decrypt.           │
└──────────────────────────────────────────────┘
```

**Master key** — A high-entropy passphrase string provisioned via Docker secret. Its only job is decrypting the config file's encrypted fields. It is NOT used directly for `client_secrets` encryption. The master key is consumed by `crypto.ts` as the `password` parameter to PBKDF2 (100k iterations for keyVersion 1, 200k for keyVersion 2+) — it must be a string with sufficient entropy (minimum 32 bytes of randomness, base64-encoded to ~44 characters). Generate via `crypto.generateEncryptionKey()` which returns a base64-encoded 32-byte random string suitable for this purpose. The key file contains only this string (no version prefix, no formatting).

**Data encryption keys** — The `encryptionKeys` field in the config file (itself encrypted until the master key decrypts it) contains the multi-key format `v1:base64,v2:base64`. These are the keys `crypto.ts` uses for `client_secrets` encrypt/decrypt, following the rotation protocol in [storage/services.md](storage/services.md).

**Why two keys?** Rotating the master key requires re-encrypting the config file and redeploying the Docker secret — a heavier operation. Rotating data encryption keys requires only updating the config file and re-encrypting `client_secrets` rows — no Docker secret change. Separating the two allows independent rotation schedules.

### Config File Format

JSON. This aligns with TypeBox (validates JSON natively), the `EncryptedData` format (already JSON), and the existing `MCPServerConfig` schema pattern.

Example:

```json
{
  "logLevel": "INFO",
  "development": false,
  "mcpServers": {
    "local-tools": {
      "command": "/usr/local/bin/mcp-server",
      "args": ["--port", "3000"]
    }
  },
  "operationDirectories": ["/app/ops"],
  "http": {
    "host": "0.0.0.0",
    "port": 3000
  },
  "postgres": {
    "_encrypted": {
      "keyVersion": 1,
      "salt": "base64...",
      "iv": "base64...",
      "data": "base64..."
    }
  },
  "redis": {
    "_encrypted": {
      "keyVersion": 1,
      "salt": "base64...",
      "iv": "base64...",
      "data": "base64..."
    }
  },
  "encryptionKeys": {
    "_encrypted": {
      "keyVersion": 1,
      "salt": "base64...",
      "iv": "base64...",
      "data": "base64..."
    }
  },
  "auth": {
    "apiKeyCacheTtl": 300,
    "sessionTokenTtl": 3600
  }
}
```

**Encrypted field convention**: Any field that is an object with `_encrypted` as its sole key is an encrypted value. The config loader:
1. Detects `{ "_encrypted": EncryptedData }` pattern
2. Decrypts with `crypto.decrypt(value._encrypted, masterKey)`
3. Parses the resulting plaintext as JSON
4. Replaces the field with the parsed value

This means the plaintext shape of `postgres` after decryption is whatever the `PostgresConfig` TypeBox schema expects. The encryption wrapper is orthogonal to the schema — `PostgresConfig` validates the *decrypted* value.

**`keyVersion` semantics in config-file `EncryptedData`**: The `keyVersion` field in config-file `_encrypted` objects controls PBKDF2 iteration count (100k for v1, 200k for v2 — see `crypto.ts:45`). This is **distinct** from `keyVersion` in `client_secrets` rows, which tracks which *data encryption key* encrypted the value. When the master key is rotated, all `_encrypted` fields are re-encrypted with `keyVersion: 1` by default — the master key itself has no version tracking (it's a single key, not a multi-key ring). If PBKDF2 iterations need to increase in the future, `keyVersion` can be bumped, but this is a crypto parameter change, not a key rotation event.

### Config Schema Hierarchy

```
BaseConfig (shared: hub + spoke)
    ├── $schema              Optional(string) — TypeBox schema URI
    ├── logLevel             "DEBUG" | "INFO" | "WARN" | "ERROR"
    ├── mcpServers           Record<string, MCPServerConfig>
    └── operationDirectories string[] (optional)

HubConfig extends BaseConfig
    ├── http                 { host, port }
    ├── postgres             PostgresConfig (encrypted in file)
    ├── redis                RedisConfig (encrypted in file)
    ├── encryptionKeys       string — "v1:base64,v2:base64" (encrypted in file)
    └── auth                 AuthConfig

SpokeConfig extends BaseConfig
    └── hub                  { url, auth } (auth details TBD in spoke-runner.md)
```

## Subsystem Configuration Reference

This section specifies every subsystem's config needs and the mechanism for satisfying them. If a subsystem needs a value, it's listed here with a clear source. No env vars, no ad-hoc mechanisms.

### Logger

**Source**: `HubConfig.logLevel` (from `BaseConfig.logLevel`)

**Config shape**:

```ts
// Part of BaseConfig
logLevel: Type.Optional(Type.Union([
  Type.Literal("DEBUG"),
  Type.Literal("INFO"),
  Type.Literal("WARN"),
  Type.Literal("ERROR"),
])),
// Default: "INFO" if not specified
```

**Initialization** (hub startup Step 3):
- `configureLogger()` reads `HubConfig.logLevel`
- Production: structured JSON to stdout (for Docker log aggregation)
- Development: pretty-print to stdout (detected by `HubConfig.development === true`)
- Logger sinks are configured once at startup; log level is NOT reloadable without restart
- No env vars. `NODE_ENV` is NOT used — use `HubConfig.development` flag and `HubConfig.logLevel`.

**Why no NODE_ENV**: `NODE_ENV` is an env var convention from Node.js. We're on Deno. Using `logLevel` and `development` in the config file gives explicit control and avoids the `NODE_ENV=production` / `NODE_ENV=development` ambiguity (e.g., `NODE_ENV=test` — what logging for that?). The config file is the single source of truth.

### Operations System

**Source**: `HubConfig.operationDirectories`, `HubConfig.mcpServers`, and `client_secrets` in the database.

**Config shapes**:

```ts
// BaseConfig.operationDirectories
operationDirectories: Type.Optional(Type.Array(Type.String())),
// Default: [] if not specified
// The hub always scans its own src/ops/ directory.
// Additional directories from this config field are appended.

// BaseConfig.mcpServers
mcpServers: Type.Optional(Type.Record(Type.String(), MCPServerConfig)),
// Default: {} if not specified. No MCP servers is valid — the hub still
// provides its own operations and MCP server endpoint.
// Key is the server name (used as namespace for operations).
```

**MCPServerConfig** (from `@alkdev/operations`):

```ts
MCPServerConfig = Type.Union([
  // stdio transport: spawn a process
  Type.Object({
    command: Type.String(),
    args: Type.Optional(Type.Array(Type.String())),
    env: Type.Optional(Type.Record(Type.String(), Type.String())),
    cwd: Type.Optional(Type.String()),
  }),
  // HTTP transport: connect to a URL
  Type.Object({
    url: Type.String(),
    headers: Type.Optional(Type.Record(Type.String(), Type.String())),
  }),
]);
```

**Important: `env` in MCPServerConfig is NOT hub env vars.** The `env` field passes environment variables to the MCP server *child process* (spawned via `command`). These are `process.env` for the child process, NOT `Deno.env` for the hub. The hub's own config never reads env vars for secrets.

**HTTPServiceConfig auth** (used by `from_openapi.ts` for OpenAPI-imported operations):

```ts
auth?: {
  type: "bearer" | "apiKey" | "basic";
  token?: string;           // Direct token value (from client_secrets)
  tokenEnv?: string;        // DEPRECATED — will be removed.
  headerName?: string;
  prefix?: string;
};
```

The `tokenEnv` field was used to reference env var names for API tokens. This is being removed because:
1. It violates the "no secrets in env vars" rule
2. The `clients` + `client_secrets` tables are the canonical source for outbound auth tokens
3. At runtime, the hub resolves `secretKey` references from `clients.config` to decrypted values from `client_secrets`, then passes them as `token` — never as env var names

**Migration path**: The `tokenEnv` field will be removed from `HTTPServiceConfig.auth`. Any code currently using `Deno.env.get(config.auth.tokenEnv)` should instead resolve the token from `client_secrets` via the `secretKey` wiring. The `from_openapi.ts` line `Deno.env.get(config.auth.tokenEnv)` is a bug, not a feature — it's the exact pattern this config system is designed to eliminate.

### Storage (Postgres)

**Source**: `HubConfig.postgres` (encrypted in config file)

**PostgresConfig** (decrypted shape):

```ts
const PostgresConfig = Type.Object({
  host: Type.String({ default: "127.0.0.1" }),
  port: Type.Number({ default: 5432 }),
  database: Type.String({ default: "alkdev" }),
  user: Type.String(),
  password: Type.String(),
  ssl: Type.Optional(Type.Boolean()),  // true = enable SSL with default CA verification; detailed config TBD
  maxConnections: Type.Optional(Type.Number({ default: 10 })),
});
```

The entire `PostgresConfig` is encrypted as one blob in the config file. This avoids having a plaintext `host` next to an encrypted `password` — the postgres connection details are treated as a unit.

**Connection pool creation** (hub startup Step 4):

```ts
function createPool(pgConfig: PostgresConfig): Pool {
  return new Pool({
    host: pgConfig.host,
    port: pgConfig.port,
    database: pgConfig.database,
    user: pgConfig.user,
    password: pgConfig.password,
    ssl: pgConfig.ssl,
    max: pgConfig.maxConnections,
  });
}
```

No env vars. No `DATABASE_URL`. The pool is created from `HubConfig.postgres` and nothing else.

**Drizzle Kit migrations** (development/CLI tool, NOT hub runtime):

The `drizzle-kit` CLI needs a database URL for migrations. This is a *development tooling concern*, NOT a runtime concern. The hub's runtime migrations use the programmatic migrator with `HubConfig.postgres`. For `drizzle-kit` CLI use:

```ts
// drizzle.config.ts
export default defineConfig({
  out: "./migrations",
  schema: "./schema.ts",
  dialect: "postgresql",
  dbCredentials: {
    // DO NOT use Deno.env.get("DATABASE_URL") or similar.
    // Instead, use a local development config file:
    url: loadDevDbUrl(),
  },
});
```

Where `loadDevDbUrl()` reads from a developer-local config file (e.g., `.alkhub/dev-db.json` or a decrypted local copy of the config). This file is gitignored and NEVER committed. The `alkhub-config decrypt` CLI can produce it. If a developer needs a quick DB URL for drizzle-kit, they run:

```bash
alkhub-config decrypt --master-key <master-key-path> --field postgres --config config.json
# Prints: {"host":"127.0.0.1","port":5432,"database":"alkdev","user":"hub","password":"***"}
# Developer assembles URL from the decrypted fields for drizzle-kit.
```

**The rule is simple: the hub's `drizzle.config.ts` does NOT call `Deno.env.get()` for database credentials.** It reads from a local dev config file or a decrypted field.

### Storage (Redis)

**Source**: `HubConfig.redis` (encrypted in config file)

**RedisConfig** (decrypted shape):

```ts
const RedisConfig = Type.Object({
  host: Type.String({ default: "127.0.0.1" }),
  port: Type.Number({ default: 6379 }),
  password: Type.Optional(Type.String()),
  db: Type.Optional(Type.Number({ default: 0 })),
});
```

Same pattern — encrypted as one blob. Redis connection created from `HubConfig.redis` only.

**Redis usage in the hub**:
- PubSub event transport (`createRedisEventTarget({ publishClient, subscribeClient, prefix: "alk:events:" })`)
- API key verification cache (keypal, with `apiKeyCacheTtl` from `HubConfig.auth`)
- Session token cache
- Spoke health tracking

The hub creates two Redis connections: one for publishing, one for subscribing (Redis pub/sub requires a dedicated subscriber connection).

### PubSub / EventTarget

**Source**: `HubConfig.redis` (for `createRedisEventTarget`)

The PubSub system itself doesn't have separate config — it uses the Redis connection from `HubConfig.redis`. The choice of transport (in-process vs. Redis vs. WebSocket) is determined by the deployment topology:

| Transport | When | Config needed |
|-----------|------|---------------|
| In-process (`EventTarget`) | Testing, single-process | None (default) |
| Redis (`createRedisEventTarget`) | Production hub | `HubConfig.redis` |
| WebSocket (`createWebSocketEventTarget` from `@alkdev/pubsub/event-target-websocket-client`) | Hub↔spoke | From spoke WebSocket connection |

No env vars. No separate pubsub config section.

### Auth (Keypal)

**Source**: `HubConfig.auth` and `client_secrets` database table.

**AuthConfig**:

```ts
const AuthConfig = Type.Object({
  apiKeyCacheTtl: Type.Number({ default: 300 }),     // seconds
  sessionTokenTtl: Type.Number({ default: 3600 }),   // seconds
});
```

AuthConfig is NOT encrypted — these are tuning parameters, not secrets. The actual API keys and tokens live in the `api_keys` table and `client_secrets` table, not in config.

**Note**: The `development` flag lives on `HubConfig` directly (see Open Questions #6, resolved), NOT on `AuthConfig`. It controls logger formatting (pretty-print vs JSON), strictness of error handling, and other global dev-vs-prod behaviors.

### Encryption Keys

**Source**: `HubConfig.encryptionKeys` (encrypted in config file)

**Decrypted shape**: `"v1:base64key,v2:base64key"`

- The first key is the **current** key — used for all new encryptions
- All keys are available for **decryption** — enables key rotation
- Generated via `crypto.generateEncryptionKey()`
- Key version is an integer; `v` prefix is a format marker, not part of the version number
- Versions MUST be monotonically increasing starting from 1 (no gaps)
- A `resolveEncryptionKeys` call failing to parse is a startup failure

### HTTP Server

**Source**: `HubConfig.http`

```ts
const HttpConfig = Type.Object({
  host: Type.String({ default: "0.0.0.0" }),
  port: Type.Number({ default: 3000 }),
});
```

### MCP Server (Inbound — Hub Exposes Operations as MCP Server)

**Source**: `HubConfig.http` (MCP rides on the same Hono HTTP server via `@hono/mcp`)

The MCP server middleware doesn't have a separate config section. It uses the Hono app's routes and the operation registry. See [mcp-server.md](mcp-server.md).

### Client Secrets (Outbound Auth to External Services)

**Source**: `clients` table (config) + `client_secrets` table (encrypted credentials)

This is NOT in the config file. Client configs and their secrets are stored in the database. The config file's `encryptionKeys` provides the data encryption keys to decrypt `client_secrets` at runtime.

See [storage/services.md](storage/services.md) for the `secretKey` wiring pattern.

### Agent Sessions (AI SDK)

**Source**: `clients` table (LLM provider configs) + `client_secrets` table (API keys)

LLM provider keys (Anthropic, OpenAI, etc.) are stored as `client_secrets`, NOT in config or env vars. The session system resolves provider configurations from the database at runtime.

### Test Configuration

**Source**: Test config file (JSON, same format as `HubConfig`).

Test database configuration uses the same `loadConfig` → `HubConfig` path. For tests:

```ts
// src/storage/test/helpers/db.ts
import { loadConfig } from "@alkdev/operations/config/loader.ts";

// Test config path is a non-sensitive convenience value.
// ALKHUB_TEST_CONFIG_PATH is acceptable as an env var because
// it contains a FILE PATH, not a secret.
const configPath = Deno.env.get("ALKHUB_TEST_CONFIG_PATH") 
  ?? "./test-config.json";
const masterKeyPath = "./test-master-key.txt";

const config = await loadConfig(configPath, masterKeyPath);
```

Test config files contain encrypted fields just like production. The test master key is a throwaway key committed to the test fixtures (safe because it's only used for test data).

**Acceptable env vars for tests only**: `ALKHUB_TEST_CONFIG_PATH` (file path, not secret), `ALKHUB_TEST_MASTER_KEY_PATH` (file path, not secret). Credentials remain encrypted in the config file.

## Design Decisions

### Threat Model

The config system is designed to resist the following threats:

1. **Cross-container secret leakage via `/proc/PID/environ`**: A process on the same host (or in another container with the same UID) reads environment variables of the hub process. Mitigated by: no secrets in env vars; master key in tmpfs Docker secret (not in `/proc/PID/environ`).

2. **Config file exposure**: The config file is stored in version control or on a compromised filesystem. Mitigated by: sensitive fields are AES-256-GCM encrypted; ciphertext reveals nothing without the master key; config file can be public.

3. **Accidental secret logging**: A developer adds `console.log(config)` or the logger dumps the full config object. Mitigated by: `loadConfig` MUST NOT log the config contents; logging redaction policy should mask known sensitive fields.

4. **Within-container secret access**: A process inside the container reads `/run/secrets/hub_master_key`. Mitigated by: tmpfs is mode 0400 uid 0; the hub process runs as root or with appropriate group membership. Container breakout is outside the threat model — if an attacker has root inside the container, all bets are off.

**Not in scope**: Physical access to the host, kernel exploits, compromised Docker daemon. These require infrastructure-level mitigations beyond the config system.

### D1: Config file over environment variables

**Context**: Most Node.js/Deno services use env vars for configuration, including sensitive values like DATABASE_URL.

**Decision**: Use a config file (JSON) for all structural configuration. Use Docker secrets for the master key. No sensitive values in env vars.

**Rationale**: Env vars are readable via `/proc/PID/environ` by any process with the same UID on the host. In a Docker environment with multiple containers on one host, this is a real attack surface. Config files with encrypted sensitive values are safe to version-control; the ciphertext reveals nothing without the master key.

**Trade-off**: Slightly more complex deployment (mount config file + secret, rather than just `docker run -e ...`). Acceptable because the hub is a long-running service deployed infrequently, not a throwaway container.

**Reference**: See [ADR-008](../decisions/ADR-008-secrets-encrypted-at-rest-with-key-versioning.md) for the original secrets-at-rest decision (revised for Docker secret pattern).

### D2: Whole-value encryption, not field-level

**Context**: The config file could encrypt individual sensitive fields (e.g., only `postgres.password`) while leaving `postgres.host` plaintext.

**Decision**: Encrypt the entire `postgres` and `redis` config sections as single encrypted blobs. The `_encrypted` wrapper replaces the whole field.

**Rationale**: Connection details are a unit — `host` + `port` + `user` + `password` together describe a connection. Encrypting only the password leaks the topology (which hosts, which ports, which databases). Whole-value encryption is simpler (one `EncryptedData` per section, not five) and more secure (nothing about the connection is visible without the master key).

**Trade-off**: Changing a non-sensitive value like `postgres.port` requires re-encrypting the entire section. This is rare and handled by the `alkhub-config` tool.

### D3: Two-layer keys (master + data) instead of one

**Context**: The master key could also serve as the data encryption key for `client_secrets`, eliminating the two-layer model.

**Decision**: Separate the master key (decrypts config file only) from data encryption keys (used for `client_secrets`).

**Rationale**: Independent rotation schedules. The master key is tied to the Docker deployment and is rotated rarely (requires redeploying the secret). Data encryption keys are rotated by updating the config file and re-encrypting `client_secrets` rows — no Docker secret change. Rotating the data key without touching the master key is a straightforward operation; merging the two would force a Docker secret redeployment for every key rotation.

**Trade-off**: Two keys to manage instead of one. The additional complexity is contained (the config file's `encryptionKeys` field is just another encrypted value), and the operational benefit of independent rotation is significant.

### D4: JSON config file format

**Context**: Config files could be JSON, YAML, TOML, or another format.

**Decision**: JSON.

**Rationale**: TypeBox validates JSON natively. `EncryptedData` objects are JSON. No parser dependency needed — `JSON.parse` is built-in. YAML/TOML require extra dependencies and add ambiguity (type coercion, multi-document, etc.) for no benefit here. The config file is machine-generated (via `alkhub-config` tool) and machine-read (by the config loader), so human-editing convenience is secondary.

**Trade-off**: JSON doesn't support comments. If operators need to document config choices, they should use a separate notes file or a `_comment` field (ignored by the schema). The `alkhub-config` tool can add `_comment` fields.

### D5: `_encrypted` wrapper pattern

**Context**: Encrypted values in the config file need a way to be distinguished from plaintext values.

**Decision**: Use `{ "_encrypted": EncryptedData }` as the marker. Any field whose value is an object with `_encrypted` as its sole key is treated as encrypted.

**Rationale**: Explicit, unambiguous, doesn't overlap with any valid config schema shape. The underscore prefix avoids collision with future config field names. The config loader can recursively walk the config object and decrypt all `_encrypted` values in a single pass before validating against the TypeBox schema.

**Trade-off**: Adds a nesting level to encrypted fields. `config.postgres._encrypted` instead of `config.postgres`. This is cosmetic and handled by the config loader — the rest of the codebase never sees the `_encrypted` wrapper.

### D6: MCPServerConfig.env is for child processes, not the hub

**Context**: `MCPServerConfig` has an `env` field that passes environment variables to MCP server child processes. `HTTPServiceConfig.auth` has a `tokenEnv` field that references an env var name.

**Decision**: `MCPServerConfig.env` is acceptable — these env vars are set in the MCP server process's environment, NOT the hub's. `HTTPServiceConfig.auth.tokenEnv` is deprecated and will be removed. The hub resolves outbound auth tokens from `client_secrets`, never from env vars.

**Rationale**: The `env` field in `MCPServerConfig` spawns child processes with specific env vars (e.g., an MCP server that needs `DEBUG=1`). These don't leak into the hub's process — they're scoped to the child. But `tokenEnv` reads from the hub's own `Deno.env`, which IS the `/proc/PID/environ` attack surface we're avoiding. The correct pattern is `secretKey` → `client_secrets` resolution, not env var lookup.

**Trade-off**: MCP server configs may need secrets (like an OpenAI API key for a websearch MCP server). These should be resolved from `client_secrets` and passed in the `env` field, not read from the hub's env. The MCP client loader resolves `secretKey` references and injects them into the MCP server child process's `env`.

### D7: No DATABASE_URL or connection string env vars

**Context**: The storage README example used `Deno.env.get("ALKHUB_DRIZZLE_KIT_URL")` as a fallback for drizzle-kit migrations. This contradicted the "no env vars for secrets" rule and confused implementers.

**Decision**: Remove the `Deno.env.get()` fallback from `drizzle.config.ts`. The only source for database credentials is `HubConfig.postgres` (encrypted in config file) or a developer-local decrypted config file (gitignored). For drizzle-kit CLI usage, developers use `alkhub-config decrypt --field postgres` or a local dev config file.

**Rationale**: Even development/CLI tooling should not normalize env vars for secrets. If the tooling reads env vars, developers will use them in production too. The "it's just for dev" exception becomes the production pattern.

**Trade-off**: Slightly more setup for developers running drizzle-kit (need a local config file instead of `export DATABASE_URL=...`). This is an intentional speed bump — it forces awareness that credentials are real and need proper handling.

**Reference**: See [ADR-008](../decisions/ADR-008-secrets-encrypted-at-rest-with-key-versioning.md) for the secrets-at-rest decision.

## Interfaces

### `loadConfig(filePath: string, masterKeyPath: string): Promise<HubConfig>`

The primary config loading function. Used by the hub at startup (see [hub-startup.md](hub-startup.md)).

```
1. Read master key from masterKeyPath (single line, trimmed)
   - Fail if file not found, empty, or whitespace-only after trim
2. Read config file from filePath
   - Fail if file not found or unreadable
3. Parse JSON
   - Fail if invalid JSON
4. Walk the object recursively; for each {_encrypted: EncryptedData} value:
   a. Validate EncryptedData has all required fields (keyVersion, salt, iv, data)
      - Fail if any field is missing
   b. crypto.decrypt(value._encrypted, masterKey)
      - Fail if decryption fails (wrong master key or corrupted data)
      - Error MUST identify which config field failed
   c. Parse decrypted string as JSON
      - Fail if decrypted plaintext is not valid JSON
   d. Fail if decrypted value is itself {_encrypted: ...} (prevents infinite recursion)
   e. Fail if the object has _encrypted AND other keys (sole-key rule)
   f. Replace the field with the parsed value
   - Array elements MAY contain {_encrypted: ...} objects
5. Validate merged plaintext against HubConfig TypeBox schema (Value.Assert)
   - Fail if required fields are missing, types mismatch, etc.
   - Error includes all TypeBox validation failures (not just the first)
6. Validate encryptionKeys field specifically:
   - Must decrypt to a non-empty string
   - Must match format "vN:base64key,vM:base64key,..."
   - Versions must be positive integers
   - No duplicate versions
   - Keys must be valid base64
7. Return validated HubConfig
```

On any failure: throw `ConfigLoadError` with field-level details. The hub startup (hub-startup.md) catches this and exits with a diagnostic message.

**Master key in-memory lifecycle**: The master key is needed only during Step 4 (decryption). After all `_encrypted` fields are resolved and validated, the master key SHOULD be zeroed from memory. **Caveat**: JavaScript strings are immutable and cannot be zeroed in place. The implementation should read the master key into a `Uint8Array` (via `Deno.readFile`) and zero that buffer after decryption. The string form of the master key may persist in V8's heap until GC. This is an acceptable trade-off given the single-process, short-lived exposure — V8's GC will collect the string once no references remain, and the `Uint8Array` buffer is explicitly zeroed. The data encryption keys (from `encryptionKeys`) MUST remain in memory for the process lifetime — they're used by `client_secrets` operations and the key rotation sweep. The `EncryptionKeyRing` object holds these; the master key buffer is discarded.

**Logging redaction**: The decrypted `HubConfig` object contains plaintext secrets (postgres password, redis password). It MUST NOT be logged at any level. `loadConfig` should log only: "Config loaded from `<path>`, N encrypted fields decrypted" — never the config contents. Any structured logging of config values must redact fields marked as sensitive in the schema.

### `resolveEncryptionKeys(raw: string): EncryptionKeyRing`

Parses the `v1:base64,v2:base64` format into a structured key ring. Called by `loadConfig` at Step 6 after decrypting the `encryptionKeys` field — the config loader validates the format and returns the parsed key ring as part of the `HubConfig` result.

```ts
interface EncryptionKeyRing {
  currentVersion: number;
  currentKey: string;
  keys: Map<number, string>;  // version → base64 key
  getKey(version: number): string | undefined;
}
```

Used by `client_secrets` operations and the key rotation sweep. See [storage/services.md](storage/services.md) for the rotation protocol.

### `resolveSecretRefs(config: Record<string, unknown>, secrets: Map<string, string>): Record<string, unknown>`

Resolves `secretKey` references in client config objects to actual values from `client_secrets`. Used by the MCP client loader and OpenAPI operation builder at startup.

```ts
// Given a client config:
// { auth: { type: "apiKey", secretKey: "gitea_token" } }
// And secrets: Map { "gitea_token" => "decrypted_token_value" }
// Returns:
// { auth: { type: "apiKey", token: "decrypted_token_value" } }
```

**Behavior**: Recursively walks the config object. For each string value that matches a key in the `secrets` map (found via `secretKey` field in an `auth` object), replaces it with the decrypted secret value. Returns a new object; does not mutate the input.

**Error handling**: If a `secretKey` reference points to a key that doesn't exist in `client_secrets` and the client is `enabled: true`, `resolveSecretRefs` throws `SecretRefError`. If the client is disabled, the missing secret is logged as a warning and the reference is left unresolved. See Open Question #7.

This replaces the `tokenEnv` pattern — secrets are resolved from the database, not from env vars.

### `alkhub-config` CLI (deployment tool)

Subcommands:
- `encrypt --master-key <path> --field <name> --value <json> --config <path>` — Encrypt a field in the config file
- `decrypt --master-key <path> --field <name> --config <path>` — Decrypt and display a field (for verification)
- `re-encrypt --old-master-key <path> --new-master-key <path> --config <path>` — Rotate master key: decrypt all fields with old key, re-encrypt with new key
- `generate-key` — Generate a new data encryption key (base64, 32 bytes) for use in the `encryptionKeys` field
- `add-encryption-key --master-key <path> --config <path> --version <N>` — Append a new key version to the `encryptionKeys` field (preserves existing versions, generates new key)
- `init --master-key <path>` — Create a new config file with encrypted fields

## TypeBox Schemas

The full TypeBox schema for `HubConfig`, assembled from the subsystem schemas above:

```ts
import { Type, type Static } from "@alkdev/typebox";

// --- BaseConfig (shared: hub + spoke) ---

export const MCPServerConfig = Type.Union([
  Type.Object({
    command: Type.String(),
    args: Type.Optional(Type.Array(Type.String())),
    env: Type.Optional(Type.Record(Type.String(), Type.String())),
    cwd: Type.Optional(Type.String()),
  }),
  Type.Object({
    url: Type.String(),
    headers: Type.Optional(Type.Record(Type.String(), Type.String())),
  }),
]);

export const BaseConfig = Type.Object({
  $schema: Type.Optional(Type.String()),
  logLevel: Type.Optional(Type.Union([
    Type.Literal("DEBUG"),
    Type.Literal("INFO"),
    Type.Literal("WARN"),
    Type.Literal("ERROR"),
  ])),
  mcpServers: Type.Optional(Type.Record(Type.String(), MCPServerConfig)),
  operationDirectories: Type.Optional(Type.Array(Type.String())),
});

// --- HubConfig ---

export const PostgresConfig = Type.Object({
  host: Type.String({ default: "127.0.0.1" }),
  port: Type.Number({ default: 5432 }),
  database: Type.String({ default: "alkdev" }),
  user: Type.String(),
  password: Type.String(),
  ssl: Type.Optional(Type.Boolean()),  // true = enable SSL with default CA verification; detailed config TBD
  maxConnections: Type.Optional(Type.Number({ default: 10 })),
});

export const RedisConfig = Type.Object({
  host: Type.String({ default: "127.0.0.1" }),
  port: Type.Number({ default: 6379 }),
  password: Type.Optional(Type.String()),
  db: Type.Optional(Type.Number({ default: 0 })),
});

export const HttpConfig = Type.Object({
  host: Type.String({ default: "0.0.0.0" }),
  port: Type.Number({ default: 3000 }),
});

export const AuthConfig = Type.Object({
  apiKeyCacheTtl: Type.Number({ default: 300 }),
  sessionTokenTtl: Type.Number({ default: 3600 }),
});

export const HubConfig = Type.Intersect([
  BaseConfig,
  Type.Object({
    http: Type.Optional(HttpConfig),
    postgres: PostgresConfig,       // encrypted in file, decrypted shape here
    redis: Type.Optional(RedisConfig), // encrypted in file, decrypted shape here
    /** Multi-key encryption format: "v1:base64,v2:base64,..." */
    encryptionKeys: Type.String(),  // encrypted in file
    auth: Type.Optional(AuthConfig),
    /** Development mode: enables pretty-print logging, stricter error handling. NOT an env var. */
    development: Type.Optional(Type.Boolean({ default: false })),
  }),
]);

// --- SpokeConfig ---

export const SpokeConfig = Type.Intersect([
  BaseConfig,
  Type.Object({
    hub: Type.Object({
      url: Type.String(),                          // wss://api.alk.dev/ws
      auth: Type.Object({
        tokenFile: Type.String(),                   // path to Docker secret / mounted file
      }),
    }),
  }),
]);

export type BaseConfig = Static<typeof BaseConfig>;
export type HubConfig = Static<typeof HubConfig>;
export type SpokeConfig = Static<typeof SpokeConfig>;
export type PostgresConfig = Static<typeof PostgresConfig>;
export type RedisConfig = Static<typeof RedisConfig>;
export type HttpConfig = Static<typeof HttpConfig>;
export type AuthConfig = Static<typeof AuthConfig>;
```

Note: The TypeBox schemas above define the *decrypted* shapes. In the config file, `postgres`, `redis`, and `encryptionKeys` are `_encrypted` objects. The `loadConfig` function decrypts them before validating against these schemas. The schema validation runs on the fully-decrypted config.

**Important**: The `encryptionKeys` field is typed as `Type.String()` in the schema, which validates it only as "is a string." Runtime format validation (`v1:base64,v2:base64`, monotonic versions, valid base64) is performed by `resolveEncryptionKeys` during `loadConfig` Step 6. TypeBox cannot express these constraints natively — the format validation happens after TypeBox validation.

**Note on `mcpServers`**: This field is optional with a default of `{}` (empty object). A hub with no MCP servers to connect to is valid — the hub still provides its own operations and MCP server endpoint. The `operationDirectories` field is similarly optional with a default of `[]` (the hub always scans `src/ops/` regardless).

## Master Key Provisioning

### Docker Secret Approach

The hub runs in Docker. The master key is provisioned as a Docker secret:

```bash
# Create the secret (once)
echo -n "your-master-key-base64" | docker secret create hub_master_key -

# Reference in docker-compose.yml or docker run
services:
  hub:
    image: alkdev/hub:latest
    secrets:
      - hub_master_key
    volumes:
      - /opt/alkhub/config.json:/etc/alkhub/config.json:ro

secrets:
  hub_master_key:
    external: true
```

If not using Docker Swarm, an equivalent tmpfs mount:

```bash
docker run -d \
  --name alkdev-hub \
  --tmpfs /run/secrets:mode=0400,uid=0 \
  -v /opt/alkhub/master-key:/run/secrets/hub_master_key:ro \
  -v /opt/alkhub/config.json:/etc/alkhub/config.json:ro \
  alkdev/hub:latest
```

**Properties**:
- File is tmpfs-backed — never written to container's writable layer
- Read-only mount — process cannot modify the secret
- Not visible in `docker inspect` environment section
- Not accessible via `/proc/PID/environ`

### Config File Encryption Tool

A CLI tool (`alkhub-config`) for encrypting values in the config file:

```bash
# Encrypt the postgres config section
alkhub-config encrypt \
  --master-key <master-key-path> \
  --field postgres \
  --value '{"host":"127.0.0.1","port":5432,"database":"alkdev","user":"hub","password":"***"}' \
  --config /etc/alkhub/config.json

# Rotate: decrypt with old master key, re-encrypt with new
alkhub-config re-encrypt \
  --old-master-key <old-master-key-path> \
  --new-master-key <new-master-key-path> \
  --config /etc/alkhub/config.json
```

This tool is part of the deployment workflow, not the runtime. Operators use it to prepare config files. The hub itself only needs the decrypt path.

## Spoke Config Notes

The spoke's config is separate from the hub's. It shares `BaseConfig` but does NOT use the `_encrypted` wrapper pattern — the spoke doesn't have a master key. Spoke auth material (API key, registration token) comes from a Docker secret or local file specific to that spoke's deployment.

The `SpokeConfig` auth field format depends on the spoke authentication model (see [spoke-runner.md](spoke-runner.md) Open Question #4). The config system should support:

```ts
// SpokeConfig (possible shape, subject to spoke auth design)
const SpokeConfig = Type.Intersect([
  BaseConfig,
  Type.Object({
    hub: Type.Object({
      url: Type.String(),                          // wss://api.alk.dev/ws
      auth: Type.Object({
        tokenFile: Type.String(),                   // path to Docker secret / mounted file
      }),
    }),
  }),
]);
```

This is a sketch — the spoke auth model needs to be specified before this is stabilized. The key point: the spoke reads its auth token from a file reference, not from an env var, and not from an encrypted config field.

## Approved Environment Variables

This is the exhaustive list of environment variables the hub and its tooling may read. Any env var not on this list is a bug.

| Variable | Context | Purpose | Secret? |
|----------|---------|---------|---------|
| `ALKHUB_CONFIG_PATH` | `main.ts` | Path to config file (default: `/etc/alkhub/config.json`) | No — file path |
| `ALKHUB_MASTER_KEY_PATH` | `main.ts` | Path to master key file (default: `/run/secrets/hub_master_key`) | No — file path |
| `ALKHUB_TEST_CONFIG_PATH` | Test only | Path to test config file | No — file path |
| `ALKHUB_TEST_MASTER_KEY_PATH` | Test only | Path to test master key file | No — file path |
| `DENO_DIR` | Deno runtime | Deno cache directory (standard Deno env var) | No |

**Not on this list** (and therefore bugs if found):
- `DATABASE_URL` — use `HubConfig.postgres` (encrypted in config file)
- `REDIS_URL` — use `HubConfig.redis` (encrypted in config file)
- `NODE_ENV` — use `HubConfig.logLevel` + `HubConfig.development` (if added)
- `ALKHUB_DRIZZLE_KIT_URL` — use decrypted local config file for drizzle-kit
- Any variable containing API keys, passwords, or tokens

## Config File Location

**Production**: `/etc/alkhub/config.json` (mounted read-only from host)
**Default master key**: `/run/secrets/hub_master_key` (Docker secret, tmpfs)

The `main.ts` entry point resolves paths:

```ts
const configPath = Deno.env.get("ALKHUB_CONFIG_PATH") || "/etc/alkhub/config.json";
const masterKeyPath = Deno.env.get("ALKHUB_MASTER_KEY_PATH") || "/run/secrets/hub_master_key";
const hub = await startHub({ configPath, masterKeyPath });
```

Both path env vars are non-sensitive convenience defaults — they contain file paths, not secrets. `startHub` receives explicit paths and has no env var dependency internally.

## Constraints

1. **No env vars for secrets or important config** — Non-sensitive convenience vars only (see Approved Environment Variables table). Anything that would be damaging if exposed via `/proc` must come from Docker secrets or encrypted config fields.
2. **Config is read-once at startup** — The config file is loaded and validated once. Runtime changes require a restart. This may be relaxed in a future phase for non-sensitive fields (see Open Questions).
3. **Master key loss = total data loss** — If the master key is lost, all encrypted config values are unrecoverable. If data encryption keys are lost, all `client_secrets` values are unrecoverable. This is standard for symmetric encryption. Mitigated by: storing master key in infrastructure secrets (not in the database), backing up config files.
4. **Config file must be valid JSON** — No YAML, no TOML. The `alkhub-config` tool enforces this.
5. **`_encrypted` wrapper is the only encryption marker** — No alternative encryption formats in config files. All encrypted values use the same `EncryptedData` structure from `crypto.ts`.
6. **Config file is mounted read-only** — The hub never writes to its config file at runtime. The `alkhub-config` CLI is a separate deployment tool.
7. **TypeBox validation runs on the fully-decrypted config** — The schema validates the plaintext shape. Encrypted fields are opaque to the schema until decrypted.
8. **PBKDF2 startup latency** — Each `crypto.decrypt` call runs PBKDF2 (100k+ iterations). With ~3 encrypted fields (postgres, redis, encryptionKeys), expect ~300ms total decryption time on modern hardware. This is acceptable for a one-time startup cost. If it becomes a problem, a future optimization could cache the derived key per (password, salt) pair, but this increases in-memory secret exposure.
9. **Drizzle Kit CLI uses local dev config, not env vars** — The `drizzle.config.ts` file does NOT fall back to env vars for database URLs. It reads from a local dev config or a decrypted field.
10. **`MCPServerConfig.env` is for child processes only** — These env vars are set in the MCP server process, NOT in the hub process. The hub never reads `Deno.env` for secrets.
11. **`HTTPServiceConfig.auth.tokenEnv` is deprecated** — Will be removed. Outbound auth tokens are resolved from `client_secrets` via `secretKey` wiring, not from env vars.

## Open Questions

1. **Config reload without restart** — For non-sensitive fields (logLevel, auth cache TTLs), a SIGHUP or API call could trigger re-reading the config file. For encrypted fields, this would require the master key to remain in memory (which we explicitly avoid after startup — see `loadConfig` § Master key in-memory lifecycle). **Current decision**: restart required for any config change. Relaxing this for non-encrypted fields is a future enhancement that would need to weigh the implementation complexity against the operational benefit.

2. **Config file generation workflow** — The `alkhub-config` tool requires the master key to encrypt values. In CI/CD, how does the pipeline get the master key? Options: (a) CI has access to the master key secret, (b) config files are pre-encrypted and stored in a private repo, (c) encryption happens at deploy time on the host. Needs operational clarity.

3. **Spoke auth field format** — Blocked on [spoke-runner.md](spoke-runner.md) WebSocket auth design. The config system supports a `tokenFile` reference, but the actual auth protocol (token in first message vs. query string vs. subprotocol) is TBD.

4. **Multiple config file layers** — Should the config loader support a base config + overlay pattern (e.g., `/etc/alkhub/config.json` + `/etc/alkhub/config.local.json`)? Useful for dev vs. prod. Could be a future enhancement.

5. **Config schema version** — The existing `BaseConfig` already supports `$schema: Type.Optional(Type.String())`. Config files generated by `alkhub-config init` SHOULD include a `$schema` field pointing to the TypeBox schema URI. This supports forward compatibility and editor validation. Implementation detail: the `alkhub-config` tool generates this; the config loader ignores it during validation.

6. **~~`development` mode flag~~**: **Resolved.** Added `development: Type.Optional(Type.Boolean({ default: false }))` to `HubConfig` directly (NOT in `AuthConfig`). Controls logger formatting (pretty-print vs. JSON) and error handling strictness. Replaces any `NODE_ENV` convention.

7. **Secret reference resolution ordering** — When `resolveSecretRefs` is called at startup, should it fail if a referenced `secretKey` doesn't exist in `client_secrets` yet? Or should it lazily resolve on first use? **Current preference**: fail at startup for clients that are `enabled: true`. If a client is disabled, its secrets don't need to exist.

## References

- [hub-startup.md](hub-startup.md) — Startup sequence that consumes this config
- [spoke-runner.md](spoke-runner.md) — Spoke auth model, WebSocket auth
- [storage/services.md](storage/services.md) — `client_secrets` encryption, key rotation protocol, `secretKey` wiring
- [storage/README.md](storage/README.md) — Storage patterns, DB connection
- [infrastructure.md](infrastructure.md) — Docker deployment, server layout
- [pubsub-redis.md](pubsub-redis.md) — Redis EventTarget adapter (uses `HubConfig.redis`)
- [operations.md](operations.md) — Operations system (uses `HubConfig.operationDirectories`, `HubConfig.mcpServers`)
- `@alkdev/operations` — Operations, call protocol (PendingRequestMap, CallHandler), config types
- `src/crypto.ts` — encrypt, decrypt, generateEncryptionKey, EncryptedData