alkdev/alktype
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refine BAST pivot: validation model, POC scope, resolve open questions

Browse Source 
- Rewrite Validator Split around BAST-native validator for
  validate_bytes (walks BAST, checks value-domain constraints, no
  external JSON Schema needed); validate_json uses standard
  jsonschema::Validator from consumer-provided JSON Schema
- Add targeted POC: BAST-native validator replacing 19 custom keyword
  validators on the bytes path, verified via existing test suite
- Fix meta-schema: require count on arrays (variable-element arrays
  deferred per OQ-001), add optional fields array to UnionDef for
  field-name discriminators
- Remove lying no-count array example, replace with deferred note
- Document dead enum constraint (materialized index never matches
  string-membered enum); BAST-native validator fixes it via index
  bounds check
- Resolve all 7 OQs + 3 spec gaps as D-BAST-001 through D-BAST-008
- Update Migration Path, Risks table, Engine internals to reflect
  BAST-native validator
- Clarify 'no pocs needed' was an overcorrection: layout swap needs
  no POC, but the validation model does

Verification: docs-only change, no code affected
This commit is contained in:
glm-5.2 committed 2026-08-15 09:05:55 +00:00
1 parent 82bc8f29c0
commit 19f8162f1a
1 file changed
+363 -209
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/bast-pivot.md
+363 -209
View File
@@ -260,20 +260,12 @@ adoption creates migration cost.
}
```
#### Array of variable-length elements (count-prefixed)
#### Array of variable-length elements (deferred — see Decisions)
```json
{
"$defs": {
"StringList": {
"kind": "struct",
"fields": [
{ "name": "items", "kind": { "kind": "array", "element": "string" } }
]
}
}
}
```
Arrays of variable-length elements (e.g., `{ "kind": "array", "element": "string" }`
without a `count`) are **not supported in v1**. The engine rejects them today
(OQ-001), and the meta-schema requires `count` on all array types. This is a
known limitation, not a gap — see [D-BAST-004](#d-bast-004-arrays-of-variable-length-elements-deferred).
#### Record (string-keyed map)
@@ -393,14 +385,14 @@ offline use.
"additionalProperties": false
},
{
"description": "Array type",
"description": "Array type (fixed-size only in v1 — count is required)",
"type": "object",
"properties": {
"kind": { "const": "array" },
"element": { "$ref": "#/$defs/TypeRef" },
"count": { "type": "integer", "minimum": 0 }
},
"required": ["kind", "element"],
"required": ["kind", "element", "count"],
"additionalProperties": false
},
{
@@ -420,6 +412,10 @@ offline use.
"properties": {
"kind": { "const": "union" },
"endian": { "enum": ["little", "big"] },
"fields": {
"type": "array",
"items": { "$ref": "#/$defs/FieldDef" }
},
"discriminator": {
"oneOf": [
{
@@ -578,10 +574,11 @@ over unchanged.
| `normalize_refs()` | Removed — BAST `$ref` values are always full JSON Pointers |
| `inline_union_variant_refs()` | Removed — union mapping values are resolved lazily via `$ref` |
| `parse_endian()`, `parse_align()`, `parse_encoding()`, `parse_discriminator()` | Adapted to read from BAST field/struct properties instead of keyword-value objects |
| Custom keyword validators (19 `jsonschema::Keyword` impls) | Removed. These were the only thing using `jsonschema`'s custom keyword API. |
| Custom keyword validators (19 `jsonschema::Keyword` impls) | Removed. Replaced by the BAST-native validator (see below). |
| `build_validator()` | Repurposed. Still builds a `jsonschema::Validator`, but from a standard JSON Schema (no custom keywords). Used for JSON payload validation (call's `OperationSpec` schemas, etc.) and for validating BAST documents against the BAST meta-schema. |
| `validate_json()` | Unchanged in signature. Validates a `Value` against the engine's compiled `jsonschema::Validator` (now built from a standard JSON Schema instead of one with custom keywords). |
| `validate_bytes()` | Unchanged in concept — materialize `Value` from bytes, then validate. The materialization step walks BAST instead of custom-keyword JSON. |
| BAST-native validator | **New.** A recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`: integer ranges, float finiteness, `maxLength`, timestamp shape, enum index bounds, union variant dispatch. Replaces the 19 custom keyword validators for the `validate_bytes` path. See [The Validator Split](#the-validator-split). |
| `validate_json()` | Unchanged in signature. Validates a `Value` against a `jsonschema::Validator` compiled from a standard JSON Schema the consumer provides. No longer uses custom keywords. |
| `validate_bytes()` | Unchanged in concept — materialize `Value` from bytes, then validate. The validation step uses the BAST-native validator (not `jsonschema`) to check value-domain constraints from the BAST document. |
### Public API
@@ -589,7 +586,7 @@ over unchanged.
|------|--------|
| `AlkTypeKind` enum | Unchanged — same 19 variants, same methods |
| `Endian`, `VariableEncoding`, `DiscriminatorKind` | Unchanged |
| `AlkTypeEngine` | `compile()` takes a BAST document + root type name instead of a custom-keyword JSON Schema. `validate_json()` may change. `validate_bytes()` unchanged. |
| `AlkTypeEngine` | `compile()` takes a BAST document + root type name instead of a custom-keyword JSON Schema. `validate_json()` validates against a consumer-provided JSON Schema. `validate_bytes()` validates against the BAST document via the BAST-native validator. |
| `LayoutMode`, `OffsetMap`, `ByteRange` | Unchanged |
| `LayoutBuilder`, `PackedLayout`, `FieldPosition` | Unchanged |
| `SequentialReader`, `FieldValue` | Unchanged |
@@ -613,105 +610,215 @@ over unchanged.
- BAST meta-schema (embedded in the crate, published at a stable URL)
- BAST document parser — validates a BAST document against the meta-schema, then extracts type definitions
- `AlkTypeEngine::compile()` takes `(bast_document: &Value, root_name: &str, mode: LayoutMode)` — the root name selects which `$defs` entry is the top-level type
- BAST-native validator — a recursive walker over the BAST type tree that checks value-domain constraints on a materialized `Value`. Replaces the 19 custom keyword validators for the `validate_bytes` path. Enforces: integer ranges, float finiteness, string/bytes `maxLength`, RFC 3339 timestamp shape, enum index bounds, union variant dispatch (recursing into variants). See [The Validator Split](#the-validator-split).
- BAST meta-schema validation at compile time (optional but recommended — the engine can skip it and trust the caller, or validate as a guard)
## The Validator Split
A key architectural clarification: BAST separates two concerns that the
current format conflates.
current format conflates, and in doing so reveals that the current
engine has **two distinct validation paths** that custom keywords
paper over with a single mechanism.
### Current model (conflated)
```
┌─────────────────────────────────────────────┐
│ JSON Schema with AlkType:* custom keywords │
│ ┌───────────────────────────────────────┐ │
│ ┌───────────────────────────────┐ │
│ │ Binary layout info (AlkType:Uint32) │ │
│ │ JSON validation info (type, enum) │ │
│ └───────────────────────────────────────┘ │
│ └───────────────────────────────┘ │
└─────────────────────────────────────────────┘
│
▼
AlkTypeEngine::compile()
│
├──► Layout (offset map / layout builder)
└──► Validator (jsonschema with custom keywords)
└──► Validator (jsonschema with 19 custom keywords)
│
├──► validate_json(&Value) — JSON value validation
└──► validate_bytes(&[u8]) — materialize, then validate
```
One document serves two roles. The engine extracts both layout and
validation from the same JSON tree.
validation from the same JSON tree. A single `jsonschema::Validator`
(with custom keywords) serves both `validate_json` and `validate_bytes`.
### BAST model (separated)
### BAST model (three layers, two validators)
The current engine's two validation paths have different needs, and
BAST makes the split explicit:
```
┌──────────────────────┐ ┌──────────────────────────┐
│ BAST document │ │ JSON Schema document │
│ (binary layout) │ │ (JSON validation) │
│ │ │ │
│ kind: "struct" │ │ type: "object" │
│ fields: [ │ │ properties: { │
│ { name, kind } │ │ id: { type: "integer"} │
│ ] │ │ } │
│ endian: "big" │ │ required: ["id"] │
└──────────┬───────────┘ └────────────┬─────────────┘
│ │
▼ ▼
AlkTypeEngine::compile() build_validator(&json_schema)
│ │
▼ ▼
Layout (offset map / jsonschema::Validator
layout builder) (standard JSON Schema)
│ │
▼ ▼
validate_bytes(&[u8]) validate_json(&Value)
┌──────────────────────┐
│ BAST document │
│ (binary layout + │
│ value constraints) │
└──────────┬───────────┘
│
▼
AlkTypeEngine::compile()
│
┌────────────────┼────────────────┐
▼ ▼
┌─────────────────┐ ┌──────────────────┐
│ Layout │ │ BAST-native │
│ (offset map / │ │ validator │
│ layout builder)│ │ (walks BAST, │
└────────┬────────┘ │ checks value │
│ │ constraints) │
▼ └────────┬──────────┘
┌─────────────────┐ │
│ Materializer │ │
│ (bytes → Value, │ │
│ guarantees │ │
│ structure) │ │
└────────┬────────┘ │
│ │
▔──────────────┬─────────────────────┘
▼
validate_bytes(&[u8])
(materialize, then check
value constraints against
the BAST schema — no external
JSON Schema needed)
┌──────────────────────────┐
│ JSON Schema document │ (separate concern)
│ (JSON validation) │
│ type: "object" │
│ properties: { │
│ id: { type: "integer"}│
│ } │
│ required: ["id"] │
└────────────┬─────────────┘
│
▼
build_validator(&json_schema)
│
▼
jsonschema::Validator
(standard JSON Schema,
no custom keywords)
│
▼
validate_json(&Value)
(validates consumer-supplied
JSON against a standard
JSON Schema — BAST not
involved)
```
Two documents, two validators, two concerns. The BAST document
describes binary layout. The JSON Schema document describes JSON data
shape. They can be linked (a BAST struct can reference a JSON Schema by
`$id` for validation) but they are separate documents.
### Why two validators, not one
Both paths use `jsonschema` under the hood — the BAST path uses it to
validate BAST documents against the BAST meta-schema at compile time;
the JSON path uses it to validate JSON payloads against standard JSON
Schema documents. The only thing removed is the custom keyword
registration path (`jsonschema::options().with_keyword(...)`).
The two paths have fundamentally different inputs and guarantees:
**`validate_bytes(&[u8])` — bytes in, BAST is the validator.**
The materializer produces a `Value` tree from bytes. By construction,
this `Value` is *structurally correct*: all declared fields are present
(because the materializer iterates the field list), types are correct
(because `read_u32` produces a `Value::Number`), bounds are checked
(via `data_access::check_bounds`), UTF-8 is valid (via `from_utf8`),
the discriminator is in the mapping, and the boolean byte is 0 or 1.
What the materializer does NOT check — and what the 19 custom keyword
validators currently check afterward — are **value-domain constraints
expressed in the schema**:
| Constraint | Current enforcer | BAST-native validator |
|------------|------------------|----------------------|
| Integer range (Int8..Uint64) | Custom keyword | BAST `kind` + range check |
| Float finiteness (Float32/64) | Custom keyword | BAST `kind` + `is_finite()` |
| String `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property |
| Bytes `maxLength` | Custom keyword (reads parent) | BAST field `maxLength` property |
| RFC 3339 timestamp shape | Custom keyword | BAST `kind: "timestamp"` + shape check |
| Union variant dispatch | `UnionValidator` (OQ-008) | BAST union — recurse into variant |
| Enum value membership | Built-in `enum` keyword (**broken on bytes path** — see below) | BAST `kind: "enum"` + index-in-range |
All of these are expressible directly from the BAST document — no
external JSON Schema needed. The BAST-native validator is a recursive
walker over the BAST type tree, checking the materialized `Value`
against each type's constraints. Union dispatch is just recursion:
read `__discriminator`, look up the variant's BAST definition, recurse.
This **recovers the OQ-008 union dispatch behavior** without custom
keywords and without requiring the consumer to hand-author
discriminator dispatch in a separate JSON Schema. The BAST document
already knows the union's variants and their fields.
**`validate_json(&Value)` — JSON in, JSON Schema is the validator.**
The consumer provides a JSON `Value` (e.g., an incoming JSON-RPC request
on alkcall's channel 0). The BAST document is irrelevant — BAST
describes bytes, not JSON shape. The right validator for a JSON value
is a standard `jsonschema::Validator` built from a standard JSON
Schema document (which the consumer provides, or which is generated
from BAST via future codegen). This is the path alkcall uses today for
its `OperationSpec` JSON validation.
### Bug fix: enum validation on the bytes path
The current engine has a **dead constraint**: when `validate_bytes`
materializes an `AlkType:Enum` field, it reads a raw `u32` index and
emits `Value::Number(index)`. The built-in `enum` keyword lists string
members (e.g., `["Ok", "Error"]`). A numeric index never matches a
string-membered `enum`, so enum membership is silently unenforced on
the bytes path. The BAST-native validator fixes this: for `kind: "enum"`,
it checks that the materialized index is within the `values` array
bounds (0..len-1), which is the correct validation for a binary enum
encoded as an index.
### What this means for consumers
**alkcall** today uses alktype for two roles:
1. Binary layout (channels chunk header) — `AlkTypeEngine::compile()` in
packed mode
1. Binary layout (channels chunk header) — `AlkTypeEngine::compile()`
in packed mode
2. JSON validation (call's `OperationSpec` schemas) — `build_validator()`
or `jsonschema` directly
Under BAST:
1. Binary layout — `AlkTypeEngine::compile(bast_doc, "ChunkHeader",
Packed)` — same flow, different input format
2. JSON validation — `build_validator(&json_schema)` — same API,
now builds a standard `jsonschema::Validator` (no custom keywords).
Consumers that don't use binary at all (e.g., adapters around remote
JSON Schema APIs) use this path exclusively.
1. Binary layout + validation — `AlkTypeEngine::compile(bast_doc,
"ChunkHeader", Packed)` then `engine.validate_bytes(&buf)`. The
engine uses the BAST-native validator — no external JSON Schema
needed for binary data. All value constraints (`maxLength`, ranges,
enum bounds, union dispatch) are enforced from the BAST document.
2. JSON validation — `build_validator(&json_schema)` — builds a
standard `jsonschema::Validator` from a standard JSON Schema
document (no custom keywords). Consumers that don't use binary at
all (e.g., adapters around remote JSON Schema APIs) use this path
exclusively.
The builder API can produce both formats:
- `Schema::struct_().field(...).build()` → BAST JSON
- `Schema::object().field(...).build()` → standard JSON Schema
The builder API produces both formats:
- `Schema::struct_().field(...).build()` → BAST JSON (binary layout)
- `Schema::object().field(...).build()` → standard JSON Schema (JSON
validation)
Both live in the same crate, same wasm target, same builder surface.
This is the point of alktype: one small wasm-compatible codebase that
handles both binary layout and JSON validation for protocol crates like
alkcall, alktty, and future channel implementations.
### Validation of binary data
`validate_bytes()` still works: materialize a `Value` tree from binary
bytes using the BAST layout, then validate that `Value` against a JSON
Schema. The JSON Schema can be:
- Derived from the BAST definition (a codegen step, future)
- Provided separately by the consumer
- A standard JSON Schema that the consumer already has for JSON
validation of the same logical type
`validate_bytes()` works as follows:
1. **Materialize** a `Value` tree from binary bytes using the BAST
layout. The materializer guarantees structural correctness (bounds,
UTF-8, bool byte, discriminator lookup, all fields present).
2. **Validate** the materialized `Value` against value-domain
constraints from the BAST document, using the BAST-native
validator. This checks integer ranges, float finiteness, `maxLength`,
timestamp shape, enum index bounds, and recurses into union variants.
The materialization step walks the BAST layout (unchanged from current
behavior — it walks the schema to compute offsets and read fields). The
validation step uses a standard `jsonschema::Validator` (no custom
keywords needed — the `Value` tree is already typed by the
materialization).
No external JSON Schema is required for `validate_bytes`. The BAST
document is the complete specification of the binary format — it
describes both the layout (how to read) and the constraints (what
values are valid). This is the "schema is the format" principle from
ADR-001, now fully realized: the BAST document is the binary format,
and it is also the validation spec.
An optional external JSON Schema can be layered on top for constraints
BAST doesn't express (e.g., cross-field consistency, regex patterns on
string content). This is additive, not load-bearing — the binary path
works without it.
## Relationship to JSON Schema and TypeBox
@@ -728,6 +835,11 @@ This means the entire JSON Schema tooling ecosystem works with BAST:
- Documentation: JSON Schema generators can produce human-readable docs
from the meta-schema
Note: the BAST meta-schema validates the *structure* of a BAST document
(is it well-formed?). The BAST-native validator validates *binary data*
against the BAST document (are the bytes a valid instance of this
format?). These are different validators for different inputs.
### TypeBox interop
TypeBox's `Type.Module({...})` pattern maps naturally to BAST's
@@ -744,10 +856,19 @@ Same TypeBox source, two output formats, two validators.
### Not a replacement for JSON Schema
BAST does not replace JSON Schema for JSON data validation. A BAST
document cannot validate a JSON payload. It describes binary data
layouts. For JSON validation, consumers use standard JSON Schema
documents (which may be derived from BAST definitions via codegen, or
authored separately).
document cannot validate a JSON payload — it describes binary data
layouts and value-domain constraints for bytes. For JSON validation,
consumers use standard JSON Schema documents (which may be derived
from BAST definitions via codegen, or authored separately). The
`validate_json` path on `AlkTypeEngine` accepts a consumer-provided
JSON Schema and uses a standard `jsonschema::Validator` — BAST is not
involved.
This is the split: `validate_bytes` is BAST-native (the BAST document
is both the layout spec and the validation spec for bytes);
`validate_json` is JSON-Schema-native (a standard JSON Schema is the
validation spec for JSON values). One crate, two validators, two
input types.
## Codegen (Future)
@@ -814,15 +935,22 @@ through JSON Schema trees do not.
1. Define the BAST meta-schema (the JSON Schema that validates BAST
documents)
2. Implement BAST document parsing in the engine (replace custom keyword
2. **Run the BAST-native validator POC** — implement the validator,
wire it into `validate_bytes`, confirm the existing test suite passes.
This de-risks the validation model before the full pivot.
3. Implement BAST document parsing in the engine (replace custom keyword
detection with `kind` field parsing)
3. Update `AlkTypeEngine::compile()` to accept a BAST document + root
4. Update `AlkTypeEngine::compile()` to accept a BAST document + root
type name
4. Update the builder API to produce BAST JSON (public methods unchanged)
5. Remove custom keyword validators, `normalize_refs()`,
5. Implement the BAST-native validator (production version, informed
by the POC)
6. Update `validate_json()` to accept a consumer-provided JSON Schema
and build a standard `jsonschema::Validator` (no custom keywords)
7. Update the builder API to produce BAST JSON (public methods unchanged)
8. Remove custom keyword validators, `normalize_refs()`,
`inline_union_variant_refs()`, and the `get_alktype_kind*` functions
6. Update all tests to use BAST format
7. Update architecture docs (ADRs, schema-layer.md, etc.)
9. Update all tests to use BAST format
10. Update architecture docs (ADRs, schema-layer.md, etc.)
### Phase 2: Downstream adoption
@@ -838,156 +966,179 @@ through JSON Schema trees do not.
`kind` values
3. External Handlebars templates in `src/codegen/templates/`
## Spec Gaps
## POC Scope
No POCs are needed for this pivot. It is a backend swap (custom
keywords → `kind`-based format) on top of a proven layout engine, not
new protocol invention. The layout engine's byte-identity is already
proven (alknet-typedef-poc, alktype-builder-poc) and the layout code is
unchanged, so there is nothing empirical left to de-risk. What remains
are three spec gaps where the draft promises more than the engine
delivers, or expresses less than the engine supports. Each is a
decision, not an unknown.
The layout swap needs no POC — it is a backend swap (custom keywords →
`kind`-based format) on top of a proven layout engine. The layout
engine's byte-identity is already proven (alknet-typedef-poc,
alktype-builder-poc) and the layout code is unchanged, so there is
nothing empirical to de-risk there.
### Gap 1: `validate_bytes` semantics after keyword validator removal
One targeted POC **is** needed to de-risk the validation model. The
risk is specific and falsifiable: can a BAST-native validator — a
recursive walker over the BAST type tree — fully replace the 19 custom
keyword validators on the `validate_bytes` path, including the OQ-008
union variant dispatch, without regression?
The current engine's `UnionValidator` dispatches to variant schemas at
validation time (OQ-008), so `validate_bytes` on a union checks variant
field constraints (e.g. `maxLength` on a `Bytes` field inside a
variant). Removing the 19 custom keyword validators deletes this
dispatch. The spec must state what `validate_bytes` validates:
### POC: BAST-native validator for `validate_bytes`
- Structural readability only (bounds, discriminator lookup, UTF-8) —
constraint checks move to a separately-provided JSON Schema
- Full constraint validation via a JSON Schema the consumer provides
(OQ-BAST-006's leaning) — but variant constraints then require the
consumer to hand-author `__discriminator` dispatch, a regression
from today's behavior
**Hypothesis:** A recursive walker over the BAST type tree can enforce
all value-domain constraints that the 19 custom keyword validators
currently enforce, recovering the OQ-008 union variant dispatch
behavior, and fixing the enum-membership dead constraint on the bytes
path — all without `jsonschema` custom keywords and without requiring
the consumer to provide an external JSON Schema.
### Gap 2: Arrays of variable-length elements
**Scope:**
1. Implement the BAST-native validator as a new module
(`src/bast_validation.rs` or similar)
2. The validator walks a materialized `Value` tree against the BAST
type definitions, checking:
- Integer ranges (Int8..Uint64)
- Float finiteness (Float32/64)
- String `maxLength` (UTF-8 byte length)
- Bytes `maxLength` (byte length)
- RFC 3339 timestamp shape (non-strict, matching current behavior)
- Enum index bounds (0..values.len()-1 — **fixes the dead constraint**)
- Union variant dispatch (read `__discriminator`, look up variant
BAST definition, recurse)
- Boolean validity (materializer already checks, but the validator
should confirm)
3. Wire it into `validate_bytes()` as the validation step (replacing
the `jsonschema::Validator` call)
4. Run the **existing test suite** — the tests encode all current
expected validation behavior. If they pass, the POC succeeds.
The example in §"The BAST Format" shows
`{ "kind": "array", "element": "string" }` (count-prefixed). The engine
explicitly rejects arrays of variable-length elements ("TArray of
variable-length element kind ... is not supported"). Either the example
is out of scope for v1 (remove it and note the restriction in the
meta-schema), or variable-element arrays are a new feature with their
own layout semantics.
**Success criteria:**
- All existing `validate_bytes` tests pass without modification to their
assertions (test *inputs* will change to BAST format, but the
expected validation outcomes must be identical)
- The union variant dispatch tests (OQ-008) pass — `maxLength` on a
`Bytes` field inside a union variant is enforced
- The enum index-bounds validation works (new behavior — currently
broken, so this is a fix, not a regression)
### Gap 3: Field-name discriminator unions
**Failure path:** If the POC reveals that the BAST-native validator
cannot cleanly express some constraint (e.g., a constraint that relies
on JSON Schema's structural keywords in a way that's hard to
reimplement), the fallback is the "structural-only + external JSON
Schema" model from the original Gap 1 — but this is unlikely given that
the materializer already guarantees structure, leaving only value-domain
checks.
The meta-schema's `UnionDef` has no `fields` array, but the engine's
field-name discriminator reads the discriminator field from the union's
`properties` object. As sketched, BAST cannot express a feature the
engine already supports. The meta-schema needs a field-name-
discriminator shape (e.g. a `fields` array on `UnionDef`), or
field-name discriminators are dropped for v1.
**Out of scope for this POC:**
- `validate_json` — this path uses a standard `jsonschema::Validator`
from a consumer-provided JSON Schema, not the BAST-native validator.
No POC needed; it's a standard `jsonschema` usage.
- BAST document parsing / meta-schema validation — the parser is
straightforward JSON walking; no empirical risk.
- Layout computation — unchanged, already proven.
## Open Questions
## Decisions
### OQ-BAST-001: Root type selection
The following were open questions in earlier drafts. Each is now
resolved. They are recorded here as decisions, not re-litigated.
How does the engine know which `$defs` entry is the root type? Options:
- Explicit: `AlkTypeEngine::compile(bast_doc, "ChunkHeader", Packed)`
- Convention: the first entry in `$defs` (fragile — depends on JSON
key order)
- Marker: a `"$root": "ChunkHeader"` property on the BAST document
### D-BAST-001: Root type selection
**Leaning**: Explicit. The root type name is a required parameter to
`compile()`. This is unambiguous and matches how consumers think about
it ("compile the ChunkHeader schema").
**Decision:** Explicit. The root type name is a required parameter to
`compile()`: `AlkTypeEngine::compile(bast_doc, "ChunkHeader", Packed)`.
This is unambiguous and matches how consumers think about it ("compile
the ChunkHeader schema"). Convention (first entry in `$defs`) is fragile
and depends on JSON key order; a `$root` marker is redundant with an
explicit parameter.
### OQ-BAST-002: JSON validation schema linkage
### D-BAST-002: Primitive type string set
How does a BAST struct reference a JSON Schema for `validate_bytes`?
Options:
- Separate parameter: `engine.validate_bytes(buf, Some(&json_schema))`
- Embedded reference: `{ "kind": "struct", "validation": { "$ref": "https://..." } }`
- Convention: same `$id` base, different fragment
**Decision:** Lowercase (`"uint32"`, `"int8"`, `"float64"`, `"bool"`,
`"string"`, `"bytes"`, `"timestamp"`). Matches JSON Schema's own
convention (`"string"`, `"integer"`, `"boolean"`), is easier to type,
and is the convention in the TypeBox research examples. The
`AlkTypeKind` enum variants remain PascalCase in Rust — the mapping is
a simple `from_str()` impl.
**Leaning**: Separate parameter for v1. The JSON Schema is a separate
document; the engine doesn't need to know about it at compile time.
`validate_bytes()` accepts an optional `&jsonschema::Validator` that
the consumer provides. This keeps BAST focused on binary layout.
### D-BAST-003: Top-level `$defs` requirement
### OQ-BAST-003: Primitive type string set
**Decision:** Always `$defs`. Every BAST document has the same
top-level shape: `{ "$defs": { ... } }`. Single-type documents are a
special case with one entry. The `$defs` block is the namespace; the
root type name (D-BAST-001) selects the entry point. A bare struct at
the top level would be a special case with different parsing logic and
no home for additional definitions.
The current proposal uses lowercase strings for primitives:
`"uint32"`, `"int8"`, `"float64"`, `"bool"`, `"string"`, `"bytes"`,
`"timestamp"`. Alternatives:
- PascalCase: `"Uint32"`, `"Int8"` (matches `AlkTypeKind` variant names)
- UPPER_CASE: `"UINT32"`, `"INT8"`
- Prefixed: `"bast:uint32"` (namespaced, but verbose)
### D-BAST-004: Arrays of variable-length elements (deferred)
**Leaning**: Lowercase. Matches JSON Schema's own convention
(`"string"`, `"integer"`, `"boolean"`), is easier to type, and is the
convention in the TypeBox research examples. The `AlkTypeKind` enum
variants remain PascalCase in Rust — the mapping is a simple
`from_str()` impl.
**Decision:** Arrays of variable-length elements are **not supported
in v1**. The meta-schema requires `count` on all array types, making
arrays fixed-size only. The no-count example has been removed. This
matches the engine's current behavior (it rejects arrays of
variable-length elements) and aligns with OQ-001 (deferred, blocked on
a concrete consumer needing interleaved variable-stride arrays).
### OQ-BAST-004: Array count — fixed vs variable
Variable-length collections are still available via `record` (a
count-prefixed string-keyed map), which the engine supports. If a
consumer needs a variable-length array of fixed-size elements, they
can use a record with integer-stringified keys as a workaround, or
wait for OQ-001 to be addressed.
When is an array fixed-size (no count prefix) vs variable-size (count
prefix in binary)? Options:
- Explicit `count` field: `{ "kind": "array", "element": "uint32", "count": 3 }` → fixed
- Absent `count`: `{ "kind": "array", "element": "uint32" }` → variable (count-prefixed)
- Separate `minItems`/`maxItems` like current JSON Schema convention
### D-BAST-005: Field-name discriminator unions
**Leaning**: Explicit `count` for fixed, absent for variable. This is
clearer than `minItems == maxItems` and matches the BAST principle of
explicit layout information.
**Decision:** Supported. The meta-schema includes an optional `fields`
array on `UnionDef`. When `discriminator.kind == "field"`, the
`fields` array provides the union's field definitions (including the
discriminator field). When `discriminator.kind == "byte"`, `fields` is
absent — the union's layout is purely the variant layout. This
preserves a feature the engine already supports. The meta-schema
update is in [the meta-schema sketch above](#meta-schema-sketch).
### OQ-BAST-005: Top-level `$defs` requirement
### D-BAST-006: `validate_bytes` validation model
Should a BAST document always have a top-level `$defs` block, or can
a single struct be the root? Options:
- Always `$defs`: `{ "$defs": { "ChunkHeader": { "kind": "struct", ... } } }`
- Bare struct: `{ "kind": "struct", "fields": [...] }` (no `$defs`)
**Decision:** BAST-native validator. The `validate_bytes` path uses a
recursive walker over the BAST type tree to check value-domain
constraints on the materialized `Value` — no external JSON Schema
needed. This recovers the OQ-008 union variant dispatch behavior (the
validator recurses into the variant's BAST definition) and fixes the
enum-membership dead constraint (the validator checks the materialized
index against the `values` array bounds). See [The Validator
Split](#the-validator-split) and the [POC](#poc-bast-native-validator-for-validate_bytes).
**Leaning**: Always `$defs`. Consistency — every BAST document has the
same top-level shape. Single-type documents are a special case of the
general form. The `$defs` block is the namespace; the root type name
selects the entry point.
An optional external JSON Schema can be layered on top for constraints
BAST doesn't express (cross-field consistency, regex patterns on string
content). This is additive, not load-bearing.
### OQ-BAST-006: `validate_json` on `AlkTypeEngine`
### D-BAST-007: `validate_json` validation model
With custom keyword validators removed, what does
`AlkTypeEngine::validate_json()` do? Options:
- Remove it — the engine is for binary layout; JSON validation is a
separate concern
- Keep it with a separately-provided JSON Schema — the engine holds a
`jsonschema::Validator` compiled from a JSON Schema the consumer
provides at compile time
- Keep it as a convenience that validates the materialized `Value`
against the BAST structure itself (type checks only, no range
constraints)
**Decision:** Standard JSON Schema validator. `validate_json` on
`AlkTypeEngine` validates a consumer-provided JSON `Value` against a
`jsonschema::Validator` compiled from a standard JSON Schema document
the consumer provides at compile time. No custom keywords. The BAST
document is not involved in this path — BAST describes bytes, not JSON
shape. This preserves the `validate_json` / `validate_bytes` symmetry
from ADR-010, but the two paths now use different validators (standard
`jsonschema` for JSON, BAST-native for bytes), reflecting their
different inputs and guarantees.
**Leaning**: Keep it with a separately-provided JSON Schema. The engine
already compiles a validator at load time (ADR-004). The validator just
comes from a standard JSON Schema instead of custom keywords. This
preserves the `validate_json` / `validate_bytes` symmetry from ADR-010.
The JSON Schema may be authored separately or derived from BAST via
future codegen. For alkcall's channel 0 (JSON-RPC), the JSON Schema is
the `OperationSpec` schema, authored independently of any BAST
document.
### OQ-BAST-007: Builder API — two output formats
### D-BAST-008: Builder API — two output formats
The builder API currently produces one JSON format (custom keywords).
Under BAST, it needs to produce two:
1. BAST JSON (for `Schema::struct_()`, `Schema::uint32()`, etc.)
2. Standard JSON Schema (for `Schema::object()`, `Schema::string_()`, etc.)
**Decision:** One builder, two build methods. The construction API is
the same (field names, types, annotations); only the output format
differs. `Schema::struct_().field(...).build()` → BAST JSON (binary
layout). `Schema::object().field(...).build()` → standard JSON Schema
(JSON validation). The builder already distinguishes AlkType kinds from
JSON Schema types via naming conventions (`string()` vs `string_()`).
Should these be two separate builder types, or one builder with a mode
flag? Options:
- Two builders: `BastBuilder` and `JsonSchemaBuilder` (or `Schema::bast`
and `Schema::json`)
- One builder with mode: `Schema::new(Mode::Bast)` / `Schema::new(Mode::JsonSchema)`
- One builder, two build methods: `schema.build_bast()` /
`schema.build_json_schema()`
**Leaning**: One builder, two build methods. The construction API is the
same (field names, types, annotations); only the output format differs.
`Schema::struct_().field(...).build()` → BAST.
`Schema::object().field(...).build()` → standard JSON Schema. The
builder already distinguishes AlkType kinds from JSON Schema types via
naming conventions (`string()` vs `string_()`).
Both output formats live in the same crate. This is the point of
alktype: one small wasm-compatible codebase that handles both binary
layout and JSON validation for protocol crates. alkcall uses both —
channel 0 is JSON (standard JSON Schema), binary channels use BAST.
Future crates (alktty, tunnels, sftp, git) will use BAST for their
binary formats. The codegen feature (future) will generate
readers/writers from BAST documents for these crates.
## Risks and Mitigations
@@ -995,9 +1146,12 @@ naming conventions (`string()` vs `string_()`).
|------|-----------|
| BAST format doesn't cover all 19 type kinds | The format is designed to cover all 19. The meta-schema is the spec — if a kind can't be expressed, the meta-schema is wrong. |
| `$ref` resolution complexity moves from engine to schema authoring | BAST `$ref` values are always full JSON Pointers (`#/$defs/Name`). No normalization, no bare names. Resolution is a single hash lookup. |
| Losing `jsonschema`'s structural validation (required fields, etc.) | BAST is for binary layout, not JSON validation. Structural constraints belong in the JSON Schema document, not the BAST document. |
| BAST-native validator misses a constraint the custom keywords enforced | The [POC](#poc-bast-native-validator-for-validate_bytes) runs the existing test suite, which encodes all current expected validation behavior. If a constraint is missed, a test fails before the pivot lands. |
| Losing OQ-008 union variant dispatch | The BAST-native validator recurses into the variant's BAST definition on `__discriminator` lookup — same behavior, no custom keywords. Covered by the POC. |
| Enum membership broken on bytes path | Already broken today (dead constraint). The BAST-native validator fixes it by checking the materialized index against the `values` array bounds. Net improvement. |
| `validate_json` loses custom keyword validation | `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema. Consumers that relied on custom keywords for JSON validation need to provide equivalent standard JSON Schema keywords. No real consumers exist yet. |
| Builder API output format change breaks consumers | No real consumers exist yet (v0.1.0 has zero adoption). The builder's public methods are unchanged; only the JSON output format changes. |
| Meta-schema maintenance burden | The meta-schema is small (~150 lines) and changes rarely. It's embedded in the crate and published at a stable URL. |
| Meta-schema maintenance burden | The meta-schema is small and changes rarely. It's embedded in the crate and published at a stable URL. |
## References