Sync architecture docs and ADRs to BAST pivot (steps 9-10)
Step 9 (convert tests to BAST format) was a no-op: steps 4-8 converted
the tests as they went. The only remaining reference in
src/tests was the intentional rejection test at
src/schema.rs:462 (asserting the old keyword form is rejected). Full
suite passes: 389 tests (312 lib + 77 integration).
Step 10 (sync architecture docs and ADRs):
Descriptive docs rewritten/updated for BAST:
- schema-layer.md: rewritten for the BAST parser (BastDoc/BastDef/
BastType typed tree, AlkTypeKind enum with to_bast_str/from_bast_str,
what was removed). Points at bast-format.md for the normative format.
- validation.md: rewritten for the two-validator model
(bast_validation for validate_bytes, standard jsonschema for
validate_json). Documents the repurposed build_validator, the
AlkTypeError::Validation uniform payload (D-BAST-009), and what is
removed.
- builder.md: updated all output examples to BAST JSON
(struct_() -> { kind: struct, fields: [...] }; object() -> standard
JSON Schema). Documents build_doc, count(), and the field-name union
fields requirement (D-BAST-005).
- overview.md: updated for BAST (what/why, schema-is-the-format table,
dependencies, architecture pointers, design decisions table).
- README.md (architecture index): updated document table, ADR table
(new ADR-BAST + ADR-VAL-SPLIT, superseded ADR-001), OQ table
(OQ-007/OQ-008 resolutions updated for BAST-native validator), and
key design principles (#1, #2, #7, #10 reworded for BAST).
- data-access.md: updated tunion function signatures to BastUnion and
the variant resolution to return BastType (resolve_typeref for refs).
- layout-engine.md: updated construct signatures
(LayoutBuilder::new(bast_doc, root_name), OffsetMap::compute(&doc),
SequentialReader::new(bast_doc, root_name)), the recursive-walk
description (BAST typed tree), and composite-kind headings
(TStruct/TUnion/TArray -> struct/union/array). Added D-BAST-004
note on array count requirement.
New ADRs:
- ADR-BAST (bast-bast-format.md): the BAST format, meta-schema,
//kind vocabulary, design principles, what is removed, the
enum index bounds bug fix. Supersedes ADR-001's format-specific
content; records D-BAST-001..009.
- ADR-VAL-SPLIT (val-split-two-validator-model.md): the two-validator
model (BAST-native for validate_bytes, standard jsonschema for
validate_json), the repurposed build_validator, the uniform
AlkTypeError::Validation payload. Refines ADR-004's validation
strategy and ADR-010's validation step; records D-BAST-006/007/009.
Amended ADRs (supersession/amendment notes added; original decision
text preserved as historical record):
- ADR-001: format-specific content superseded by ADR-BAST;
purpose/scope and schema-is-the-format principle retained.
- ADR-002: unchanged under the pivot; one-line note that the input
format changed but the modes didn't.
- ADR-003: annotation semantics retained; annotation location moved
to BAST type-level properties (amended by ADR-BAST).
- ADR-004: AlkTypeError enum retained (D-BAST-009); validation
strategy section refined by ADR-VAL-SPLIT.
- ADR-009: builder API surface retained; build() output format
amended to BAST / standard JSON Schema by ADR-BAST (D-BAST-008).
- ADR-010: validate_bytes two-step concept retained; validation step
amended to the BAST-native validator by ADR-VAL-SPLIT.
Other:
- Cargo.toml description: JSON Schema with AlkType:* custom keywords
-> BAST document.
- bast-pivot.md research record: status draft -> implemented, with a
pointer to the ADRs that superseded its decisions.
- bast-implementation.md plan: status draft -> complete, with a note
that step 9 was a no-op and step 10 is this commit.
- open-questions.md: OQ-006/OQ-007/OQ-008 resolutions updated for the
BAST-native validator.
- questions/008-unionvalidator-variant-dispatch.md: added a
post-BAST-pivot note pointing to the current bast_validation
implementation; v0.1.0 resolution text preserved as historical
record.
Verification:
- cargo test --release: 389 pass (312 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean
- cross-reference check: every relative link in the new/updated docs
resolves (verified by script).
This commit is contained in:
glm-5.2 committed
1 parent
54fd112fde
commit
62270b03ca
20 files changed
+1642
-1075
No files matched your search
+271
-287
@@ -1,63 +1,147 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-08-11
|
||||
status: accepted
|
||||
last_updated: 2026-08-15
|
||||
---
|
||||
|
||||
# alktype — Validation
|
||||
|
||||
The validation layer: custom keyword validators for all 19 `AlkType:*`
|
||||
kinds, the `AlkTypeError` enum, load-time vs access-time validation
|
||||
strategy, and the `AlkTypeEngine` as the compiled form of a schema.
|
||||
The validation layer: two validators for two input types, the
|
||||
`AlkTypeError` enum, the load-time-build / access-time-check strategy,
|
||||
and the `AlkTypeEngine` as the compiled form of a BAST document.
|
||||
|
||||
## The Validator Split
|
||||
|
||||
BAST separates two concerns that the v0.1.0 format conflated, and in
|
||||
doing so reveals that the engine has **two distinct validation paths**
|
||||
with different inputs and guarantees. This is the validator split,
|
||||
decided in [D-BAST-006](../research/bast-pivot.md#d-bast-006-validate_bytes-validation-model),
|
||||
[D-BAST-007](../research/bast-pivot.md#d-bast-007-validate_json-validation-model),
|
||||
and
|
||||
[D-BAST-009](../research/bast-pivot.md#d-bast-009-alktypeerrorvalidation-payload-shape),
|
||||
and recorded in [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md).
|
||||
|
||||
| Path | Input | Validator | Schema source |
|
||||
|------|-------|-----------|---------------|
|
||||
| `validate_bytes(&[u8])` | Raw bytes | BAST-native validator (`bast_validation`) | The BAST document (binary layout + value constraints) |
|
||||
| `validate_json(&Value)` | Parsed JSON `Value` | Standard `jsonschema::Validator` | A consumer-provided standard JSON Schema |
|
||||
|
||||
### `validate_bytes` — bytes in, BAST is the validator
|
||||
|
||||
The materializer produces a `serde_json::Value` tree from bytes
|
||||
(walking the layout engine). By construction, this `Value` is
|
||||
*structurally correct*: all declared fields are present (the
|
||||
materializer iterates the field list), types are correct (`read_u32`
|
||||
produces `Value::Number`), bounds are checked (via
|
||||
`data_access::check_bounds`), UTF-8 is valid (via `from_utf8`), the
|
||||
discriminator is in the mapping, and the boolean byte is 0 or 1.
|
||||
|
||||
What the materializer does NOT check — and what the BAST-native
|
||||
validator checks afterward — are **value-domain constraints expressed
|
||||
in the BAST document**. The BAST-native validator
|
||||
(`src/bast_validation.rs`) is a recursive walker over the BAST typed
|
||||
tree ([`crate::bast::BastDoc`]/[`BastType`]) that checks exactly these:
|
||||
|
||||
| Constraint | Validator arm |
|
||||
|------------|---------------|
|
||||
| Integer range (Int8..Uint64) | `validate_int`/`validate_uint` with `as_i64`/`as_u64` + range check |
|
||||
| Int64/Uint64 (full range) | `validate_int64`/`validate_uint64` (JSON precision caveat per ADR-005) |
|
||||
| Float finiteness (Float32/64) | `validate_float` with `as_f64().is_finite()` |
|
||||
| String `maxLength` (byte length) | `check_string` reads the field-level `maxLength` |
|
||||
| Bytes `maxLength` (array length) | `check_bytes` accepts `Value::String` and `Value::Array` forms |
|
||||
| RFC 3339 timestamp shape | `validate_timestamp` — non-strict check (matching v0.1.0) |
|
||||
| Enum index bounds | `validate_enum` checks `idx < values.len()` — **fixes the v0.1.0 dead constraint** |
|
||||
| Union variant dispatch | `validate_union` reads `__discriminator`, looks up the variant, recurses via `validate_typeref` |
|
||||
| Struct fields | `validate_struct` walks `fields`, requires each declared field present, recurses |
|
||||
| Array count | `validate_array` checks `arr.len() == count` and recurses per element |
|
||||
| Record values | `validate_record` recurses into each value's `values` type |
|
||||
| Boolean | `validate_bool` (materializer already rejects non-0/1 bytes) |
|
||||
|
||||
No external JSON Schema is required for `validate_bytes`. The BAST
|
||||
document is the complete specification of the binary format — it
|
||||
describes both the layout (how to read) and the constraints (what
|
||||
values are valid). This is the "schema is the format" principle from
|
||||
ADR-001, now fully realized.
|
||||
|
||||
An optional external JSON Schema can be layered on top for constraints
|
||||
BAST doesn't express (cross-field consistency, regex patterns on string
|
||||
content). This is additive, not load-bearing.
|
||||
|
||||
### `validate_json` — JSON in, JSON Schema is the validator
|
||||
|
||||
The consumer provides a JSON `Value` (e.g., an incoming JSON-RPC
|
||||
request). The BAST document is irrelevant — BAST describes bytes, not
|
||||
JSON shape. The right validator for a JSON value is a standard
|
||||
`jsonschema::Validator` built from a standard JSON Schema document the
|
||||
consumer provides at `AlkTypeEngine::compile` time. This is the path
|
||||
alkcall uses for its `OperationSpec` JSON validation. No custom
|
||||
keywords; BAST is not involved.
|
||||
|
||||
If no JSON Schema was supplied to `compile`, the JSON-validation
|
||||
methods return `AlkTypeError::Schema` (`validate_json`) or `false`
|
||||
(`is_valid_json`).
|
||||
|
||||
### `AlkTypeError::Validation` payload shape (D-BAST-009)
|
||||
|
||||
The `validate_bytes` path no longer uses `jsonschema`, so its error
|
||||
payload is constructed via `jsonschema::ValidationError::custom` purely
|
||||
to keep the `Validation` variant's type unchanged. The rationale is
|
||||
consumer ergonomics on the *combined* path: consumers like alkcall use
|
||||
both `validate_json` (channel 0, JSON-RPC) and `validate_bytes` (binary
|
||||
channels) and handle `AlkTypeError::Validation` in one place. A single
|
||||
uniform payload type means one match arm covers both sources.
|
||||
|
||||
The alternative (`Validation(String)`) would force `validate_json` to
|
||||
flatten its structured errors (instance path, schema path, keyword) to
|
||||
a `String` via `Display` — the more information-rich path loses data to
|
||||
accommodate the less rich one. That is the wrong direction.
|
||||
|
||||
### What is removed
|
||||
|
||||
Under the BAST pivot, the v0.1.0 validation machinery is removed from
|
||||
the `validate_bytes` path:
|
||||
|
||||
- All 19 `jsonschema::Keyword` implementations (~200 lines of validator
|
||||
factories) — replaced by the BAST-native validator (~250 lines, a
|
||||
flat match with no factories, no trait objects, no sub-validator
|
||||
pre-computation).
|
||||
- `inline_union_variant_refs()` — union variant refs are resolved lazily
|
||||
by the validator and materializer.
|
||||
- The custom-keyword `build_validator` path — `build_validator` is
|
||||
repurposed to build a *standard* `jsonschema::Validator` from a
|
||||
consumer-provided JSON Schema (no custom keywords). See
|
||||
[`build_validator`](#build_validator).
|
||||
|
||||
The `jsonschema` crate **remains a direct dependency** for
|
||||
`validate_json` and for validating BAST documents against the BAST
|
||||
meta-schema. The only thing removed is the custom keyword integration
|
||||
path. The `validate_bytes` path no longer touches `jsonschema` — a
|
||||
small wasm binary-size win in addition to the architecture
|
||||
simplification.
|
||||
|
||||
## Validation Strategy
|
||||
|
||||
Validation is delegated to the `jsonschema` crate (v0.46.5, Draft
|
||||
2020-12). The alktype engine does not implement its own validation —
|
||||
it registers custom keyword validators for each `AlkType:*` kind and
|
||||
lets `jsonschema` handle the structural validation (object properties,
|
||||
required fields, array items, enum values).
|
||||
The strategy is decided in [ADR-004](decisions/004-error-handling-validation-strategy.md)
|
||||
and refined by [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md):
|
||||
|
||||
The strategy is decided in [ADR-004](decisions/004-error-handling-validation-strategy.md):
|
||||
|
||||
1. **Load time:** Parse the schema JSON, build the layout engine, build the
|
||||
jsonschema validator. This is the `AlkTypeEngine::compile(schema)` constructor.
|
||||
1. **Load time:** Parse the BAST document into the typed tree, compute
|
||||
the layout engine, and (optionally) build the standard
|
||||
`jsonschema::Validator` for the JSON-validation path. This is the
|
||||
`AlkTypeEngine::compile` constructor.
|
||||
2. **Access time:** Use the compiled engine for repeated read/write
|
||||
operations. Validation is opt-in per operation.
|
||||
|
||||
### What validation validates
|
||||
|
||||
The jsonschema validator operates on `serde_json::Value` instances — it
|
||||
validates JSON representations of data, not raw byte buffers. This is
|
||||
the correct separation of concerns:
|
||||
|
||||
- **JSON validation** (jsonschema): validates that a JSON document
|
||||
conforms to the schema. Used for validating hand-written schemas,
|
||||
TypeBox output, JSON payloads, or the JSON representation of a binary
|
||||
struct after deserialization.
|
||||
- **Binary access validation** (data access layer): the read/write
|
||||
functions perform type-level validation at access time — range checks
|
||||
for integers, UTF-8 validity for strings, buffer bounds checking.
|
||||
These return `AlkTypeError::Access` with field paths.
|
||||
|
||||
The "schema is the format" principle means the same schema describes
|
||||
both the JSON shape and the binary layout. The jsonschema validator
|
||||
checks the JSON shape; the data access layer checks the binary layout.
|
||||
A consumer that wants to validate a binary buffer end-to-end reads the
|
||||
buffer into a `Value` tree via the data access layer, then validates
|
||||
that `Value` against the jsonschema validator. This is a two-step
|
||||
process, not a single `validate(buffer)` call.
|
||||
|
||||
### The `AlkTypeEngine` struct
|
||||
|
||||
The `AlkTypeEngine` is the compiled form of a schema. It supports both
|
||||
layout modes (ADR-002) via an internal `Layout` enum:
|
||||
The `AlkTypeEngine` is the compiled form of a BAST document. It
|
||||
supports both layout modes (ADR-002) via an internal `Layout` enum:
|
||||
|
||||
```rust
|
||||
pub struct AlkTypeEngine {
|
||||
layout: Layout, // packed or aligned (private enum)
|
||||
validator: jsonschema::Validator, // compiled once at load time
|
||||
endian: Endian, // parsed from the schema's "endian" annotation
|
||||
schema: Value, // the normalized schema (refs resolved)
|
||||
layout: Layout, // packed or aligned (private enum)
|
||||
json_validator: Option<jsonschema::Validator>, // None when no JSON Schema supplied
|
||||
endian: Endian, // parsed from the root struct's "endian"
|
||||
bast_doc: Value, // retained for sequential_reader/read_field
|
||||
root_name: String, // the selected $defs entry
|
||||
}
|
||||
|
||||
// Private — the consumer selects via LayoutMode at compile time.
|
||||
@@ -68,205 +152,105 @@ enum Layout {
|
||||
```
|
||||
|
||||
The consumer selects the mode at construction time via `LayoutMode`
|
||||
(see [layout-engine.md](layout-engine.md) §"Mode Selection"). The `Layout`
|
||||
enum is private — the engine exposes mode-appropriate accessors instead:
|
||||
(see [layout-engine.md](layout-engine.md) §"Mode Selection"). The
|
||||
`Layout` enum is private — the engine exposes mode-appropriate
|
||||
accessors instead:
|
||||
|
||||
```rust
|
||||
impl AlkTypeEngine {
|
||||
pub fn compile(schema: &mut Value, mode: LayoutMode) -> Result<Self, AlkTypeError>;
|
||||
pub fn compile(
|
||||
bast_doc: &Value,
|
||||
root_name: &str,
|
||||
mode: LayoutMode,
|
||||
json_schema: Option<&Value>,
|
||||
) -> Result<Self, AlkTypeError>;
|
||||
pub fn mode(&self) -> LayoutMode;
|
||||
pub fn endian(&self) -> Endian;
|
||||
pub fn offset_map(&self) -> Option<&OffsetMap>; // Some in aligned mode
|
||||
pub fn layout_builder(&self) -> Option<&LayoutBuilder>; // Some in packed mode
|
||||
pub fn sequential_reader(&self) -> Option<SequentialReader>; // owned fresh reader (ADR-007)
|
||||
pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>; // ADR-004
|
||||
pub fn is_valid_json(&self, instance: &Value) -> bool; // ADR-004
|
||||
pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError>; // ADR-010
|
||||
pub fn read_field<'a>(&self, buffer: &'a [u8], field_path: &str)
|
||||
-> Result<FieldValue<'a>, AlkTypeError>; // aligned mode
|
||||
pub fn write_field(&self, buffer: &mut [u8], field_path: &str,
|
||||
value: &FieldValue<'_>) -> Result<(), AlkTypeError>; // aligned mode
|
||||
pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>; // D-BAST-007
|
||||
pub fn is_valid_json(&self, instance: &Value) -> bool; // D-BAST-007
|
||||
pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError>; // D-BAST-006
|
||||
}
|
||||
```
|
||||
|
||||
`compile` takes `&mut Value` because it normalizes `$ref` values in place
|
||||
(via [`normalize_refs`](schema-layer.md#ref-resolution-and-normalization))
|
||||
before computing the layout and building the validator. The `schema`
|
||||
field retains the normalized schema for `read_field`'s kind lookup and
|
||||
for `sequential_reader()`'s factory construction. The validator is
|
||||
mode-agnostic (it operates on `Value`, not raw bytes).
|
||||
`compile` takes `&Value` (not `&mut Value`) — BAST needs no in-place
|
||||
`normalize_refs`. `root_name` selects which `$defs` entry is the
|
||||
top-level type (D-BAST-001). `json_schema` is the optional
|
||||
consumer-provided standard JSON Schema for the `validate_json` path
|
||||
(D-BAST-007); pass `None` when JSON validation is not needed. The
|
||||
engine retains a clone of the BAST `Value` so `sequential_reader` and
|
||||
`read_field` can re-parse the typed tree on demand without lifetime
|
||||
entanglement with the caller's `Value`.
|
||||
|
||||
The `Layout::Packed` variant stores only the `LayoutBuilder` (write-side).
|
||||
The `SequentialReader` (read-side) is not stored — it has mutable cursor
|
||||
state that the consumer owns, so `sequential_reader()` constructs a fresh
|
||||
reader on each call (ADR-007).
|
||||
The `Layout::Packed` variant stores only the `LayoutBuilder`
|
||||
(write-side). The `SequentialReader` (read-side) is not stored — it has
|
||||
mutable cursor state that the consumer owns, so `sequential_reader()`
|
||||
constructs a fresh reader on each call (ADR-007).
|
||||
|
||||
The `read_field`/`write_field` methods on `AlkTypeEngine` are the
|
||||
aligned-mode data-access API — see [data-access.md](data-access.md)
|
||||
§"Higher-level read/write".
|
||||
|
||||
## Custom Keyword Validators
|
||||
## `build_validator`
|
||||
|
||||
Each `AlkType:*` kind gets a `Keyword` implementation registered via
|
||||
`jsonschema::options().with_keyword(...)`. The validators check leaf
|
||||
type constraints; `jsonschema` handles all structural validation.
|
||||
|
||||
### Numeric type validators
|
||||
|
||||
**`AlkType:Float32` / `AlkType:Float64`:**
|
||||
- Value must be a finite number.
|
||||
- For `Float32`: value must be representable as `f32` (no precision loss
|
||||
beyond `f32`'s mantissa).
|
||||
|
||||
**`AlkType:Int8` / `AlkType:Int16` / `AlkType:Int32`:**
|
||||
- Value must be an integer within the type's range.
|
||||
- Int8: -128..127, Int16: -32768..32767, Int32: -2147483648..2147483647.
|
||||
|
||||
**`AlkType:Uint8` / `AlkType:Uint16` / `AlkType:Uint32`:**
|
||||
- Value must be a non-negative integer within the type's range.
|
||||
- Uint8: 0..255, Uint16: 0..65535, Uint32: 0..4294967295.
|
||||
|
||||
### String and binary validators
|
||||
|
||||
**`AlkType:String`:**
|
||||
- Value must be a valid UTF-8 string.
|
||||
- If `maxLength` is specified in the schema, the string's byte length
|
||||
must not exceed it.
|
||||
|
||||
**`AlkType:Bytes`:**
|
||||
- Value must be a string (the JSON form for `validate_json` consumers)
|
||||
or an array of integers 0..=255 (the materialized form for
|
||||
`validate_bytes`). JSON has no native byte type; the string form is
|
||||
the JSON convention, the array form is the round-trippable form for
|
||||
non-UTF-8 bytes (see [OQ-007](questions/007-bytes-materialization-lossy-utf8.md)).
|
||||
- If `maxLength` is specified, the byte length must not exceed it. For
|
||||
the string form, this is the string's byte length; for the array
|
||||
form, this is the array length (one entry per byte).
|
||||
- **Binary representation:** In the binary layout, `TBytes` is raw bytes
|
||||
with no encoding (not base64, not hex). The JSON representation (for
|
||||
validation) uses a string or array; the binary representation (for
|
||||
data access) uses `&[u8]` directly.
|
||||
|
||||
**`AlkType:Enum`:**
|
||||
- The `AlkType:Enum` custom keyword signals that the type is an enum for
|
||||
*layout* purposes (the engine needs to know it's a fixed-size u32 index,
|
||||
not a variable-length string). The built-in `enum` keyword provides the
|
||||
value list and handles value-membership validation. The custom keyword
|
||||
validator is a no-op beyond the built-in check — it exists solely for
|
||||
the layout engine to recognize the type.
|
||||
|
||||
**`AlkType:Timestamp`:**
|
||||
- Value must be a valid RFC 3339 timestamp string (the internet profile
|
||||
of ISO 8601, e.g., `"2026-07-20T15:30:00Z"`).
|
||||
|
||||
### Composite type validators
|
||||
|
||||
**`AlkType:Struct`:**
|
||||
- Value must be an object.
|
||||
- Each property must match its declared `AlkType:*` kind.
|
||||
- Required fields must be present.
|
||||
- The `jsonschema` crate's built-in `properties` and `required` keywords
|
||||
handle the structural checks — the custom keyword only needs to
|
||||
validate that each field's value matches its `AlkType:*` kind.
|
||||
|
||||
**`AlkType:Union`:**
|
||||
- The instance must be an object with a `__discriminator` field
|
||||
carrying the mapping key (stringified discriminator value for
|
||||
byte-offset discriminators, string value for field-name
|
||||
discriminators). This is the shape the materializer produces for
|
||||
`validate_bytes`; `validate_json` consumers produce the same shape
|
||||
when validating a union instance.
|
||||
- The `UnionValidator` builds a sub-validator for each variant at
|
||||
factory time (when the parent validator tree is constructed) and
|
||||
dispatches on `__discriminator` at validation time, validating the
|
||||
full instance (including the variant fields) against the selected
|
||||
variant's schema. This closes the OQ-008 gap: variant field
|
||||
constraints (e.g. `maxLength` on a `Bytes` field inside a variant)
|
||||
are checked.
|
||||
- `$ref`s in the union's `mapping` are inlined by
|
||||
`schema::inline_union_variant_refs` during `AlkTypeEngine::compile`
|
||||
(before `build_validator`), so the `union_factory` sees full inline
|
||||
variant schemas. See [OQ-008](questions/008-unionvalidator-variant-dispatch.md).
|
||||
|
||||
**`AlkType:Array`:**
|
||||
- Value must be an array.
|
||||
- Each element must match the array's declared element type.
|
||||
- If `minItems`/`maxItems` is specified, the array length must be within
|
||||
bounds.
|
||||
|
||||
### Other validators
|
||||
|
||||
**`AlkType:Boolean`:**
|
||||
- Value must be `true` or `false`.
|
||||
|
||||
**`AlkType:Record`:**
|
||||
- Value must be an object.
|
||||
- All values must match the record's declared value type (specified via
|
||||
the `"values"` property in the schema, e.g.,
|
||||
`"values": { "AlkType:Float32": true }`).
|
||||
|
||||
### Validator implementation pattern
|
||||
|
||||
Each custom keyword implementation is ~10 lines. Example for
|
||||
`AlkType:Float32`:
|
||||
`src/validation.rs` exposes one function:
|
||||
|
||||
```rust
|
||||
struct Float32Validator;
|
||||
|
||||
impl Keyword for Float32Validator {
|
||||
fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> {
|
||||
match instance {
|
||||
Value::Number(n) if n.as_f64().map_or(false, |f| f.is_finite()) => Ok(()),
|
||||
_ => Err(ValidationError::custom("expected finite f32-compatible number")),
|
||||
}
|
||||
}
|
||||
fn is_valid(&self, instance: &Value) -> bool {
|
||||
instance.as_f64().map_or(false, |f| f.is_finite())
|
||||
}
|
||||
}
|
||||
pub fn build_validator(schema: &Value) -> Result<jsonschema::Validator, AlkTypeError>;
|
||||
```
|
||||
|
||||
Registration:
|
||||
Under the pivot this is **repurposed** (D-BAST-007): it builds a
|
||||
*standard* `jsonschema::Validator` from a consumer-provided plain JSON
|
||||
Schema — no custom keywords, no BAST involvement. The engine calls it
|
||||
internally during `compile` when `json_schema` is `Some`. Consumers
|
||||
that only need a one-off validator may call `jsonschema::options().build(schema)`
|
||||
directly; `build_validator` exists so the engine's error mapping
|
||||
(`jsonschema` build error → `AlkTypeError::Schema`) is reused.
|
||||
|
||||
```rust
|
||||
let validator = jsonschema::options()
|
||||
.with_keyword("AlkType:Float32", |parent, value, path| {
|
||||
Ok(Box::new(Float32Validator))
|
||||
})
|
||||
.build(&schema)?;
|
||||
```
|
||||
|
||||
The factory closure receives the parent schema object, the keyword's
|
||||
value, and the schema path. This enables cross-keyword awareness — for
|
||||
example, a `AlkType:Struct` validator can inspect the parent's
|
||||
`properties` to validate each field against its declared `AlkType:*` kind.
|
||||
The v0.1.0 custom-keyword `build_validator` (registered 19
|
||||
`with_keyword(...)` factories) is removed.
|
||||
|
||||
## AlkTypeError
|
||||
|
||||
A single `AlkTypeError` enum covers all error conditions across the
|
||||
engine's three phases (schema parsing, offset computation, read/write)
|
||||
plus validation. Decided in [ADR-004](decisions/004-error-handling-validation-strategy.md).
|
||||
engine's phases (schema parsing, offset computation, read/write) plus
|
||||
validation. Decided in [ADR-004](decisions/004-error-handling-validation-strategy.md);
|
||||
the variant shapes are unchanged under the pivot (D-BAST-009).
|
||||
|
||||
```rust
|
||||
pub enum AlkTypeError {
|
||||
/// Schema parsing errors (invalid JSON, missing keywords, unknown AlkType kinds).
|
||||
/// Schema parsing errors (malformed BAST, dangling $ref, unknown kind).
|
||||
Schema(String),
|
||||
/// Offset computation errors (field not found, unsupported type).
|
||||
Offset { field_path: String, reason: String },
|
||||
/// Read/write errors (buffer too short, invalid UTF-8, value out of range).
|
||||
Access { field_path: String, reason: String },
|
||||
/// Validation errors (delegated to jsonschema).
|
||||
Validation(ValidationError<'static>),
|
||||
/// Validation errors (both paths — D-BAST-009 uniform payload).
|
||||
Validation(jsonschema::ValidationError<'static>),
|
||||
}
|
||||
```
|
||||
|
||||
- **`Schema`** — for errors during `AlkTypeEngine::compile()`. Invalid
|
||||
JSON, missing required keywords, unknown `AlkType:*` kinds.
|
||||
- **`Schema`** — for errors during `AlkTypeEngine::compile()` or any
|
||||
BAST-walking path. Malformed BAST, missing `$defs`, unknown `kind`
|
||||
string, dangling `$ref`, empty `mapping`, etc.
|
||||
- **`Offset`** — for errors during offset computation. Field not found
|
||||
in the schema, type not supported for offset computation, recursive
|
||||
depth exceeded. Carries the field path.
|
||||
- **`Access`** — for errors during read/write. Buffer too short, invalid
|
||||
UTF-8 in a string field, value out of range for the target type.
|
||||
Carries the field path.
|
||||
- **`Validation`** — wraps `jsonschema`'s `ValidationError`. The
|
||||
`'static` lifetime is correct — the validator owns its schema reference
|
||||
and lives for the lifetime of the `AlkTypeEngine`.
|
||||
in the BAST tree, type not supported for offset computation. Carries
|
||||
the field path.
|
||||
- **`Access`** — for errors during read/write. Buffer too short,
|
||||
invalid UTF-8 in a string field, value out of range for the target
|
||||
type. Carries the field path.
|
||||
- **`Validation`** — wraps a `jsonschema::ValidationError<'static>`.
|
||||
On the `validate_json` path, this is the `jsonschema` crate's own
|
||||
structured error. On the `validate_bytes` path, it is constructed via
|
||||
`jsonschema::ValidationError::custom` from the BAST-native
|
||||
validator's path + reason string. The `'static` lifetime is correct —
|
||||
the payload owns its data.
|
||||
|
||||
### Field-path-carrying errors
|
||||
|
||||
@@ -287,20 +271,54 @@ you exactly which field failed and why.
|
||||
### Load time: `AlkTypeEngine::compile()`
|
||||
|
||||
The expensive work happens once at schema load time:
|
||||
1. Normalize `$ref` values in the schema (`normalize_refs`).
|
||||
2. Parse the schema's `"endian"` annotation.
|
||||
3. Compute the layout (`LayoutBuilder`/`SequentialReader` for packed, `OffsetMap` for aligned).
|
||||
4. Build the jsonschema validator (`jsonschema::options().with_keyword(...).build(&schema)?`).
|
||||
|
||||
The result is a `AlkTypeEngine` that can be used for repeated operations.
|
||||
1. Parse the BAST document into the typed tree (`BastDoc::new`).
|
||||
2. Parse the root struct's `"endian"` annotation.
|
||||
3. Compute the layout (`LayoutBuilder` for packed, `OffsetMap` for
|
||||
aligned).
|
||||
4. If `json_schema` is `Some`, build the standard
|
||||
`jsonschema::Validator` via `validation::build_validator`.
|
||||
|
||||
The result is an `AlkTypeEngine` that can be used for repeated
|
||||
operations. The BAST-native validator is not pre-built — it is a
|
||||
recursive walker that runs on the materialized `Value` at access time,
|
||||
re-using the `BastDoc` (re-parsed on demand from the retained
|
||||
`bast_doc`).
|
||||
|
||||
### Access time: `engine.validate_bytes(&[u8])`
|
||||
|
||||
For binary-layout schemas, `validate_bytes` runs the two phases in
|
||||
sequence (D-BAST-006):
|
||||
|
||||
1. **Materialize `Value` from bytes.** `materialize::materialize_packed`
|
||||
or `materialize::materialize_aligned` walks the buffer against the
|
||||
BAST typed tree and the engine's `Endian`, producing a
|
||||
`serde_json::Value` tree. Composites are recursed into (`Struct` →
|
||||
object of field values; `Array` → array of element values; `Union`
|
||||
→ dispatch then recurse; `Record` → object of key/value entries).
|
||||
The read phase reuses the existing data-access functions and returns
|
||||
`AlkTypeError::Access` (with field paths) on read failures.
|
||||
2. **Validate the `Value`.** The materialized `Value` is passed to
|
||||
`bast_validation::validate_value(&doc, &value)`, producing
|
||||
`AlkTypeError::Validation` on the first violated value-domain
|
||||
constraint.
|
||||
|
||||
Mode dispatch:
|
||||
|
||||
- **Packed mode** — materializes fields in declaration order.
|
||||
- **Aligned mode** — uses the `OffsetMap` to read fields at their
|
||||
computed offsets.
|
||||
|
||||
Both modes produce the same `Value` form; the BAST-native validator is
|
||||
mode-agnostic.
|
||||
|
||||
### Access time: `engine.validate_json(&Value)` / `engine.is_valid_json(&Value)`
|
||||
|
||||
Validation is opt-in per operation. The consumer calls
|
||||
`engine.validate_json(instance)` when validation is desired, or
|
||||
`engine.is_valid_json(instance)` for a boolean check. The jsonschema
|
||||
validator is already compiled — these are fast checks against the
|
||||
compiled validator.
|
||||
`engine.is_valid_json(instance)` for a boolean check. The
|
||||
`jsonschema::Validator` is already compiled — these are fast checks
|
||||
against the compiled validator.
|
||||
|
||||
```rust
|
||||
pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>;
|
||||
@@ -308,79 +326,37 @@ pub fn is_valid_json(&self, instance: &Value) -> bool;
|
||||
```
|
||||
|
||||
The argument is a `serde_json::Value` (the JSON representation of the
|
||||
data), not a raw byte buffer — see §"What validation validates" above.
|
||||
To validate a binary buffer end-to-end, the consumer reads it into a
|
||||
`Value` tree via the data access layer, then validates that `Value`.
|
||||
data), not a raw byte buffer. `validate_json` validates against the
|
||||
consumer-provided JSON Schema supplied at `compile` time (D-BAST-007);
|
||||
the BAST document is not involved. If no JSON Schema was supplied,
|
||||
`validate_json` returns `AlkTypeError::Schema` and `is_valid_json`
|
||||
returns `false`.
|
||||
|
||||
High-throughput paths can skip validation. Security-sensitive paths
|
||||
(parsing incoming frames from untrusted peers) can validate every frame.
|
||||
The choice is the consumer's.
|
||||
(parsing incoming frames from untrusted peers) can validate every
|
||||
frame. The choice is the consumer's.
|
||||
|
||||
### Access time: `engine.validate_bytes(&[u8])` — binary buffer validation
|
||||
|
||||
For binary-layout schemas (schemas declaring `AlkType:*` kinds), the
|
||||
engine offers a single-call form of the two-step dance: walk the bytes
|
||||
against the layout to materialize a `Value` tree, then validate that
|
||||
`Value` against the compiled jsonschema validator. Decided in
|
||||
[ADR-010](decisions/010-generalized-validation-validate-bytes.md).
|
||||
|
||||
```rust
|
||||
pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError>;
|
||||
```
|
||||
|
||||
`validate_bytes` runs the existing machinery in sequence:
|
||||
|
||||
1. **Materialize `Value` from bytes.** A new internal helper
|
||||
(`materialize_value`, alongside `SequentialReader::read_field_value`
|
||||
in `src/sequential_reader.rs`) walks the buffer against the schema
|
||||
and the engine's `Endian`, producing a `serde_json::Value` tree.
|
||||
Composites are recursed into (`Struct` → object of field values;
|
||||
`Array` → array of element values; `Union` → dispatch then recurse;
|
||||
`Record` → object of key/value entries). The read phase reuses the
|
||||
existing data-access functions and returns `AlkTypeError::Access`
|
||||
(with field paths) on read failures.
|
||||
2. **Validate the `Value`.** The materialized `Value` is passed to the
|
||||
existing `self.validator.validate(&value)`, producing
|
||||
`AlkTypeError::Validation` on failure.
|
||||
|
||||
Mode dispatch:
|
||||
|
||||
- **Packed mode** — walks with a fresh `SequentialReader` (the engine
|
||||
is already a reader factory per ADR-007), materializing fields in
|
||||
declaration order.
|
||||
- **Aligned mode** — uses the `OffsetMap` to read fields at their
|
||||
computed offsets, then materializes composites by recursing into the
|
||||
offset map's nested entries.
|
||||
|
||||
Both modes produce the same `Value` form; the validator is
|
||||
mode-agnostic (it operates on `Value`, not bytes — ADR-004).
|
||||
|
||||
#### When to use which entry point
|
||||
### When to use which entry point
|
||||
|
||||
| Entry point | Schema form | Input form | When |
|
||||
|-------------|--------------|------------|------|
|
||||
| `validate_json(&Value)` | Any (AlkType or plain JSON Schema) | Already-parsed `serde_json::Value` | Call's JSON payloads (`OperationSpec.input_schema`); TypeBox output; anything off `serde_json::from_slice` / `from_str` |
|
||||
| `validate_bytes(&[u8])` | AlkType binary-layout schema | Raw `&[u8]` buffer | Channels' 8-byte chunk header; future binary call frames; SFTP packet buffers; metatensor index structs |
|
||||
| `validate_json(&Value)` | Consumer-provided standard JSON Schema | Already-parsed `serde_json::Value` | Call's JSON payloads (`OperationSpec.input_schema`); anything off `serde_json::from_slice` / `from_str` |
|
||||
| `validate_bytes(&[u8])` | BAST document (binary layout) | Raw `&[u8]` buffer | Channels' 8-byte chunk header; future binary call frames; SFTP packet buffers; metatensor index structs |
|
||||
|
||||
`validate_bytes` requires the engine's schema to declare `AlkType:*`
|
||||
kinds — it materializes `Value` via the layout engine, which needs
|
||||
binary-layout semantics. A pure JSON Schema (call's `input_schema`,
|
||||
no AlkType kinds) compiled via `AlkTypeEngine::compile` would fail at
|
||||
the materialize step (no `AlkType:Struct` at the root). For pure JSON
|
||||
payloads, the consumer uses `serde_json::from_slice` then
|
||||
`validate_json`. See [ADR-010](decisions/010-generalized-validation-validate-bytes.md)
|
||||
§"Not a binary-payload validator for JSON-only schemas".
|
||||
`validate_bytes` requires the engine's root type to be a struct (the
|
||||
layout engine enforces this) — it materializes `Value` via the layout
|
||||
engine, which needs binary-layout semantics. For pure JSON payloads,
|
||||
the consumer uses `serde_json::from_slice` then `validate_json`. See
|
||||
[ADR-010](decisions/010-generalized-validation-validate-bytes.md) and
|
||||
[ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md).
|
||||
|
||||
#### What `validate_bytes` is not
|
||||
|
||||
- **Not a new validation engine.** It runs the existing `jsonschema`
|
||||
validator against the existing materialized `Value`. No new
|
||||
validator code, no parallel validation path (ADR-001).
|
||||
- **Not framing-aware.** It validates the bytes of *one* schema
|
||||
instance. It does not strip length prefixes, parse
|
||||
`[length: u32][payload]` framing, or handle multiple frames in a
|
||||
buffer. That's the consumer's job. alktype validates what one
|
||||
schema describes; it does not parse the wire envelope around it.
|
||||
buffer. That's the consumer's job. alktype validates what one schema
|
||||
describes; it does not parse the wire envelope around it.
|
||||
- **Not a `Validator` trait.** Two methods on one struct, not a trait
|
||||
abstraction. See [ADR-010](decisions/010-generalized-validation-validate-bytes.md)
|
||||
§"Not a `Validator` trait abstraction".
|
||||
@@ -390,24 +366,24 @@ payloads, the consumer uses `serde_json::from_slice` then
|
||||
Validation and data access are independent operations on the same data.
|
||||
The consumer can:
|
||||
|
||||
1. Validate the JSON representation of a buffer to ensure it conforms to
|
||||
the schema.
|
||||
1. Validate the bytes of a buffer to ensure it conforms to the BAST
|
||||
document's value constraints.
|
||||
2. Read fields from the binary buffer at computed offsets.
|
||||
3. Both — validate the JSON representation first, then read the binary
|
||||
buffer (defense in depth).
|
||||
3. Both — validate first, then read (defense in depth).
|
||||
|
||||
The engine does not couple validation and access. A consumer that trusts
|
||||
its data source can skip validation and go straight to read/write. A
|
||||
consumer that parses untrusted input can validate the JSON
|
||||
representation first, then access the binary buffer.
|
||||
The engine does not couple validation and access. A consumer that
|
||||
trusts its data source can skip validation and go straight to
|
||||
read/write. A consumer that parses untrusted input can validate first,
|
||||
then access the binary buffer.
|
||||
|
||||
## Design Decisions
|
||||
|
||||
| Decision | ADR | Summary |
|
||||
|----------|-----|---------|
|
||||
| Error handling and validation | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
|
||||
| Two-validator model (BAST-native + standard jsonschema) | [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md) | `validate_bytes` uses the BAST-native validator; `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema; D-BAST-006/007/009 |
|
||||
| Error handling and validation strategy | [ADR-004](decisions/004-error-handling-validation-strategy.md) | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors; jsonschema `ValidationError` wrapping |
|
||||
| Generalized validation — `validate_bytes` | [ADR-010](decisions/010-generalized-validation-validate-bytes.md) | Single-call binary-buffer validation (materialize `Value` from bytes, then validate); two methods on one struct, not a trait |
|
||||
| Purpose and scope | [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | Why jsonschema not a custom engine |
|
||||
| BAST format | [ADR-BAST](decisions/bast-bast-format.md) | The BAST document is the complete binary-format spec (layout + value constraints) |
|
||||
|
||||
## Open Questions
|
||||
|
||||
@@ -418,11 +394,19 @@ see [builder.md](builder.md).
|
||||
|
||||
## References
|
||||
|
||||
- `@alkdev/alknet: docs/research/alknet-typedef/findings.md`
|
||||
§"Validation" — the POC's custom keyword validators for all 17 kinds
|
||||
- [`bast-format.md` §Validation Model](bast-format.md#validation-model)
|
||||
— the normative validation model
|
||||
- [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md) — the
|
||||
two-validator decision
|
||||
- [ADR-004](decisions/004-error-handling-validation-strategy.md) —
|
||||
error handling and validation strategy
|
||||
- [schema-layer.md](schema-layer.md) — the 19 AlkType kinds that the
|
||||
validators check
|
||||
- [data-access.md](data-access.md) — read/write functions that operate
|
||||
on the same buffers
|
||||
- [ADR-010](decisions/010-generalized-validation-validate-bytes.md) —
|
||||
`validate_bytes` (the collapsed two-step dance)
|
||||
- [schema-layer.md](schema-layer.md) — the BAST parser that the
|
||||
BAST-native validator walks
|
||||
- [data-access.md](data-access.md) — read/write functions and the
|
||||
materializer that produce the `Value` the validator checks
|
||||
- [`src/bast_validation.rs`](../../src/bast_validation.rs) — the
|
||||
BAST-native validator implementation
|
||||
- [`src/validation.rs`](../../src/validation.rs) — the `build_validator`
|
||||
helper
|
||||
Reference in new issue
Block a user