alktype/docs/architecture/validation.md

status, last_updated

status	last_updated
accepted	2026-08-31

alktype — Validation

The validation layer: two validators for two input types, the AlkTypeError enum, the load-time-build / access-time-check strategy, and the AlkTypeEngine as the compiled form of a BAST document.

The Validator Split

BAST separates two concerns that the v0.1.0 format conflated, and in doing so reveals that the engine has two distinct validation paths with different inputs and guarantees. This is the validator split, decided in D-BAST-006, D-BAST-007, and D-BAST-009, and recorded in ADR-VAL-SPLIT.

Path	Input	Validator	Schema source
validate_bytes(&[u8])	Raw bytes	Compiled ValidationPlan walk	The BAST document (binary layout + value constraints)
validate_json(&Value)	Parsed JSON Value	Standard jsonschema::Validator	A consumer-provided standard JSON Schema

validate_bytes — bytes in, BAST is the validator

The materializer produces a serde_json::Value tree from bytes (walking the layout engine). By construction, this Value is structurally correct: all declared fields are present (the materializer iterates the field list), types are correct (read_u32 produces Value::Number), bounds are checked (via data_access::check_bounds), UTF-8 is valid (via from_utf8), the discriminator is in the mapping, and the boolean byte is 0 or 1.

What the materializer does NOT check — and what the validation half checks afterward — are value-domain constraints expressed in the BAST document. Since ADR-012 §3 (0.3.0), those constraints are not walked interpretively per buffer: they are compiled once into a ValidationPlan (src/validation_plan.rs) at AlkTypeEngine::compile time, and each validate_bytes call walks the compiled constraint tree against the materialized Value — no $ref re-resolution, no schema re-parse, no per-node path formatting (error paths render only on failure). The plan's constraint nodes implement exactly the table below (the constraint set is unchanged from the retired interpretive walker):

Constraint	Plan node (ValidNode)
Integer range (Int8..Uint32)	Int { min, max } / Uint { max }
Int64/Uint64 (full range)	I64 / U64 (JSON precision caveat per ADR-005)
Float finiteness (Float32/64)	Float with as_f64().is_finite()
String maxLength (byte length)	Str { max_len } — maxLength baked in from the owning field at compile time
Bytes maxLength (array length)	Bytes { max_len } — accepts the Value::Array form (the materializer emits bytes as an array of u8)
Enum index bounds	Enum { count } checks idx < count — fixes the v0.1.0 dead constraint
Union variant dispatch	Union { variants } reads __discriminator, dispatches on the compiled variant nodes
Struct fields	Struct { fields } requires each declared field present, recurses
Array count	Array { count, element } checks arr.len() == count and recurses per element
Record values	Record { values } recurses into each value
Boolean	Bool (materializer already rejects non-0/1 bytes)

The plan is a public type (ValidationPlan, Debug + Clone + PartialEq + Eq + Hash + Send + Sync): engine.validation_plan() exposes it for consumers that validate their own materialized Value trees or want its fingerprint() for caching / schema handshakes (ADR-012 §1). The one-shot bast_validation::validate_value(&doc, &value) remains as a convenience wrapper (compile + validate) for callers holding a BAST document without an engine.

No external JSON Schema is required for validate_bytes. The BAST document is the complete specification of the binary format — it describes both the layout (how to read) and the constraints (what values are valid). This is the "schema is the format" principle from ADR-001, now fully realized.

An optional external JSON Schema can be layered on top for constraints BAST doesn't express (cross-field consistency, regex patterns on string content). This is additive, not load-bearing.

validate_json — JSON in, JSON Schema is the validator

The consumer provides a JSON Value (e.g., an incoming JSON-RPC request). The BAST document is irrelevant — BAST describes bytes, not JSON shape. The right validator for a JSON value is a standard jsonschema::Validator built from a standard JSON Schema document the consumer provides at AlkTypeEngine::compile time. This is the path alkcall uses for its OperationSpec JSON validation. No custom keywords; BAST is not involved.

If no JSON Schema was supplied to compile, the JSON-validation methods return AlkTypeError::Schema (validate_json) or false (is_valid_json).

AlkTypeError::Validation payload shape (D-BAST-009)

The validate_bytes path no longer uses jsonschema, so its error payload is constructed via jsonschema::ValidationError::custom purely to keep the Validation variant's type unchanged. The rationale is consumer ergonomics on the combined path: consumers like alkcall use both validate_json (channel 0, JSON-RPC) and validate_bytes (binary channels) and handle AlkTypeError::Validation in one place. A single uniform payload type means one match arm covers both sources.

The alternative (Validation(String)) would force validate_json to flatten its structured errors (instance path, schema path, keyword) to a String via Display — the more information-rich path loses data to accommodate the less rich one. That is the wrong direction.

What is removed

Under the BAST pivot, the v0.1.0 validation machinery is removed from the validate_bytes path:

• All 19 jsonschema::Keyword implementations (~200 lines of validator factories) — replaced by the BAST-native validator (~250 lines, a flat match with no factories, no trait objects, no sub-validator pre-computation).
• inline_union_variant_refs() — union variant refs are resolved lazily by the validator and materializer.
• The custom-keyword build_validator path — build_validator is repurposed to build a standard jsonschema::Validator from a consumer-provided JSON Schema (no custom keywords). See build_validator.

The jsonschema crate remains a direct dependency for validate_json and for validating BAST documents against the BAST meta-schema. The only thing removed is the custom keyword integration path. The validate_bytes path no longer touches jsonschema — a small wasm binary-size win in addition to the architecture simplification.

Validation Strategy

The strategy is decided in ADR-004 and refined by ADR-VAL-SPLIT:

1. Load time: Parse the BAST document into the typed tree, compile the ValidationPlan (the value-domain constraint tree), compute the layout engine, and (optionally) build the standard jsonschema::Validator for the JSON-validation path. This is the AlkTypeEngine::compile constructor.
2. Access time: Use the compiled engine for repeated read/write operations. Validation is opt-in per operation: the validation half walks the compiled ValidationPlan, never the BAST document.

The AlkTypeEngine struct

The AlkTypeEngine is the compiled form of a BAST document. It supports both layout modes (ADR-002) via an internal Layout enum:

pub struct AlkTypeEngine {
    layout: Layout,                          // packed or aligned (private enum)
    json_validator: Option<jsonschema::Validator>, // None when no JSON Schema supplied
    validation_plan: Arc<ValidationPlan>,    // compiled value-domain constraints (ADR-012 §3)
    endian: Endian,                          // parsed from the root struct's "endian"
    bast_doc: Value,                         // retained for sequential_reader/read_field
    root_name: String,                       // the selected $defs entry
}

// Private — the consumer selects via LayoutMode at compile time.
enum Layout {
    Packed { builder: LayoutBuilder },
    Aligned { offset_map: OffsetMap },
}

The consumer selects the mode at construction time via LayoutMode (see layout-engine.md §"Mode Selection"). The Layout enum is private — the engine exposes mode-appropriate accessors instead:

impl AlkTypeEngine {
    pub fn compile(
        bast_doc: &Value,
        root_name: &str,
        mode: LayoutMode,
        json_schema: Option<&Value>,
    ) -> Result<Self, AlkTypeError>;
    pub fn mode(&self) -> LayoutMode;
    pub fn endian(&self) -> Endian;
    pub fn offset_map(&self) -> Option<&OffsetMap>;          // Some in aligned mode
    pub fn layout_builder(&self) -> Option<&LayoutBuilder>;  // Some in packed mode
    pub fn sequential_reader(&self) -> Option<SequentialReader>; // owned fresh reader (ADR-007)
    pub fn read_field<'a>(&self, buffer: &'a [u8], field_path: &str)
        -> Result<FieldValue<'a>, AlkTypeError>;            // aligned mode
    pub fn write_field(&self, buffer: &mut [u8], field_path: &str,
        value: &FieldValue<'_>) -> Result<(), AlkTypeError>; // aligned mode
    pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>;   // D-BAST-007
    pub fn is_valid_json(&self, instance: &Value) -> bool;                        // D-BAST-007
    pub fn validate_bytes(&self, buffer: &[u8]) -> Result<(), AlkTypeError>;     // D-BAST-006
    pub fn validation_plan(&self) -> &Arc<ValidationPlan>;  // compiled constraints (ADR-012 §3)
}

compile takes &Value (not &mut Value) — BAST needs no in-place normalize_refs. root_name selects which $defs entry is the top-level type (D-BAST-001). json_schema is the optional consumer-provided standard JSON Schema for the validate_json path (D-BAST-007); pass None when JSON validation is not needed. The engine retains a clone of the BAST Value so sequential_reader and read_field can re-parse the typed tree on demand without lifetime entanglement with the caller's Value.

The Layout::Packed variant stores only the LayoutBuilder (write-side). The SequentialReader (read-side) is not stored — it has mutable cursor state that the consumer owns, so sequential_reader() constructs a fresh reader on each call (ADR-007).

The read_field/write_field methods on AlkTypeEngine are the aligned-mode data-access API — see data-access.md §"Higher-level read/write".

build_validator

src/validation.rs exposes one function:

pub fn build_validator(schema: &Value) -> Result<jsonschema::Validator, AlkTypeError>;

Under the pivot this is repurposed (D-BAST-007): it builds a standard jsonschema::Validator from a consumer-provided plain JSON Schema — no custom keywords, no BAST involvement. The engine calls it internally during compile when json_schema is Some. Consumers that only need a one-off validator may call jsonschema::options().build(schema) directly; build_validator exists so the engine's error mapping (jsonschema build error → AlkTypeError::Schema) is reused.

The v0.1.0 custom-keyword build_validator (registered 19 with_keyword(...) factories) is removed.

AlkTypeError

A single AlkTypeError enum covers all error conditions across the engine's phases (schema parsing, offset computation, read/write) plus validation. Decided in ADR-004; the variant shapes are unchanged under the pivot (D-BAST-009).

pub enum AlkTypeError {
    /// Schema parsing errors (malformed BAST, dangling $ref, unknown kind).
    Schema(String),
    /// Offset computation errors (field not found, unsupported type).
    Offset { field_path: String, reason: String },
    /// Read/write errors (buffer too short, invalid UTF-8, value out of range).
    Access { field_path: String, reason: String },
    /// Validation errors (both paths — D-BAST-009 uniform payload).
    Validation(jsonschema::ValidationError<'static>),
}

• Schema — for errors during AlkTypeEngine::compile() or any BAST-walking path. Malformed BAST, missing $defs, unknown kind string, dangling $ref, empty mapping, etc.
• Offset — for errors during offset computation. Field not found in the BAST tree, type not supported for offset computation. Carries the field path.
• Access — for errors during read/write. Buffer too short, invalid UTF-8 in a string field, value out of range for the target type. Carries the field path.
• Validation — wraps a jsonschema::ValidationError<'static>. On the validate_json path, this is the jsonschema crate's own structured error. On the validate_bytes path, it is constructed via jsonschema::ValidationError::custom from the BAST-native validator's path + reason string. The 'static lifetime is correct — the payload owns its data.

Field-path-carrying errors

Read/write and offset errors include the field path for debugging:

Err(AlkTypeError::Access {
    field_path: "header.version".to_string(),
    reason: "buffer too short: need 4 bytes at offset 12, have 2".to_string(),
})

This makes debugging binary format issues tractable — the error tells you exactly which field failed and why.

Validation Timing

Load time: AlkTypeEngine::compile()

The expensive work happens once at schema load time:

1. Parse the BAST document into the typed tree (BastDoc::new).
2. Parse the root struct's "endian" annotation.
3. Compile the ValidationPlan — the value-domain constraint tree, with eager $ref resolution. Its compile walk rejects cyclic $ref graphs with a clean Schema error before the layout computation: the layout walkers' struct/union recursion has no cycle guard, so ordering the plan compile first is what keeps a self-referential (malicious or accidental) document a handleable error, not a stack overflow.
4. Compute the layout (LayoutBuilder for packed, OffsetMap for aligned).
5. If json_schema is Some, build the standard jsonschema::Validator via validation::build_validator.

The result is an AlkTypeEngine that can be used for repeated operations. The validation half is pre-built: the engine holds an Arc<ValidationPlan> and walks it per buffer without re-touching the BAST document (ADR-012 §3).

Access time: engine.validate_bytes(&[u8])

For binary-layout schemas, validate_bytes runs the two phases in sequence (D-BAST-006):

1. Materialize Value from bytes. materialize::materialize_packed or materialize::materialize_aligned walks the buffer against the BAST typed tree and the engine's Endian, producing a serde_json::Value tree. Composites are recursed into (Struct → object of field values; Array → array of element values; Union → dispatch then recurse; Record → object of key/value entries). The read phase reuses the existing data-access functions and returns AlkTypeError::Access (with field paths) on read failures.
2. Validate the Value against the ValidationPlan. The materialized Value is walked against the compiled constraint tree (engine.validation_plan().validate(&value)), producing AlkTypeError::Validation on the first violated value-domain constraint. This is the ADR-012 §3 end state: the only per-buffer schema-touching step is the materialize half (the bytes must be decoded against the tree); the validation half is plan-fast.

Mode dispatch:

• Packed mode — materializes fields in declaration order.
• Aligned mode — uses the OffsetMap to read fields at their computed offsets.

Both modes produce the same Value form; the validation plan is mode-agnostic.

Access time: engine.validate_json(&Value) / engine.is_valid_json(&Value)

Validation is opt-in per operation. The consumer calls engine.validate_json(instance) when validation is desired, or engine.is_valid_json(instance) for a boolean check. The jsonschema::Validator is already compiled — these are fast checks against the compiled validator.

pub fn validate_json(&self, instance: &Value) -> Result<(), AlkTypeError>;
pub fn is_valid_json(&self, instance: &Value) -> bool;

The argument is a serde_json::Value (the JSON representation of the data), not a raw byte buffer. validate_json validates against the consumer-provided JSON Schema supplied at compile time (D-BAST-007); the BAST document is not involved. If no JSON Schema was supplied, validate_json returns AlkTypeError::Schema and is_valid_json returns false.

High-throughput paths can skip validation. Security-sensitive paths (parsing incoming frames from untrusted peers) can validate every frame. The choice is the consumer's.

When to use which entry point

Entry point	Schema form	Input form	When
validate_json(&Value)	Consumer-provided standard JSON Schema	Already-parsed serde_json::Value	Call's JSON payloads (OperationSpec.input_schema); anything off serde_json::from_slice / from_str
validate_bytes(&[u8])	BAST document (binary layout)	Raw &[u8] buffer	Channels' 8-byte chunk header; future binary call frames; SFTP packet buffers; metatensor index structs

validate_bytes requires the engine's root type to be a struct (the layout engine enforces this) — it materializes Value via the layout engine, which needs binary-layout semantics. For pure JSON payloads, the consumer uses serde_json::from_slice then validate_json. See ADR-010 and ADR-VAL-SPLIT.

What validate_bytes is not

• Not framing-aware. It validates the bytes of one schema instance. It does not strip length prefixes, parse [length: u32][payload] framing, or handle multiple frames in a buffer. That's the consumer's job. alktype validates what one schema describes; it does not parse the wire envelope around it.
• Not a Validator trait. Two methods on one struct, not a trait abstraction. See ADR-010 §"Not a Validator trait abstraction".

Relationship to Read/Write

Validation and data access are independent operations on the same data. The consumer can:

1. Validate the bytes of a buffer to ensure it conforms to the BAST document's value constraints.
2. Read fields from the binary buffer at computed offsets.
3. Both — validate first, then read (defense in depth).

The engine does not couple validation and access. A consumer that trusts its data source can skip validation and go straight to read/write. A consumer that parses untrusted input can validate first, then access the binary buffer.

Design Decisions

Decision	ADR	Summary
Two-validator model (BAST-native + standard jsonschema)	ADR-VAL-SPLIT	validate_bytes uses the compiled ValidationPlan; validate_json uses a standard jsonschema::Validator from a consumer-provided JSON Schema; D-BAST-006/007/009
Error handling and validation strategy	ADR-004	AlkTypeError enum; load-time build, access-time check; field-path-carrying errors; jsonschema ValidationError wrapping
Generalized validation — validate_bytes	ADR-010	Single-call binary-buffer validation (materialize Value from bytes, then validate); two methods on one struct, not a trait
Compiled ValidationPlan	ADR-012	The value-domain constraint tree is compiled once at compile (eager $ref resolution, cycle rejection) and walked per buffer; Hash + Eq + fingerprint(); retires the interpretive BastDoc walk
BAST format	ADR-BAST	The BAST document is the complete binary-format spec (layout + value constraints)

Open Questions

None specific to validation. The three alktype OQs (OQ-001, OQ-002, OQ-003) are about layout, platform support, and schema construction — not validation. OQ-003 is resolved by ADR-009; see builder.md.

References

• bast-format.md §Validation Model — the normative validation model
• ADR-VAL-SPLIT — the two-validator decision
• ADR-012 — the ValidationPlan decision (§3)
• ADR-004 — error handling and validation strategy
• ADR-010 — validate_bytes (the collapsed two-step dance)
• schema-layer.md — the BAST parser that the plan compiler consumes
• data-access.md — read/write functions and the materializer that produce the Value the plan checks
• src/validation_plan.rs — the compiled ValidationPlan implementation
• src/bast_validation.rs — the one-shot wrapper (validate_value) and shared error helpers
• src/validation.rs — the build_validator helper