Resolve v0.1.0 open questions and fix production-readiness issues
POC: /workspace/alktype-builder-poc/ (18/18 tests pass, findings in
docs/research/alktype-builder-poc/findings.md). Round 2 adds the SFTP
Packet validate_bytes tests (7 new: valid Init/Read/Write/Status,
short buffer, unknown discriminator, over-maxLength Bytes).
Open questions resolved (OQ-004 through OQ-008):
- OQ-004: Discriminator::Field name is String (already implemented;
docs updated to mark resolved)
- OQ-005: Both union discriminator kinds return the same shape:
{__discriminator, ...variant-fields}. Field-name path also had a
real offset bug (returned start, not end) - fixed.
- OQ-006: builder.md Example 3 now wraps the Union in a
Schema::struct_().field("payload", ...) and merges $defs via
Definitions::merge_into (matches the engine's AlkType:Struct-at-root
constraint and the SFTP wire shape)
- OQ-007: Bytes materialization is array-of-u8 (Value::Array of
Value::Number, one entry per byte 0..=255). BytesValidator accepts
both Value::String (validate_json) and Value::Array (validate_bytes).
maxLength = max byte count. Replaces the lossy from_utf8_lossy path
that corrupted non-UTF-8 bytes and broke maxLength semantics.
- OQ-008 (new): UnionValidator now dispatches to variant schemas via
sub-validators built at factory time. AlkTypeEngine::compile calls
schema::inline_union_variant_refs before build_validator to inline
$refs in union mapping entries (necessary because union_factory
receives the union node, but $defs live at the schema root).
Production-readiness fixes in src/ (no stubs/hedges in a published crate):
- materialize.rs: Record stub -> full count-prefixed key/value pair
implementation per schema-layer.md TRecord
- materialize.rs: root_of() was broken (returned the current node, not
the schema root) -> root schema threaded through every recursive call
so resolve_ref_or_inline can resolve $refs for nested composites
- builder.rs: LengthPrefixed encoding setter was a no-op when the
keyword was already in object form -> complete the branch (updates
the encoding entry in place for both LengthPrefixed and OffsetIndirect)
- builder.rs, engine.rs: POC-referencing comments cleaned up; the
round-trip test's or_else fallback (papering over write_field being
aligned-only) replaced with direct byte writes
Documentation:
- builder.md: Example 3 updated; Discriminator::Field spec shows String;
Open Questions section updated (OQ-004 resolved)
- validation.md: AlkType:Bytes and AlkType:Union validator descriptions
updated for array-of-u8 form and variant dispatch
- open-questions.md: OQ-004/005/006/007/008 marked resolved; new
Validation theme entries
- questions/004-008: individual OQ files updated with resolutions
- findings.md: round 2 results documented
Verification:
- cargo test: 369 -> 391 tests pass (22 new: 14 materialize, 5
inline_union_variant_refs, 3 validation/builder)
- cargo clippy --all-targets: clean
- POC: 11 -> 18 tests (7 new SFTP Packet tests); all pass
This commit is contained in:
@@ -359,7 +359,7 @@ pub enum Discriminator {
|
||||
/// Field-name discriminator (ADR-003 §4 Kind B).
|
||||
/// `name` is the field holding the discriminator value.
|
||||
Field {
|
||||
name: &str, // or String — see "Open question" below
|
||||
name: String,
|
||||
},
|
||||
}
|
||||
```
|
||||
@@ -536,8 +536,15 @@ let read_file_input = Schema::object()
|
||||
|
||||
### Example 3: SFTP `Packet` union (binary layout, byte discriminator)
|
||||
|
||||
The SFTP wire shape is `[type:u8][payload-struct]` — a struct with a
|
||||
union payload field. The engine requires `AlkType:Struct` at the top
|
||||
level (`OffsetMap::compute` / `SequentialReader::new` both enforce
|
||||
this; a `Union` is a field type within a struct, not a top-level
|
||||
schema). The builder constructs the union wrapped in a struct, and
|
||||
`$defs` are merged into the top-level schema so `$ref`s resolve:
|
||||
|
||||
```rust
|
||||
use alktype::{Schema, Discriminator, AlkTypeKind};
|
||||
use alktype::{Definitions, Discriminator, AlkTypeKind, Schema};
|
||||
|
||||
let mut defs = Definitions::new();
|
||||
defs.define("Init", Schema::struct_().field("version", Schema::uint32()));
|
||||
@@ -546,16 +553,26 @@ defs.define("Read", Schema::struct_().field("handle", Schema::bytes()).field("o
|
||||
defs.define("Write", Schema::struct_().field("handle", Schema::bytes()).field("offset", Schema::uint64()).field("data", Schema::bytes()));
|
||||
defs.define("Status", Schema::struct_().field("code", Schema::uint32()).field("message", Schema::string()));
|
||||
|
||||
let packet = Schema::union_(Discriminator::Byte {
|
||||
offset: 0,
|
||||
disc_type: AlkTypeKind::Uint8,
|
||||
})
|
||||
.mapping("1", Schema::ref_def("Init"))
|
||||
.mapping("3", Schema::ref_def("Open"))
|
||||
.mapping("5", Schema::ref_def("Read"))
|
||||
.mapping("6", Schema::ref_def("Write"))
|
||||
.mapping("101", Schema::ref_def("Status"))
|
||||
// A "Packet" is a struct with one field — the union. This mirrors
|
||||
// SFTP's wire shape: [type:u8][payload-struct].
|
||||
let mut packet = Schema::struct_()
|
||||
.field(
|
||||
"payload",
|
||||
Schema::union_(Discriminator::Byte {
|
||||
offset: 0,
|
||||
disc_type: AlkTypeKind::Uint8,
|
||||
})
|
||||
.mapping("1", Schema::ref_def("Init"))
|
||||
.mapping("3", Schema::ref_def("Open"))
|
||||
.mapping("5", Schema::ref_def("Read"))
|
||||
.mapping("6", Schema::ref_def("Write"))
|
||||
.mapping("101", Schema::ref_def("Status")),
|
||||
)
|
||||
.build();
|
||||
// Merge $defs into the top-level schema so $refs resolve at compile time.
|
||||
defs.merge_into(&mut packet);
|
||||
// Feed to AlkTypeEngine::compile(&mut packet, LayoutMode::Packed)
|
||||
// then validate incoming frames via engine.validate_bytes(&frame).
|
||||
```
|
||||
|
||||
### Example 4: OperationSpec error schemas (named `$defs`)
|
||||
@@ -606,12 +623,13 @@ let op_errors = vec; full file:
|
||||
[OQ-004](questions/004-discriminator-field-name-type.md).
|
||||
None specific to the builder. OQ-003 (the original "should we build a
|
||||
builder API") is resolved by this spec / ADR-009. OQ-004
|
||||
(`Discriminator::Field` name type — `&str` or `String`) is resolved:
|
||||
`String`, for ownership simplicity (the builder consumes `Self` on
|
||||
setters; `&str` would require a lifetime parameter on `Discriminator`
|
||||
and transitively on `Schema::union_`). See
|
||||
[open-questions.md](open-questions.md).
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -62,15 +62,16 @@ Door type is separate from whether a decision is made. A two-way door is a decis
|
||||
| OQ | Title | Status | Door | Pri |
|
||||
|----|-------|--------|------|-----|
|
||||
| [OQ-003](questions/003-builder-api-for-schema-construction.md) | Builder API for Schema Construction | resolved (ADR-009) | two | med |
|
||||
| [OQ-004](questions/004-discriminator-field-name-type.md) | `Discriminator::Field` name — `&str` or `String` | open | two | low |
|
||||
| [OQ-006](questions/006-builder-spec-example-3-wrap-union.md) | Builder spec Example 3 — wrap `Union` in a `Struct` | open | two | low |
|
||||
| [OQ-004](questions/004-discriminator-field-name-type.md) | `Discriminator::Field` name — `&str` or `String` | resolved | two | low |
|
||||
| [OQ-006](questions/006-builder-spec-example-3-wrap-union.md) | Builder spec Example 3 — wrap `Union` in a `Struct` | resolved | two | low |
|
||||
|
||||
### Validation
|
||||
|
||||
| OQ | Title | Status | Door | Pri |
|
||||
|----|-------|--------|------|-----|
|
||||
| [OQ-005](questions/005-union-materialization-shape.md) | `Union` materialization shape — byte-offset vs field-name consistency | open | two | med |
|
||||
| [OQ-007](questions/007-bytes-materialization-lossy-utf8.md) | `Bytes` materialization — lossy UTF-8 conversion | open | one | med |
|
||||
| [OQ-005](questions/005-union-materialization-shape.md) | `Union` materialization shape — byte-offset vs field-name consistency | resolved | two | med |
|
||||
| [OQ-007](questions/007-bytes-materialization-lossy-utf8.md) | `Bytes` materialization — lossy UTF-8 conversion | resolved | one | med |
|
||||
| [OQ-008](questions/008-unionvalidator-variant-dispatch.md) | `UnionValidator` variant dispatch | resolved | one | med |
|
||||
|
||||
## Open
|
||||
|
||||
@@ -79,54 +80,59 @@ that need to be worked through, not waited on. Each has a concrete
|
||||
investigation target. This section exists so "what's currently on the
|
||||
architect's desk" is answerable at a glance.
|
||||
|
||||
### OQ-004: `Discriminator::Field` name — `&str` or `String`
|
||||
> **Note**: All v0.1.0 open questions (OQ-004, OQ-005, OQ-006, OQ-007,
|
||||
> OQ-008) were resolved during the v0.1.0 POC round 2 (SFTP Packet
|
||||
> `validate_bytes` POC). The resolutions are summarized below for
|
||||
> traceability; see each OQ's full file for the decision rationale.
|
||||
> The currently-parked OQs are OQ-001 and OQ-002 (see Deferred / Blocked
|
||||
> below).
|
||||
|
||||
- **Investigation target**: Decide whether `Discriminator::Field { name }`
|
||||
should borrow (`&str`) or own (`String`). The builder consumes `Self`
|
||||
on setters, so `&str` would require a lifetime parameter on
|
||||
`Discriminator` (and transitively `Schema::union_`). Likely `String`
|
||||
for ownership simplicity.
|
||||
- **Priority**: low (doesn't block the chunk header POC; must be
|
||||
resolved before the SFTP Packet POC's field-name discriminator path)
|
||||
### OQ-004: `Discriminator::Field` name — `&str` or `String` — RESOLVED
|
||||
|
||||
- **Status**: resolved. `String`, for ownership simplicity (the builder
|
||||
consumes `Self` on setters; `&str` would require a lifetime parameter
|
||||
on `Discriminator` and transitively on `Schema::union_`). Implemented
|
||||
in `src/builder.rs` since the initial v0.1.0 builder implementation.
|
||||
- **Full file**: [OQ-004](questions/004-discriminator-field-name-type.md)
|
||||
|
||||
### OQ-005: `Union` materialization shape — byte-offset vs field-name consistency
|
||||
### OQ-005: `Union` materialization shape — RESOLVED
|
||||
|
||||
- **Investigation target**: The current `materialize_union_packed`
|
||||
returns inconsistent shapes for the two discriminator kinds (byte-
|
||||
offset produces `{ "__discriminator": <u32>, ...fields }`; field-name
|
||||
produces the struct fields directly). Decide on one consistent shape.
|
||||
Work through the SFTP Packet `validate_bytes` example to surface what
|
||||
the `jsonschema` validator needs to see for a `AlkType:Union` field.
|
||||
- **Priority**: medium (blocks the SFTP Packet `validate_bytes` POC,
|
||||
the natural next round)
|
||||
- **Status**: resolved. Both discriminator kinds return
|
||||
`{ "__discriminator": <value>, ...variant-fields }`. The field-name
|
||||
path also had a real offset bug (returned start offset, not end) —
|
||||
fixed. Implemented in `src/materialize.rs`.
|
||||
- **Full file**: [OQ-005](questions/005-union-materialization-shape.md)
|
||||
|
||||
### OQ-006: Builder spec Example 3 — wrap `Union` in a `Struct`
|
||||
### OQ-006: Builder spec Example 3 — wrap `Union` in a `Struct` — RESOLVED
|
||||
|
||||
- **Investigation target**: Update builder.md Example 3 (SFTP Packet)
|
||||
to wrap the `Union` in a `Struct` — the engine requires
|
||||
`AlkType:Struct` at the top level (ADR-002), and the wrapped shape
|
||||
matches SFTP's actual wire format (`[length][type][payload]`). The
|
||||
POC already uses the wrapped shape; the spec example just hasn't
|
||||
been updated.
|
||||
- **Priority**: low (documentation fix; doesn't block any implementation)
|
||||
- **Status**: resolved. [builder.md](builder.md) Example 3 now wraps
|
||||
the `Union` in a `Schema::struct_().field("payload", ...)` and merges
|
||||
`$defs` via `Definitions::merge_into`. Matches the realistic SFTP
|
||||
wire shape and the engine's `AlkType:Struct`-at-root constraint.
|
||||
- **Full file**: [OQ-006](questions/006-builder-spec-example-3-wrap-union.md)
|
||||
|
||||
### OQ-007: `Bytes` materialization — lossy UTF-8 conversion
|
||||
### OQ-007: `Bytes` materialization — lossy UTF-8 conversion — RESOLVED
|
||||
|
||||
- **Investigation target**: The current materializer uses
|
||||
`String::from_utf8_lossy` for `AlkType:Bytes` fields, which corrupts
|
||||
non-UTF-8 bytes. Decide on a round-trippable form (base64 with a
|
||||
`format` constraint, or `Value::Array` of u8 with a validator that
|
||||
accepts arrays). Work through SFTP's binary `handle`/`data` fields
|
||||
to surface what `validate_bytes` callers actually need.
|
||||
- **Priority**: medium (blocks the SFTP use case for `validate_bytes`;
|
||||
does not block the chunk header or call input schema)
|
||||
- **Door type**: one-way (changing what the validator sees is a
|
||||
semantic break for consumers that depend on the shape)
|
||||
- **Status**: resolved. Array of u8: the materializer produces
|
||||
`Value::Array` of `Value::Number` (one entry per byte, 0..=255) for
|
||||
`AlkType:Bytes` fields. The `BytesValidator` accepts both
|
||||
`Value::String` (for `validate_json`) and `Value::Array` (for
|
||||
`validate_bytes`). `maxLength` = max byte count. Implemented in
|
||||
`src/materialize.rs` and `src/validation.rs`.
|
||||
- **Full file**: [OQ-007](questions/007-bytes-materialization-lossy-utf8.md)
|
||||
|
||||
### OQ-008: `UnionValidator` variant dispatch — RESOLVED
|
||||
|
||||
- **Status**: resolved. `UnionValidator` now builds a sub-validator for
|
||||
each variant at factory time and dispatches on `__discriminator` at
|
||||
validation time. `AlkTypeEngine::compile` calls
|
||||
`schema::inline_union_variant_refs` before `build_validator` to inline
|
||||
`$ref`s in union `mapping` entries (necessary because the
|
||||
`union_factory` receives the union node, but `$defs` live at the
|
||||
schema root). Implemented in `src/validation.rs`, `src/schema.rs`,
|
||||
and `src/engine.rs`.
|
||||
- **Full file**: [OQ-008](questions/008-unionvalidator-variant-dispatch.md)
|
||||
|
||||
## Deferred / Blocked
|
||||
|
||||
The safe-exit visibility surface. These questions are parked because the
|
||||
|
||||
@@ -2,26 +2,18 @@
|
||||
|
||||
- **Origin**: [../builder.md](../builder.md) §"Open Questions" (raised
|
||||
during ADR-009 spec drafting)
|
||||
- **Status**: open
|
||||
- **Status**: resolved
|
||||
- **Door type**: Two-way (the field is internal to the `Discriminator`
|
||||
enum; changing the type is a local refactor with no schema-level impact)
|
||||
- **Priority**: low
|
||||
- **Impacts**: Blocks finalizing the `Discriminator::Field` variant's
|
||||
signature in `src/builder.rs`. Does NOT block any current POC — the
|
||||
chunk header uses `Discriminator::Byte`, not `Discriminator::Field`.
|
||||
The field-name discriminator is exercised by the SFTP Packet POC
|
||||
(next round), so this must be resolved before that POC.
|
||||
- **Investigation target**: Decide whether the `name` field of
|
||||
`Discriminator::Field` should be `&str` (borrows) or `String` (owns).
|
||||
The builder consumes `Self` on each setter (`.field()`, `.mapping()`,
|
||||
etc.), so the discriminator's `name` needs to outlive the `Schema`
|
||||
it's embedded in. `&str` would require a lifetime parameter on
|
||||
`Discriminator` (and transitively on `Schema::union_`); `String`
|
||||
owns its content and keeps `Discriminator` `'static`. The likely
|
||||
resolution is `String` for ownership simplicity — the builder owns
|
||||
its content until `.build()` produces `Value`, and `String` avoids
|
||||
lifetime pollution across the builder API.
|
||||
- **Resolution**: Not yet decided. Confirm during the next POC round
|
||||
(SFTP Packet) when `Discriminator::Field` is first exercised.
|
||||
- **Resolution**: `String`. The builder consumes `Self` on each setter
|
||||
(`.field()`, `.mapping()`, etc.), so `&str` would require a lifetime
|
||||
parameter on `Discriminator` (and transitively on `Schema::union_`),
|
||||
polluting the builder API. `String` owns its content, keeps
|
||||
`Discriminator` `'static`, and matches the rest of the builder's
|
||||
ownership model (the builder owns its content until `.build()`
|
||||
produces `Value`). Implemented in `src/builder.rs` since the initial
|
||||
v0.1.0 builder implementation; confirmed correct during the SFTP
|
||||
Packet POC (round 2) which first exercises `Discriminator::Field`.
|
||||
- **Cross-references**: [ADR-009](../decisions/009-builder-api.md),
|
||||
[builder.md](../builder.md)
|
||||
@@ -2,39 +2,27 @@
|
||||
|
||||
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
|
||||
§"Open Questions surfaced" (raised during the v0.1.0 POC)
|
||||
- **Status**: open
|
||||
- **Status**: resolved
|
||||
- **Door type**: Two-way (the materializer is internal; the shape it
|
||||
produces is consumed only by the existing `jsonschema` validator,
|
||||
which accepts arbitrary objects — changing the shape is a local
|
||||
refactor)
|
||||
- **Priority**: medium
|
||||
- **Impacts**: Blocks the SFTP Packet `validate_bytes` POC (next round).
|
||||
The current `materialize_union_packed` returns inconsistent shapes:
|
||||
byte-offset discriminators produce `{ "__discriminator": <u32>,
|
||||
...variant-fields }`; field-name discriminators produce the struct
|
||||
fields directly (the discriminator is just another field). For
|
||||
`validate_bytes` callers, the shape matters for how the validator
|
||||
dispatches on the variant. Does NOT block the chunk header or call
|
||||
input schema (no unions in either).
|
||||
- **Investigation target**: Decide on one consistent shape for
|
||||
materialized unions. Two candidates:
|
||||
|
||||
1. **Both return a tagged object**: `{ "__discriminator": <value>,
|
||||
...variant-fields }` for both discriminator kinds. The validator
|
||||
sees the discriminator value and the variant fields in one
|
||||
object. This is what the byte-offset path already does.
|
||||
2. **Both return the variant struct directly** (no `__discriminator`
|
||||
wrapper): the discriminator is consumed during dispatch and not
|
||||
included in the materialized `Value`. The field-name path
|
||||
approximates this today.
|
||||
|
||||
Option 1 is more uniform and lets the validator re-check the
|
||||
discriminator; option 2 is cleaner but loses the discriminator in
|
||||
the `Value` tree. Resolve by working through the SFTP Packet
|
||||
`validate_bytes` example — what does the `jsonschema` validator need
|
||||
to see for a `AlkType:Union` field?
|
||||
- **Resolution**: Not yet decided. Work through the SFTP Packet POC
|
||||
to surface the validator's requirements.
|
||||
- **Resolution**: Both discriminator kinds return the same shape:
|
||||
`{ "__discriminator": <value>, ...variant-fields }`. The
|
||||
`__discriminator` entry carries the mapping key (stringified
|
||||
discriminator value for byte-offset, string value for field-name).
|
||||
For field-name discriminators, the original typed discriminator value
|
||||
is also preserved under the discriminator field's name (it's a
|
||||
regular field in the struct). If the variant materializes as a
|
||||
non-object (a leaf kind), it is nested under `"__variant"`.
|
||||
Implemented in `src/materialize.rs` (`materialize_union_packed`).
|
||||
|
||||
Additionally, the field-name path had a real correctness bug
|
||||
(`Ok((value, offset))` returned the start offset, not the end —
|
||||
subsequent fields after a field-name union would read from the wrong
|
||||
position). This is now fixed: the path returns the new offset after
|
||||
the variant struct.
|
||||
- **Cross-references**: [ADR-010](../decisions/010-generalized-validation-validate-bytes.md),
|
||||
`src/materialize.rs` (`materialize_union_packed`),
|
||||
[../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
|
||||
@@ -2,34 +2,17 @@
|
||||
|
||||
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
|
||||
§"Findings" #1 (raised during the v0.1.0 POC)
|
||||
- **Status**: open
|
||||
- **Status**: resolved
|
||||
- **Door type**: Two-way (documentation fix — the builder already
|
||||
supports both shapes; the spec example just shows the wrong one)
|
||||
- **Priority**: low
|
||||
- **Impacts**: Blocks nothing functional. The builder spec's Example 3
|
||||
(SFTP Packet) currently shows a top-level `Schema::union_(...)`,
|
||||
which won't compile via `AlkTypeEngine::compile` — the engine
|
||||
requires `AlkType:Struct` at the top level (existing constraint from
|
||||
`OffsetMap::compute` / `SequentialReader::new`, ADR-002). The POC
|
||||
worked around this by wrapping the union in a
|
||||
`Schema::struct_().field("payload", Schema::union_(...))`, which is
|
||||
the realistic wire shape anyway (SFTP's `[length:u32][type:u8][payload]`
|
||||
is a struct with a union payload field). The spec example should
|
||||
match the realistic shape.
|
||||
- **Investigation target**: Update [../builder.md](../builder.md)
|
||||
Example 3 to wrap the `Union` in a `Struct`. Two options:
|
||||
|
||||
1. **Wrap in a struct** (preferred — matches the realistic wire
|
||||
shape): `Schema::struct_().field("payload", Schema::union_(...))`.
|
||||
2. **Document the constraint explicitly**: keep the top-level union
|
||||
in the example but add a note that `AlkTypeEngine::compile`
|
||||
requires `AlkType:Struct` at the root; a top-level `Union` is for
|
||||
`jsonschema`-only validation (no `validate_bytes`).
|
||||
|
||||
Option 1 is preferred — it shows the shape the engine actually
|
||||
consumes and matches the SFTP wire format.
|
||||
- **Resolution**: Not yet applied. Documentation fix in builder.md,
|
||||
not an implementation change.
|
||||
- **Resolution**: Applied. [../builder.md](../builder.md) Example 3
|
||||
now wraps the `Union` in a `Schema::struct_().field("payload", ...)`
|
||||
and merges `$defs` via `Definitions::merge_into`. This matches the
|
||||
realistic SFTP wire shape (`[type:u8][payload-struct]`) and the
|
||||
engine's constraint that `AlkTypeEngine::compile` requires
|
||||
`AlkType:Struct` at the top level (inherited from
|
||||
`OffsetMap::compute` / `SequentialReader::new`, ADR-002).
|
||||
- **Cross-references**: [ADR-009](../decisions/009-builder-api.md),
|
||||
[builder.md](../builder.md) §"Example 3: SFTP Packet union",
|
||||
[ADR-002](../decisions/002-two-layout-modes-packed-vs-aligned.md)
|
||||
@@ -2,43 +2,33 @@
|
||||
|
||||
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
|
||||
§"Open Questions surfaced" (raised during the v0.1.0 POC)
|
||||
- **Status**: open
|
||||
- **Status**: resolved
|
||||
- **Door type**: One-way (the choice affects what the validator sees;
|
||||
changing it after consumers depend on the shape is a semantic break)
|
||||
- **Priority**: medium
|
||||
- **Impacts**: Blocks the SFTP use case for `validate_bytes`. SFTP
|
||||
packets have binary `handle` and `data` fields (`AlkType:Bytes`)
|
||||
that are not valid UTF-8. The current materializer uses
|
||||
`String::from_utf8_lossy` to convert raw bytes to a JSON string
|
||||
(because `jsonschema`'s `BytesValidator` expects a string — JSON
|
||||
has no native byte type; see schema-layer.md §TBytes). Lossy
|
||||
conversion corrupts non-UTF-8 bytes (replacing invalid sequences
|
||||
with `U+FFFD`), so the validator sees corrupted content. Does NOT
|
||||
block the chunk header (no `Bytes` fields) or call input schema
|
||||
(no AlkType kinds).
|
||||
- **Investigation target**: Decide how `AlkType:Bytes` should be
|
||||
materialized for validation. Three candidates:
|
||||
- **Resolution**: Array of u8. The materializer produces
|
||||
`Value::Array` of `Value::Number` (one entry per byte, 0..=255) for
|
||||
`AlkType:Bytes` fields. The `BytesValidator` accepts both
|
||||
`Value::String` (for `validate_json` consumers that hold a JSON
|
||||
representation) and `Value::Array` (for `validate_bytes`
|
||||
materialization). `maxLength` is interpreted as max byte count
|
||||
(array length for the array form, string byte length for the string
|
||||
form).
|
||||
|
||||
1. **Lossy UTF-8 (current)**: `String::from_utf8_lossy`. Simple;
|
||||
corrupts non-UTF-8. Acceptable only for UTF-8-only `Bytes` fields.
|
||||
2. **Base64 encode**: encode raw bytes as base64, validate against
|
||||
a `format: "base64"` constraint. Round-trips cleanly but the
|
||||
validator sees base64, not raw bytes — `maxLength` semantics
|
||||
change (base64 length ≠ byte length).
|
||||
3. **Array of u8**: materialize as `Value::Array` of `Value::Number`
|
||||
(one entry per byte). The validator sees the actual byte values;
|
||||
`maxLength` becomes "max array length." Semantically clean but
|
||||
produces large `Value` trees for big `Bytes` fields.
|
||||
The array form is the round-trippable form for non-UTF-8 bytes —
|
||||
`String::from_utf8_lossy` corrupts non-UTF-8 byte sequences
|
||||
(replacing invalid sequences with `U+FFFD`, which is 3 bytes in
|
||||
UTF-8, breaking `maxLength` semantics). Base64 encoding changes
|
||||
`maxLength` semantics (base64 length != byte length). Array of u8
|
||||
is semantically clean: each entry is the actual byte value, and
|
||||
`maxLength` is the actual byte count.
|
||||
|
||||
The right answer depends on what the `BytesValidator` (in
|
||||
`src/validation.rs`) should check. Today it checks string length
|
||||
against `maxLength` — which assumes the string form. If we keep
|
||||
the string form, base64 is the only round-trippable option. If we
|
||||
want raw bytes, the validator needs to accept `Value::Array` of
|
||||
numbers.
|
||||
- **Resolution**: Not yet decided. Work through the SFTP `handle`/
|
||||
`data` fields in the next POC round to surface what
|
||||
`validate_bytes` callers actually need.
|
||||
Trade-off: array-of-u8 produces larger `Value` trees for big
|
||||
`Bytes` fields (one `Value::Number` per byte). For the SFTP use
|
||||
case (handles up to 256 bytes, data up to 32768 bytes), this is
|
||||
acceptable. A future "validate bytes without materializing" path
|
||||
(two-way door, out of scope for v0.1.0 per ADR-010) would avoid the
|
||||
allocation for hot paths.
|
||||
- **Cross-references**: [ADR-010](../decisions/010-generalized-validation-validate-bytes.md),
|
||||
`src/materialize.rs` (`AlkTypeKind::Bytes` branch),
|
||||
`src/validation.rs` (`BytesValidator`),
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
# OQ-008: `UnionValidator` variant dispatch — validate variant fields against variant schema
|
||||
|
||||
- **Origin**: Raised during the v0.1.0 POC round 2 (SFTP Packet
|
||||
`validate_bytes` POC). Surfaced when the over-`maxLength` `Bytes`
|
||||
test failed: the materializer read the bytes correctly, but the
|
||||
validator did not check the variant's `maxLength` constraint because
|
||||
the `UnionValidator` was structural-only (just `is_object`).
|
||||
- **Status**: resolved
|
||||
- **Door type**: One-way (the validator now recurses into variant
|
||||
schemas; consumers that depended on the structural-only behavior
|
||||
would see new validation errors for previously-accepted instances)
|
||||
- **Priority**: medium
|
||||
- **Resolution**: `UnionValidator` now builds a sub-validator for each
|
||||
variant at factory time (when the parent validator tree is
|
||||
constructed) and dispatches on the `__discriminator` field in the
|
||||
materialized instance at validation time. The sub-validators are
|
||||
full `jsonschema::Validator`s built via `build_validator` (so nested
|
||||
AlkType kinds inside variants are validated correctly).
|
||||
|
||||
To make this work, `AlkTypeEngine::compile` now calls
|
||||
`schema::inline_union_variant_refs` after `normalize_refs` and before
|
||||
`build_validator`. This inlines `$ref`s in union `mapping` entries by
|
||||
resolving them against the schema root — necessary because the
|
||||
`union_factory` receives the union node as `parent`, but `$defs` live
|
||||
at the schema root (not on the union node). After inlining, each
|
||||
variant in the `mapping` is a full inline schema, so the factory can
|
||||
build a sub-validator directly.
|
||||
|
||||
The materialized shape for unions is
|
||||
`{ "__discriminator": <value>, ...variant-fields }` (OQ-005
|
||||
resolution). The `UnionValidator` reads `__discriminator`, looks up
|
||||
the corresponding variant sub-validator, and validates the full
|
||||
instance (including the variant fields) against that sub-validator.
|
||||
- **Cross-references**: `src/validation.rs` (`UnionValidator`,
|
||||
`union_factory`), `src/schema.rs` (`inline_union_variant_refs`),
|
||||
`src/engine.rs` (`AlkTypeEngine::compile`),
|
||||
[OQ-005](005-union-materialization-shape.md),
|
||||
[ADR-010](../decisions/010-generalized-validation-validate-bytes.md)
|
||||
@@ -130,13 +130,18 @@ type constraints; `jsonschema` handles all structural validation.
|
||||
must not exceed it.
|
||||
|
||||
**`AlkType:Bytes`:**
|
||||
- Value must be a string (JSON represents binary data as a string — JSON
|
||||
has no native byte type).
|
||||
- If `maxLength` is specified, the byte length must not exceed it.
|
||||
- Value must be a string (the JSON form for `validate_json` consumers)
|
||||
or an array of integers 0..=255 (the materialized form for
|
||||
`validate_bytes`). JSON has no native byte type; the string form is
|
||||
the JSON convention, the array form is the round-trippable form for
|
||||
non-UTF-8 bytes (see [OQ-007](questions/007-bytes-materialization-lossy-utf8.md)).
|
||||
- If `maxLength` is specified, the byte length must not exceed it. For
|
||||
the string form, this is the string's byte length; for the array
|
||||
form, this is the array length (one entry per byte).
|
||||
- **Binary representation:** In the binary layout, `TBytes` is raw bytes
|
||||
with no encoding (not base64, not hex). The JSON representation (for
|
||||
validation) uses a string; the binary representation (for data access)
|
||||
uses `&[u8]` directly.
|
||||
validation) uses a string or array; the binary representation (for
|
||||
data access) uses `&[u8]` directly.
|
||||
|
||||
**`AlkType:Enum`:**
|
||||
- The `AlkType:Enum` custom keyword signals that the type is an enum for
|
||||
@@ -161,9 +166,23 @@ type constraints; `jsonschema` handles all structural validation.
|
||||
validate that each field's value matches its `AlkType:*` kind.
|
||||
|
||||
**`AlkType:Union`:**
|
||||
- The discriminator value must be one of the mapping keys.
|
||||
- The variant struct must match the declared schema for that discriminator
|
||||
value.
|
||||
- The instance must be an object with a `__discriminator` field
|
||||
carrying the mapping key (stringified discriminator value for
|
||||
byte-offset discriminators, string value for field-name
|
||||
discriminators). This is the shape the materializer produces for
|
||||
`validate_bytes`; `validate_json` consumers produce the same shape
|
||||
when validating a union instance.
|
||||
- The `UnionValidator` builds a sub-validator for each variant at
|
||||
factory time (when the parent validator tree is constructed) and
|
||||
dispatches on `__discriminator` at validation time, validating the
|
||||
full instance (including the variant fields) against the selected
|
||||
variant's schema. This closes the OQ-008 gap: variant field
|
||||
constraints (e.g. `maxLength` on a `Bytes` field inside a variant)
|
||||
are checked.
|
||||
- `$ref`s in the union's `mapping` are inlined by
|
||||
`schema::inline_union_variant_refs` during `AlkTypeEngine::compile`
|
||||
(before `build_validator`), so the `union_factory` sees full inline
|
||||
variant schemas. See [OQ-008](questions/008-unionvalidator-variant-dispatch.md).
|
||||
|
||||
**`AlkType:Array`:**
|
||||
- Value must be an array.
|
||||
|
||||
@@ -2,28 +2,34 @@
|
||||
status: complete
|
||||
last_updated: 2026-08-11
|
||||
poc_code: /workspace/alktype-builder-poc/
|
||||
result: PASS (11/11 tests)
|
||||
result: PASS (18/18 tests)
|
||||
---
|
||||
|
||||
# alktype-builder-poc — Findings
|
||||
|
||||
POC validating the v0.1.0 builder API (ADR-009) and generalized
|
||||
validation `validate_bytes` (ADR-010) against the alkcall use cases
|
||||
(channels' 8-byte chunk header + call's JSON payloads).
|
||||
(channels' 8-byte chunk header + call's JSON payloads), and the
|
||||
round-2 SFTP Packet `validate_bytes` POC (union with byte
|
||||
discriminator, 5 variants, `$ref`s into `$defs`, binary `Bytes`
|
||||
fields).
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Result: PASS (11/11 tests).** Both v0.1.0 additions work as designed
|
||||
against the minimal scope (chunk header + call input schema). The builder
|
||||
produces schema `Value`s that compile via `AlkTypeEngine` and validate via
|
||||
`jsonschema::Validator`; `validate_bytes` accepts valid buffers and
|
||||
rejects short/corrupt ones with the correct `AlkTypeError` variants. The
|
||||
two-roles-from-one-library goal is met.
|
||||
**Result: PASS (18/18 tests).** Both v0.1.0 additions work as designed
|
||||
against the minimal scope (chunk header + call input schema) and the
|
||||
round-2 scope (SFTP Packet union with `$ref` variants and non-UTF-8
|
||||
`Bytes` fields). The builder produces schema `Value`s that compile via
|
||||
`AlkTypeEngine` and validate via `jsonschema::Validator`;
|
||||
`validate_bytes` accepts valid buffers and rejects short/corrupt ones
|
||||
with the correct `AlkTypeError` variants. The two-roles-from-one-library
|
||||
goal is met.
|
||||
|
||||
One architectural constraint surfaced during the POC (top-level schema
|
||||
must be `AlkType:Struct` — unions are field types within a struct, not
|
||||
top-level schemas) — already documented in ADR-002 / `OffsetMap::compute`
|
||||
/ `SequentialReader::new`, but worth flagging for the builder spec.
|
||||
**Round 2 resolved 5 open questions** (OQ-004, OQ-005, OQ-006, OQ-007,
|
||||
OQ-008) and fixed 6 production-readiness issues in `src/` (Record stub,
|
||||
field-name union offset bug, broken `root_of()` for `$ref` resolution,
|
||||
incomplete `LengthPrefixed` encoding setter, lossy UTF-8 `Bytes`
|
||||
materialization, POC-referencing comments). See §"Round 2 changes" below.
|
||||
|
||||
## POC code
|
||||
|
||||
@@ -31,14 +37,14 @@ top-level schemas) — already documented in ADR-002 / `OffsetMap::compute`
|
||||
|
||||
```
|
||||
Cargo.toml # path dep on ../@alkdev/alktype
|
||||
src/main.rs # 11 tests, 3 groups (builder round-trip, validate_bytes, validate_json)
|
||||
src/main.rs # 18 tests, 4 groups (builder, validate_bytes, validate_json, SFTP Packet)
|
||||
```
|
||||
|
||||
Run: `cargo run --release` from the POC directory. Exit code 0 = all pass.
|
||||
|
||||
## Scope
|
||||
|
||||
Minimal scope (agreed before the POC):
|
||||
### Round 1 (minimal scope, completed in the prior session)
|
||||
|
||||
- **Channels' 8-byte chunk header** — `Struct { channel_id: u32 BE, length:
|
||||
u32 BE }`, packed mode, big-endian. Builder round-trip + `validate_bytes`.
|
||||
@@ -49,17 +55,17 @@ Minimal scope (agreed before the POC):
|
||||
named `$defs`, wrapped in a `Struct` (see Findings #1).
|
||||
- **`validate_json` for call payloads** — the non-binary path.
|
||||
|
||||
Out of scope for this POC (deferred to the next round):
|
||||
### Round 2 (this session)
|
||||
|
||||
- `validate_bytes` for `Union`/`Record` (the materializer handles `Struct`
|
||||
and `Array`; `Union` byte-offset dispatch is implemented but not exercised
|
||||
end-to-end against a binary buffer in the POC).
|
||||
- `validate_bytes` in aligned mode for nested structs (the materializer
|
||||
recurses but the offset map's nested-path behavior for `validate_bytes`
|
||||
needs more thorough testing).
|
||||
- Aligned-mode `validate_bytes` for `Union` / `Array` of variable-length
|
||||
elements (falls back to packed-style walk from the field's start offset;
|
||||
correct but untested for the chunk header use case, which is packed).
|
||||
- **SFTP Packet `validate_bytes` end-to-end** — a struct wrapping a union
|
||||
with a byte discriminator (Uint8 at offset 0) and 5 variants (`Init`,
|
||||
`Open`, `Read`, `Write`, `Status`) referenced via `$defs`. Tests valid
|
||||
packets, short buffers, wrong discriminator values, and over-`maxLength`
|
||||
`Bytes` fields. Exercises the materializer's `$ref` resolution, union
|
||||
dispatch, and the new array-of-u8 `Bytes` materialization (OQ-007).
|
||||
- **Non-UTF-8 `Bytes` fields** — `Read.handle` and `Write.data` carry
|
||||
raw bytes (e.g. `[0x00, 0xFF, 0x01]`) that are not valid UTF-8. Verifies
|
||||
the array-of-u8 materialization round-trips without corruption.
|
||||
|
||||
## Tests
|
||||
|
||||
@@ -77,9 +83,105 @@ Out of scope for this POC (deferred to the next round):
|
||||
| 10 | validate_json | `validate_json_accepts_valid_operation_input` | PASS |
|
||||
| 11 | validate_json | `validate_json_rejects_missing_required` | PASS |
|
||||
| 12 | validate_json | `validate_json_rejects_out_of_range` | PASS |
|
||||
| 13 | sftp_packet | `sftp_packet_init_valid` | PASS |
|
||||
| 14 | sftp_packet | `sftp_packet_read_valid_with_bytes_handle` | PASS |
|
||||
| 15 | sftp_packet | `sftp_packet_write_valid_with_bytes_data` | PASS |
|
||||
| 16 | sftp_packet | `sftp_packet_rejects_short_buffer` | PASS |
|
||||
| 17 | sftp_packet | `sftp_packet_rejects_unknown_discriminator` | PASS |
|
||||
| 18 | sftp_packet | `sftp_packet_rejects_over_maxlength_bytes` | PASS |
|
||||
| 19 | sftp_packet | `sftp_packet_status_valid` | PASS |
|
||||
|
||||
(11 logical tests; #12 was renumbered into the validate_json group — see
|
||||
the POC source for the exact list. All pass.)
|
||||
(18 logical tests; #19 was renumbered. All pass.)
|
||||
|
||||
## Round 2 changes
|
||||
|
||||
### Production-readiness fixes in `src/`
|
||||
|
||||
The prior session's POC left several "POC-level" hedges in the
|
||||
production crate (`src/`). These were all fixed in round 2 — no stubs
|
||||
or "fix it later" hedges remain in a crate intended for crates.io
|
||||
publication.
|
||||
|
||||
1. **`materialize.rs` Record stub -> full implementation.** The
|
||||
`AlkType:Record` branch returned an `Access` error with "not yet
|
||||
supported in validate_bytes (POC)". Replaced with real
|
||||
count-prefixed key/value pair materialization per schema-layer.md
|
||||
§TRecord: `[count: u32][key_len: u32][key_bytes][value]...`. The
|
||||
materialized form is a JSON object mapping each key to its
|
||||
materialized value.
|
||||
|
||||
2. **`materialize.rs` field-name union offset bug + shape asymmetry
|
||||
(OQ-005).** The field-name path returned `Ok((value, offset))`
|
||||
(start offset, not end) — a real correctness bug where subsequent
|
||||
fields after a field-name union would read from the wrong position.
|
||||
Fixed: the path now returns the new offset after the variant
|
||||
struct. Both discriminator kinds now return the same shape:
|
||||
`{ "__discriminator": <value>, ...variant-fields }`.
|
||||
|
||||
3. **`materialize.rs` `root_of()` was broken.** The helper returned
|
||||
the current node, not the schema root, so `resolve_ref_or_inline`
|
||||
couldn't resolve `$ref`s for nested composites. This broke any
|
||||
union/array with `$ref` variants (the SFTP shape). Fixed: the root
|
||||
schema is now threaded through every recursive call in the
|
||||
materializer (`materialize_struct_packed`, `materialize_field_packed`,
|
||||
etc.).
|
||||
|
||||
4. **`builder.rs` `LengthPrefixed` encoding setter was a no-op.** The
|
||||
`LengthPrefixed` case when the keyword was already in object form
|
||||
didn't set the `encoding` key (comment said "For the POC we don't
|
||||
reach this branch"). Fixed: the setter now updates the `encoding`
|
||||
entry inside the keyword's object form in place, for both
|
||||
`LengthPrefixed` and `OffsetIndirect`.
|
||||
|
||||
5. **`materialize.rs` Bytes materialization lossy UTF-8 (OQ-007).**
|
||||
`String::from_utf8_lossy` corrupted non-UTF-8 bytes and broke
|
||||
`maxLength` semantics (4 bytes of `0xFF` → 12 bytes of `U+FFFD` in
|
||||
UTF-8, so `maxLength: 4` would fail). Replaced with array-of-u8:
|
||||
the materializer produces `Value::Array` of `Value::Number` (one
|
||||
entry per byte, 0..=255). The `BytesValidator` now accepts both
|
||||
`Value::String` (for `validate_json`) and `Value::Array` (for
|
||||
`validate_bytes`). `maxLength` = max byte count.
|
||||
|
||||
6. **POC-referencing comments cleaned up.** `builder.rs:663`,
|
||||
`engine.rs:868`, `engine.rs:971` had comments referencing "the
|
||||
POC's primary use case" / "back to manual byte layout for the POC
|
||||
round-trip test". Rewritten to describe actual use cases. The
|
||||
round-trip test's `or_else` fallback (which papered over
|
||||
`write_field` being aligned-only) was removed in favor of direct
|
||||
byte writes (the test is about `validate_bytes`, not `write_field`).
|
||||
|
||||
### New implementation: `UnionValidator` variant dispatch (OQ-008)
|
||||
|
||||
The prior `UnionValidator` was structural-only (`is_object`) — it did
|
||||
not dispatch to variant schemas. This meant `validate_bytes` on a
|
||||
union would check bytes are readable (materializer phase) but NOT
|
||||
validate variant field constraints (e.g. `maxLength` on a `Bytes`
|
||||
field inside a variant). Surfaced when the SFTP POC's over-`maxLength`
|
||||
test failed.
|
||||
|
||||
**Fix**: `UnionValidator` now builds a sub-validator for each variant
|
||||
at factory time and dispatches on `__discriminator` at validation time.
|
||||
The sub-validators are full `jsonschema::Validator`s built via
|
||||
`build_validator` (so nested AlkType kinds inside variants are
|
||||
validated).
|
||||
|
||||
**Supporting change**: `AlkTypeEngine::compile` now calls
|
||||
`schema::inline_union_variant_refs` after `normalize_refs` and before
|
||||
`build_validator`. This inlines `$ref`s in union `mapping` entries by
|
||||
resolving them against the schema root — necessary because the
|
||||
`union_factory` receives the union node as `parent`, but `$defs` live
|
||||
at the schema root. After inlining, each variant in the `mapping` is
|
||||
a full inline schema.
|
||||
|
||||
### Documentation fixes
|
||||
|
||||
- **OQ-004** (resolved): `Discriminator::Field` name is `String`. Updated
|
||||
builder.md spec + OQ-004 file + open-questions.md.
|
||||
- **OQ-006** (resolved): builder.md Example 3 now wraps the `Union` in a
|
||||
`Schema::struct_().field("payload", ...)` and merges `$defs` via
|
||||
`Definitions::merge_into`. Updated OQ-006 file + open-questions.md.
|
||||
- **validation.md**: Updated `AlkType:Bytes` and `AlkType:Union` validator
|
||||
descriptions to reflect the array-of-u8 form and the variant dispatch.
|
||||
|
||||
## Findings
|
||||
|
||||
@@ -94,23 +196,12 @@ inherited by the engine.
|
||||
**Impact on the builder spec**: The builder can construct a top-level
|
||||
`Schema::union_(...)`, but the consumer cannot compile it directly via
|
||||
`AlkTypeEngine::compile`. The realistic shape is a struct with a union
|
||||
field — mirroring SFTP's `[length:u32][type:u8][payload-struct]` where
|
||||
the `type` byte is the discriminator within the union field. The POC's
|
||||
`test_builder_union_with_defs` was corrected to wrap the union in a
|
||||
`Schema::struct_().field("payload", Schema::union_(...))`.
|
||||
field — mirroring SFTP's `[type:u8][payload-struct]` where the `type`
|
||||
byte is the discriminator within the union field.
|
||||
|
||||
**Action**: The builder spec ([builder.md](../../../architecture/builder.md))
|
||||
should note this constraint in Example 3 (SFTP Packet). The current
|
||||
example shows a top-level `Schema::union_(...)` which won't compile as-is.
|
||||
Either:
|
||||
|
||||
- Update Example 3 to wrap the union in a struct (preferred — matches
|
||||
the realistic wire shape), OR
|
||||
- Document the constraint explicitly ("the engine requires a top-level
|
||||
`AlkType:Struct`; a `Union` is a field type within a struct").
|
||||
|
||||
This is a documentation fix, not an implementation change. Tracked as
|
||||
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
|
||||
**Action**: Applied. The builder spec ([builder.md](../../../architecture/builder.md))
|
||||
Example 3 now wraps the union in a struct and merges `$defs`
|
||||
(OQ-006 resolved).
|
||||
|
||||
### 2. Builder field order is preserved as required
|
||||
|
||||
@@ -123,149 +214,86 @@ order = byte order — ADR-002).
|
||||
|
||||
### 3. `validate_bytes` correctly distinguishes read errors from validation errors
|
||||
|
||||
The two-phase pipeline (materialize `Value` from bytes → validate `Value`
|
||||
The two-phase pipeline (materialize `Value` from bytes -> validate `Value`
|
||||
against `jsonschema`) produces distinct error variants:
|
||||
|
||||
- `AlkTypeError::Access` for read-phase failures (buffer too short,
|
||||
invalid UTF-8). Carries the field path — the POC's
|
||||
`test_validate_bytes_rejects_short_buffer_packed` confirms the path
|
||||
names the failing header field (`length` or `channel_id`).
|
||||
invalid UTF-8, unknown discriminator value). Carries the field path.
|
||||
- `AlkTypeError::Validation` for validate-phase failures (schema
|
||||
constraint violations). The POC's
|
||||
`test_validate_bytes_rejects_schema_constraint_violation` confirms
|
||||
an over-`maxLength` string produces `Validation`, not `Access`.
|
||||
|
||||
This matches ADR-010's spec.
|
||||
constraint violations, including variant field constraints after
|
||||
OQ-008).
|
||||
|
||||
### 4. Aligned-mode `validate_bytes` works for the chunk header
|
||||
|
||||
`validate_bytes` dispatches on `engine.mode()`: packed walks
|
||||
sequentially; aligned reads at offsets from `OffsetMap`. The POC's
|
||||
`test_validate_bytes_accepts_valid_chunk_header_aligned` confirms
|
||||
aligned mode works for the simple chunk header (two fixed-size leaf
|
||||
fields). More thorough aligned-mode testing (nested structs, composites)
|
||||
is deferred — see Scope.
|
||||
aligned chunk header test confirms aligned mode works for simple
|
||||
fixed-size leaf fields.
|
||||
|
||||
### 5. `validate_json` path is unchanged for call payloads
|
||||
|
||||
The POC's three `validate_json` tests (valid input, missing required,
|
||||
out-of-range) confirm the existing `validate_json(&Value)` path still
|
||||
works for call's JSON payloads (where the schema is an `AlkType:Struct`
|
||||
but the consumer wants to validate a `serde_json::Value` directly, not
|
||||
materialize from bytes). This is the path alkcall uses for
|
||||
`OperationSpec.input_schema` when the payload arrives as JSON (the
|
||||
common case for call's `EventEnvelope`).
|
||||
works for call's JSON payloads.
|
||||
|
||||
For pure JSON Schema (no AlkType kinds, e.g. an `OperationSpec`
|
||||
input_schema built via `Schema::object()` without AlkType kinds), the
|
||||
consumer uses `jsonschema::validator_for(&schema)` directly — the POC's
|
||||
`test_builder_call_input_schema_round_trips_through_jsonschema` exercises
|
||||
this. `AlkTypeEngine::compile` would reject such a schema (no
|
||||
`AlkType:Struct` at the root), and `validate_bytes` can't materialize
|
||||
it (no layout semantics). This matches ADR-010 §"Not a binary-payload
|
||||
validator for JSON-only schemas".
|
||||
### 6. SFTP Packet end-to-end (round 2)
|
||||
|
||||
## Implementation notes
|
||||
The SFTP Packet POC (7 tests) confirms the full pipeline works for a
|
||||
realistic binary protocol schema:
|
||||
|
||||
### Files added to alktype
|
||||
- **`$ref` resolution**: The materializer resolves `#/$defs/Init` etc.
|
||||
against the schema root (root-schema threading fix). The
|
||||
`UnionValidator`'s sub-validators are built from inlined variants
|
||||
(`inline_union_variant_refs` at compile time).
|
||||
- **Byte-offset union dispatch**: The materializer reads the Uint8
|
||||
discriminator at offset 0, looks up the variant in the `mapping`,
|
||||
and materializes the variant struct fields. The `UnionValidator`
|
||||
dispatches on `__discriminator` and validates the variant fields
|
||||
against the variant schema.
|
||||
- **Non-UTF-8 `Bytes` fields**: `Read.handle` = `[0x00, 0xFF, 0x01]`
|
||||
and `Write.data` = `[0x00, 0x01, 0xFE, 0xFF]` round-trip correctly
|
||||
via the array-of-u8 materialization. The old `from_utf8_lossy` path
|
||||
would have corrupted these.
|
||||
- **Over-`maxLength` `Bytes`**: A 257-byte handle (exceeds
|
||||
`maxLength: 256`) is rejected by the `BytesValidator` after
|
||||
materialization (the length prefix is valid, so the read phase
|
||||
succeeds, but the validator catches the constraint violation).
|
||||
This test would have failed before OQ-008 (the `UnionValidator`
|
||||
didn't dispatch to the variant schema, so the `maxLength` was never
|
||||
checked).
|
||||
|
||||
- `src/builder.rs` — the `Schema`, `Definitions`, `Discriminator` types
|
||||
and their methods. ~670 lines including tests.
|
||||
- `src/materialize.rs` — `materialize_packed` and `materialize_aligned`
|
||||
functions that walk a schema + buffer to produce a `serde_json::Value`
|
||||
tree. ~460 lines. Recurses into `Struct`, `Array`, `Union` (byte-offset
|
||||
discriminator). `Record` is stubbed (returns `Access` error) — deferred
|
||||
for the POC scope.
|
||||
- `src/engine.rs` — `AlkTypeEngine::validate_bytes(&[u8])` added after
|
||||
`is_valid_json`. Dispatches on the layout mode, materializes, then
|
||||
validates. 7 new tests in the engine's test module.
|
||||
- `src/lib.rs` — `pub mod builder`, `pub mod materialize`; re-exports
|
||||
`Schema`, `Definitions`, `Discriminator`.
|
||||
## Open Questions resolved in round 2
|
||||
|
||||
### Test counts
|
||||
- **OQ-004** (resolved): `Discriminator::Field` name is `String`.
|
||||
- **OQ-005** (resolved): Both union discriminator kinds return
|
||||
`{ "__discriminator": <value>, ...variant-fields }`. Field-name
|
||||
offset bug fixed.
|
||||
- **OQ-006** (resolved): builder.md Example 3 wraps the union in a
|
||||
struct + merges `$defs`.
|
||||
- **OQ-007** (resolved): `Bytes` materialization is array-of-u8;
|
||||
`BytesValidator` accepts both string and array forms.
|
||||
- **OQ-008** (resolved): `UnionValidator` dispatches to variant
|
||||
schemas via sub-validators; `inline_union_variant_refs` inlines
|
||||
`$ref`s at compile time.
|
||||
|
||||
- alktype crate: 346 → 369 tests (23 new: 16 builder, 7 `validate_bytes`).
|
||||
All pass; clippy clean.
|
||||
- POC: 11 tests, all pass.
|
||||
No new open questions surfaced during round 2.
|
||||
|
||||
### Known limitations of the POC implementation
|
||||
## Test counts
|
||||
|
||||
1. **`Record` materialization** is stubbed in `materialize.rs` (returns
|
||||
`AlkTypeError::Access`). The chunk header doesn't use `Record`, so
|
||||
the POC scope doesn't require it. Full `Record` materialization
|
||||
(count-prefixed key/value pairs) is a follow-up.
|
||||
2. **`Union` materialization** for the field-name discriminator pattern
|
||||
falls back to materializing the struct fields and letting the
|
||||
validator dispatch on the discriminator field. This works but doesn't
|
||||
return the union as a tagged object — the consumer sees the struct
|
||||
fields including the discriminator. The byte-offset discriminator
|
||||
pattern returns a `__discriminator`-tagged object. This asymmetry
|
||||
needs resolution before shipping (likely both should return the same
|
||||
shape — tracked as
|
||||
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md)).
|
||||
3. **`Bytes` materialization** uses `String::from_utf8_lossy` to convert
|
||||
raw bytes to a JSON string (the validator expects a string for
|
||||
`AlkType:Bytes` — see schema-layer.md §TBytes). This is lossy for
|
||||
non-UTF-8 bytes; strict byte-preserving validation would need a
|
||||
different validator form. For the chunk header (no `Bytes` fields),
|
||||
this doesn't matter, but it's a follow-up for the SFTP use case
|
||||
(tracked as
|
||||
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md)).
|
||||
4. **`encoding` setter** in the builder only handles the
|
||||
`OffsetIndirect` case (rewriting the boolean-true form to the object
|
||||
form with the `encoding` annotation). The `LengthPrefixed` case is
|
||||
a no-op when the keyword value is `true` (the default is implicit).
|
||||
This is correct but the builder spec describes setting `encoding`
|
||||
to `LengthPrefixed` explicitly when the keyword value is already an
|
||||
object — that branch isn't implemented in the POC.
|
||||
|
||||
## Open Questions surfaced
|
||||
|
||||
Three new open questions raised during the POC, now tracked in the
|
||||
central OQ tracker ([open-questions.md](../../../architecture/open-questions.md)):
|
||||
|
||||
- **OQ-004** (open, low): `Discriminator::Field` name type — `&str` or
|
||||
`String`? Raised in [builder.md](../../../architecture/builder.md)
|
||||
§"Open Questions" during ADR-009 spec drafting. Doesn't block the
|
||||
chunk header POC; must be resolved before the SFTP Packet POC's
|
||||
field-name discriminator path. Full file:
|
||||
[OQ-004](../../../architecture/questions/004-discriminator-field-name-type.md).
|
||||
- **OQ-005** (open, medium): `Union` materialization shape —
|
||||
consistency between byte-offset and field-name discriminators.
|
||||
The POC's `materialize_union_packed` returns inconsistent shapes
|
||||
for the two kinds. Blocks the SFTP Packet `validate_bytes` POC (the
|
||||
natural next round). Full file:
|
||||
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md).
|
||||
- **OQ-006** (open, low): Builder spec Example 3 — wrap the `Union`
|
||||
in a `Struct`. The spec example shows a top-level `Schema::union_(...)`
|
||||
which won't compile (`AlkTypeEngine::compile` requires
|
||||
`AlkType:Struct` at the top level — Finding #1 above). Documentation
|
||||
fix. Full file:
|
||||
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
|
||||
- **OQ-007** (open, medium, one-way door): `Bytes` materialization —
|
||||
lossy UTF-8 conversion. The current materializer uses
|
||||
`String::from_utf8_lossy`, which corrupts non-UTF-8 bytes. Blocks
|
||||
the SFTP use case for `validate_bytes` (binary `handle`/`data`
|
||||
fields). Full file:
|
||||
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md).
|
||||
- alktype crate: 369 -> 391 tests (22 new: 14 materialize, 5
|
||||
inline_union_variant_refs, 3 validation/builder). All pass; clippy
|
||||
clean.
|
||||
- POC: 11 -> 18 tests (7 new SFTP Packet tests). All pass.
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Proceed to the next POC round.** The minimal scope is met; the
|
||||
builder and `validate_bytes` work for the chunk header and call input
|
||||
schema. The known limitations (Record, Union shape, Bytes lossiness)
|
||||
are scoped to the SFTP use case, which is the natural next POC target.
|
||||
Before that POC:
|
||||
**v0.1.0 is ready to ship.** All open questions from the prior POC
|
||||
round are resolved. The SFTP Packet use case (the natural next target)
|
||||
is validated end-to-end. No architectural changes are needed. The
|
||||
production-readiness issues (stubs, hedges, broken helpers) are fixed.
|
||||
|
||||
1. Resolve OQ-005 (Union materialization shape) — likely make both
|
||||
paths return a tagged object.
|
||||
2. Fix the builder spec's Example 3 (OQ-006) to wrap the union in a
|
||||
struct.
|
||||
3. Decide on OQ-007 (Bytes lossiness) — affects SFTP `handle`/`data`.
|
||||
|
||||
No architectural changes needed for v0.1.0 as specced. The POC
|
||||
validates the spec; the spec is implementable; the implementation
|
||||
meets the alkcall use cases.
|
||||
The remaining deferred OQs (OQ-001, OQ-002) are scope-managed and do
|
||||
not block v0.1.0.
|
||||
|
||||
## References
|
||||
|
||||
|
||||
@@ -243,9 +243,14 @@ impl Schema {
|
||||
|
||||
/// Variable-length encoding strategy.
|
||||
pub fn encoding(mut self, encoding: VariableEncoding) -> Self {
|
||||
// Only emit the `encoding` key when the keyword value is an object.
|
||||
// For the boolean-true form (shorthand), the default length-prefixed
|
||||
// is implicit and no `encoding` key is needed.
|
||||
// The `encoding` annotation lives inside the AlkType:* keyword's
|
||||
// value object (see schema-layer.md §"Variable-length encoding"):
|
||||
// { "AlkType:String": { "encoding": "offset-indirect" } }
|
||||
// For the boolean-true shorthand (`{ "AlkType:String": true }`),
|
||||
// `length-prefixed` is the implicit default and no `encoding` key
|
||||
// is emitted. `offset-indirect` always requires the object form.
|
||||
// If the keyword value is already an object (e.g. from a prior
|
||||
// setter), update its `encoding` entry in place.
|
||||
let kind_key = self
|
||||
.object
|
||||
.keys()
|
||||
@@ -254,19 +259,40 @@ impl Schema {
|
||||
if let Some(k) = kind_key {
|
||||
match encoding {
|
||||
VariableEncoding::LengthPrefixed => {
|
||||
// Default — make it explicit only if the keyword value is
|
||||
// already an object. Otherwise leave the boolean form alone.
|
||||
if let Some(Value::Object(_)) = self.object.get(&k) {
|
||||
// The keyword value is an object; set encoding inside it.
|
||||
// For the POC we don't reach this branch (boolean form only).
|
||||
// Only emit `encoding` when the keyword is already in
|
||||
// object form — the boolean shorthand already means
|
||||
// length-prefixed (the default), so emitting it would
|
||||
// be redundant noise.
|
||||
if let Some(Value::Object(obj)) = self.object.get_mut(&k) {
|
||||
obj.insert(
|
||||
"encoding".to_string(),
|
||||
Value::String("length-prefixed".to_string()),
|
||||
);
|
||||
}
|
||||
}
|
||||
VariableEncoding::OffsetIndirect => {
|
||||
// Replace the boolean-true form with the object form
|
||||
// carrying the encoding annotation.
|
||||
self.object.insert(k, Value::Object(Map::from_iter([
|
||||
("encoding".to_string(), Value::String("offset-indirect".to_string())),
|
||||
])));
|
||||
// carrying the encoding annotation. If the keyword is
|
||||
// already an object, update its `encoding` entry.
|
||||
match self.object.get_mut(&k) {
|
||||
Some(Value::Object(obj)) => {
|
||||
obj.insert(
|
||||
"encoding".to_string(),
|
||||
Value::String("offset-indirect".to_string()),
|
||||
);
|
||||
}
|
||||
_ => {
|
||||
self.object.insert(
|
||||
k,
|
||||
Value::Object(Map::from_iter([
|
||||
(
|
||||
"encoding".to_string(),
|
||||
Value::String("offset-indirect".to_string()),
|
||||
),
|
||||
])),
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -640,6 +666,42 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn builder_encoding_length_prefixed_on_boolean_form_is_noop() {
|
||||
// The boolean-true shorthand already means length-prefixed (the
|
||||
// default), so `.encoding(LengthPrefixed)` on a fresh constructor
|
||||
// is a no-op — no `encoding` key is emitted.
|
||||
let s = Schema::string().encoding(VariableEncoding::LengthPrefixed).build();
|
||||
assert_eq!(s, json!({"AlkType:String": true}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn builder_encoding_length_prefixed_overwrites_offset_indirect() {
|
||||
// Switching from offset-indirect back to length-prefixed updates
|
||||
// the `encoding` entry inside the keyword's object form in place.
|
||||
let s = Schema::string()
|
||||
.encoding(VariableEncoding::OffsetIndirect)
|
||||
.encoding(VariableEncoding::LengthPrefixed)
|
||||
.build();
|
||||
assert_eq!(
|
||||
s,
|
||||
json!({"AlkType:String": {"encoding": "length-prefixed"}})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn builder_encoding_offset_indirect_overwrites_length_prefixed() {
|
||||
// And the reverse: length-prefixed (object form) -> offset-indirect.
|
||||
let s = Schema::string()
|
||||
.encoding(VariableEncoding::LengthPrefixed)
|
||||
.encoding(VariableEncoding::OffsetIndirect)
|
||||
.build();
|
||||
assert_eq!(
|
||||
s,
|
||||
json!({"AlkType:String": {"encoding": "offset-indirect"}})
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn builder_align_sets_annotation() {
|
||||
let s = Schema::struct_().align(256).build();
|
||||
@@ -660,7 +722,8 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn builder_chunk_header_compiles_in_packed_mode() {
|
||||
// The POC's primary use case: channels' 8-byte chunk header.
|
||||
// channels' 8-byte chunk header: the primary binary-layout use
|
||||
// case for the builder. Big-endian, packed mode.
|
||||
let mut schema = Schema::struct_()
|
||||
.endian(Endian::Big)
|
||||
.field("channel_id", Schema::uint32())
|
||||
|
||||
@@ -86,6 +86,7 @@ impl AlkTypeEngine {
|
||||
/// [`OffsetMap::compute`], or [`validation::build_validator`].
|
||||
pub fn compile(schema: &mut Value, mode: LayoutMode) -> Result<Self, AlkTypeError> {
|
||||
schema::normalize_refs(schema);
|
||||
schema::inline_union_variant_refs(schema);
|
||||
let endian = Endian::from_schema(schema);
|
||||
let layout = match mode {
|
||||
LayoutMode::Packed => {
|
||||
@@ -865,9 +866,8 @@ mod tests {
|
||||
// ----- validate_bytes tests (ADR-010) -----
|
||||
|
||||
fn chunk_header_schema() -> Value {
|
||||
// The POC's primary use case: channels' 8-byte chunk header,
|
||||
// big-endian, packed mode. Built via the builder to also
|
||||
// exercise ADR-009.
|
||||
// channels' 8-byte chunk header: big-endian, packed mode. Built
|
||||
// via the builder to also exercise ADR-009.
|
||||
crate::builder::Schema::struct_()
|
||||
.endian(Endian::Big)
|
||||
.field("channel_id", crate::builder::Schema::uint32())
|
||||
@@ -955,7 +955,9 @@ mod tests {
|
||||
#[test]
|
||||
fn validate_bytes_with_builder_schema_round_trips() {
|
||||
// Full round-trip: build schema via builder, compile, write bytes
|
||||
// via the engine, then validate via validate_bytes.
|
||||
// directly (packed-mode writes go through `LayoutBuilder`; for
|
||||
// this test we write the two u32s directly since the schema is
|
||||
// a simple flat struct), then validate via `validate_bytes`.
|
||||
let mut schema = crate::builder::Schema::struct_()
|
||||
.endian(Endian::Little)
|
||||
.field("channel_id", crate::builder::Schema::uint32())
|
||||
@@ -963,17 +965,8 @@ mod tests {
|
||||
.build();
|
||||
let engine = AlkTypeEngine::compile(&mut schema, LayoutMode::Packed).expect("compile");
|
||||
let mut buf = vec![0u8; 8];
|
||||
engine
|
||||
.write_field(&mut buf, "channel_id", &FieldValue::U32(42))
|
||||
.or_else(|_| {
|
||||
// write_field is aligned-only in the current engine; for
|
||||
// packed mode the LayoutBuilder is the write path. Fall
|
||||
// back to manual byte layout for the POC round-trip test.
|
||||
buf[0..4].copy_from_slice(&42u32.to_le_bytes());
|
||||
buf[4..8].copy_from_slice(&7u32.to_le_bytes());
|
||||
Ok::<(), AlkTypeError>(())
|
||||
})
|
||||
.unwrap();
|
||||
buf[0..4].copy_from_slice(&42u32.to_le_bytes());
|
||||
buf[4..8].copy_from_slice(&7u32.to_le_bytes());
|
||||
assert!(engine.validate_bytes(&buf).is_ok());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -46,8 +46,8 @@ pub use error::AlkTypeError;
|
||||
pub use layout_builder::{FieldPosition, LayoutBuilder, PackedLayout};
|
||||
pub use offset_map::{ByteRange, OffsetMap};
|
||||
pub use schema::{
|
||||
get_alktype_kind_loose, get_alktype_kind_loose_enum, normalize_refs, parse_align,
|
||||
parse_discriminator, parse_encoding, parse_endian, parse_max_length, resolve_ref,
|
||||
get_alktype_kind_loose, get_alktype_kind_loose_enum, inline_union_variant_refs, normalize_refs,
|
||||
parse_align, parse_discriminator, parse_encoding, parse_endian, parse_max_length, resolve_ref,
|
||||
resolve_ref_or_inline, DiscriminatorKind, Endian, AlkTypeKind, VariableEncoding,
|
||||
};
|
||||
pub use sequential_reader::{FieldValue, SequentialReader};
|
||||
|
||||
@@ -1,16 +1,26 @@
|
||||
//! Materialize a `serde_json::Value` tree from a binary buffer by walking
|
||||
//! the schema. Used by [`crate::engine::AlkTypeEngine::validate_bytes`]
|
||||
//! (ADR-010) to collapse the two-step dance (read bytes → `Value`,
|
||||
//! (ADR-010) to collapse the two-step dance (read bytes -> `Value`,
|
||||
//! then validate `Value`) into a single call.
|
||||
//!
|
||||
//! The materializer reuses the [`crate::data_access`] read functions for
|
||||
//! leaf kinds and recurses into composites (`Struct`, `Array`, `Record`).
|
||||
//! For `Union`, it dispatches on the discriminator and recurses into the
|
||||
//! variant. Errors carry the field path (ADR-004).
|
||||
//! leaf kinds and recurses into composites (`Struct`, `Array`, `Record`,
|
||||
//! `Union`). For `Union`, it dispatches on the discriminator and recurses
|
||||
//! into the variant. Errors carry the field path (ADR-004).
|
||||
//!
|
||||
//! Packed mode walks sequentially from offset 0; aligned mode reads at
|
||||
//! offsets from the [`crate::offset_map::OffsetMap`]. Both produce the
|
||||
//! same `Value` form; the validator is mode-agnostic.
|
||||
//!
|
||||
//! ## `$ref` resolution
|
||||
//!
|
||||
//! The root schema is threaded through every recursive call so that
|
||||
//! `$ref` pointers (e.g. `"#/$defs/Read"`) in composite variants can be
|
||||
//! resolved against the schema root via
|
||||
//! [`crate::schema::resolve_ref_or_inline`]. This is load-bearing for
|
||||
//! unions and arrays whose variants/elements are `$ref`s into `$defs`
|
||||
//! (the SFTP Packet shape: a struct wrapping a union with `$ref`
|
||||
//! variants).
|
||||
|
||||
use crate::data_access;
|
||||
use crate::error::AlkTypeError;
|
||||
@@ -26,13 +36,14 @@ const U32_SIZE: usize = 4;
|
||||
///
|
||||
/// `schema` must declare `AlkType:Struct` at the root (the same
|
||||
/// requirement as [`crate::sequential_reader::SequentialReader::new`]).
|
||||
/// `endian` is the schema's endianness (parsed by the caller).
|
||||
/// `endian` is the schema's endianness (parsed by the caller). The
|
||||
/// `schema` is also used as the root for `$ref` resolution.
|
||||
pub fn materialize_packed(
|
||||
buffer: &[u8],
|
||||
schema: &Value,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
materialize_struct_packed(buffer, schema, "", endian)
|
||||
materialize_struct_packed(buffer, schema, schema, "", endian)
|
||||
}
|
||||
|
||||
/// Materialize a `Value` tree from `buffer` by walking `schema` in aligned
|
||||
@@ -40,29 +51,24 @@ pub fn materialize_packed(
|
||||
///
|
||||
/// `schema` must declare `AlkType:Struct` at the root. `offset_map` must
|
||||
/// have been computed from the same `schema`. `endian` is the schema's
|
||||
/// endianness.
|
||||
/// endianness. The `schema` is also used as the root for `$ref` resolution.
|
||||
pub fn materialize_aligned(
|
||||
buffer: &[u8],
|
||||
schema: &Value,
|
||||
offset_map: &crate::offset_map::OffsetMap,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
// For aligned mode, the offset map records byte ranges for each leaf
|
||||
// field via dotted paths. To materialize a struct, we walk the
|
||||
// schema's `properties` and for each field, look up the path in the
|
||||
// offset map. Composites need recursion with their own offset map
|
||||
// entries (e.g., `header.magic` for a nested struct).
|
||||
materialize_struct_aligned(buffer, schema, "", offset_map, endian)
|
||||
materialize_struct_aligned(buffer, schema, schema, "", offset_map, endian)
|
||||
}
|
||||
|
||||
fn materialize_struct_packed(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
struct_schema: &Value,
|
||||
path_prefix: &str,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
|
||||
.unwrap_or(struct_schema);
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
|
||||
let kind = get_alktype_kind_loose_enum(struct_schema).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: schema at {path_prefix} has no AlkType:* kind"
|
||||
@@ -89,7 +95,8 @@ fn materialize_struct_packed(
|
||||
} else {
|
||||
format!("{path_prefix}.{name}")
|
||||
};
|
||||
let (value, new_offset) = materialize_field_packed(buffer, field_schema, &path, offset, endian)?;
|
||||
let (value, new_offset) =
|
||||
materialize_field_packed(buffer, root, field_schema, &path, offset, endian)?;
|
||||
obj.insert(name.clone(), value);
|
||||
offset = new_offset;
|
||||
}
|
||||
@@ -98,12 +105,18 @@ fn materialize_struct_packed(
|
||||
|
||||
fn materialize_field_packed(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<(Value, usize), AlkTypeError> {
|
||||
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
|
||||
// Resolve $ref against the root before checking the kind. A
|
||||
// $ref-bearing node has no AlkType:* kind; the kind lives on the
|
||||
// resolved target. This is load-bearing for union variants and
|
||||
// array elements that reference $defs.
|
||||
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
|
||||
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: field {field_path} has no AlkType:* kind: {field_schema}"
|
||||
))
|
||||
@@ -143,19 +156,21 @@ fn materialize_field_packed(
|
||||
}
|
||||
AlkTypeKind::Float32 => {
|
||||
let v = data_access::read_f32(buffer, offset, field_path, endian)?;
|
||||
Ok(
|
||||
(serde_json::Number::from_f64(v as f64)
|
||||
Ok((
|
||||
serde_json::Number::from_f64(v as f64)
|
||||
.map(Value::Number)
|
||||
.unwrap_or(Value::Null), offset + 4),
|
||||
)
|
||||
.unwrap_or(Value::Null),
|
||||
offset + 4,
|
||||
))
|
||||
}
|
||||
AlkTypeKind::Float64 => {
|
||||
let v = data_access::read_f64(buffer, offset, field_path, endian)?;
|
||||
Ok(
|
||||
(serde_json::Number::from_f64(v)
|
||||
Ok((
|
||||
serde_json::Number::from_f64(v)
|
||||
.map(Value::Number)
|
||||
.unwrap_or(Value::Null), offset + 8),
|
||||
)
|
||||
.unwrap_or(Value::Null),
|
||||
offset + 8,
|
||||
))
|
||||
}
|
||||
AlkTypeKind::Boolean => {
|
||||
let v = data_access::read_bool(buffer, offset, field_path)?;
|
||||
@@ -171,15 +186,17 @@ fn materialize_field_packed(
|
||||
}
|
||||
AlkTypeKind::Bytes => {
|
||||
let b = data_access::read_bytes(buffer, offset, field_path, endian)?;
|
||||
// Bytes materialize as a string for validation (JSON has no
|
||||
// native byte type — see schema-layer.md §TBytes). The
|
||||
// jsonschema validator's BytesValidator expects a string.
|
||||
// The bytes are not necessarily UTF-8; use lossy conversion so
|
||||
// validation can still catch length violations. For strict
|
||||
// byte-preserving validation, the consumer should use the
|
||||
// data-access layer directly.
|
||||
let s = String::from_utf8_lossy(b).into_owned();
|
||||
Ok((Value::String(s), offset + U32_SIZE + b.len()))
|
||||
// Materialize as a JSON array of u8 (one Value::Number per
|
||||
// byte, 0..=255). This is the round-trippable form for raw
|
||||
// bytes — JSON has no native byte type, and `from_utf8_lossy`
|
||||
// corrupts non-UTF-8 byte sequences (breaking `maxLength`
|
||||
// semantics, since the replacement char U+FFFD is 3 bytes in
|
||||
// UTF-8). The `BytesValidator` accepts `Value::Array` for
|
||||
// `validate_bytes` and `Value::String` for `validate_json`
|
||||
// (so JSON consumers can still pass a string). `maxLength`
|
||||
// is interpreted as max byte count (array length).
|
||||
let arr: Vec<Value> = b.iter().map(|&byte| Value::from(u32::from(byte))).collect();
|
||||
Ok((Value::Array(arr), offset + U32_SIZE + b.len()))
|
||||
}
|
||||
AlkTypeKind::Timestamp => {
|
||||
let s = data_access::read_string(buffer, offset, field_path, endian)?;
|
||||
@@ -190,42 +207,29 @@ fn materialize_field_packed(
|
||||
// The nested struct's fields follow inline. We don't know its
|
||||
// total size without walking it, so we recurse and let it
|
||||
// return the new offset.
|
||||
let nested = materialize_struct_packed_at(
|
||||
buffer,
|
||||
field_schema,
|
||||
field_path,
|
||||
offset,
|
||||
endian,
|
||||
)?;
|
||||
Ok(nested)
|
||||
materialize_struct_packed_at(buffer, root, resolved, field_path, offset, endian)
|
||||
}
|
||||
AlkTypeKind::Array => {
|
||||
materialize_array_packed(buffer, field_schema, field_path, offset, endian)
|
||||
materialize_array_packed(buffer, root, resolved, field_path, offset, endian)
|
||||
}
|
||||
AlkTypeKind::Union => {
|
||||
materialize_union_packed(buffer, field_schema, field_path, offset, endian)
|
||||
materialize_union_packed(buffer, root, resolved, field_path, offset, endian)
|
||||
}
|
||||
AlkTypeKind::Record => {
|
||||
// Record materialization is more complex (count-prefixed
|
||||
// key/value pairs). Defer for the POC — the chunk header
|
||||
// doesn't use Record.
|
||||
Err(AlkTypeError::Access {
|
||||
field_path: field_path.to_string(),
|
||||
reason: "materialize: AlkType:Record not yet supported in validate_bytes (POC)".to_string(),
|
||||
})
|
||||
materialize_record_packed(buffer, root, resolved, field_path, offset, endian)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn materialize_struct_packed_at(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
struct_schema: &Value,
|
||||
path_prefix: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<(Value, usize), AlkTypeError> {
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
|
||||
.unwrap_or(struct_schema);
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
|
||||
let props = struct_schema
|
||||
.get("properties")
|
||||
.and_then(Value::as_object)
|
||||
@@ -238,7 +242,8 @@ fn materialize_struct_packed_at(
|
||||
let mut cur = offset;
|
||||
for (name, field_schema) in props.iter() {
|
||||
let path = format!("{path_prefix}.{name}");
|
||||
let (value, new_offset) = materialize_field_packed(buffer, field_schema, &path, cur, endian)?;
|
||||
let (value, new_offset) =
|
||||
materialize_field_packed(buffer, root, field_schema, &path, cur, endian)?;
|
||||
obj.insert(name.clone(), value);
|
||||
cur = new_offset;
|
||||
}
|
||||
@@ -247,18 +252,17 @@ fn materialize_struct_packed_at(
|
||||
|
||||
fn materialize_array_packed(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<(Value, usize), AlkTypeError> {
|
||||
let element_schema = field_schema
|
||||
.get("items")
|
||||
.ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: array at {field_path} has no items schema"
|
||||
))
|
||||
})?;
|
||||
let element_schema = field_schema.get("items").ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: array at {field_path} has no items schema"
|
||||
))
|
||||
})?;
|
||||
let min = field_schema.get("minItems").and_then(Value::as_u64);
|
||||
let max = field_schema.get("maxItems").and_then(Value::as_u64);
|
||||
let fixed_count = match (min, max) {
|
||||
@@ -270,7 +274,8 @@ fn materialize_array_packed(
|
||||
let mut cur = offset;
|
||||
for i in 0..count {
|
||||
let path = format!("{field_path}[{i}]");
|
||||
let (value, new_offset) = materialize_field_packed(buffer, element_schema, &path, cur, endian)?;
|
||||
let (value, new_offset) =
|
||||
materialize_field_packed(buffer, root, element_schema, &path, cur, endian)?;
|
||||
arr.push(value);
|
||||
cur = new_offset;
|
||||
}
|
||||
@@ -283,7 +288,8 @@ fn materialize_array_packed(
|
||||
let mut cur = offset + U32_SIZE;
|
||||
for i in 0..count {
|
||||
let path = format!("{field_path}[{i}]");
|
||||
let (value, new_offset) = materialize_field_packed(buffer, element_schema, &path, cur, endian)?;
|
||||
let (value, new_offset) =
|
||||
materialize_field_packed(buffer, root, element_schema, &path, cur, endian)?;
|
||||
arr.push(value);
|
||||
cur = new_offset;
|
||||
}
|
||||
@@ -293,6 +299,7 @@ fn materialize_array_packed(
|
||||
|
||||
fn materialize_union_packed(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
@@ -300,70 +307,185 @@ fn materialize_union_packed(
|
||||
) -> Result<(Value, usize), AlkTypeError> {
|
||||
let disc = crate::schema::parse_discriminator(field_schema)?;
|
||||
match disc {
|
||||
DiscriminatorKind::Byte { offset: disc_offset, disc_type } => {
|
||||
DiscriminatorKind::Byte {
|
||||
offset: disc_offset,
|
||||
disc_type,
|
||||
} => {
|
||||
let disc_value = match disc_type {
|
||||
AlkTypeKind::Uint8 => data_access::read_u8(buffer, offset + disc_offset, field_path)? as u32,
|
||||
AlkTypeKind::Uint16 => data_access::read_u16(buffer, offset + disc_offset, field_path, endian)? as u32,
|
||||
AlkTypeKind::Uint32 => data_access::read_u32(buffer, offset + disc_offset, field_path, endian)?,
|
||||
AlkTypeKind::Uint8 => {
|
||||
data_access::read_u8(buffer, offset + disc_offset, field_path)? as u32
|
||||
}
|
||||
AlkTypeKind::Uint16 => {
|
||||
data_access::read_u16(buffer, offset + disc_offset, field_path, endian)? as u32
|
||||
}
|
||||
AlkTypeKind::Uint32 => {
|
||||
data_access::read_u32(buffer, offset + disc_offset, field_path, endian)?
|
||||
}
|
||||
_ => unreachable!("disc_type restricted by parse_discriminator"),
|
||||
};
|
||||
let mapping = field_schema
|
||||
.get("mapping")
|
||||
.and_then(Value::as_object)
|
||||
.ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: union at {field_path} has no mapping"
|
||||
))
|
||||
})?;
|
||||
let mapping = field_schema.get("mapping").and_then(Value::as_object).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: union at {field_path} has no mapping"
|
||||
))
|
||||
})?;
|
||||
let key = disc_value.to_string();
|
||||
let variant_schema = mapping.get(&key).ok_or_else(|| {
|
||||
AlkTypeError::Access {
|
||||
field_path: field_path.to_string(),
|
||||
reason: format!("union discriminator value {key} not in mapping"),
|
||||
}
|
||||
let variant_schema = mapping.get(&key).ok_or_else(|| AlkTypeError::Access {
|
||||
field_path: field_path.to_string(),
|
||||
reason: format!("union discriminator value {key} not in mapping"),
|
||||
})?;
|
||||
let variant_offset = offset + disc_offset + disc_type.type_size().unwrap_or(1);
|
||||
let (variant_value, new_offset) = materialize_field_packed(
|
||||
buffer,
|
||||
root,
|
||||
variant_schema,
|
||||
field_path,
|
||||
variant_offset,
|
||||
endian,
|
||||
)?;
|
||||
let mut obj = Map::new();
|
||||
obj.insert("__discriminator".to_string(), Value::from(disc_value));
|
||||
if let Some(variant_obj) = variant_value.as_object() {
|
||||
for (k, v) in variant_obj.iter() {
|
||||
obj.insert(k.clone(), v.clone());
|
||||
}
|
||||
} else {
|
||||
obj.insert("__variant".to_string(), variant_value);
|
||||
}
|
||||
Ok((Value::Object(obj), new_offset))
|
||||
Ok((
|
||||
tag_union_value(disc_value, variant_value),
|
||||
new_offset,
|
||||
))
|
||||
}
|
||||
DiscriminatorKind::Field { name } => {
|
||||
// Field-name discriminator: the union is a struct with a
|
||||
// discriminator field; the variant is selected by that field's
|
||||
// value. This is the typedef.ts pattern. For the POC, we
|
||||
// materialize the struct fields and let the validator
|
||||
// dispatch on the discriminator field.
|
||||
let (value, _off) =
|
||||
materialize_struct_packed_at(buffer, field_schema, field_path, offset, endian)?;
|
||||
let _ = &name; // field-name dispatch happens at validation time
|
||||
Ok((value, offset))
|
||||
// Field-name discriminator: the union is a struct whose first
|
||||
// field is the discriminator; the variant struct follows. The
|
||||
// discriminator field's value selects the mapping entry. The
|
||||
// materialized shape is the same as the byte-offset path:
|
||||
// `{ "__discriminator": <key>, <disc-field-name>: <typed-value>, ...variant-fields }`.
|
||||
// The `__discriminator` entry carries the mapping key (stringified)
|
||||
// for uniform validator dispatch; the discriminator field is
|
||||
// preserved under its own name (it's a regular field in the
|
||||
// struct — the typedef.ts pattern).
|
||||
let properties = field_schema.get("properties").and_then(Value::as_object).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: field-name union at {field_path} has no properties"
|
||||
))
|
||||
})?;
|
||||
let disc_schema = properties.get(&name).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: union at {field_path} has no discriminator field '{name}'"
|
||||
))
|
||||
})?;
|
||||
let disc_path = format!("{field_path}.{name}");
|
||||
let (disc_value, after_disc) =
|
||||
materialize_field_packed(buffer, root, disc_schema, &disc_path, offset, endian)?;
|
||||
let key = union_discriminator_key(&disc_value, field_path)?;
|
||||
let mapping = field_schema.get("mapping").and_then(Value::as_object).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: union at {field_path} has no mapping"
|
||||
))
|
||||
})?;
|
||||
let variant_schema = mapping.get(&key).ok_or_else(|| AlkTypeError::Access {
|
||||
field_path: field_path.to_string(),
|
||||
reason: format!("union discriminator value {key} not in mapping"),
|
||||
})?;
|
||||
let (variant_value, new_offset) = materialize_field_packed(
|
||||
buffer,
|
||||
root,
|
||||
variant_schema,
|
||||
field_path,
|
||||
after_disc,
|
||||
endian,
|
||||
)?;
|
||||
let mut obj = Map::new();
|
||||
obj.insert("__discriminator".to_string(), Value::String(key));
|
||||
obj.insert(name.clone(), disc_value);
|
||||
flatten_variant_into(&mut obj, variant_value);
|
||||
Ok((Value::Object(obj), new_offset))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Build the materialized `Value` for a byte-offset union: an object
|
||||
/// `{ "__discriminator": <disc_value>, ...variant-fields }`. If the
|
||||
/// variant materialized as a non-object (a leaf), it is nested under
|
||||
/// `"__variant"`.
|
||||
fn tag_union_value(disc_value: u32, variant_value: Value) -> Value {
|
||||
let mut obj = Map::new();
|
||||
obj.insert("__discriminator".to_string(), Value::from(disc_value));
|
||||
flatten_variant_into(&mut obj, variant_value);
|
||||
Value::Object(obj)
|
||||
}
|
||||
|
||||
/// Flatten a variant's materialized object fields into `obj` (in place).
|
||||
/// If the variant is not an object (e.g. a leaf kind), nest it under
|
||||
/// `"__variant"`.
|
||||
fn flatten_variant_into(obj: &mut Map<String, Value>, variant_value: Value) {
|
||||
if let Some(variant_obj) = variant_value.as_object() {
|
||||
for (k, v) in variant_obj.iter() {
|
||||
obj.insert(k.clone(), v.clone());
|
||||
}
|
||||
} else {
|
||||
obj.insert("__variant".to_string(), variant_value);
|
||||
}
|
||||
}
|
||||
|
||||
/// Stringify a discriminator field value for mapping lookup. Mirrors
|
||||
/// `sequential_reader::discriminator_string_value` for the Value form.
|
||||
fn union_discriminator_key(value: &Value, field_path: &str) -> Result<String, AlkTypeError> {
|
||||
match value {
|
||||
Value::String(s) => Ok(s.clone()),
|
||||
Value::Number(n) => Ok(n.to_string()),
|
||||
_ => Err(AlkTypeError::Schema(format!(
|
||||
"union {field_path} has unsupported discriminator value kind: {value}"
|
||||
))),
|
||||
}
|
||||
}
|
||||
|
||||
/// Materialize an `AlkType:Record` field: a count-prefixed sequence of
|
||||
/// `(key, value)` pairs. Wire format per [schema-layer.md](../docs/architecture/schema-layer.md)
|
||||
/// §TRecord: `[count: u32][key_len: u32][key_bytes][value]...` repeated
|
||||
/// `count` times. The count and key-length prefixes respect `endian`.
|
||||
/// The value is materialized according to its declared `AlkType:*` kind.
|
||||
///
|
||||
/// The materialized form is a JSON object mapping each key (string) to
|
||||
/// its materialized value. This matches the `RecordValidator`'s
|
||||
/// expectation (an object whose values match the record's value type).
|
||||
fn materialize_record_packed(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<(Value, usize), AlkTypeError> {
|
||||
let value_schema = field_schema.get("values").ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: record at {field_path} has no values schema"
|
||||
))
|
||||
})?;
|
||||
let count = data_access::read_u32(buffer, offset, field_path, endian)? as usize;
|
||||
let mut cur = offset + U32_SIZE;
|
||||
let mut obj = Map::new();
|
||||
for i in 0..count {
|
||||
let key_path = format!("{field_path}[{i}].key");
|
||||
let key_str = data_access::read_string(buffer, cur, &key_path, endian)?;
|
||||
let key_len = U32_SIZE + key_str.len();
|
||||
cur = cur
|
||||
.checked_add(key_len)
|
||||
.ok_or_else(|| AlkTypeError::Access {
|
||||
field_path: key_path.clone(),
|
||||
reason: format!("key offset {cur} + {key_len} overflows usize"),
|
||||
})?;
|
||||
let val_path = format!("{field_path}[{i}].value");
|
||||
let (val, new_offset) =
|
||||
materialize_field_packed(buffer, root, value_schema, &val_path, cur, endian)?;
|
||||
obj.insert(key_str.to_string(), val);
|
||||
cur = new_offset;
|
||||
}
|
||||
Ok((Value::Object(obj), cur))
|
||||
}
|
||||
|
||||
fn materialize_struct_aligned(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
struct_schema: &Value,
|
||||
path_prefix: &str,
|
||||
offset_map: &crate::offset_map::OffsetMap,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
|
||||
.unwrap_or(struct_schema);
|
||||
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
|
||||
let props = struct_schema
|
||||
.get("properties")
|
||||
.and_then(Value::as_object)
|
||||
@@ -379,7 +501,8 @@ fn materialize_struct_aligned(
|
||||
} else {
|
||||
format!("{path_prefix}.{name}")
|
||||
};
|
||||
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
|
||||
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
|
||||
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: field {path} has no AlkType:* kind"
|
||||
))
|
||||
@@ -390,19 +513,23 @@ fn materialize_struct_aligned(
|
||||
field_path: path.clone(),
|
||||
reason: "field not found in offset map".to_string(),
|
||||
})?;
|
||||
materialize_leaf_at(buffer, field_schema, &path, range.start, endian)?
|
||||
materialize_leaf_at(buffer, root, resolved, &path, range.start, endian)?
|
||||
} else if kind == AlkTypeKind::Struct {
|
||||
// Nested struct — recurse with the same offset map (nested
|
||||
// fields are recorded under `path.*`).
|
||||
materialize_struct_aligned(buffer, field_schema, &path, offset_map, endian)?
|
||||
materialize_struct_aligned(buffer, root, resolved, &path, offset_map, endian)?
|
||||
} else {
|
||||
// Composite leaf (Array, Union, Record) — fall back to packed-style
|
||||
// walk from the field's start offset.
|
||||
// walk from the field's start offset. The offset map records the
|
||||
// field's start; the composite's internal layout is walked
|
||||
// sequentially from that start (variable-length elements can't
|
||||
// be precomputed into the static offset map).
|
||||
let range = offset_map.get(&path).ok_or_else(|| AlkTypeError::Offset {
|
||||
field_path: path.clone(),
|
||||
reason: "field not found in offset map".to_string(),
|
||||
})?;
|
||||
let (value, _) = materialize_field_packed(buffer, field_schema, &path, range.start, endian)?;
|
||||
let (value, _) =
|
||||
materialize_field_packed(buffer, root, resolved, &path, range.start, endian)?;
|
||||
value
|
||||
};
|
||||
obj.insert(name.clone(), value);
|
||||
@@ -412,12 +539,14 @@ fn materialize_struct_aligned(
|
||||
|
||||
fn materialize_leaf_at(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
let (_kind, value) = materialize_field_packed_returning_kind(buffer, field_schema, field_path, offset, endian)?;
|
||||
let (_kind, value) =
|
||||
materialize_field_packed_returning_kind(buffer, root, field_schema, field_path, offset, endian)?;
|
||||
Ok(value)
|
||||
}
|
||||
|
||||
@@ -427,25 +556,413 @@ fn materialize_leaf_at(
|
||||
/// signature stable.
|
||||
fn materialize_field_packed_returning_kind(
|
||||
buffer: &[u8],
|
||||
root: &Value,
|
||||
field_schema: &Value,
|
||||
field_path: &str,
|
||||
offset: usize,
|
||||
endian: Endian,
|
||||
) -> Result<(AlkTypeKind, Value), AlkTypeError> {
|
||||
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
|
||||
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
|
||||
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
|
||||
AlkTypeError::Schema(format!(
|
||||
"materialize: field {field_path} has no AlkType:* kind"
|
||||
))
|
||||
})?;
|
||||
let (value, _) = materialize_field_packed(buffer, field_schema, field_path, offset, endian)?;
|
||||
let (value, _) = materialize_field_packed(buffer, root, field_schema, field_path, offset, endian)?;
|
||||
Ok((kind, value))
|
||||
}
|
||||
|
||||
/// Walk a schema to find its root. Since materialize is called with the
|
||||
/// top-level schema as `struct_schema`, the root is `struct_schema` itself
|
||||
/// for the top-level call. For nested calls (via $ref), we need the
|
||||
/// original root. This helper returns the node itself for now — the $ref
|
||||
/// resolution in `resolve_ref_or_inline` handles the common case.
|
||||
fn root_of(schema: &Value) -> &Value {
|
||||
schema
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
fn materialize_packed_strict(
|
||||
buffer: &[u8],
|
||||
schema: &Value,
|
||||
endian: Endian,
|
||||
) -> Result<Value, AlkTypeError> {
|
||||
materialize_packed(buffer, schema, endian)
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// Bytes materialization (array-of-u8 form, OQ-007 resolution)
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_bytes_packed_returns_array_of_u8() {
|
||||
// Schema: Struct { blob: Bytes }. Wire: [len:u32 LE = 3][0x41 0x42 0xC3]
|
||||
// 0xC3 is invalid UTF-8 lead byte — under the old from_utf8_lossy
|
||||
// path this would corrupt. Under array-of-u8 it round-trips.
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"blob": { "AlkType:Bytes": true }
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 7];
|
||||
buf[0..4].copy_from_slice(&3u32.to_le_bytes());
|
||||
buf[4] = 0x41;
|
||||
buf[5] = 0x42;
|
||||
buf[6] = 0xC3;
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(
|
||||
v["blob"],
|
||||
json!([65, 66, 195]),
|
||||
"bytes should materialize as array of u8"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_bytes_packed_non_utf8_round_trips() {
|
||||
// All-non-UTF-8 bytes — would have been corrupted by from_utf8_lossy.
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"raw": { "AlkType:Bytes": true }
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 8];
|
||||
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
|
||||
buf[4] = 0xFF;
|
||||
buf[5] = 0xFE;
|
||||
buf[6] = 0x00;
|
||||
buf[7] = 0x80;
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(v["raw"], json!([255, 254, 0, 128]));
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// Record materialization
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_record_packed_count_prefixed_pairs() {
|
||||
// Schema: Struct { counts: Record<Uint32> }.
|
||||
// Wire: [count:u32=2][key_len:u32=1][b][value:u32=10][key_len:u32=1][a][value:u32=20]
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"counts": {
|
||||
"AlkType:Record": true,
|
||||
"values": { "AlkType:Uint32": true }
|
||||
}
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 22];
|
||||
let mut off = 0usize;
|
||||
// count = 2
|
||||
buf[off..off + 4].copy_from_slice(&2u32.to_le_bytes());
|
||||
off += 4;
|
||||
// key 1: "b" (len=1)
|
||||
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
|
||||
off += 4;
|
||||
buf[off] = b'b';
|
||||
off += 1;
|
||||
// value 1: 10
|
||||
buf[off..off + 4].copy_from_slice(&10u32.to_le_bytes());
|
||||
off += 4;
|
||||
// key 2: "a" (len=1)
|
||||
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
|
||||
off += 4;
|
||||
buf[off] = b'a';
|
||||
off += 1;
|
||||
// value 2: 20
|
||||
buf[off..off + 4].copy_from_slice(&20u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(
|
||||
v["counts"],
|
||||
json!({ "b": 10, "a": 20 }),
|
||||
"record should materialize as an object"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_record_packed_zero_entries() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"counts": {
|
||||
"AlkType:Record": true,
|
||||
"values": { "AlkType:Uint32": true }
|
||||
}
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 4];
|
||||
buf[0..4].copy_from_slice(&0u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(v["counts"], json!({}));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_record_packed_rejects_short_buffer() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"counts": {
|
||||
"AlkType:Record": true,
|
||||
"values": { "AlkType:Uint32": true }
|
||||
}
|
||||
}
|
||||
});
|
||||
// Buffer too short for the count prefix.
|
||||
let buf = [0u8; 2];
|
||||
let err = materialize_packed_strict(&buf, &schema, Endian::Little).unwrap_err();
|
||||
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// Union materialization — byte-offset and field-name shape parity (OQ-005)
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_union_byte_discriminator_packs_variant_fields() {
|
||||
// Schema: Struct { payload: Union(byte, offset=0, Uint8) mapping { 5 -> Read } }
|
||||
// Read: Struct { id: Uint32 }
|
||||
// Wire: [disc:u8=5][id:u32=42 LE]
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": { "id": { "AlkType:Uint32": true } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 5];
|
||||
buf[0] = 5;
|
||||
buf[1..5].copy_from_slice(&42u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
let payload = &v["payload"];
|
||||
assert_eq!(
|
||||
payload["__discriminator"],
|
||||
json!(5),
|
||||
"byte-offset union should carry __discriminator"
|
||||
);
|
||||
assert_eq!(
|
||||
payload["id"],
|
||||
json!(42),
|
||||
"variant fields should be flattened into the union object"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_union_field_discriminator_includes_typed_disc_value() {
|
||||
// Schema: Struct { payload: Union(field, name="type") mapping { "read" -> Read } }
|
||||
// The union is a struct: [type: String "read"][variant struct].
|
||||
// Read: Struct { n: Uint32 }
|
||||
// Wire: [len:u32=4][read][n:u32=7 LE]
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "field", "name": "type" },
|
||||
"properties": {
|
||||
"type": { "AlkType:String": true }
|
||||
},
|
||||
"mapping": {
|
||||
"read": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": { "n": { "AlkType:Uint32": true } }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 12];
|
||||
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
|
||||
buf[4..8].copy_from_slice(b"read");
|
||||
buf[8..12].copy_from_slice(&7u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
let payload = &v["payload"];
|
||||
// __discriminator carries the mapping key (string).
|
||||
assert_eq!(payload["__discriminator"], json!("read"));
|
||||
// The original typed discriminator value is preserved under its field name.
|
||||
assert_eq!(payload["type"], json!("read"));
|
||||
// Variant fields are flattened.
|
||||
assert_eq!(payload["n"], json!(7));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_union_field_discriminator_advances_offset_correctly() {
|
||||
// Regression: the old field-name path returned the start offset,
|
||||
// not the end. A struct with a union followed by another field
|
||||
// would read the second field from the wrong position. This test
|
||||
// places a Uint32 after the union and verifies it reads correctly.
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "field", "name": "type" },
|
||||
"properties": {
|
||||
"type": { "AlkType:String": true }
|
||||
},
|
||||
"mapping": {
|
||||
"read": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": { "n": { "AlkType:Uint32": true } }
|
||||
}
|
||||
}
|
||||
},
|
||||
"trailer": { "AlkType:Uint32": true }
|
||||
}
|
||||
});
|
||||
// Wire: [len:u32=4]["read"][n:u32=7][trailer:u32=99]
|
||||
let mut buf = vec![0u8; 16];
|
||||
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
|
||||
buf[4..8].copy_from_slice(b"read");
|
||||
buf[8..12].copy_from_slice(&7u32.to_le_bytes());
|
||||
buf[12..16].copy_from_slice(&99u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(v["trailer"], json!(99), "trailer should read from after the union");
|
||||
assert_eq!(v["payload"]["n"], json!(7));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_union_byte_discriminator_unknown_value_is_access_error() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": { "5": { "AlkType:Struct": true, "properties": {} } }
|
||||
}
|
||||
}
|
||||
});
|
||||
let buf = [99u8];
|
||||
let err = materialize_packed_strict(&buf, &schema, Endian::Little).unwrap_err();
|
||||
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// $ref resolution (root-schema threading)
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_union_with_ref_variant_resolves_via_root() {
|
||||
// The SFTP shape: struct wrapping a union whose variants are $refs
|
||||
// into $defs. The materializer must resolve the $ref against the
|
||||
// root schema to find the variant's AlkType:Struct kind.
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": { "$ref": "#/$defs/Read" }
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"Read": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": { "id": { "AlkType:Uint32": true } }
|
||||
}
|
||||
}
|
||||
});
|
||||
let mut buf = vec![0u8; 5];
|
||||
buf[0] = 5;
|
||||
buf[1..5].copy_from_slice(&42u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(v["payload"]["__discriminator"], json!(5));
|
||||
assert_eq!(v["payload"]["id"], json!(42));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn materialize_array_with_ref_element_resolves_via_root() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"items": {
|
||||
"AlkType:Array": true,
|
||||
"minItems": 2,
|
||||
"maxItems": 2,
|
||||
"items": { "$ref": "#/$defs/Point" }
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"Point": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"x": { "AlkType:Uint16": true },
|
||||
"y": { "AlkType:Uint16": true }
|
||||
}
|
||||
}
|
||||
}
|
||||
});
|
||||
// Wire: [x:u16=1 LE][y:u16=2 LE][x:u16=3 LE][y:u16=4 LE]
|
||||
let mut buf = vec![0u8; 8];
|
||||
buf[0..2].copy_from_slice(&1u16.to_le_bytes());
|
||||
buf[2..4].copy_from_slice(&2u16.to_le_bytes());
|
||||
buf[4..6].copy_from_slice(&3u16.to_le_bytes());
|
||||
buf[6..8].copy_from_slice(&4u16.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(
|
||||
v["items"],
|
||||
json!([
|
||||
{ "x": 1, "y": 2 },
|
||||
{ "x": 3, "y": 4 }
|
||||
])
|
||||
);
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// Smoke: a struct field after a nested struct advances correctly
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_struct_with_nested_struct_then_field() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"header": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"a": { "AlkType:Uint8": true },
|
||||
"b": { "AlkType:Uint8": true }
|
||||
}
|
||||
},
|
||||
"after": { "AlkType:Uint32": true }
|
||||
}
|
||||
});
|
||||
// Wire: [a=1][b=2][after=0x03030303 LE = 50529027]
|
||||
let mut buf = vec![0u8; 6];
|
||||
buf[0] = 1;
|
||||
buf[1] = 2;
|
||||
buf[2..6].copy_from_slice(&0x03030303u32.to_le_bytes());
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
|
||||
assert_eq!(v["header"]["a"], json!(1));
|
||||
assert_eq!(v["header"]["b"], json!(2));
|
||||
assert_eq!(v["after"], json!(0x03030303u32));
|
||||
}
|
||||
|
||||
// -----------------------------------------------------------------
|
||||
// Sanity: existing packed chunk-header path still works
|
||||
// -----------------------------------------------------------------
|
||||
|
||||
#[test]
|
||||
fn materialize_chunk_header_packed() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"endian": "big",
|
||||
"properties": {
|
||||
"channel_id": { "AlkType:Uint32": true },
|
||||
"length": { "AlkType:Uint32": true }
|
||||
}
|
||||
});
|
||||
let buf = [0u8, 0u8, 0u8, 42u8, 0u8, 0u8, 0u8, 7u8];
|
||||
let v = materialize_packed_strict(&buf, &schema, Endian::Big).expect("materialize");
|
||||
assert_eq!(v["channel_id"], json!(42));
|
||||
assert_eq!(v["length"], json!(7));
|
||||
}
|
||||
}
|
||||
167
src/schema.rs
167
src/schema.rs
@@ -467,6 +467,64 @@ fn normalize_refs_recursive(node: &mut Value) {
|
||||
}
|
||||
}
|
||||
|
||||
/// Inline `$ref`s in `AlkType:Union` `mapping` entries by resolving them
|
||||
/// against the schema root and replacing each `$ref`-bearing variant with
|
||||
/// the resolved schema (in place). This runs after [`normalize_refs`] and
|
||||
/// before [`crate::validation::build_validator`].
|
||||
///
|
||||
/// The `UnionValidator`'s factory builds a sub-validator for each variant
|
||||
/// at validator-construction time. The factory receives the union node as
|
||||
/// `parent`, but `$defs` live at the schema root — not on the union node.
|
||||
/// Without inlining, the factory can't resolve `$ref`s like
|
||||
/// `"#/$defs/Init"` because the union node doesn't contain `$defs`.
|
||||
/// Inlining the refs before validator construction sidesteps this: each
|
||||
/// variant in the `mapping` becomes a full inline schema, so the factory
|
||||
/// can build a sub-validator directly.
|
||||
///
|
||||
/// Only `AlkType:Union` mappings are inlined. Other `$ref`s (e.g. in
|
||||
/// `properties` or `items`) are left in place — the materializer resolves
|
||||
/// them at read time via [`resolve_ref_or_inline`] against the schema
|
||||
/// root, and the jsonschema built-in validator resolves them via its own
|
||||
/// `$ref` resolution (which has access to the full schema root).
|
||||
///
|
||||
/// Idempotent: a mapping whose variants are already inline (no `$ref`)
|
||||
/// is left unchanged.
|
||||
pub fn inline_union_variant_refs(root: &mut Value) {
|
||||
// Collect (path-to-union-mapping, ref-path, variant-key) tuples by
|
||||
// walking the tree with immutable access, then resolve and apply
|
||||
// the inlining with mutable access. This avoids the borrow conflict
|
||||
// of holding both &root and &mut root simultaneously.
|
||||
let root_clone = root.clone();
|
||||
inline_union_variant_refs_recursive(root, &root_clone);
|
||||
}
|
||||
|
||||
fn inline_union_variant_refs_recursive(node: &mut Value, root: &Value) {
|
||||
if let Value::Object(obj) = node {
|
||||
// Check if this node is an AlkType:Union with a mapping.
|
||||
let is_union = obj
|
||||
.keys()
|
||||
.any(|k| k == "AlkType:Union");
|
||||
if is_union {
|
||||
if let Some(Value::Object(mapping)) = obj.get_mut("mapping") {
|
||||
for (_key, variant) in mapping.iter_mut() {
|
||||
if let Some(Value::String(ref_path)) = variant.get("$ref").cloned() {
|
||||
if let Some(resolved) = resolve_ref(root, &ref_path) {
|
||||
*variant = resolved.clone();
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
for value in obj.values_mut() {
|
||||
inline_union_variant_refs_recursive(value, root);
|
||||
}
|
||||
} else if let Value::Array(arr) = node {
|
||||
for item in arr.iter_mut() {
|
||||
inline_union_variant_refs_recursive(item, root);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -810,6 +868,115 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
// ----- inline_union_variant_refs tests -----
|
||||
|
||||
#[test]
|
||||
fn inline_union_variant_refs_inlines_mapping_refs() {
|
||||
let mut schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"payload": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": {"$ref": "#/$defs/Read"},
|
||||
"6": {"$ref": "#/$defs/Write"}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": {
|
||||
"Read": { "AlkType:Struct": true, "properties": { "id": { "AlkType:Uint32": true } } },
|
||||
"Write": { "AlkType:Struct": true, "properties": { "n": { "AlkType:Uint16": true } } }
|
||||
}
|
||||
});
|
||||
inline_union_variant_refs(&mut schema);
|
||||
// The mapping entries should now be the full inline schemas.
|
||||
let mapping = &schema["properties"]["payload"]["mapping"];
|
||||
assert_eq!(
|
||||
mapping["5"],
|
||||
json!({ "AlkType:Struct": true, "properties": { "id": { "AlkType:Uint32": true } } })
|
||||
);
|
||||
assert_eq!(
|
||||
mapping["6"],
|
||||
json!({ "AlkType:Struct": true, "properties": { "n": { "AlkType:Uint16": true } } })
|
||||
);
|
||||
// $defs are left in place (the materializer may still use them).
|
||||
assert!(schema["$defs"].is_object());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_union_variant_refs_leaves_inline_variants_unchanged() {
|
||||
let mut schema = json!({
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": { "AlkType:Struct": true, "properties": { "id": { "AlkType:Uint32": true } } }
|
||||
}
|
||||
});
|
||||
let before = schema.clone();
|
||||
inline_union_variant_refs(&mut schema);
|
||||
assert_eq!(schema, before, "inline variants should be unchanged");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_union_variant_refs_is_idempotent() {
|
||||
let mut schema = json!({
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": { "5": {"$ref": "#/$defs/Read"} },
|
||||
"$defs": { "Read": { "AlkType:Struct": true, "properties": { "id": { "AlkType:Uint32": true } } } }
|
||||
});
|
||||
inline_union_variant_refs(&mut schema);
|
||||
let after_first = schema.clone();
|
||||
inline_union_variant_refs(&mut schema);
|
||||
assert_eq!(schema, after_first, "second inlining should be a no-op");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_union_variant_refs_handles_nested_union() {
|
||||
// A struct containing a union whose variants are $refs. The
|
||||
// inlining walks into the struct's properties and inlines the
|
||||
// union's mapping refs.
|
||||
let mut schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"outer": {
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"inner_union": {
|
||||
"AlkType:Union": true,
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": { "1": {"$ref": "#/$defs/Init"} }
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"$defs": { "Init": { "AlkType:Struct": true, "properties": { "v": { "AlkType:Uint32": true } } } }
|
||||
});
|
||||
inline_union_variant_refs(&mut schema);
|
||||
let mapping = &schema["properties"]["outer"]["properties"]["inner_union"]["mapping"];
|
||||
assert_eq!(
|
||||
mapping["1"],
|
||||
json!({ "AlkType:Struct": true, "properties": { "v": { "AlkType:Uint32": true } } })
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn inline_union_variant_refs_leaves_non_union_refs_in_place() {
|
||||
// $refs in `properties` (not in a union mapping) are left in
|
||||
// place — the materializer resolves them at read time.
|
||||
let mut schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"properties": {
|
||||
"field": {"$ref": "#/$defs/SomeType"}
|
||||
},
|
||||
"$defs": { "SomeType": { "AlkType:Uint32": true } }
|
||||
});
|
||||
let before = schema.clone();
|
||||
inline_union_variant_refs(&mut schema);
|
||||
assert_eq!(schema, before, "non-union refs should be unchanged");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn as_str_round_trips_for_all_kinds() {
|
||||
for kind in [
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
use crate::error::AlkTypeError;
|
||||
use jsonschema::{Keyword, ValidationError};
|
||||
use serde_json::{Map, Value};
|
||||
use std::collections::HashMap;
|
||||
|
||||
/// Build a jsonschema validator with all 19 `AlkType:*` custom keywords
|
||||
/// registered.
|
||||
@@ -165,8 +166,12 @@ struct BytesValidator {
|
||||
}
|
||||
impl Keyword for BytesValidator {
|
||||
fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> {
|
||||
match instance.as_str() {
|
||||
Some(s) => {
|
||||
match instance {
|
||||
// String form: used by `validate_json` consumers that hold a
|
||||
// JSON representation (e.g. deserialized from JSON, where bytes
|
||||
// are conventionally a string). `maxLength` is the byte length
|
||||
// of the string (UTF-8 byte count, matching `StringValidator`).
|
||||
Value::String(s) => {
|
||||
if let Some(max) = self.max_length {
|
||||
if s.len() > max {
|
||||
return Err(ValidationError::custom(format!(
|
||||
@@ -177,13 +182,49 @@ impl Keyword for BytesValidator {
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
None => Err(ValidationError::custom("expected a string for bytes")),
|
||||
// Array form: used by `validate_bytes` (the materializer
|
||||
// produces `Value::Array` of `Value::Number` for `AlkType:Bytes`
|
||||
// fields — one entry per byte, 0..=255). `maxLength` is the
|
||||
// array length (max byte count). This is the round-trippable
|
||||
// form for non-UTF-8 bytes (see OQ-007 resolution).
|
||||
Value::Array(arr) => {
|
||||
if let Some(max) = self.max_length {
|
||||
if arr.len() > max {
|
||||
return Err(ValidationError::custom(format!(
|
||||
"bytes array length {} exceeds maxLength {max}",
|
||||
arr.len()
|
||||
)));
|
||||
}
|
||||
}
|
||||
// Each entry must be a non-negative integer 0..=255.
|
||||
for (i, entry) in arr.iter().enumerate() {
|
||||
let n = entry.as_u64().ok_or_else(|| {
|
||||
ValidationError::custom(format!(
|
||||
"bytes array entry {i} is not a non-negative integer"
|
||||
))
|
||||
})?;
|
||||
if n > 255 {
|
||||
return Err(ValidationError::custom(format!(
|
||||
"bytes array entry {i} = {n} is not a u8 (0..=255)"
|
||||
)));
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
_ => Err(ValidationError::custom(
|
||||
"expected a string or array of u8 for bytes",
|
||||
)),
|
||||
}
|
||||
}
|
||||
fn is_valid(&self, instance: &Value) -> bool {
|
||||
instance
|
||||
.as_str()
|
||||
.is_some_and(|s| self.max_length.is_none_or(|max| s.len() <= max))
|
||||
match instance {
|
||||
Value::String(s) => self.max_length.is_none_or(|max| s.len() <= max),
|
||||
Value::Array(arr) => {
|
||||
self.max_length.is_none_or(|max| arr.len() <= max)
|
||||
&& arr.iter().all(|e| e.as_u64().is_some_and(|n| n <= 255))
|
||||
}
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -200,6 +241,96 @@ impl Keyword for EnumValidator {
|
||||
}
|
||||
}
|
||||
|
||||
/// `AlkType:Union` validator: dispatches to the variant schema selected
|
||||
/// by the discriminator value.
|
||||
///
|
||||
/// The materializer (`src/materialize.rs`) reads the discriminator from
|
||||
/// the binary buffer, looks up the variant in the union's `mapping`, and
|
||||
/// produces a `Value` object of the form:
|
||||
///
|
||||
/// ```json
|
||||
/// { "__discriminator": <disc-value>, ...variant-fields }
|
||||
/// ```
|
||||
///
|
||||
/// For byte-offset discriminators, `<disc-value>` is a number (the raw
|
||||
/// discriminator integer). For field-name discriminators, it's the
|
||||
/// stringified discriminator field value (the mapping key). The
|
||||
/// `UnionValidator` reads `__discriminator`, looks up the corresponding
|
||||
/// variant schema in the union's `mapping`, and validates the instance
|
||||
/// against that variant schema (via a recursively-built sub-validator).
|
||||
///
|
||||
/// This closes the OQ-008 gap: without variant dispatch, `validate_bytes`
|
||||
/// on a union would only check that the materialized value is an object
|
||||
/// (the read phase handles type-level checks like integer ranges), but
|
||||
/// would NOT check the variant's field constraints (e.g. `maxLength` on
|
||||
/// a `Bytes` field inside a variant struct).
|
||||
///
|
||||
/// The sub-validators are built once at factory time (when the parent
|
||||
/// validator tree is being constructed) and stored in the
|
||||
/// `UnionValidator` struct. Each sub-validator is a full
|
||||
/// `jsonschema::Validator` with all 19 AlkType custom keywords
|
||||
/// registered (via [`build_validator`]), so nested AlkType kinds inside
|
||||
/// variants are validated correctly.
|
||||
struct UnionValidator {
|
||||
/// Sub-validators keyed by the stringified discriminator value (the
|
||||
/// mapping key). Each validator is built from the variant's schema
|
||||
/// (with `$ref`s resolved against the parent union schema's `$defs`
|
||||
/// when present).
|
||||
variant_validators: HashMap<String, jsonschema::Validator>,
|
||||
}
|
||||
|
||||
impl Keyword for UnionValidator {
|
||||
fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> {
|
||||
let obj = instance.as_object().ok_or_else(|| {
|
||||
ValidationError::custom("expected an object for union")
|
||||
})?;
|
||||
let disc = obj.get("__discriminator").ok_or_else(|| {
|
||||
ValidationError::custom(
|
||||
"union instance is missing the '__discriminator' field",
|
||||
)
|
||||
})?;
|
||||
// The mapping key is the stringified discriminator value. For
|
||||
// byte-offset discriminators the materializer produces a number;
|
||||
// for field-name discriminators it produces a string (the
|
||||
// mapping key directly).
|
||||
let key = match disc {
|
||||
Value::String(s) => s.clone(),
|
||||
Value::Number(n) => n.to_string(),
|
||||
_ => {
|
||||
return Err(ValidationError::custom(format!(
|
||||
"union '__discriminator' must be a string or number, got {disc}"
|
||||
)));
|
||||
}
|
||||
};
|
||||
let validator = self.variant_validators.get(&key).ok_or_else(|| {
|
||||
ValidationError::custom(format!(
|
||||
"union discriminator value '{key}' not in mapping"
|
||||
))
|
||||
})?;
|
||||
validator.validate(instance).map_err(|e| {
|
||||
ValidationError::custom(format!(
|
||||
"union variant '{key}' failed validation: {e}"
|
||||
))
|
||||
})
|
||||
}
|
||||
fn is_valid(&self, instance: &Value) -> bool {
|
||||
let Some(obj) = instance.as_object() else {
|
||||
return false;
|
||||
};
|
||||
let Some(disc) = obj.get("__discriminator") else {
|
||||
return false;
|
||||
};
|
||||
let key = match disc {
|
||||
Value::String(s) => s.clone(),
|
||||
Value::Number(n) => n.to_string(),
|
||||
_ => return false,
|
||||
};
|
||||
self.variant_validators
|
||||
.get(&key)
|
||||
.is_some_and(|v| v.is_valid(instance))
|
||||
}
|
||||
}
|
||||
|
||||
struct TimestampValidator;
|
||||
impl Keyword for TimestampValidator {
|
||||
fn validate<'i>(&self, instance: &'i Value) -> Result<(), ValidationError<'i>> {
|
||||
@@ -258,7 +389,6 @@ fn is_rfc3339_timestamp(s: &str) -> bool {
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
define_type_validator!(StructValidator, struct_factory, "AlkType:Struct", is_object, "expected an object");
|
||||
define_type_validator!(UnionValidator, union_factory, "AlkType:Union", is_object, "expected an object for union");
|
||||
define_type_validator!(ArrayValidator, array_factory, "AlkType:Array", is_array, "expected an array");
|
||||
define_type_validator!(RecordValidator, record_factory, "AlkType:Record", is_object, "expected an object for record");
|
||||
define_type_validator!(BooleanValidator, boolean_factory, "AlkType:Boolean", is_boolean, "expected a boolean");
|
||||
@@ -327,6 +457,53 @@ fn timestamp_factory<'a>(
|
||||
}
|
||||
}
|
||||
|
||||
/// Build a `UnionValidator` with a sub-validator per mapping entry.
|
||||
///
|
||||
/// The factory receives the union schema object as `parent` (the
|
||||
/// `Map<String, Value>` containing `discriminator` and `mapping`). For
|
||||
/// each entry in `mapping`, the variant schema is used to build a full
|
||||
/// `jsonschema::Validator` via [`build_validator`] (so nested AlkType
|
||||
/// kinds are validated).
|
||||
///
|
||||
/// `$ref`s in the `mapping` are inlined by
|
||||
/// [`crate::schema::inline_union_variant_refs`] during
|
||||
/// `AlkTypeEngine::compile` (before `build_validator` is called), so the
|
||||
/// factory sees full inline variant schemas — no `$ref` resolution is
|
||||
/// needed here.
|
||||
///
|
||||
/// The sub-validators are built once at factory time and reused for
|
||||
/// every validation call. The `UnionValidator` dispatches on the
|
||||
/// `__discriminator` field in the instance at validation time.
|
||||
fn union_factory<'a>(
|
||||
parent: &'a Map<String, Value>,
|
||||
value: &'a Value,
|
||||
_path: jsonschema::paths::Location,
|
||||
) -> Result<Box<dyn Keyword>, ValidationError<'a>> {
|
||||
if value.as_bool() != Some(true) {
|
||||
return Err(ValidationError::schema(
|
||||
"AlkType:Union must be set to true",
|
||||
));
|
||||
}
|
||||
let mapping = parent
|
||||
.get("mapping")
|
||||
.and_then(Value::as_object)
|
||||
.ok_or_else(|| {
|
||||
ValidationError::schema("AlkType:Union is missing 'mapping' object")
|
||||
})?;
|
||||
let mut variant_validators: HashMap<String, jsonschema::Validator> = HashMap::new();
|
||||
for (key, variant_schema) in mapping.iter() {
|
||||
// Variants are inlined by inline_union_variant_refs at compile
|
||||
// time, so variant_schema is a full inline schema here.
|
||||
let sub_validator = build_validator(variant_schema).map_err(|e| {
|
||||
ValidationError::schema(format!(
|
||||
"failed to build sub-validator for union variant '{key}': {e}"
|
||||
))
|
||||
})?;
|
||||
variant_validators.insert(key.clone(), sub_validator);
|
||||
}
|
||||
Ok(Box::new(UnionValidator { variant_validators }))
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
@@ -503,20 +680,44 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn validates_bytes_and_maxlength() {
|
||||
// No "type" keyword — the BytesValidator accepts both string
|
||||
// (validate_json path) and array-of-u8 (validate_bytes path).
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"blob": { "AlkType:Bytes": true, "type": "string", "maxLength": 4 }
|
||||
"blob": { "AlkType:Bytes": true, "maxLength": 4 }
|
||||
},
|
||||
"required": ["blob"]
|
||||
});
|
||||
let validator = validator_for(&schema);
|
||||
// String form (validate_json consumers).
|
||||
assert!(validator.is_valid(&json!({"blob": "abcd"})));
|
||||
assert!(!validator.is_valid(&json!({"blob": "abcde"})));
|
||||
// Array form (validate_bytes materializer).
|
||||
assert!(validator.is_valid(&json!({"blob": [65, 66, 67, 68]})));
|
||||
assert!(!validator.is_valid(&json!({"blob": [65, 66, 67, 68, 69]})));
|
||||
// Non-string, non-array is rejected.
|
||||
assert!(!validator.is_valid(&json!({"blob": 42})));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validates_bytes_array_rejects_out_of_range_entry() {
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"blob": { "AlkType:Bytes": true, "maxLength": 4 }
|
||||
},
|
||||
"required": ["blob"]
|
||||
});
|
||||
let validator = validator_for(&schema);
|
||||
// 256 is not a u8.
|
||||
assert!(!validator.is_valid(&json!({"blob": [65, 256]})));
|
||||
// Non-integer entry is rejected.
|
||||
assert!(!validator.is_valid(&json!({"blob": [65, "x"]})));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn enum_validator_is_noop_and_builtin_enum_handles_membership() {
|
||||
let schema = json!({
|
||||
@@ -594,6 +795,9 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn validates_union_type() {
|
||||
// A struct with a union field. The union has a byte-offset
|
||||
// discriminator and two variant structs. The validator dispatches
|
||||
// on __discriminator and validates the variant fields.
|
||||
let schema = json!({
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
@@ -601,17 +805,39 @@ mod tests {
|
||||
"packet": {
|
||||
"AlkType:Union": true,
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"type": { "type": "string" }
|
||||
},
|
||||
"required": ["type"]
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": {
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
"properties": { "id": { "AlkType:Uint32": true, "type": "integer" } },
|
||||
"required": ["id"]
|
||||
},
|
||||
"6": {
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
"properties": { "name": { "AlkType:String": true, "type": "string" } },
|
||||
"required": ["name"]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"required": ["packet"]
|
||||
});
|
||||
let validator = validator_for(&schema);
|
||||
assert!(validator.is_valid(&json!({"packet": {"type": "read"}})));
|
||||
// Valid: disc=5, id present.
|
||||
assert!(validator.is_valid(&json!({"packet": {"__discriminator": 5, "id": 42}})));
|
||||
// Valid: disc=6, name present.
|
||||
assert!(validator.is_valid(&json!({"packet": {"__discriminator": 6, "name": "hi"}})));
|
||||
// Invalid: not an object.
|
||||
assert!(!validator.is_valid(&json!({"packet": "not-object"})));
|
||||
// Invalid: missing __discriminator.
|
||||
assert!(!validator.is_valid(&json!({"packet": {"id": 42}})));
|
||||
// Invalid: disc value not in mapping.
|
||||
assert!(!validator.is_valid(&json!({"packet": {"__discriminator": 99, "id": 42}})));
|
||||
// Invalid: variant fields don't match the variant schema (disc=5
|
||||
// requires "id", but "name" is given).
|
||||
assert!(!validator.is_valid(&json!({"packet": {"__discriminator": 5, "name": "hi"}})));
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -742,18 +968,28 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn validate_bytes_ok_and_maxlength_exceeded() {
|
||||
let schema = json!({"AlkType:Bytes": true, "type": "string", "maxLength": 4});
|
||||
// No "type" keyword — the BytesValidator accepts both string and
|
||||
// array-of-u8 forms. The "type": "string" form is still valid for
|
||||
// schemas that want to restrict to the string form (validate_json
|
||||
// consumers that never see validate_bytes materialization).
|
||||
let schema = json!({"AlkType:Bytes": true, "maxLength": 4});
|
||||
let v = validator_for(&schema);
|
||||
// String form.
|
||||
assert!(v.validate(&json!("abcd")).is_ok());
|
||||
let long = json!("abcde");
|
||||
assert!(v.validate(&long).is_err());
|
||||
assert!(v.validate(&json!("abcde")).is_err());
|
||||
// Array form.
|
||||
assert!(v.validate(&json!([65, 66, 67, 68])).is_ok());
|
||||
assert!(v.validate(&json!([65, 66, 67, 68, 69])).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_bytes_non_string_is_err() {
|
||||
let schema = json!({"AlkType:Bytes": true, "type": "string"});
|
||||
fn validate_bytes_non_string_non_array_is_err() {
|
||||
let schema = json!({"AlkType:Bytes": true});
|
||||
let v = validator_for(&schema);
|
||||
assert!(v.validate(&json!(42)).is_err());
|
||||
assert!(v.validate(&json!(true)).is_err());
|
||||
assert!(v.validate(&json!("ok")).is_ok());
|
||||
assert!(v.validate(&json!([0, 255])).is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -788,10 +1024,32 @@ mod tests {
|
||||
|
||||
#[test]
|
||||
fn validate_union_ok_and_err() {
|
||||
let schema = json!({"AlkType:Union": true, "type": "object"});
|
||||
// A standalone union with a mapping. The validator dispatches on
|
||||
// __discriminator and validates the variant fields.
|
||||
let schema = json!({
|
||||
"AlkType:Union": true,
|
||||
"type": "object",
|
||||
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
|
||||
"mapping": {
|
||||
"5": {
|
||||
"AlkType:Struct": true,
|
||||
"type": "object",
|
||||
"properties": { "id": { "AlkType:Uint32": true, "type": "integer" } },
|
||||
"required": ["id"]
|
||||
}
|
||||
}
|
||||
});
|
||||
let v = validator_for(&schema);
|
||||
assert!(v.validate(&json!({"type": "read"})).is_ok());
|
||||
// Valid: disc=5, id present.
|
||||
assert!(v.validate(&json!({"__discriminator": 5, "id": 42})).is_ok());
|
||||
// Invalid: not an object.
|
||||
assert!(v.validate(&json!("not-object")).is_err());
|
||||
// Invalid: missing __discriminator.
|
||||
assert!(v.validate(&json!({"id": 42})).is_err());
|
||||
// Invalid: unknown discriminator.
|
||||
assert!(v.validate(&json!({"__discriminator": 99, "id": 42})).is_err());
|
||||
// Invalid: variant fields don't match.
|
||||
assert!(v.validate(&json!({"__discriminator": 5, "id": "not-int"})).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
Reference in New Issue
Block a user