alkdev/alktype
6
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Resolve v0.1.0 open questions and fix production-readiness issues

Browse Source 
POC: /workspace/alktype-builder-poc/ (18/18 tests pass, findings in
docs/research/alktype-builder-poc/findings.md). Round 2 adds the SFTP
Packet validate_bytes tests (7 new: valid Init/Read/Write/Status,
short buffer, unknown discriminator, over-maxLength Bytes).

Open questions resolved (OQ-004 through OQ-008):
- OQ-004: Discriminator::Field name is String (already implemented;
  docs updated to mark resolved)
- OQ-005: Both union discriminator kinds return the same shape:
  {__discriminator, ...variant-fields}. Field-name path also had a
  real offset bug (returned start, not end) - fixed.
- OQ-006: builder.md Example 3 now wraps the Union in a
  Schema::struct_().field("payload", ...) and merges $defs via
  Definitions::merge_into (matches the engine's AlkType:Struct-at-root
  constraint and the SFTP wire shape)
- OQ-007: Bytes materialization is array-of-u8 (Value::Array of
  Value::Number, one entry per byte 0..=255). BytesValidator accepts
  both Value::String (validate_json) and Value::Array (validate_bytes).
  maxLength = max byte count. Replaces the lossy from_utf8_lossy path
  that corrupted non-UTF-8 bytes and broke maxLength semantics.
- OQ-008 (new): UnionValidator now dispatches to variant schemas via
  sub-validators built at factory time. AlkTypeEngine::compile calls
  schema::inline_union_variant_refs before build_validator to inline
  $refs in union mapping entries (necessary because union_factory
  receives the union node, but $defs live at the schema root).

Production-readiness fixes in src/ (no stubs/hedges in a published crate):
- materialize.rs: Record stub -> full count-prefixed key/value pair
  implementation per schema-layer.md TRecord
- materialize.rs: root_of() was broken (returned the current node, not
  the schema root) -> root schema threaded through every recursive call
  so resolve_ref_or_inline can resolve $refs for nested composites
- builder.rs: LengthPrefixed encoding setter was a no-op when the
  keyword was already in object form -> complete the branch (updates
  the encoding entry in place for both LengthPrefixed and OffsetIndirect)
- builder.rs, engine.rs: POC-referencing comments cleaned up; the
  round-trip test's or_else fallback (papering over write_field being
  aligned-only) replaced with direct byte writes

Documentation:
- builder.md: Example 3 updated; Discriminator::Field spec shows String;
  Open Questions section updated (OQ-004 resolved)
- validation.md: AlkType:Bytes and AlkType:Union validator descriptions
  updated for array-of-u8 form and variant dispatch
- open-questions.md: OQ-004/005/006/007/008 marked resolved; new
  Validation theme entries
- questions/004-008: individual OQ files updated with resolutions
- findings.md: round 2 results documented

Verification:
- cargo test: 369 -> 391 tests pass (22 new: 14 materialize, 5
  inline_union_variant_refs, 3 validation/builder)
- cargo clippy --all-targets: clean
- POC: 11 -> 18 tests (7 new SFTP Packet tests); all pass
This commit is contained in:
glm-5.2 committed 2026-08-11 07:28:24 +00:00
1 parent c6893eece8
commit c0217d91a8
15 files changed
+1553 -493
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

src/materialize.rs
+633 -116
View File
@@ -1,16 +1,26 @@
//! Materialize a `serde_json::Value` tree from a binary buffer by walking
//! the schema. Used by [`crate::engine::AlkTypeEngine::validate_bytes`]
//! (ADR-010) to collapse the two-step dance (read bytes → `Value`,
//! (ADR-010) to collapse the two-step dance (read bytes -> `Value`,
//! then validate `Value`) into a single call.
//!
//! The materializer reuses the [`crate::data_access`] read functions for
//! leaf kinds and recurses into composites (`Struct`, `Array`, `Record`).
//! For `Union`, it dispatches on the discriminator and recurses into the
//! variant. Errors carry the field path (ADR-004).
//! leaf kinds and recurses into composites (`Struct`, `Array`, `Record`,
//! `Union`). For `Union`, it dispatches on the discriminator and recurses
//! into the variant. Errors carry the field path (ADR-004).
//!
//! Packed mode walks sequentially from offset 0; aligned mode reads at
//! offsets from the [`crate::offset_map::OffsetMap`]. Both produce the
//! same `Value` form; the validator is mode-agnostic.
//!
//! ## `$ref` resolution
//!
//! The root schema is threaded through every recursive call so that
//! `$ref` pointers (e.g. `"#/$defs/Read"`) in composite variants can be
//! resolved against the schema root via
//! [`crate::schema::resolve_ref_or_inline`]. This is load-bearing for
//! unions and arrays whose variants/elements are `$ref`s into `$defs`
//! (the SFTP Packet shape: a struct wrapping a union with `$ref`
//! variants).
use crate::data_access;
use crate::error::AlkTypeError;
@@ -26,13 +36,14 @@ const U32_SIZE: usize = 4;
///
/// `schema` must declare `AlkType:Struct` at the root (the same
/// requirement as [`crate::sequential_reader::SequentialReader::new`]).
/// `endian` is the schema's endianness (parsed by the caller).
/// `endian` is the schema's endianness (parsed by the caller). The
/// `schema` is also used as the root for `$ref` resolution.
pub fn materialize_packed(
buffer: &[u8],
schema: &Value,
endian: Endian,
) -> Result<Value, AlkTypeError> {
materialize_struct_packed(buffer, schema, "", endian)
materialize_struct_packed(buffer, schema, schema, "", endian)
}
/// Materialize a `Value` tree from `buffer` by walking `schema` in aligned
@@ -40,29 +51,24 @@ pub fn materialize_packed(
///
/// `schema` must declare `AlkType:Struct` at the root. `offset_map` must
/// have been computed from the same `schema`. `endian` is the schema's
/// endianness.
/// endianness. The `schema` is also used as the root for `$ref` resolution.
pub fn materialize_aligned(
buffer: &[u8],
schema: &Value,
offset_map: &crate::offset_map::OffsetMap,
endian: Endian,
) -> Result<Value, AlkTypeError> {
// For aligned mode, the offset map records byte ranges for each leaf
// field via dotted paths. To materialize a struct, we walk the
// schema's `properties` and for each field, look up the path in the
// offset map. Composites need recursion with their own offset map
// entries (e.g., `header.magic` for a nested struct).
materialize_struct_aligned(buffer, schema, "", offset_map, endian)
materialize_struct_aligned(buffer, schema, schema, "", offset_map, endian)
}
fn materialize_struct_packed(
buffer: &[u8],
root: &Value,
struct_schema: &Value,
path_prefix: &str,
endian: Endian,
) -> Result<Value, AlkTypeError> {
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
.unwrap_or(struct_schema);
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
let kind = get_alktype_kind_loose_enum(struct_schema).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: schema at {path_prefix} has no AlkType:* kind"
@@ -89,7 +95,8 @@ fn materialize_struct_packed(
} else {
format!("{path_prefix}.{name}")
};
let (value, new_offset) = materialize_field_packed(buffer, field_schema, &path, offset, endian)?;
let (value, new_offset) =
materialize_field_packed(buffer, root, field_schema, &path, offset, endian)?;
obj.insert(name.clone(), value);
offset = new_offset;
}
@@ -98,12 +105,18 @@ fn materialize_struct_packed(
fn materialize_field_packed(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
endian: Endian,
) -> Result<(Value, usize), AlkTypeError> {
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
// Resolve $ref against the root before checking the kind. A
// $ref-bearing node has no AlkType:* kind; the kind lives on the
// resolved target. This is load-bearing for union variants and
// array elements that reference $defs.
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: field {field_path} has no AlkType:* kind: {field_schema}"
))
@@ -143,19 +156,21 @@ fn materialize_field_packed(
}
AlkTypeKind::Float32 => {
let v = data_access::read_f32(buffer, offset, field_path, endian)?;
Ok(
(serde_json::Number::from_f64(v as f64)
Ok((
serde_json::Number::from_f64(v as f64)
.map(Value::Number)
.unwrap_or(Value::Null), offset + 4),
)
.unwrap_or(Value::Null),
offset + 4,
))
}
AlkTypeKind::Float64 => {
let v = data_access::read_f64(buffer, offset, field_path, endian)?;
Ok(
(serde_json::Number::from_f64(v)
Ok((
serde_json::Number::from_f64(v)
.map(Value::Number)
.unwrap_or(Value::Null), offset + 8),
)
.unwrap_or(Value::Null),
offset + 8,
))
}
AlkTypeKind::Boolean => {
let v = data_access::read_bool(buffer, offset, field_path)?;
@@ -171,15 +186,17 @@ fn materialize_field_packed(
}
AlkTypeKind::Bytes => {
let b = data_access::read_bytes(buffer, offset, field_path, endian)?;
// Bytes materialize as a string for validation (JSON has no
// native byte type — see schema-layer.md §TBytes). The
// jsonschema validator's BytesValidator expects a string.
// The bytes are not necessarily UTF-8; use lossy conversion so
// validation can still catch length violations. For strict
// byte-preserving validation, the consumer should use the
// data-access layer directly.
let s = String::from_utf8_lossy(b).into_owned();
Ok((Value::String(s), offset + U32_SIZE + b.len()))
// Materialize as a JSON array of u8 (one Value::Number per
// byte, 0..=255). This is the round-trippable form for raw
// bytes — JSON has no native byte type, and `from_utf8_lossy`
// corrupts non-UTF-8 byte sequences (breaking `maxLength`
// semantics, since the replacement char U+FFFD is 3 bytes in
// UTF-8). The `BytesValidator` accepts `Value::Array` for
// `validate_bytes` and `Value::String` for `validate_json`
// (so JSON consumers can still pass a string). `maxLength`
// is interpreted as max byte count (array length).
let arr: Vec<Value> = b.iter().map(|&byte| Value::from(u32::from(byte))).collect();
Ok((Value::Array(arr), offset + U32_SIZE + b.len()))
}
AlkTypeKind::Timestamp => {
let s = data_access::read_string(buffer, offset, field_path, endian)?;
@@ -190,42 +207,29 @@ fn materialize_field_packed(
// The nested struct's fields follow inline. We don't know its
// total size without walking it, so we recurse and let it
// return the new offset.
let nested = materialize_struct_packed_at(
buffer,
field_schema,
field_path,
offset,
endian,
)?;
Ok(nested)
materialize_struct_packed_at(buffer, root, resolved, field_path, offset, endian)
}
AlkTypeKind::Array => {
materialize_array_packed(buffer, field_schema, field_path, offset, endian)
materialize_array_packed(buffer, root, resolved, field_path, offset, endian)
}
AlkTypeKind::Union => {
materialize_union_packed(buffer, field_schema, field_path, offset, endian)
materialize_union_packed(buffer, root, resolved, field_path, offset, endian)
}
AlkTypeKind::Record => {
// Record materialization is more complex (count-prefixed
// key/value pairs). Defer for the POC — the chunk header
// doesn't use Record.
Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "materialize: AlkType:Record not yet supported in validate_bytes (POC)".to_string(),
})
materialize_record_packed(buffer, root, resolved, field_path, offset, endian)
}
}
}
fn materialize_struct_packed_at(
buffer: &[u8],
root: &Value,
struct_schema: &Value,
path_prefix: &str,
offset: usize,
endian: Endian,
) -> Result<(Value, usize), AlkTypeError> {
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
.unwrap_or(struct_schema);
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
let props = struct_schema
.get("properties")
.and_then(Value::as_object)
@@ -238,7 +242,8 @@ fn materialize_struct_packed_at(
let mut cur = offset;
for (name, field_schema) in props.iter() {
let path = format!("{path_prefix}.{name}");
let (value, new_offset) = materialize_field_packed(buffer, field_schema, &path, cur, endian)?;
let (value, new_offset) =
materialize_field_packed(buffer, root, field_schema, &path, cur, endian)?;
obj.insert(name.clone(), value);
cur = new_offset;
}
@@ -247,18 +252,17 @@ fn materialize_struct_packed_at(
fn materialize_array_packed(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
endian: Endian,
) -> Result<(Value, usize), AlkTypeError> {
let element_schema = field_schema
.get("items")
.ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: array at {field_path} has no items schema"
))
})?;
let element_schema = field_schema.get("items").ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: array at {field_path} has no items schema"
))
})?;
let min = field_schema.get("minItems").and_then(Value::as_u64);
let max = field_schema.get("maxItems").and_then(Value::as_u64);
let fixed_count = match (min, max) {
@@ -270,7 +274,8 @@ fn materialize_array_packed(
let mut cur = offset;
for i in 0..count {
let path = format!("{field_path}[{i}]");
let (value, new_offset) = materialize_field_packed(buffer, element_schema, &path, cur, endian)?;
let (value, new_offset) =
materialize_field_packed(buffer, root, element_schema, &path, cur, endian)?;
arr.push(value);
cur = new_offset;
}
@@ -283,7 +288,8 @@ fn materialize_array_packed(
let mut cur = offset + U32_SIZE;
for i in 0..count {
let path = format!("{field_path}[{i}]");
let (value, new_offset) = materialize_field_packed(buffer, element_schema, &path, cur, endian)?;
let (value, new_offset) =
materialize_field_packed(buffer, root, element_schema, &path, cur, endian)?;
arr.push(value);
cur = new_offset;
}
@@ -293,6 +299,7 @@ fn materialize_array_packed(
fn materialize_union_packed(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
@@ -300,70 +307,185 @@ fn materialize_union_packed(
) -> Result<(Value, usize), AlkTypeError> {
let disc = crate::schema::parse_discriminator(field_schema)?;
match disc {
DiscriminatorKind::Byte { offset: disc_offset, disc_type } => {
DiscriminatorKind::Byte {
offset: disc_offset,
disc_type,
} => {
let disc_value = match disc_type {
AlkTypeKind::Uint8 => data_access::read_u8(buffer, offset + disc_offset, field_path)? as u32,
AlkTypeKind::Uint16 => data_access::read_u16(buffer, offset + disc_offset, field_path, endian)? as u32,
AlkTypeKind::Uint32 => data_access::read_u32(buffer, offset + disc_offset, field_path, endian)?,
AlkTypeKind::Uint8 => {
data_access::read_u8(buffer, offset + disc_offset, field_path)? as u32
}
AlkTypeKind::Uint16 => {
data_access::read_u16(buffer, offset + disc_offset, field_path, endian)? as u32
}
AlkTypeKind::Uint32 => {
data_access::read_u32(buffer, offset + disc_offset, field_path, endian)?
}
_ => unreachable!("disc_type restricted by parse_discriminator"),
};
let mapping = field_schema
.get("mapping")
.and_then(Value::as_object)
.ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: union at {field_path} has no mapping"
))
})?;
let mapping = field_schema.get("mapping").and_then(Value::as_object).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: union at {field_path} has no mapping"
))
})?;
let key = disc_value.to_string();
let variant_schema = mapping.get(&key).ok_or_else(|| {
AlkTypeError::Access {
field_path: field_path.to_string(),
reason: format!("union discriminator value {key} not in mapping"),
}
let variant_schema = mapping.get(&key).ok_or_else(|| AlkTypeError::Access {
field_path: field_path.to_string(),
reason: format!("union discriminator value {key} not in mapping"),
})?;
let variant_offset = offset + disc_offset + disc_type.type_size().unwrap_or(1);
let (variant_value, new_offset) = materialize_field_packed(
buffer,
root,
variant_schema,
field_path,
variant_offset,
endian,
)?;
let mut obj = Map::new();
obj.insert("__discriminator".to_string(), Value::from(disc_value));
if let Some(variant_obj) = variant_value.as_object() {
for (k, v) in variant_obj.iter() {
obj.insert(k.clone(), v.clone());
}
} else {
obj.insert("__variant".to_string(), variant_value);
}
Ok((Value::Object(obj), new_offset))
Ok((
tag_union_value(disc_value, variant_value),
new_offset,
))
}
DiscriminatorKind::Field { name } => {
// Field-name discriminator: the union is a struct with a
// discriminator field; the variant is selected by that field's
// value. This is the typedef.ts pattern. For the POC, we
// materialize the struct fields and let the validator
// dispatch on the discriminator field.
let (value, _off) =
materialize_struct_packed_at(buffer, field_schema, field_path, offset, endian)?;
let _ = &name; // field-name dispatch happens at validation time
Ok((value, offset))
// Field-name discriminator: the union is a struct whose first
// field is the discriminator; the variant struct follows. The
// discriminator field's value selects the mapping entry. The
// materialized shape is the same as the byte-offset path:
// `{ "__discriminator": <key>, <disc-field-name>: <typed-value>, ...variant-fields }`.
// The `__discriminator` entry carries the mapping key (stringified)
// for uniform validator dispatch; the discriminator field is
// preserved under its own name (it's a regular field in the
// struct — the typedef.ts pattern).
let properties = field_schema.get("properties").and_then(Value::as_object).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: field-name union at {field_path} has no properties"
))
})?;
let disc_schema = properties.get(&name).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: union at {field_path} has no discriminator field '{name}'"
))
})?;
let disc_path = format!("{field_path}.{name}");
let (disc_value, after_disc) =
materialize_field_packed(buffer, root, disc_schema, &disc_path, offset, endian)?;
let key = union_discriminator_key(&disc_value, field_path)?;
let mapping = field_schema.get("mapping").and_then(Value::as_object).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: union at {field_path} has no mapping"
))
})?;
let variant_schema = mapping.get(&key).ok_or_else(|| AlkTypeError::Access {
field_path: field_path.to_string(),
reason: format!("union discriminator value {key} not in mapping"),
})?;
let (variant_value, new_offset) = materialize_field_packed(
buffer,
root,
variant_schema,
field_path,
after_disc,
endian,
)?;
let mut obj = Map::new();
obj.insert("__discriminator".to_string(), Value::String(key));
obj.insert(name.clone(), disc_value);
flatten_variant_into(&mut obj, variant_value);
Ok((Value::Object(obj), new_offset))
}
}
}
/// Build the materialized `Value` for a byte-offset union: an object
/// `{ "__discriminator": <disc_value>, ...variant-fields }`. If the
/// variant materialized as a non-object (a leaf), it is nested under
/// `"__variant"`.
fn tag_union_value(disc_value: u32, variant_value: Value) -> Value {
let mut obj = Map::new();
obj.insert("__discriminator".to_string(), Value::from(disc_value));
flatten_variant_into(&mut obj, variant_value);
Value::Object(obj)
}
/// Flatten a variant's materialized object fields into `obj` (in place).
/// If the variant is not an object (e.g. a leaf kind), nest it under
/// `"__variant"`.
fn flatten_variant_into(obj: &mut Map<String, Value>, variant_value: Value) {
if let Some(variant_obj) = variant_value.as_object() {
for (k, v) in variant_obj.iter() {
obj.insert(k.clone(), v.clone());
}
} else {
obj.insert("__variant".to_string(), variant_value);
}
}
/// Stringify a discriminator field value for mapping lookup. Mirrors
/// `sequential_reader::discriminator_string_value` for the Value form.
fn union_discriminator_key(value: &Value, field_path: &str) -> Result<String, AlkTypeError> {
match value {
Value::String(s) => Ok(s.clone()),
Value::Number(n) => Ok(n.to_string()),
_ => Err(AlkTypeError::Schema(format!(
"union {field_path} has unsupported discriminator value kind: {value}"
))),
}
}
/// Materialize an `AlkType:Record` field: a count-prefixed sequence of
/// `(key, value)` pairs. Wire format per [schema-layer.md](../docs/architecture/schema-layer.md)
/// §TRecord: `[count: u32][key_len: u32][key_bytes][value]...` repeated
/// `count` times. The count and key-length prefixes respect `endian`.
/// The value is materialized according to its declared `AlkType:*` kind.
///
/// The materialized form is a JSON object mapping each key (string) to
/// its materialized value. This matches the `RecordValidator`'s
/// expectation (an object whose values match the record's value type).
fn materialize_record_packed(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
endian: Endian,
) -> Result<(Value, usize), AlkTypeError> {
let value_schema = field_schema.get("values").ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: record at {field_path} has no values schema"
))
})?;
let count = data_access::read_u32(buffer, offset, field_path, endian)? as usize;
let mut cur = offset + U32_SIZE;
let mut obj = Map::new();
for i in 0..count {
let key_path = format!("{field_path}[{i}].key");
let key_str = data_access::read_string(buffer, cur, &key_path, endian)?;
let key_len = U32_SIZE + key_str.len();
cur = cur
.checked_add(key_len)
.ok_or_else(|| AlkTypeError::Access {
field_path: key_path.clone(),
reason: format!("key offset {cur} + {key_len} overflows usize"),
})?;
let val_path = format!("{field_path}[{i}].value");
let (val, new_offset) =
materialize_field_packed(buffer, root, value_schema, &val_path, cur, endian)?;
obj.insert(key_str.to_string(), val);
cur = new_offset;
}
Ok((Value::Object(obj), cur))
}
fn materialize_struct_aligned(
buffer: &[u8],
root: &Value,
struct_schema: &Value,
path_prefix: &str,
offset_map: &crate::offset_map::OffsetMap,
endian: Endian,
) -> Result<Value, AlkTypeError> {
let struct_schema = resolve_ref_or_inline(struct_schema, root_of(struct_schema))
.unwrap_or(struct_schema);
let struct_schema = resolve_ref_or_inline(struct_schema, root).unwrap_or(struct_schema);
let props = struct_schema
.get("properties")
.and_then(Value::as_object)
@@ -379,7 +501,8 @@ fn materialize_struct_aligned(
} else {
format!("{path_prefix}.{name}")
};
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: field {path} has no AlkType:* kind"
))
@@ -390,19 +513,23 @@ fn materialize_struct_aligned(
field_path: path.clone(),
reason: "field not found in offset map".to_string(),
})?;
materialize_leaf_at(buffer, field_schema, &path, range.start, endian)?
materialize_leaf_at(buffer, root, resolved, &path, range.start, endian)?
} else if kind == AlkTypeKind::Struct {
// Nested struct — recurse with the same offset map (nested
// fields are recorded under `path.*`).
materialize_struct_aligned(buffer, field_schema, &path, offset_map, endian)?
materialize_struct_aligned(buffer, root, resolved, &path, offset_map, endian)?
} else {
// Composite leaf (Array, Union, Record) — fall back to packed-style
// walk from the field's start offset.
// walk from the field's start offset. The offset map records the
// field's start; the composite's internal layout is walked
// sequentially from that start (variable-length elements can't
// be precomputed into the static offset map).
let range = offset_map.get(&path).ok_or_else(|| AlkTypeError::Offset {
field_path: path.clone(),
reason: "field not found in offset map".to_string(),
})?;
let (value, _) = materialize_field_packed(buffer, field_schema, &path, range.start, endian)?;
let (value, _) =
materialize_field_packed(buffer, root, resolved, &path, range.start, endian)?;
value
};
obj.insert(name.clone(), value);
@@ -412,12 +539,14 @@ fn materialize_struct_aligned(
fn materialize_leaf_at(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
endian: Endian,
) -> Result<Value, AlkTypeError> {
let (_kind, value) = materialize_field_packed_returning_kind(buffer, field_schema, field_path, offset, endian)?;
let (_kind, value) =
materialize_field_packed_returning_kind(buffer, root, field_schema, field_path, offset, endian)?;
Ok(value)
}
@@ -427,25 +556,413 @@ fn materialize_leaf_at(
/// signature stable.
fn materialize_field_packed_returning_kind(
buffer: &[u8],
root: &Value,
field_schema: &Value,
field_path: &str,
offset: usize,
endian: Endian,
) -> Result<(AlkTypeKind, Value), AlkTypeError> {
let kind = get_alktype_kind_loose_enum(field_schema).ok_or_else(|| {
let resolved = resolve_ref_or_inline(field_schema, root).unwrap_or(field_schema);
let kind = get_alktype_kind_loose_enum(resolved).ok_or_else(|| {
AlkTypeError::Schema(format!(
"materialize: field {field_path} has no AlkType:* kind"
))
})?;
let (value, _) = materialize_field_packed(buffer, field_schema, field_path, offset, endian)?;
let (value, _) = materialize_field_packed(buffer, root, field_schema, field_path, offset, endian)?;
Ok((kind, value))
}
/// Walk a schema to find its root. Since materialize is called with the
/// top-level schema as `struct_schema`, the root is `struct_schema` itself
/// for the top-level call. For nested calls (via $ref), we need the
/// original root. This helper returns the node itself for now — the $ref
/// resolution in `resolve_ref_or_inline` handles the common case.
fn root_of(schema: &Value) -> &Value {
schema
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
fn materialize_packed_strict(
buffer: &[u8],
schema: &Value,
endian: Endian,
) -> Result<Value, AlkTypeError> {
materialize_packed(buffer, schema, endian)
}
// -----------------------------------------------------------------
// Bytes materialization (array-of-u8 form, OQ-007 resolution)
// -----------------------------------------------------------------
#[test]
fn materialize_bytes_packed_returns_array_of_u8() {
// Schema: Struct { blob: Bytes }. Wire: [len:u32 LE = 3][0x41 0x42 0xC3]
// 0xC3 is invalid UTF-8 lead byte — under the old from_utf8_lossy
// path this would corrupt. Under array-of-u8 it round-trips.
let schema = json!({
"AlkType:Struct": true,
"properties": {
"blob": { "AlkType:Bytes": true }
}
});
let mut buf = vec![0u8; 7];
buf[0..4].copy_from_slice(&3u32.to_le_bytes());
buf[4] = 0x41;
buf[5] = 0x42;
buf[6] = 0xC3;
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(
v["blob"],
json!([65, 66, 195]),
"bytes should materialize as array of u8"
);
}
#[test]
fn materialize_bytes_packed_non_utf8_round_trips() {
// All-non-UTF-8 bytes — would have been corrupted by from_utf8_lossy.
let schema = json!({
"AlkType:Struct": true,
"properties": {
"raw": { "AlkType:Bytes": true }
}
});
let mut buf = vec![0u8; 8];
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
buf[4] = 0xFF;
buf[5] = 0xFE;
buf[6] = 0x00;
buf[7] = 0x80;
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(v["raw"], json!([255, 254, 0, 128]));
}
// -----------------------------------------------------------------
// Record materialization
// -----------------------------------------------------------------
#[test]
fn materialize_record_packed_count_prefixed_pairs() {
// Schema: Struct { counts: Record<Uint32> }.
// Wire: [count:u32=2][key_len:u32=1][b][value:u32=10][key_len:u32=1][a][value:u32=20]
let schema = json!({
"AlkType:Struct": true,
"properties": {
"counts": {
"AlkType:Record": true,
"values": { "AlkType:Uint32": true }
}
}
});
let mut buf = vec![0u8; 22];
let mut off = 0usize;
// count = 2
buf[off..off + 4].copy_from_slice(&2u32.to_le_bytes());
off += 4;
// key 1: "b" (len=1)
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
off += 4;
buf[off] = b'b';
off += 1;
// value 1: 10
buf[off..off + 4].copy_from_slice(&10u32.to_le_bytes());
off += 4;
// key 2: "a" (len=1)
buf[off..off + 4].copy_from_slice(&1u32.to_le_bytes());
off += 4;
buf[off] = b'a';
off += 1;
// value 2: 20
buf[off..off + 4].copy_from_slice(&20u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(
v["counts"],
json!({ "b": 10, "a": 20 }),
"record should materialize as an object"
);
}
#[test]
fn materialize_record_packed_zero_entries() {
let schema = json!({
"AlkType:Struct": true,
"properties": {
"counts": {
"AlkType:Record": true,
"values": { "AlkType:Uint32": true }
}
}
});
let mut buf = vec![0u8; 4];
buf[0..4].copy_from_slice(&0u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(v["counts"], json!({}));
}
#[test]
fn materialize_record_packed_rejects_short_buffer() {
let schema = json!({
"AlkType:Struct": true,
"properties": {
"counts": {
"AlkType:Record": true,
"values": { "AlkType:Uint32": true }
}
}
});
// Buffer too short for the count prefix.
let buf = [0u8; 2];
let err = materialize_packed_strict(&buf, &schema, Endian::Little).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
}
// -----------------------------------------------------------------
// Union materialization — byte-offset and field-name shape parity (OQ-005)
// -----------------------------------------------------------------
#[test]
fn materialize_union_byte_discriminator_packs_variant_fields() {
// Schema: Struct { payload: Union(byte, offset=0, Uint8) mapping { 5 -> Read } }
// Read: Struct { id: Uint32 }
// Wire: [disc:u8=5][id:u32=42 LE]
let schema = json!({
"AlkType:Struct": true,
"properties": {
"payload": {
"AlkType:Union": true,
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
"mapping": {
"5": {
"AlkType:Struct": true,
"properties": { "id": { "AlkType:Uint32": true } }
}
}
}
}
});
let mut buf = vec![0u8; 5];
buf[0] = 5;
buf[1..5].copy_from_slice(&42u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
let payload = &v["payload"];
assert_eq!(
payload["__discriminator"],
json!(5),
"byte-offset union should carry __discriminator"
);
assert_eq!(
payload["id"],
json!(42),
"variant fields should be flattened into the union object"
);
}
#[test]
fn materialize_union_field_discriminator_includes_typed_disc_value() {
// Schema: Struct { payload: Union(field, name="type") mapping { "read" -> Read } }
// The union is a struct: [type: String "read"][variant struct].
// Read: Struct { n: Uint32 }
// Wire: [len:u32=4][read][n:u32=7 LE]
let schema = json!({
"AlkType:Struct": true,
"properties": {
"payload": {
"AlkType:Union": true,
"discriminator": { "kind": "field", "name": "type" },
"properties": {
"type": { "AlkType:String": true }
},
"mapping": {
"read": {
"AlkType:Struct": true,
"properties": { "n": { "AlkType:Uint32": true } }
}
}
}
}
});
let mut buf = vec![0u8; 12];
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
buf[4..8].copy_from_slice(b"read");
buf[8..12].copy_from_slice(&7u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
let payload = &v["payload"];
// __discriminator carries the mapping key (string).
assert_eq!(payload["__discriminator"], json!("read"));
// The original typed discriminator value is preserved under its field name.
assert_eq!(payload["type"], json!("read"));
// Variant fields are flattened.
assert_eq!(payload["n"], json!(7));
}
#[test]
fn materialize_union_field_discriminator_advances_offset_correctly() {
// Regression: the old field-name path returned the start offset,
// not the end. A struct with a union followed by another field
// would read the second field from the wrong position. This test
// places a Uint32 after the union and verifies it reads correctly.
let schema = json!({
"AlkType:Struct": true,
"properties": {
"payload": {
"AlkType:Union": true,
"discriminator": { "kind": "field", "name": "type" },
"properties": {
"type": { "AlkType:String": true }
},
"mapping": {
"read": {
"AlkType:Struct": true,
"properties": { "n": { "AlkType:Uint32": true } }
}
}
},
"trailer": { "AlkType:Uint32": true }
}
});
// Wire: [len:u32=4]["read"][n:u32=7][trailer:u32=99]
let mut buf = vec![0u8; 16];
buf[0..4].copy_from_slice(&4u32.to_le_bytes());
buf[4..8].copy_from_slice(b"read");
buf[8..12].copy_from_slice(&7u32.to_le_bytes());
buf[12..16].copy_from_slice(&99u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(v["trailer"], json!(99), "trailer should read from after the union");
assert_eq!(v["payload"]["n"], json!(7));
}
#[test]
fn materialize_union_byte_discriminator_unknown_value_is_access_error() {
let schema = json!({
"AlkType:Struct": true,
"properties": {
"payload": {
"AlkType:Union": true,
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
"mapping": { "5": { "AlkType:Struct": true, "properties": {} } }
}
}
});
let buf = [99u8];
let err = materialize_packed_strict(&buf, &schema, Endian::Little).unwrap_err();
assert!(matches!(err, AlkTypeError::Access { .. }), "got {err:?}");
}
// -----------------------------------------------------------------
// $ref resolution (root-schema threading)
// -----------------------------------------------------------------
#[test]
fn materialize_union_with_ref_variant_resolves_via_root() {
// The SFTP shape: struct wrapping a union whose variants are $refs
// into $defs. The materializer must resolve the $ref against the
// root schema to find the variant's AlkType:Struct kind.
let schema = json!({
"AlkType:Struct": true,
"properties": {
"payload": {
"AlkType:Union": true,
"discriminator": { "kind": "byte", "offset": 0, "type": "AlkType:Uint8" },
"mapping": {
"5": { "$ref": "#/$defs/Read" }
}
}
},
"$defs": {
"Read": {
"AlkType:Struct": true,
"properties": { "id": { "AlkType:Uint32": true } }
}
}
});
let mut buf = vec![0u8; 5];
buf[0] = 5;
buf[1..5].copy_from_slice(&42u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(v["payload"]["__discriminator"], json!(5));
assert_eq!(v["payload"]["id"], json!(42));
}
#[test]
fn materialize_array_with_ref_element_resolves_via_root() {
let schema = json!({
"AlkType:Struct": true,
"properties": {
"items": {
"AlkType:Array": true,
"minItems": 2,
"maxItems": 2,
"items": { "$ref": "#/$defs/Point" }
}
},
"$defs": {
"Point": {
"AlkType:Struct": true,
"properties": {
"x": { "AlkType:Uint16": true },
"y": { "AlkType:Uint16": true }
}
}
}
});
// Wire: [x:u16=1 LE][y:u16=2 LE][x:u16=3 LE][y:u16=4 LE]
let mut buf = vec![0u8; 8];
buf[0..2].copy_from_slice(&1u16.to_le_bytes());
buf[2..4].copy_from_slice(&2u16.to_le_bytes());
buf[4..6].copy_from_slice(&3u16.to_le_bytes());
buf[6..8].copy_from_slice(&4u16.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(
v["items"],
json!([
{ "x": 1, "y": 2 },
{ "x": 3, "y": 4 }
])
);
}
// -----------------------------------------------------------------
// Smoke: a struct field after a nested struct advances correctly
// -----------------------------------------------------------------
#[test]
fn materialize_struct_with_nested_struct_then_field() {
let schema = json!({
"AlkType:Struct": true,
"properties": {
"header": {
"AlkType:Struct": true,
"properties": {
"a": { "AlkType:Uint8": true },
"b": { "AlkType:Uint8": true }
}
},
"after": { "AlkType:Uint32": true }
}
});
// Wire: [a=1][b=2][after=0x03030303 LE = 50529027]
let mut buf = vec![0u8; 6];
buf[0] = 1;
buf[1] = 2;
buf[2..6].copy_from_slice(&0x03030303u32.to_le_bytes());
let v = materialize_packed_strict(&buf, &schema, Endian::Little).expect("materialize");
assert_eq!(v["header"]["a"], json!(1));
assert_eq!(v["header"]["b"], json!(2));
assert_eq!(v["after"], json!(0x03030303u32));
}
// -----------------------------------------------------------------
// Sanity: existing packed chunk-header path still works
// -----------------------------------------------------------------
#[test]
fn materialize_chunk_header_packed() {
let schema = json!({
"AlkType:Struct": true,
"endian": "big",
"properties": {
"channel_id": { "AlkType:Uint32": true },
"length": { "AlkType:Uint32": true }
}
});
let buf = [0u8, 0u8, 0u8, 42u8, 0u8, 0u8, 0u8, 7u8];
let v = materialize_packed_strict(&buf, &schema, Endian::Big).expect("materialize");
assert_eq!(v["channel_id"], json!(42));
assert_eq!(v["length"], json!(7));
}
}