fix: W3-3 — read_field/write_field misread aligned maxLength reservations

The running validate_pair campaign found a third crash: in aligned
mode a maxLength reservation (ADR-003 strategy 2, VARCHAR(N)) stores
RAW zero-padded data with no length prefix — materialize and
validate_bytes implement exactly that — but read_field read the entry
through data_access::read_string, i.e. parsed the window's first four
bytes as a u32 length prefix. Raw reservation bytes that look like a
large prefix then fail bounds with Access while validate_bytes says
Ok: the validate⇒read agreement lattice breaks on every aligned
maxLength string/bytes field (any schema declaring maxLength in
aligned mode). write_field had the same mismatch (prefix+data into a
raw window).

Engine fix:
- VariableEncoding gains MaxLengthReserved (additive variant, ADR-003
  strategy 2). OffsetMap::compute records it for maxLength fields with
  the default encoding; maxLength+offset-indirect stays OffsetIndirect
  (the pair read is intentional, the window reserves max_len bytes),
  preserving the W3-1 combination semantics.
- read_field String/Bytes arms dispatch on MaxLengthReserved → new
  data_access::read_reservation_string / read_reservation (raw window
  inside-buffer check + NUL trim — the materializer's exact semantics).
- write_field dispatches → new data_access::write_reservation (zero-
  pads the window, rejects oversized values with Access).
- materialize_aligned reads MaxLengthReserved through the same new
  read_reservation paths (single source of truth; replaces the inline
  trim logic with an identical implementation).
- offset_map compute rejects a MaxLengthReserved encoding reaching the
  walk with a clean Offset error (recorded, never declared).
- builder round-trips: MaxLengthReserved serializes via maxLength (the
  document form), never as an encoding value.
- three engine regression tests: raw-not-prefixed read, zero-pad
  write + oversize rejection, validate⇒read_field agreement.
- fuzz/shared validate_pair invariant updated: the W3-1
  shorter-than-reservation exemption now applies to offset-indirect
  only; reservations assert the full window in-bounds (fixed engine).
- corpus regenerated for generator-consistent numbering (seeds 037-044
  relabeled; W3-1/W3-2 artifacts remain 044/045-047 → now 044, 048-050
  region) — 48 seeds, replay 30/30 green.

Verification: main crate 573 tests pass; clippy -D warnings clean
(crate + shared); wasm clean; cargo fuzz build clean.
This commit is contained in:
glm-5.3-flash committed 2026-09-30 08:20:13 +00:00
1 parent b7ead99724
commit a0dd3d2de4
15 files changed
+307 -34

No files matched your search

Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
Binary file not shown.
+14 -13
View File
@@ -408,19 +408,20 @@ fn drive_pair(engine: &AlkTypeEngine, doc: &Value, root: &str, buffer: &[u8]) {
) {
continue;
}
// W3-1 (pinned contract, not engine behavior to
// change): an offset-indirect entry's range is
// the pair/reservation window (a declared
// maxLength contributes its full size), but the
// {data_offset, data_length} pair points
// absolutely into the whole buffer — the data
// may live anywhere in the buffer and the
// buffer may be shorter than the reservation
// (the wave-1 data_access bounds partition is
// the contract: the pointed-to window sits
// inside the buffer, nothing about the
// reservation). For every other encoding a
// successful read implies range.end ≤ len.
// W3-1 (pinned contract): an offset-indirect
// entry's range is the absolute pair window
// and the {data_offset, data_length} pair
// points anywhere in the buffer, so only the
// pointed-to window must sit inside the buffer.
// W3-3 fix (engine change): maxLength
// reservations now read as raw NUL-trimmed
// windows — the read requires range.end ≤ len,
// so a validated buffer shorter than the
// reservation window fails both validate and
// the read; the exemption is unnecessary and
// the full assertion applies. For every other
// encoding a successful read implies
// range.end ≤ len.
let indirect =
entry.meta.encoding == alktype::VariableEncoding::OffsetIndirect;
if !indirect {
+5
View File
@@ -667,6 +667,11 @@ fn build_field_value(f: &FieldBuilder) -> Value {
match enc {
VariableEncoding::LengthPrefixed => "length-prefixed",
VariableEncoding::OffsetIndirect => "offset-indirect",
// The reservation strategy is expressed via
// `maxLength` in the document form, never as an
// encoding value (ADR-003 strategy 2); a builder
// cannot emit it as `encoding`.
VariableEncoding::MaxLengthReserved => "length-prefixed",
}
.to_string(),
),
+89
View File
@@ -416,6 +416,95 @@ pub fn write_bytes_indirect(
Ok(8)
}
// ---------------------------------------------------------------------------
// Variable-length read/write (maxLength reservation — ADR-003 strategy 2)
// ---------------------------------------------------------------------------
/// Read a `maxLength` reservation leaf (aligned mode, ADR-003 strategy 2
/// — the `VARCHAR(N)` pattern).
///
/// The entry's byte range holds the raw zero-padded data with *no*
/// length prefix; trailing NUL bytes are trimmed, matching the
/// materializer's reservation semantics (`materialize_variable_aligned`).
/// The reserved window must be fully inside the buffer for the read to
/// succeed — a shorter buffer exposes the same truncation a
/// `read_string`-style access would, naming `field_path`.
///
/// # Errors
///
/// - [`AlkTypeError::Access`] if the reservation window
/// `[range.start..range.end)` is unavailable in the buffer, or the
/// trimmed content is not valid UTF-8 (`read_reservation_string`).
pub fn read_reservation<'a>(
buffer: &'a [u8],
range: crate::offset_map::ByteRange,
field_path: &str,
) -> Result<&'a [u8], AlkTypeError> {
check_bounds(buffer.len(), range.start, range.end, field_path)?;
let data = buffer
.get(range.start..range.end)
.ok_or_else(|| access_err(field_path, "reservation window unavailable"))?;
let trimmed_len = data
.iter()
.rposition(|&b| b != 0)
.map_or(0, |i| i + 1);
Ok(&data[..trimmed_len])
}
/// Read a `maxLength` reservation leaf as a UTF-8 string.
///
/// The NUL-trimmed reservation content must be valid UTF-8.
///
/// # Errors
///
/// - [`AlkTypeError::Access`] if the window is out of bounds or the
/// trimmed content is not valid UTF-8.
pub fn read_reservation_string<'a>(
buffer: &'a [u8],
range: crate::offset_map::ByteRange,
field_path: &str,
) -> Result<&'a str, AlkTypeError> {
let bytes = read_reservation(buffer, range, field_path)?;
std::str::from_utf8(bytes).map_err(|e| {
access_err(
field_path,
format!("invalid UTF-8 in maxLength reservation: {e}"),
)
})
}
/// Write a `maxLength` reservation leaf.
///
/// Writes `value` at the window start; the remainder of the window is
/// zeroed (the reservation's zero-padding). `value.len()` must not
/// exceed the window; anything larger is a clean `Access` error — the
/// wire form has no length prefix to consult, so oversized data cannot
/// be represented.
pub fn write_reservation(
buffer: &mut [u8],
range: crate::offset_map::ByteRange,
value: &[u8],
field_path: &str,
) -> Result<(), AlkTypeError> {
if value.len() > range.len() {
return Err(access_err(
field_path,
format!(
"reservation window {} too small for {} bytes of data",
range.len(),
value.len()
),
));
}
check_bounds(buffer.len(), range.start, range.end, field_path)?;
let dest = buffer
.get_mut(range.start..range.end)
.ok_or_else(|| access_err(field_path, "reservation window unavailable"))?;
dest.fill(0);
dest[..value.len()].copy_from_slice(value);
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
+144 -14
View File
@@ -434,6 +434,9 @@ impl AlkTypeEngine {
VariableEncoding::OffsetIndirect => {
data_access::read_string_indirect(buffer, range.start, field_path, endian)?
}
VariableEncoding::MaxLengthReserved => {
data_access::read_reservation_string(buffer, range, field_path)?
}
VariableEncoding::LengthPrefixed => {
data_access::read_string(buffer, range.start, field_path, endian)?
}
@@ -445,6 +448,9 @@ impl AlkTypeEngine {
VariableEncoding::OffsetIndirect => {
data_access::read_bytes_indirect(buffer, range.start, field_path, endian)?
}
VariableEncoding::MaxLengthReserved => {
data_access::read_reservation(buffer, range, field_path)?
}
VariableEncoding::LengthPrefixed => {
data_access::read_bytes(buffer, range.start, field_path, endian)?
}
@@ -545,29 +551,48 @@ impl AlkTypeEngine {
data_access::write_enum(buffer, range.start, *v, field_path, endian)
}
FieldValue::String(v) => {
if encoding == VariableEncoding::OffsetIndirect {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
match encoding {
VariableEncoding::OffsetIndirect => {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
use data_access::write_string_indirect with the \
offset map range and a data offset"
.to_string(),
});
.to_string(),
});
}
VariableEncoding::MaxLengthReserved => {
data_access::write_reservation(
buffer,
range,
v.as_bytes(),
field_path,
)?;
}
VariableEncoding::LengthPrefixed => {
data_access::write_string(buffer, range.start, v, field_path, endian)?;
}
}
data_access::write_string(buffer, range.start, v, field_path, endian)?;
Ok(())
}
FieldValue::Bytes(v) => {
if encoding == VariableEncoding::OffsetIndirect {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
match encoding {
VariableEncoding::OffsetIndirect => {
return Err(AlkTypeError::Access {
field_path: field_path.to_string(),
reason: "write_field cannot write offset-indirect fields; \
use data_access::write_bytes_indirect with the \
offset map range and a data offset"
.to_string(),
});
.to_string(),
});
}
VariableEncoding::MaxLengthReserved => {
data_access::write_reservation(buffer, range, v, field_path)?;
}
VariableEncoding::LengthPrefixed => {
data_access::write_bytes(buffer, range.start, v, field_path, endian)?;
}
}
data_access::write_bytes(buffer, range.start, v, field_path, endian)?;
Ok(())
}
FieldValue::Struct { .. } | FieldValue::Union { .. } | FieldValue::Array { .. } => {
@@ -1190,6 +1215,111 @@ mod tests {
}
}
/// W3-3 regression: an aligned `maxLength` reservation holds raw
/// zero-padded data with NO length prefix (ADR-003 strategy 2).
/// `read_field` must read the raw reservation (NUL-trimmed) —
/// reading a length prefix there misparsed raw data as a huge
/// prefix and broke the validate_bytes ⇒ read_field agreement.
#[test]
fn read_field_aligned_max_length_reservation_reads_raw_not_prefixed() {
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [ { "name": "s", "kind": "string", "maxLength": 16 } ]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let mut buf = vec![0u8; 32];
buf[0..8].copy_from_slice(b"hi there");
match engine.read_field(&buf, "s").unwrap() {
FieldValue::String(s) => assert_eq!(s, "hi there"),
other => panic!("expected String, got {other:?}"),
}
// Bytes that look like a huge length prefix when misread must
// never fool the read: 0xff bytes are raw data (invalid UTF-8
// only where they survive the NUL trim).
buf[0..2].copy_from_slice(&[0xFF, 0xFF]);
let v = engine.validate_bytes(&buf);
assert!(v.is_err());
match engine.read_field(&buf, "s") {
Err(AlkTypeError::Access { reason, .. }) => {
assert!(reason.contains("maxLength"), "reason: {reason}");
}
other => panic!("expected Access for invalid reservation UTF-8, got {other:?}"),
}
}
/// W3-3 write-side: `write_field` over an aligned maxLength
/// reservation fills the window with the value + zero padding, and
/// an oversized value is a clean Access error.
#[test]
fn write_field_aligned_max_length_reservation_zero_pads_and_rejects_oversize() {
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [ { "name": "s", "kind": "string", "maxLength": 8 } ]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
let mut buf = vec![0xA5u8; 8];
engine
.write_field(&mut buf, "s", &FieldValue::String("ab"))
.expect("an 8-byte reservation accepts 2 bytes");
assert_eq!(&buf[..2], b"ab");
assert!(buf[2..].iter().all(|&b| b == 0), "the rest of the window zero-pads");
let err = engine
.write_field(&mut buf, "s", &FieldValue::String("9bytes!!!"))
.expect_err("a 9-byte value overflows an 8-byte reservation");
assert!(matches!(err, AlkTypeError::Access { .. }));
}
/// W3-3 agreement: with the reservation read path fixed,
/// `validate_bytes` Ok implies `read_field` Ok on every leaf —
/// the exact lattice the wave-3 target asserts.
#[test]
fn aligned_max_length_validate_then_read_field_agree() {
let doc = json!({
"$defs": {
"S": {
"kind": "struct",
"fields": [
{ "name": "id", "kind": "uint32" },
{ "name": "s", "kind": "string", "maxLength": 12 }
]
}
}
});
let engine = AlkTypeEngine::compile(&doc, "S", LayoutMode::Aligned, None).expect("compile");
// Layout: id@0..4, s@4..16 (total 16). The reservation reads
// its window raw with trailing-NUL trim — a "length-prefix-
// looking" word inside the data must never be consulted. The
// window here starts with valid UTF-8 and ends in NULs; the
// 0xFFFF word sits mid-window as pure data in the middle of
// the s region is impossible (the window starts at 4), so
// keep the data clean and pin the boundary behavior: leading
// data + NUL padding reads verbatim.
let mut buf = vec![0u8; 16];
buf[0..4].copy_from_slice(&7u32.to_le_bytes());
buf[4..8].copy_from_slice(b"stri");
match engine.validate_bytes(&buf) {
Ok(()) => {}
Err(e) => panic!("validate_bytes must accept raw reservation data, got {e:?}"),
}
match engine.read_field(&buf, "s").unwrap() {
FieldValue::String(s) => assert_eq!(s, "stri"),
other => panic!("expected String, got {other:?}"),
}
// A misread as length-prefixed would have consumed the first
// four bytes s[0..4] = "stri" as a length word (0x69727473 =
// 1_770_632_051) and failed bounds — this exact buffer cannot
// read Ok through the old path, so this pin doubles as the
// regression.
}
#[test]
fn write_field_aligned_rejects_offset_indirect() {
let doc = json!({
+14
View File
@@ -1017,6 +1017,20 @@ fn materialize_variable_aligned(
"materialize: offset-indirect encoding on non-variable kind at {field_path}"
))),
},
VariableEncoding::MaxLengthReserved => match ty {
BastType::Primitive(AlkTypeKind::String) => {
let s = data_access::read_reservation_string(buffer, *range, field_path)?;
Ok(Value::String(s.to_string()))
}
BastType::Primitive(AlkTypeKind::Bytes) => {
let b = data_access::read_reservation(buffer, *range, field_path)?;
let arr: Vec<Value> = b.iter().map(|&byte| Value::from(u32::from(byte))).collect();
Ok(Value::Array(arr))
}
_ => Err(AlkTypeError::Schema(format!(
"materialize: maxLength reservation on non-variable kind at {field_path}"
))),
},
VariableEncoding::LengthPrefixed => {
if let Some(_max_len) = field.max_length() {
let data = buffer.get(start..range.end).ok_or_else(|| AlkTypeError::Access {
+33 -7
View File
@@ -541,10 +541,36 @@ impl<'d> ComputeCtx<'d> {
let encoding = field.encoding();
let max_length = field.max_length();
let (size, natural) = match (max_length, encoding) {
(Some(max_len), _) => (max_len, 1),
(None, VariableEncoding::OffsetIndirect) => (8, 4),
(None, VariableEncoding::LengthPrefixed) => (4, 4),
// Recording rule (W3-3): a `maxLength` field with the default
// (inline) encoding is the ADR-003 strategy-2 raw reservation
// — recorded as `MaxLengthReserved` so leaf consumers read the
// window raw instead of misparsing its first bytes as a length
// prefix. A field declaring BOTH `maxLength` and
// `offset-indirect` keeps `OffsetIndirect` (the pair read is
// intentional; the window reserves `max_len` bytes with the
// pair at its start).
let (size, natural, recorded) = match (max_length, encoding) {
(Some(max_len), VariableEncoding::LengthPrefixed) => {
(max_len, 1, VariableEncoding::MaxLengthReserved)
}
(Some(max_len), VariableEncoding::OffsetIndirect) => {
(max_len, 1, VariableEncoding::OffsetIndirect)
}
(None, VariableEncoding::OffsetIndirect) => (8, 4, VariableEncoding::OffsetIndirect),
(None, VariableEncoding::LengthPrefixed) => (4, 4, VariableEncoding::LengthPrefixed),
(None | Some(_), VariableEncoding::MaxLengthReserved) => {
// `MaxLengthReserved` is only ever recorded here,
// never declared in a parsed schema; a walk that
// reaches this arm means a recording bug. No-panic
// convention: reject cleanly instead of
// `unreachable!`.
return Err(AlkTypeError::Offset {
field_path: field_path.to_string(),
reason: "internal: a MaxLengthReserved encoding reached the variable \
layout walk; reservations are recorded, never declared"
.to_string(),
});
}
};
let align = field_alignment(field, struct_default_align, natural);
@@ -556,7 +582,7 @@ impl<'d> ComputeCtx<'d> {
field_path: field_path.to_string(),
reason: format!("offset {start} + size {size} overflows usize"),
})?;
self.push(field_path, start, start + size, kind, encoding, endian);
self.push(field_path, start, start + size, kind, recorded, endian);
Ok(FieldLayout { align })
}
@@ -991,7 +1017,7 @@ mod tests {
});
let m = map(&root, "S");
assert_eq!(m.get("id"), Some(&OffsetEntry { range: ByteRange { start: 0, end: 4 }, meta: LeafMeta { kind: AlkTypeKind::Uint32, encoding: VariableEncoding::LengthPrefixed, endian: Endian::Little } }));
assert_eq!(m.get("name"), Some(&OffsetEntry { range: ByteRange { start: 4, end: 260 }, meta: LeafMeta { kind: AlkTypeKind::String, encoding: VariableEncoding::LengthPrefixed, endian: Endian::Little } }));
assert_eq!(m.get("name"), Some(&OffsetEntry { range: ByteRange { start: 4, end: 260 }, meta: LeafMeta { kind: AlkTypeKind::String, encoding: VariableEncoding::MaxLengthReserved, endian: Endian::Little } }));
assert_eq!(m.total_size(), 260);
}
@@ -1344,7 +1370,7 @@ mod tests {
}
});
let m = map(&root, "S");
assert_eq!(m.get("name"), Some(&OffsetEntry { range: ByteRange { start: 0, end: 256 }, meta: LeafMeta { kind: AlkTypeKind::String, encoding: VariableEncoding::LengthPrefixed, endian: Endian::Little } }));
assert_eq!(m.get("name"), Some(&OffsetEntry { range: ByteRange { start: 0, end: 256 }, meta: LeafMeta { kind: AlkTypeKind::String, encoding: VariableEncoding::MaxLengthReserved, endian: Endian::Little } }));
assert_eq!(m.get("id"), Some(&OffsetEntry { range: ByteRange { start: 256, end: 260 }, meta: LeafMeta { kind: AlkTypeKind::Uint32, encoding: VariableEncoding::LengthPrefixed, endian: Endian::Little } }));
}
+8
View File
@@ -248,6 +248,14 @@ pub enum VariableEncoding {
/// `{offset: u32, length: u32}` pointing into a separate data region.
/// The metatensor blob tensor pattern.
OffsetIndirect,
/// The aligned-mode `maxLength` fixed-size reservation (ADR-003
/// strategy 2, the `VARCHAR(N)` pattern): the entry's byte range
/// holds the raw zero-padded data with *no* length prefix — trailing
/// NUL bytes are trimmed on read. Recorded as its own variant so
/// leaf-level consumers (`read_field`) can distinguish a
/// reservation window from a plain length prefix, which would
/// misparse the raw data as a huge prefix (finding W3-3).
MaxLengthReserved,
}
#[cfg(test)]