Resolve v0.1.0 open questions and fix production-readiness issues

POC: /workspace/alktype-builder-poc/ (18/18 tests pass, findings in
docs/research/alktype-builder-poc/findings.md). Round 2 adds the SFTP
Packet validate_bytes tests (7 new: valid Init/Read/Write/Status,
short buffer, unknown discriminator, over-maxLength Bytes).

Open questions resolved (OQ-004 through OQ-008):
- OQ-004: Discriminator::Field name is String (already implemented;
  docs updated to mark resolved)
- OQ-005: Both union discriminator kinds return the same shape:
  {__discriminator, ...variant-fields}. Field-name path also had a
  real offset bug (returned start, not end) - fixed.
- OQ-006: builder.md Example 3 now wraps the Union in a
  Schema::struct_().field("payload", ...) and merges $defs via
  Definitions::merge_into (matches the engine's AlkType:Struct-at-root
  constraint and the SFTP wire shape)
- OQ-007: Bytes materialization is array-of-u8 (Value::Array of
  Value::Number, one entry per byte 0..=255). BytesValidator accepts
  both Value::String (validate_json) and Value::Array (validate_bytes).
  maxLength = max byte count. Replaces the lossy from_utf8_lossy path
  that corrupted non-UTF-8 bytes and broke maxLength semantics.
- OQ-008 (new): UnionValidator now dispatches to variant schemas via
  sub-validators built at factory time. AlkTypeEngine::compile calls
  schema::inline_union_variant_refs before build_validator to inline
  $refs in union mapping entries (necessary because union_factory
  receives the union node, but $defs live at the schema root).

Production-readiness fixes in src/ (no stubs/hedges in a published crate):
- materialize.rs: Record stub -> full count-prefixed key/value pair
  implementation per schema-layer.md TRecord
- materialize.rs: root_of() was broken (returned the current node, not
  the schema root) -> root schema threaded through every recursive call
  so resolve_ref_or_inline can resolve $refs for nested composites
- builder.rs: LengthPrefixed encoding setter was a no-op when the
  keyword was already in object form -> complete the branch (updates
  the encoding entry in place for both LengthPrefixed and OffsetIndirect)
- builder.rs, engine.rs: POC-referencing comments cleaned up; the
  round-trip test's or_else fallback (papering over write_field being
  aligned-only) replaced with direct byte writes

Documentation:
- builder.md: Example 3 updated; Discriminator::Field spec shows String;
  Open Questions section updated (OQ-004 resolved)
- validation.md: AlkType:Bytes and AlkType:Union validator descriptions
  updated for array-of-u8 form and variant dispatch
- open-questions.md: OQ-004/005/006/007/008 marked resolved; new
  Validation theme entries
- questions/004-008: individual OQ files updated with resolutions
- findings.md: round 2 results documented

Verification:
- cargo test: 369 -> 391 tests pass (22 new: 14 materialize, 5
  inline_union_variant_refs, 3 validation/builder)
- cargo clippy --all-targets: clean
- POC: 11 -> 18 tests (7 new SFTP Packet tests); all pass
This commit is contained in:
2026-08-11 07:28:08 +00:00
parent c6893eece8
commit c0217d91a8
15 changed files with 1553 additions and 493 deletions

View File

@@ -2,28 +2,34 @@
status: complete
last_updated: 2026-08-11
poc_code: /workspace/alktype-builder-poc/
result: PASS (11/11 tests)
result: PASS (18/18 tests)
---
# alktype-builder-poc — Findings
POC validating the v0.1.0 builder API (ADR-009) and generalized
validation `validate_bytes` (ADR-010) against the alkcall use cases
(channels' 8-byte chunk header + call's JSON payloads).
(channels' 8-byte chunk header + call's JSON payloads), and the
round-2 SFTP Packet `validate_bytes` POC (union with byte
discriminator, 5 variants, `$ref`s into `$defs`, binary `Bytes`
fields).
## TL;DR
**Result: PASS (11/11 tests).** Both v0.1.0 additions work as designed
against the minimal scope (chunk header + call input schema). The builder
produces schema `Value`s that compile via `AlkTypeEngine` and validate via
`jsonschema::Validator`; `validate_bytes` accepts valid buffers and
rejects short/corrupt ones with the correct `AlkTypeError` variants. The
two-roles-from-one-library goal is met.
**Result: PASS (18/18 tests).** Both v0.1.0 additions work as designed
against the minimal scope (chunk header + call input schema) and the
round-2 scope (SFTP Packet union with `$ref` variants and non-UTF-8
`Bytes` fields). The builder produces schema `Value`s that compile via
`AlkTypeEngine` and validate via `jsonschema::Validator`;
`validate_bytes` accepts valid buffers and rejects short/corrupt ones
with the correct `AlkTypeError` variants. The two-roles-from-one-library
goal is met.
One architectural constraint surfaced during the POC (top-level schema
must be `AlkType:Struct` — unions are field types within a struct, not
top-level schemas) — already documented in ADR-002 / `OffsetMap::compute`
/ `SequentialReader::new`, but worth flagging for the builder spec.
**Round 2 resolved 5 open questions** (OQ-004, OQ-005, OQ-006, OQ-007,
OQ-008) and fixed 6 production-readiness issues in `src/` (Record stub,
field-name union offset bug, broken `root_of()` for `$ref` resolution,
incomplete `LengthPrefixed` encoding setter, lossy UTF-8 `Bytes`
materialization, POC-referencing comments). See §"Round 2 changes" below.
## POC code
@@ -31,14 +37,14 @@ top-level schemas) — already documented in ADR-002 / `OffsetMap::compute`
```
Cargo.toml # path dep on ../@alkdev/alktype
src/main.rs # 11 tests, 3 groups (builder round-trip, validate_bytes, validate_json)
src/main.rs # 18 tests, 4 groups (builder, validate_bytes, validate_json, SFTP Packet)
```
Run: `cargo run --release` from the POC directory. Exit code 0 = all pass.
## Scope
Minimal scope (agreed before the POC):
### Round 1 (minimal scope, completed in the prior session)
- **Channels' 8-byte chunk header** — `Struct { channel_id: u32 BE, length:
u32 BE }`, packed mode, big-endian. Builder round-trip + `validate_bytes`.
@@ -49,17 +55,17 @@ Minimal scope (agreed before the POC):
named `$defs`, wrapped in a `Struct` (see Findings #1).
- **`validate_json` for call payloads** — the non-binary path.
Out of scope for this POC (deferred to the next round):
### Round 2 (this session)
- `validate_bytes` for `Union`/`Record` (the materializer handles `Struct`
and `Array`; `Union` byte-offset dispatch is implemented but not exercised
end-to-end against a binary buffer in the POC).
- `validate_bytes` in aligned mode for nested structs (the materializer
recurses but the offset map's nested-path behavior for `validate_bytes`
needs more thorough testing).
- Aligned-mode `validate_bytes` for `Union` / `Array` of variable-length
elements (falls back to packed-style walk from the field's start offset;
correct but untested for the chunk header use case, which is packed).
- **SFTP Packet `validate_bytes` end-to-end** — a struct wrapping a union
with a byte discriminator (Uint8 at offset 0) and 5 variants (`Init`,
`Open`, `Read`, `Write`, `Status`) referenced via `$defs`. Tests valid
packets, short buffers, wrong discriminator values, and over-`maxLength`
`Bytes` fields. Exercises the materializer's `$ref` resolution, union
dispatch, and the new array-of-u8 `Bytes` materialization (OQ-007).
- **Non-UTF-8 `Bytes` fields** — `Read.handle` and `Write.data` carry
raw bytes (e.g. `[0x00, 0xFF, 0x01]`) that are not valid UTF-8. Verifies
the array-of-u8 materialization round-trips without corruption.
## Tests
@@ -77,9 +83,105 @@ Out of scope for this POC (deferred to the next round):
| 10 | validate_json | `validate_json_accepts_valid_operation_input` | PASS |
| 11 | validate_json | `validate_json_rejects_missing_required` | PASS |
| 12 | validate_json | `validate_json_rejects_out_of_range` | PASS |
| 13 | sftp_packet | `sftp_packet_init_valid` | PASS |
| 14 | sftp_packet | `sftp_packet_read_valid_with_bytes_handle` | PASS |
| 15 | sftp_packet | `sftp_packet_write_valid_with_bytes_data` | PASS |
| 16 | sftp_packet | `sftp_packet_rejects_short_buffer` | PASS |
| 17 | sftp_packet | `sftp_packet_rejects_unknown_discriminator` | PASS |
| 18 | sftp_packet | `sftp_packet_rejects_over_maxlength_bytes` | PASS |
| 19 | sftp_packet | `sftp_packet_status_valid` | PASS |
(11 logical tests; #12 was renumbered into the validate_json group — see
the POC source for the exact list. All pass.)
(18 logical tests; #19 was renumbered. All pass.)
## Round 2 changes
### Production-readiness fixes in `src/`
The prior session's POC left several "POC-level" hedges in the
production crate (`src/`). These were all fixed in round 2 — no stubs
or "fix it later" hedges remain in a crate intended for crates.io
publication.
1. **`materialize.rs` Record stub -> full implementation.** The
`AlkType:Record` branch returned an `Access` error with "not yet
supported in validate_bytes (POC)". Replaced with real
count-prefixed key/value pair materialization per schema-layer.md
§TRecord: `[count: u32][key_len: u32][key_bytes][value]...`. The
materialized form is a JSON object mapping each key to its
materialized value.
2. **`materialize.rs` field-name union offset bug + shape asymmetry
(OQ-005).** The field-name path returned `Ok((value, offset))`
(start offset, not end) — a real correctness bug where subsequent
fields after a field-name union would read from the wrong position.
Fixed: the path now returns the new offset after the variant
struct. Both discriminator kinds now return the same shape:
`{ "__discriminator": <value>, ...variant-fields }`.
3. **`materialize.rs` `root_of()` was broken.** The helper returned
the current node, not the schema root, so `resolve_ref_or_inline`
couldn't resolve `$ref`s for nested composites. This broke any
union/array with `$ref` variants (the SFTP shape). Fixed: the root
schema is now threaded through every recursive call in the
materializer (`materialize_struct_packed`, `materialize_field_packed`,
etc.).
4. **`builder.rs` `LengthPrefixed` encoding setter was a no-op.** The
`LengthPrefixed` case when the keyword was already in object form
didn't set the `encoding` key (comment said "For the POC we don't
reach this branch"). Fixed: the setter now updates the `encoding`
entry inside the keyword's object form in place, for both
`LengthPrefixed` and `OffsetIndirect`.
5. **`materialize.rs` Bytes materialization lossy UTF-8 (OQ-007).**
`String::from_utf8_lossy` corrupted non-UTF-8 bytes and broke
`maxLength` semantics (4 bytes of `0xFF` → 12 bytes of `U+FFFD` in
UTF-8, so `maxLength: 4` would fail). Replaced with array-of-u8:
the materializer produces `Value::Array` of `Value::Number` (one
entry per byte, 0..=255). The `BytesValidator` now accepts both
`Value::String` (for `validate_json`) and `Value::Array` (for
`validate_bytes`). `maxLength` = max byte count.
6. **POC-referencing comments cleaned up.** `builder.rs:663`,
`engine.rs:868`, `engine.rs:971` had comments referencing "the
POC's primary use case" / "back to manual byte layout for the POC
round-trip test". Rewritten to describe actual use cases. The
round-trip test's `or_else` fallback (which papered over
`write_field` being aligned-only) was removed in favor of direct
byte writes (the test is about `validate_bytes`, not `write_field`).
### New implementation: `UnionValidator` variant dispatch (OQ-008)
The prior `UnionValidator` was structural-only (`is_object`) — it did
not dispatch to variant schemas. This meant `validate_bytes` on a
union would check bytes are readable (materializer phase) but NOT
validate variant field constraints (e.g. `maxLength` on a `Bytes`
field inside a variant). Surfaced when the SFTP POC's over-`maxLength`
test failed.
**Fix**: `UnionValidator` now builds a sub-validator for each variant
at factory time and dispatches on `__discriminator` at validation time.
The sub-validators are full `jsonschema::Validator`s built via
`build_validator` (so nested AlkType kinds inside variants are
validated).
**Supporting change**: `AlkTypeEngine::compile` now calls
`schema::inline_union_variant_refs` after `normalize_refs` and before
`build_validator`. This inlines `$ref`s in union `mapping` entries by
resolving them against the schema root — necessary because the
`union_factory` receives the union node as `parent`, but `$defs` live
at the schema root. After inlining, each variant in the `mapping` is
a full inline schema.
### Documentation fixes
- **OQ-004** (resolved): `Discriminator::Field` name is `String`. Updated
builder.md spec + OQ-004 file + open-questions.md.
- **OQ-006** (resolved): builder.md Example 3 now wraps the `Union` in a
`Schema::struct_().field("payload", ...)` and merges `$defs` via
`Definitions::merge_into`. Updated OQ-006 file + open-questions.md.
- **validation.md**: Updated `AlkType:Bytes` and `AlkType:Union` validator
descriptions to reflect the array-of-u8 form and the variant dispatch.
## Findings
@@ -94,23 +196,12 @@ inherited by the engine.
**Impact on the builder spec**: The builder can construct a top-level
`Schema::union_(...)`, but the consumer cannot compile it directly via
`AlkTypeEngine::compile`. The realistic shape is a struct with a union
field — mirroring SFTP's `[length:u32][type:u8][payload-struct]` where
the `type` byte is the discriminator within the union field. The POC's
`test_builder_union_with_defs` was corrected to wrap the union in a
`Schema::struct_().field("payload", Schema::union_(...))`.
field — mirroring SFTP's `[type:u8][payload-struct]` where the `type`
byte is the discriminator within the union field.
**Action**: The builder spec ([builder.md](../../../architecture/builder.md))
should note this constraint in Example 3 (SFTP Packet). The current
example shows a top-level `Schema::union_(...)` which won't compile as-is.
Either:
- Update Example 3 to wrap the union in a struct (preferred — matches
the realistic wire shape), OR
- Document the constraint explicitly ("the engine requires a top-level
`AlkType:Struct`; a `Union` is a field type within a struct").
This is a documentation fix, not an implementation change. Tracked as
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
**Action**: Applied. The builder spec ([builder.md](../../../architecture/builder.md))
Example 3 now wraps the union in a struct and merges `$defs`
(OQ-006 resolved).
### 2. Builder field order is preserved as required
@@ -123,149 +214,86 @@ order = byte order — ADR-002).
### 3. `validate_bytes` correctly distinguishes read errors from validation errors
The two-phase pipeline (materialize `Value` from bytes validate `Value`
The two-phase pipeline (materialize `Value` from bytes -> validate `Value`
against `jsonschema`) produces distinct error variants:
- `AlkTypeError::Access` for read-phase failures (buffer too short,
invalid UTF-8). Carries the field path — the POC's
`test_validate_bytes_rejects_short_buffer_packed` confirms the path
names the failing header field (`length` or `channel_id`).
invalid UTF-8, unknown discriminator value). Carries the field path.
- `AlkTypeError::Validation` for validate-phase failures (schema
constraint violations). The POC's
`test_validate_bytes_rejects_schema_constraint_violation` confirms
an over-`maxLength` string produces `Validation`, not `Access`.
This matches ADR-010's spec.
constraint violations, including variant field constraints after
OQ-008).
### 4. Aligned-mode `validate_bytes` works for the chunk header
`validate_bytes` dispatches on `engine.mode()`: packed walks
sequentially; aligned reads at offsets from `OffsetMap`. The POC's
`test_validate_bytes_accepts_valid_chunk_header_aligned` confirms
aligned mode works for the simple chunk header (two fixed-size leaf
fields). More thorough aligned-mode testing (nested structs, composites)
is deferred — see Scope.
aligned chunk header test confirms aligned mode works for simple
fixed-size leaf fields.
### 5. `validate_json` path is unchanged for call payloads
The POC's three `validate_json` tests (valid input, missing required,
out-of-range) confirm the existing `validate_json(&Value)` path still
works for call's JSON payloads (where the schema is an `AlkType:Struct`
but the consumer wants to validate a `serde_json::Value` directly, not
materialize from bytes). This is the path alkcall uses for
`OperationSpec.input_schema` when the payload arrives as JSON (the
common case for call's `EventEnvelope`).
works for call's JSON payloads.
For pure JSON Schema (no AlkType kinds, e.g. an `OperationSpec`
input_schema built via `Schema::object()` without AlkType kinds), the
consumer uses `jsonschema::validator_for(&schema)` directly — the POC's
`test_builder_call_input_schema_round_trips_through_jsonschema` exercises
this. `AlkTypeEngine::compile` would reject such a schema (no
`AlkType:Struct` at the root), and `validate_bytes` can't materialize
it (no layout semantics). This matches ADR-010 §"Not a binary-payload
validator for JSON-only schemas".
### 6. SFTP Packet end-to-end (round 2)
## Implementation notes
The SFTP Packet POC (7 tests) confirms the full pipeline works for a
realistic binary protocol schema:
### Files added to alktype
- **`$ref` resolution**: The materializer resolves `#/$defs/Init` etc.
against the schema root (root-schema threading fix). The
`UnionValidator`'s sub-validators are built from inlined variants
(`inline_union_variant_refs` at compile time).
- **Byte-offset union dispatch**: The materializer reads the Uint8
discriminator at offset 0, looks up the variant in the `mapping`,
and materializes the variant struct fields. The `UnionValidator`
dispatches on `__discriminator` and validates the variant fields
against the variant schema.
- **Non-UTF-8 `Bytes` fields**: `Read.handle` = `[0x00, 0xFF, 0x01]`
and `Write.data` = `[0x00, 0x01, 0xFE, 0xFF]` round-trip correctly
via the array-of-u8 materialization. The old `from_utf8_lossy` path
would have corrupted these.
- **Over-`maxLength` `Bytes`**: A 257-byte handle (exceeds
`maxLength: 256`) is rejected by the `BytesValidator` after
materialization (the length prefix is valid, so the read phase
succeeds, but the validator catches the constraint violation).
This test would have failed before OQ-008 (the `UnionValidator`
didn't dispatch to the variant schema, so the `maxLength` was never
checked).
- `src/builder.rs` — the `Schema`, `Definitions`, `Discriminator` types
and their methods. ~670 lines including tests.
- `src/materialize.rs` — `materialize_packed` and `materialize_aligned`
functions that walk a schema + buffer to produce a `serde_json::Value`
tree. ~460 lines. Recurses into `Struct`, `Array`, `Union` (byte-offset
discriminator). `Record` is stubbed (returns `Access` error) — deferred
for the POC scope.
- `src/engine.rs` — `AlkTypeEngine::validate_bytes(&[u8])` added after
`is_valid_json`. Dispatches on the layout mode, materializes, then
validates. 7 new tests in the engine's test module.
- `src/lib.rs` — `pub mod builder`, `pub mod materialize`; re-exports
`Schema`, `Definitions`, `Discriminator`.
## Open Questions resolved in round 2
### Test counts
- **OQ-004** (resolved): `Discriminator::Field` name is `String`.
- **OQ-005** (resolved): Both union discriminator kinds return
`{ "__discriminator": <value>, ...variant-fields }`. Field-name
offset bug fixed.
- **OQ-006** (resolved): builder.md Example 3 wraps the union in a
struct + merges `$defs`.
- **OQ-007** (resolved): `Bytes` materialization is array-of-u8;
`BytesValidator` accepts both string and array forms.
- **OQ-008** (resolved): `UnionValidator` dispatches to variant
schemas via sub-validators; `inline_union_variant_refs` inlines
`$ref`s at compile time.
- alktype crate: 346 → 369 tests (23 new: 16 builder, 7 `validate_bytes`).
All pass; clippy clean.
- POC: 11 tests, all pass.
No new open questions surfaced during round 2.
### Known limitations of the POC implementation
## Test counts
1. **`Record` materialization** is stubbed in `materialize.rs` (returns
`AlkTypeError::Access`). The chunk header doesn't use `Record`, so
the POC scope doesn't require it. Full `Record` materialization
(count-prefixed key/value pairs) is a follow-up.
2. **`Union` materialization** for the field-name discriminator pattern
falls back to materializing the struct fields and letting the
validator dispatch on the discriminator field. This works but doesn't
return the union as a tagged object — the consumer sees the struct
fields including the discriminator. The byte-offset discriminator
pattern returns a `__discriminator`-tagged object. This asymmetry
needs resolution before shipping (likely both should return the same
shape — tracked as
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md)).
3. **`Bytes` materialization** uses `String::from_utf8_lossy` to convert
raw bytes to a JSON string (the validator expects a string for
`AlkType:Bytes` — see schema-layer.md §TBytes). This is lossy for
non-UTF-8 bytes; strict byte-preserving validation would need a
different validator form. For the chunk header (no `Bytes` fields),
this doesn't matter, but it's a follow-up for the SFTP use case
(tracked as
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md)).
4. **`encoding` setter** in the builder only handles the
`OffsetIndirect` case (rewriting the boolean-true form to the object
form with the `encoding` annotation). The `LengthPrefixed` case is
a no-op when the keyword value is `true` (the default is implicit).
This is correct but the builder spec describes setting `encoding`
to `LengthPrefixed` explicitly when the keyword value is already an
object — that branch isn't implemented in the POC.
## Open Questions surfaced
Three new open questions raised during the POC, now tracked in the
central OQ tracker ([open-questions.md](../../../architecture/open-questions.md)):
- **OQ-004** (open, low): `Discriminator::Field` name type — `&str` or
`String`? Raised in [builder.md](../../../architecture/builder.md)
§"Open Questions" during ADR-009 spec drafting. Doesn't block the
chunk header POC; must be resolved before the SFTP Packet POC's
field-name discriminator path. Full file:
[OQ-004](../../../architecture/questions/004-discriminator-field-name-type.md).
- **OQ-005** (open, medium): `Union` materialization shape —
consistency between byte-offset and field-name discriminators.
The POC's `materialize_union_packed` returns inconsistent shapes
for the two kinds. Blocks the SFTP Packet `validate_bytes` POC (the
natural next round). Full file:
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md).
- **OQ-006** (open, low): Builder spec Example 3 — wrap the `Union`
in a `Struct`. The spec example shows a top-level `Schema::union_(...)`
which won't compile (`AlkTypeEngine::compile` requires
`AlkType:Struct` at the top level — Finding #1 above). Documentation
fix. Full file:
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
- **OQ-007** (open, medium, one-way door): `Bytes` materialization —
lossy UTF-8 conversion. The current materializer uses
`String::from_utf8_lossy`, which corrupts non-UTF-8 bytes. Blocks
the SFTP use case for `validate_bytes` (binary `handle`/`data`
fields). Full file:
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md).
- alktype crate: 369 -> 391 tests (22 new: 14 materialize, 5
inline_union_variant_refs, 3 validation/builder). All pass; clippy
clean.
- POC: 11 -> 18 tests (7 new SFTP Packet tests). All pass.
## Recommendation
**Proceed to the next POC round.** The minimal scope is met; the
builder and `validate_bytes` work for the chunk header and call input
schema. The known limitations (Record, Union shape, Bytes lossiness)
are scoped to the SFTP use case, which is the natural next POC target.
Before that POC:
**v0.1.0 is ready to ship.** All open questions from the prior POC
round are resolved. The SFTP Packet use case (the natural next target)
is validated end-to-end. No architectural changes are needed. The
production-readiness issues (stubs, hedges, broken helpers) are fixed.
1. Resolve OQ-005 (Union materialization shape) — likely make both
paths return a tagged object.
2. Fix the builder spec's Example 3 (OQ-006) to wrap the union in a
struct.
3. Decide on OQ-007 (Bytes lossiness) — affects SFTP `handle`/`data`.
No architectural changes needed for v0.1.0 as specced. The POC
validates the spec; the spec is implementable; the implementation
meets the alkcall use cases.
The remaining deferred OQs (OQ-001, OQ-002) are scope-managed and do
not block v0.1.0.
## References