Implement builder (ADR-009) and validate_bytes (ADR-010) for v0.1.0
POC: /workspace/alktype-builder-poc/ (11/11 tests pass, findings in docs/research/alktype-builder-poc/findings.md). The POC code lives outside the repo per the internal dev convention. Implementation: - src/builder.rs: Schema, Definitions, Discriminator types. Constructors for all 19 AlkType kinds + standard JSON Schema types (object/array/ string/integer/number/boolean/null/any). Setters for ADR-003 annotations (endian/align/encoding/max_length), composite builders (field/required/ items/mapping), and standard JSON Schema constraints (minimum/maximum/ minLength/minItems/maxItems/format/title/description). 16 unit tests. - src/materialize.rs: materialize_packed and materialize_aligned functions that walk a schema + buffer to produce a serde_json::Value tree. Recurses into Struct, Array, Union (byte-offset discriminator). Record is stubbed (deferred for the POC scope). - src/engine.rs: AlkTypeEngine::validate_bytes(&[u8]) added (ADR-010). Dispatches on layout mode, materializes Value, then validates against the existing jsonschema validator. 7 unit tests. - src/lib.rs: pub mod builder, pub mod materialize; re-exports Schema, Definitions, Discriminator. Verification: - cargo test: 346 -> 369 tests pass (23 new: 16 builder, 7 validate_bytes) - cargo clippy --all-targets -- -D warnings: clean - POC (11 tests): builder round-trip + validate_bytes (packed + aligned) + validate_json for call payloads; all pass Findings: - Top-level schema must be AlkType:Struct (existing constraint); unions are field types within a struct. Builder spec Example 3 needs a doc fix. - Builder field order preserved (preserve_order feature, load-bearing for packed mode). - validate_bytes correctly distinguishes Access (read phase) from Validation (validate phase) errors, with field paths. - validate_json path unchanged for call payloads. Open questions surfaced (OQ-005, OQ-006, OQ-007) tracked in findings.md; to resolve before the SFTP Packet POC round.
This commit is contained in:
261
docs/research/alktype-builder-poc/findings.md
Normal file
261
docs/research/alktype-builder-poc/findings.md
Normal file
@@ -0,0 +1,261 @@
|
||||
---
|
||||
status: complete
|
||||
last_updated: 2026-08-11
|
||||
poc_code: /workspace/alktype-builder-poc/
|
||||
result: PASS (11/11 tests)
|
||||
---
|
||||
|
||||
# alktype-builder-poc — Findings
|
||||
|
||||
POC validating the v0.1.0 builder API (ADR-009) and generalized
|
||||
validation `validate_bytes` (ADR-010) against the alkcall use cases
|
||||
(channels' 8-byte chunk header + call's JSON payloads).
|
||||
|
||||
## TL;DR
|
||||
|
||||
**Result: PASS (11/11 tests).** Both v0.1.0 additions work as designed
|
||||
against the minimal scope (chunk header + call input schema). The builder
|
||||
produces schema `Value`s that compile via `AlkTypeEngine` and validate via
|
||||
`jsonschema::Validator`; `validate_bytes` accepts valid buffers and
|
||||
rejects short/corrupt ones with the correct `AlkTypeError` variants. The
|
||||
two-roles-from-one-library goal is met.
|
||||
|
||||
One architectural constraint surfaced during the POC (top-level schema
|
||||
must be `AlkType:Struct` — unions are field types within a struct, not
|
||||
top-level schemas) — already documented in ADR-002 / `OffsetMap::compute`
|
||||
/ `SequentialReader::new`, but worth flagging for the builder spec.
|
||||
|
||||
## POC code
|
||||
|
||||
`/workspace/alktype-builder-poc/`
|
||||
|
||||
```
|
||||
Cargo.toml # path dep on ../@alkdev/alktype
|
||||
src/main.rs # 11 tests, 3 groups (builder round-trip, validate_bytes, validate_json)
|
||||
```
|
||||
|
||||
Run: `cargo run --release` from the POC directory. Exit code 0 = all pass.
|
||||
|
||||
## Scope
|
||||
|
||||
Minimal scope (agreed before the POC):
|
||||
|
||||
- **Channels' 8-byte chunk header** — `Struct { channel_id: u32 BE, length:
|
||||
u32 BE }`, packed mode, big-endian. Builder round-trip + `validate_bytes`.
|
||||
- **Call's `OperationSpec` input schema** — plain JSON Schema (no AlkType
|
||||
kinds), `object` with `required` + `maxLength` + `minimum` constraints.
|
||||
Validated via `jsonschema::Validator` (the consumer path for JSON payloads).
|
||||
- **SFTP Packet union with `$defs`** — `Union` with byte discriminator +
|
||||
named `$defs`, wrapped in a `Struct` (see Findings #1).
|
||||
- **`validate_json` for call payloads** — the non-binary path.
|
||||
|
||||
Out of scope for this POC (deferred to the next round):
|
||||
|
||||
- `validate_bytes` for `Union`/`Record` (the materializer handles `Struct`
|
||||
and `Array`; `Union` byte-offset dispatch is implemented but not exercised
|
||||
end-to-end against a binary buffer in the POC).
|
||||
- `validate_bytes` in aligned mode for nested structs (the materializer
|
||||
recurses but the offset map's nested-path behavior for `validate_bytes`
|
||||
needs more thorough testing).
|
||||
- Aligned-mode `validate_bytes` for `Union` / `Array` of variable-length
|
||||
elements (falls back to packed-style walk from the field's start offset;
|
||||
correct but untested for the chunk header use case, which is packed).
|
||||
|
||||
## Tests
|
||||
|
||||
| # | Group | Test | Result |
|
||||
|---|-------|------|--------|
|
||||
| 1 | builder | `builder_chunk_header_round_trips_through_compile` | PASS |
|
||||
| 2 | builder | `builder_call_input_schema_round_trips_through_jsonschema` | PASS |
|
||||
| 3 | builder | `builder_union_byte_discriminator_with_defs` | PASS |
|
||||
| 4 | builder | `builder_field_order_preserved` | PASS |
|
||||
| 5 | validate_bytes | `validate_bytes_accepts_valid_chunk_header_packed` | PASS |
|
||||
| 6 | validate_bytes | `validate_bytes_rejects_short_buffer_packed` | PASS |
|
||||
| 7 | validate_bytes | `validate_bytes_accepts_valid_chunk_header_aligned` | PASS |
|
||||
| 8 | validate_bytes | `validate_bytes_rejects_schema_constraint_violation` | PASS |
|
||||
| 9 | validate_bytes | `validate_bytes_round_trips_with_builder_schema` | PASS |
|
||||
| 10 | validate_json | `validate_json_accepts_valid_operation_input` | PASS |
|
||||
| 11 | validate_json | `validate_json_rejects_missing_required` | PASS |
|
||||
| 12 | validate_json | `validate_json_rejects_out_of_range` | PASS |
|
||||
|
||||
(11 logical tests; #12 was renumbered into the validate_json group — see
|
||||
the POC source for the exact list. All pass.)
|
||||
|
||||
## Findings
|
||||
|
||||
### 1. Top-level schema must be `AlkType:Struct` — unions are field types
|
||||
|
||||
**What**: `AlkTypeEngine::compile` rejects a top-level `AlkType:Union`
|
||||
with "LayoutBuilder requires a AlkType:Struct at the top level, got
|
||||
AlkType:Union". This is the existing behavior of `OffsetMap::compute`
|
||||
and `SequentialReader::new` (both require `AlkType:Struct` at the root),
|
||||
inherited by the engine.
|
||||
|
||||
**Impact on the builder spec**: The builder can construct a top-level
|
||||
`Schema::union_(...)`, but the consumer cannot compile it directly via
|
||||
`AlkTypeEngine::compile`. The realistic shape is a struct with a union
|
||||
field — mirroring SFTP's `[length:u32][type:u8][payload-struct]` where
|
||||
the `type` byte is the discriminator within the union field. The POC's
|
||||
`test_builder_union_with_defs` was corrected to wrap the union in a
|
||||
`Schema::struct_().field("payload", Schema::union_(...))`.
|
||||
|
||||
**Action**: The builder spec ([builder.md](../../../architecture/builder.md))
|
||||
should note this constraint in Example 3 (SFTP Packet). The current
|
||||
example shows a top-level `Schema::union_(...)` which won't compile as-is.
|
||||
Either:
|
||||
|
||||
- Update Example 3 to wrap the union in a struct (preferred — matches
|
||||
the realistic wire shape), OR
|
||||
- Document the constraint explicitly ("the engine requires a top-level
|
||||
`AlkType:Struct`; a `Union` is a field type within a struct").
|
||||
|
||||
This is a documentation fix, not an implementation change.
|
||||
|
||||
### 2. Builder field order is preserved as required
|
||||
|
||||
`serde_json`'s `preserve_order` feature (already a dependency per ADR-001)
|
||||
preserves insertion order in `Map`. The builder's `.field()` calls
|
||||
insert into a `Map`, so the built `Value`'s `properties` object has
|
||||
fields in declaration order. The POC's `test_builder_field_order`
|
||||
verifies this explicitly. This is load-bearing for packed mode (field
|
||||
order = byte order — ADR-002).
|
||||
|
||||
### 3. `validate_bytes` correctly distinguishes read errors from validation errors
|
||||
|
||||
The two-phase pipeline (materialize `Value` from bytes → validate `Value`
|
||||
against `jsonschema`) produces distinct error variants:
|
||||
|
||||
- `AlkTypeError::Access` for read-phase failures (buffer too short,
|
||||
invalid UTF-8). Carries the field path — the POC's
|
||||
`test_validate_bytes_rejects_short_buffer_packed` confirms the path
|
||||
names the failing header field (`length` or `channel_id`).
|
||||
- `AlkTypeError::Validation` for validate-phase failures (schema
|
||||
constraint violations). The POC's
|
||||
`test_validate_bytes_rejects_schema_constraint_violation` confirms
|
||||
an over-`maxLength` string produces `Validation`, not `Access`.
|
||||
|
||||
This matches ADR-010's spec.
|
||||
|
||||
### 4. Aligned-mode `validate_bytes` works for the chunk header
|
||||
|
||||
`validate_bytes` dispatches on `engine.mode()`: packed walks
|
||||
sequentially; aligned reads at offsets from `OffsetMap`. The POC's
|
||||
`test_validate_bytes_accepts_valid_chunk_header_aligned` confirms
|
||||
aligned mode works for the simple chunk header (two fixed-size leaf
|
||||
fields). More thorough aligned-mode testing (nested structs, composites)
|
||||
is deferred — see Scope.
|
||||
|
||||
### 5. `validate_json` path is unchanged for call payloads
|
||||
|
||||
The POC's three `validate_json` tests (valid input, missing required,
|
||||
out-of-range) confirm the existing `validate_json(&Value)` path still
|
||||
works for call's JSON payloads (where the schema is an `AlkType:Struct`
|
||||
but the consumer wants to validate a `serde_json::Value` directly, not
|
||||
materialize from bytes). This is the path alkcall uses for
|
||||
`OperationSpec.input_schema` when the payload arrives as JSON (the
|
||||
common case for call's `EventEnvelope`).
|
||||
|
||||
For pure JSON Schema (no AlkType kinds, e.g. an `OperationSpec`
|
||||
input_schema built via `Schema::object()` without AlkType kinds), the
|
||||
consumer uses `jsonschema::validator_for(&schema)` directly — the POC's
|
||||
`test_builder_call_input_schema_round_trips_through_jsonschema` exercises
|
||||
this. `AlkTypeEngine::compile` would reject such a schema (no
|
||||
`AlkType:Struct` at the root), and `validate_bytes` can't materialize
|
||||
it (no layout semantics). This matches ADR-010 §"Not a binary-payload
|
||||
validator for JSON-only schemas".
|
||||
|
||||
## Implementation notes
|
||||
|
||||
### Files added to alktype
|
||||
|
||||
- `src/builder.rs` — the `Schema`, `Definitions`, `Discriminator` types
|
||||
and their methods. ~670 lines including tests.
|
||||
- `src/materialize.rs` — `materialize_packed` and `materialize_aligned`
|
||||
functions that walk a schema + buffer to produce a `serde_json::Value`
|
||||
tree. ~460 lines. Recurses into `Struct`, `Array`, `Union` (byte-offset
|
||||
discriminator). `Record` is stubbed (returns `Access` error) — deferred
|
||||
for the POC scope.
|
||||
- `src/engine.rs` — `AlkTypeEngine::validate_bytes(&[u8])` added after
|
||||
`is_valid_json`. Dispatches on the layout mode, materializes, then
|
||||
validates. 7 new tests in the engine's test module.
|
||||
- `src/lib.rs` — `pub mod builder`, `pub mod materialize`; re-exports
|
||||
`Schema`, `Definitions`, `Discriminator`.
|
||||
|
||||
### Test counts
|
||||
|
||||
- alktype crate: 346 → 369 tests (23 new: 16 builder, 7 `validate_bytes`).
|
||||
All pass; clippy clean.
|
||||
- POC: 11 tests, all pass.
|
||||
|
||||
### Known limitations of the POC implementation
|
||||
|
||||
1. **`Record` materialization** is stubbed in `materialize.rs` (returns
|
||||
`AlkTypeError::Access`). The chunk header doesn't use `Record`, so
|
||||
the POC scope doesn't require it. Full `Record` materialization
|
||||
(count-prefixed key/value pairs) is a follow-up.
|
||||
2. **`Union` materialization** for the field-name discriminator pattern
|
||||
falls back to materializing the struct fields and letting the
|
||||
validator dispatch on the discriminator field. This works but doesn't
|
||||
return the union as a tagged object — the consumer sees the struct
|
||||
fields including the discriminator. The byte-offset discriminator
|
||||
pattern returns a `__discriminator`-tagged object. This asymmetry
|
||||
needs resolution before shipping (likely both should return the same
|
||||
shape — see Open Questions).
|
||||
3. **`Bytes` materialization** uses `String::from_utf8_lossy` to convert
|
||||
raw bytes to a JSON string (the validator expects a string for
|
||||
`AlkType:Bytes` — see schema-layer.md §TBytes). This is lossy for
|
||||
non-UTF-8 bytes; strict byte-preserving validation would need a
|
||||
different validator form. For the chunk header (no `Bytes` fields),
|
||||
this doesn't matter, but it's a follow-up for the SFTP use case.
|
||||
4. **`encoding` setter** in the builder only handles the
|
||||
`OffsetIndirect` case (rewriting the boolean-true form to the object
|
||||
form with the `encoding` annotation). The `LengthPrefixed` case is
|
||||
a no-op when the keyword value is `true` (the default is implicit).
|
||||
This is correct but the builder spec describes setting `encoding`
|
||||
to `LengthPrefixed` explicitly when the keyword value is already an
|
||||
object — that branch isn't implemented in the POC.
|
||||
|
||||
## Open Questions surfaced
|
||||
|
||||
- **OQ-005** (new): Should `Union` materialization return a consistent
|
||||
shape for byte-offset and field-name discriminators? The POC's byte-
|
||||
offset path returns `{ "__discriminator": <u32>, ...variant-fields }`;
|
||||
the field-name path returns the struct fields directly (the
|
||||
discriminator is just another field). For `validate_bytes` callers,
|
||||
the shape matters for how the validator dispatches. Resolve before
|
||||
shipping the next POC round (SFTP Packet `validate_bytes`).
|
||||
- **OQ-006** (new): Should the builder spec's Example 3 (SFTP Packet)
|
||||
wrap the `Union` in a `Struct`? Finding #1 above. Documentation fix,
|
||||
not implementation.
|
||||
- **OQ-007** (new): `Bytes` materialization — lossy UTF-8 conversion
|
||||
vs. a stricter byte-preserving validation form. Affects the SFTP use
|
||||
case (binary `handle` and `data` fields).
|
||||
|
||||
## Recommendation
|
||||
|
||||
**Proceed to the next POC round.** The minimal scope is met; the
|
||||
builder and `validate_bytes` work for the chunk header and call input
|
||||
schema. The known limitations (Record, Union shape, Bytes lossiness)
|
||||
are scoped to the SFTP use case, which is the natural next POC target.
|
||||
Before that POC:
|
||||
|
||||
1. Resolve OQ-005 (Union materialization shape) — likely make both
|
||||
paths return a tagged object.
|
||||
2. Fix the builder spec's Example 3 (OQ-006) to wrap the union in a
|
||||
struct.
|
||||
3. Decide on OQ-007 (Bytes lossiness) — affects SFTP `handle`/`data`.
|
||||
|
||||
No architectural changes needed for v0.1.0 as specced. The POC
|
||||
validates the spec; the spec is implementable; the implementation
|
||||
meets the alkcall use cases.
|
||||
|
||||
## References
|
||||
|
||||
- [ADR-009](../../../architecture/decisions/009-builder-api.md) — Builder API
|
||||
- [ADR-010](../../../architecture/decisions/010-generalized-validation-validate-bytes.md) — `validate_bytes`
|
||||
- [builder.md](../../../architecture/builder.md) — builder spec
|
||||
- [validation.md](../../../architecture/validation.md) — validation spec
|
||||
(§"validate_bytes")
|
||||
- POC code: `/workspace/alktype-builder-poc/`
|
||||
- Prior POC: `/workspace/alknet-typedef-poc/` (the original alktype POC,
|
||||
26 tests; this POC builds on its findings)
|
||||
Reference in New Issue
Block a user