alkdev/alktype
4
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Centralize scattered OQs into the tracker (OQ-004 through OQ-007)

Browse Source 
Four open questions were scattered inline in builder.md and the POC
findings doc. Moved them into the central OQ tracker under
docs/architecture/questions/ and updated the index:

- OQ-004: Discriminator::Field name type (&str vs String) — raised in
  builder.md during ADR-009 spec drafting
- OQ-005: Union materialization shape (byte-offset vs field-name
  consistency) — raised in the POC findings
- OQ-006: Builder spec Example 3 wrap Union in Struct — raised in the
  POC findings (doc fix; engine requires AlkType:Struct at top level)
- OQ-007: Bytes materialization lossy UTF-8 — raised in the POC
  findings (blocks SFTP use case for validate_bytes)

Index updates:
- open-questions.md: new 'Schema Construction' and 'Validation' theme
  groups; new 'Open' section for active investigation targets (distinct
  from 'Deferred / Blocked' which holds scope-parked OQs)
- README.md: OQ table extended with OQ-004 through OQ-007
- builder.md: inline OQ-004 replaced with a tracker reference
- findings.md: inline OQ-005/006/007 replaced with tracker references

Doc-only change; 369 tests pass.
This commit is contained in:
glm-5.2
2026-08-11 06:40:28 +00:00
parent 5588278451
commit c6893eece8
8 changed files with 255 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
docs/architecture/README.md
View File

@@ -43,6 +43,10 @@ format definition; the engine is generic.
| OQ-001 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it |
| OQ-002 | `no_std` + `alloc` support | deferred(scope) | Target `std` for v1; blocked on an embedded use case |
| OQ-003 | Builder API for schema construction | resolved (ADR-009) | Resolved in v0.1.0; alkcall is the concrete consumer; see [builder.md](builder.md) |
| OQ-004 | `Discriminator::Field` name — `&str` or `String` | open | Builder API ownership question; resolve before the SFTP Packet POC's field-name discriminator path |
| OQ-005 | `Union` materialization shape — byte-offset vs field-name consistency | open | Blocks the SFTP Packet `validate_bytes` POC (next round) |
| OQ-006 | Builder spec Example 3 — wrap `Union` in a `Struct` | open | Documentation fix in builder.md; the engine requires `AlkType:Struct` at the top level |
| OQ-007 | `Bytes` materialization — lossy UTF-8 conversion | open | Blocks the SFTP use case for `validate_bytes` (binary `handle`/`data` fields) |
## Key Design Principles

6
docs/architecture/builder.md
View File

@@ -609,9 +609,9 @@ let op_errors = vec![
- **OQ-004** (open): `Discriminator::Field` name type — `&str` or
`String`? The builder consumes `Self` on each setter, so the
discriminator's `name` needs to outlive the `Schema` it's embedded
in. `&str` borrows; `String` owns. Resolve during implementation
(likely `String` for ownership simplicity, since the builder owns
its content until `.build()`).
in. `&str` borrows; `String` owns. Tracked in
[open-questions.md](open-questions.md); full file:
[OQ-004](questions/004-discriminator-field-name-type.md).
## References

68
docs/architecture/open-questions.md
View File

@@ -8,9 +8,11 @@ last_updated: 2026-08-11
Each open question lives in its own file under [`questions/`](questions/),
named `NNN-slug.md` (mirroring the ADR convention). This file is the index:
theme-grouped tables for scannability, plus a cross-theme
[Open](#open) section (active investigation targets) and a
[Deferred / Blocked](#deferred--blocked) section that surfaces the
safe-exit deferrals with their blocking conditions inline — so "what's
currently parked and why" is answerable at a glance.
currently on the architect's desk" and "what's currently parked and why"
are each answerable at a glance.
**Status values**:
- `open` — Needs to be resolved now. Has a clear path to resolution.
@@ -60,6 +62,70 @@ Door type is separate from whether a decision is made. A two-way door is a decis
| OQ | Title | Status | Door | Pri |
|----|-------|--------|------|-----|
| [OQ-003](questions/003-builder-api-for-schema-construction.md) | Builder API for Schema Construction | resolved (ADR-009) | two | med |
| [OQ-004](questions/004-discriminator-field-name-type.md) | `Discriminator::Field` name — `&str` or `String` | open | two | low |
| [OQ-006](questions/006-builder-spec-example-3-wrap-union.md) | Builder spec Example 3 — wrap `Union` in a `Struct` | open | two | low |
### Validation
| OQ | Title | Status | Door | Pri |
|----|-------|--------|------|-----|
| [OQ-005](questions/005-union-materialization-shape.md) | `Union` materialization shape — byte-offset vs field-name consistency | open | two | med |
| [OQ-007](questions/007-bytes-materialization-lossy-utf8.md) | `Bytes` materialization — lossy UTF-8 conversion | open | one | med |
## Open
Active investigation targets — questions with a clear path to resolution
that need to be worked through, not waited on. Each has a concrete
investigation target. This section exists so "what's currently on the
architect's desk" is answerable at a glance.
### OQ-004: `Discriminator::Field` name — `&str` or `String`
- **Investigation target**: Decide whether `Discriminator::Field { name }`
should borrow (`&str`) or own (`String`). The builder consumes `Self`
on setters, so `&str` would require a lifetime parameter on
`Discriminator` (and transitively `Schema::union_`). Likely `String`
for ownership simplicity.
- **Priority**: low (doesn't block the chunk header POC; must be
resolved before the SFTP Packet POC's field-name discriminator path)
- **Full file**: [OQ-004](questions/004-discriminator-field-name-type.md)
### OQ-005: `Union` materialization shape — byte-offset vs field-name consistency
- **Investigation target**: The current `materialize_union_packed`
returns inconsistent shapes for the two discriminator kinds (byte-
offset produces `{ "__discriminator": <u32>, ...fields }`; field-name
produces the struct fields directly). Decide on one consistent shape.
Work through the SFTP Packet `validate_bytes` example to surface what
the `jsonschema` validator needs to see for a `AlkType:Union` field.
- **Priority**: medium (blocks the SFTP Packet `validate_bytes` POC,
the natural next round)
- **Full file**: [OQ-005](questions/005-union-materialization-shape.md)
### OQ-006: Builder spec Example 3 — wrap `Union` in a `Struct`
- **Investigation target**: Update builder.md Example 3 (SFTP Packet)
to wrap the `Union` in a `Struct` — the engine requires
`AlkType:Struct` at the top level (ADR-002), and the wrapped shape
matches SFTP's actual wire format (`[length][type][payload]`). The
POC already uses the wrapped shape; the spec example just hasn't
been updated.
- **Priority**: low (documentation fix; doesn't block any implementation)
- **Full file**: [OQ-006](questions/006-builder-spec-example-3-wrap-union.md)
### OQ-007: `Bytes` materialization — lossy UTF-8 conversion
- **Investigation target**: The current materializer uses
`String::from_utf8_lossy` for `AlkType:Bytes` fields, which corrupts
non-UTF-8 bytes. Decide on a round-trippable form (base64 with a
`format` constraint, or `Value::Array` of u8 with a validator that
accepts arrays). Work through SFTP's binary `handle`/`data` fields
to surface what `validate_bytes` callers actually need.
- **Priority**: medium (blocks the SFTP use case for `validate_bytes`;
does not block the chunk header or call input schema)
- **Door type**: one-way (changing what the validator sees is a
semantic break for consumers that depend on the shape)
- **Full file**: [OQ-007](questions/007-bytes-materialization-lossy-utf8.md)
## Deferred / Blocked

27
docs/architecture/questions/004-discriminator-field-name-type.md Normal file
View File

@@ -0,0 +1,27 @@
# OQ-004: `Discriminator::Field` name — `&str` or `String`
- **Origin**: [../builder.md](../builder.md) §"Open Questions" (raised
during ADR-009 spec drafting)
- **Status**: open
- **Door type**: Two-way (the field is internal to the `Discriminator`
enum; changing the type is a local refactor with no schema-level impact)
- **Priority**: low
- **Impacts**: Blocks finalizing the `Discriminator::Field` variant's
signature in `src/builder.rs`. Does NOT block any current POC — the
chunk header uses `Discriminator::Byte`, not `Discriminator::Field`.
The field-name discriminator is exercised by the SFTP Packet POC
(next round), so this must be resolved before that POC.
- **Investigation target**: Decide whether the `name` field of
`Discriminator::Field` should be `&str` (borrows) or `String` (owns).
The builder consumes `Self` on each setter (`.field()`, `.mapping()`,
etc.), so the discriminator's `name` needs to outlive the `Schema`
it's embedded in. `&str` would require a lifetime parameter on
`Discriminator` (and transitively on `Schema::union_`); `String`
owns its content and keeps `Discriminator` `'static`. The likely
resolution is `String` for ownership simplicity — the builder owns
its content until `.build()` produces `Value`, and `String` avoids
lifetime pollution across the builder API.
- **Resolution**: Not yet decided. Confirm during the next POC round
(SFTP Packet) when `Discriminator::Field` is first exercised.
- **Cross-references**: [ADR-009](../decisions/009-builder-api.md),
[builder.md](../builder.md)

40
docs/architecture/questions/005-union-materialization-shape.md Normal file
View File

@@ -0,0 +1,40 @@
# OQ-005: `Union` materialization shape — consistency between byte-offset and field-name discriminators
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
§"Open Questions surfaced" (raised during the v0.1.0 POC)
- **Status**: open
- **Door type**: Two-way (the materializer is internal; the shape it
produces is consumed only by the existing `jsonschema` validator,
which accepts arbitrary objects — changing the shape is a local
refactor)
- **Priority**: medium
- **Impacts**: Blocks the SFTP Packet `validate_bytes` POC (next round).
The current `materialize_union_packed` returns inconsistent shapes:
byte-offset discriminators produce `{ "__discriminator": <u32>,
...variant-fields }`; field-name discriminators produce the struct
fields directly (the discriminator is just another field). For
`validate_bytes` callers, the shape matters for how the validator
dispatches on the variant. Does NOT block the chunk header or call
input schema (no unions in either).
- **Investigation target**: Decide on one consistent shape for
materialized unions. Two candidates:
1. **Both return a tagged object**: `{ "__discriminator": <value>,
...variant-fields }` for both discriminator kinds. The validator
sees the discriminator value and the variant fields in one
object. This is what the byte-offset path already does.
2. **Both return the variant struct directly** (no `__discriminator`
wrapper): the discriminator is consumed during dispatch and not
included in the materialized `Value`. The field-name path
approximates this today.
Option 1 is more uniform and lets the validator re-check the
discriminator; option 2 is cleaner but loses the discriminator in
the `Value` tree. Resolve by working through the SFTP Packet
`validate_bytes` example — what does the `jsonschema` validator need
to see for a `AlkType:Union` field?
- **Resolution**: Not yet decided. Work through the SFTP Packet POC
to surface the validator's requirements.
- **Cross-references**: [ADR-010](../decisions/010-generalized-validation-validate-bytes.md),
`src/materialize.rs` (`materialize_union_packed`),
[../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)

35
docs/architecture/questions/006-builder-spec-example-3-wrap-union.md Normal file
View File

@@ -0,0 +1,35 @@
# OQ-006: Builder spec Example 3 — wrap `Union` in a `Struct`
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
§"Findings" #1 (raised during the v0.1.0 POC)
- **Status**: open
- **Door type**: Two-way (documentation fix — the builder already
supports both shapes; the spec example just shows the wrong one)
- **Priority**: low
- **Impacts**: Blocks nothing functional. The builder spec's Example 3
(SFTP Packet) currently shows a top-level `Schema::union_(...)`,
which won't compile via `AlkTypeEngine::compile` — the engine
requires `AlkType:Struct` at the top level (existing constraint from
`OffsetMap::compute` / `SequentialReader::new`, ADR-002). The POC
worked around this by wrapping the union in a
`Schema::struct_().field("payload", Schema::union_(...))`, which is
the realistic wire shape anyway (SFTP's `[length:u32][type:u8][payload]`
is a struct with a union payload field). The spec example should
match the realistic shape.
- **Investigation target**: Update [../builder.md](../builder.md)
Example 3 to wrap the `Union` in a `Struct`. Two options:
1. **Wrap in a struct** (preferred — matches the realistic wire
shape): `Schema::struct_().field("payload", Schema::union_(...))`.
2. **Document the constraint explicitly**: keep the top-level union
in the example but add a note that `AlkTypeEngine::compile`
requires `AlkType:Struct` at the root; a top-level `Union` is for
`jsonschema`-only validation (no `validate_bytes`).
Option 1 is preferred — it shows the shape the engine actually
consumes and matches the SFTP wire format.
- **Resolution**: Not yet applied. Documentation fix in builder.md,
not an implementation change.
- **Cross-references**: [ADR-009](../decisions/009-builder-api.md),
[builder.md](../builder.md) §"Example 3: SFTP Packet union",
[ADR-002](../decisions/002-two-layout-modes-packed-vs-aligned.md)

45
docs/architecture/questions/007-bytes-materialization-lossy-utf8.md Normal file
View File

@@ -0,0 +1,45 @@
# OQ-007: `Bytes` materialization — lossy UTF-8 conversion
- **Origin**: [../../research/alktype-builder-poc/findings.md](../../research/alktype-builder-poc/findings.md)
§"Open Questions surfaced" (raised during the v0.1.0 POC)
- **Status**: open
- **Door type**: One-way (the choice affects what the validator sees;
changing it after consumers depend on the shape is a semantic break)
- **Priority**: medium
- **Impacts**: Blocks the SFTP use case for `validate_bytes`. SFTP
packets have binary `handle` and `data` fields (`AlkType:Bytes`)
that are not valid UTF-8. The current materializer uses
`String::from_utf8_lossy` to convert raw bytes to a JSON string
(because `jsonschema`'s `BytesValidator` expects a string — JSON
has no native byte type; see schema-layer.md §TBytes). Lossy
conversion corrupts non-UTF-8 bytes (replacing invalid sequences
with `U+FFFD`), so the validator sees corrupted content. Does NOT
block the chunk header (no `Bytes` fields) or call input schema
(no AlkType kinds).
- **Investigation target**: Decide how `AlkType:Bytes` should be
materialized for validation. Three candidates:
1. **Lossy UTF-8 (current)**: `String::from_utf8_lossy`. Simple;
corrupts non-UTF-8. Acceptable only for UTF-8-only `Bytes` fields.
2. **Base64 encode**: encode raw bytes as base64, validate against
a `format: "base64"` constraint. Round-trips cleanly but the
validator sees base64, not raw bytes — `maxLength` semantics
change (base64 length ≠ byte length).
3. **Array of u8**: materialize as `Value::Array` of `Value::Number`
(one entry per byte). The validator sees the actual byte values;
`maxLength` becomes "max array length." Semantically clean but
produces large `Value` trees for big `Bytes` fields.
The right answer depends on what the `BytesValidator` (in
`src/validation.rs`) should check. Today it checks string length
against `maxLength` — which assumes the string form. If we keep
the string form, base64 is the only round-trippable option. If we
want raw bytes, the validator needs to accept `Value::Array` of
numbers.
- **Resolution**: Not yet decided. Work through the SFTP `handle`/
`data` fields in the next POC round to surface what
`validate_bytes` callers actually need.
- **Cross-references**: [ADR-010](../decisions/010-generalized-validation-validate-bytes.md),
`src/materialize.rs` (`AlkTypeKind::Bytes` branch),
`src/validation.rs` (`BytesValidator`),
[schema-layer.md](../schema-layer.md) §"TBytes"

50
docs/research/alktype-builder-poc/findings.md
View File

@@ -109,7 +109,8 @@ Either:
- Document the constraint explicitly ("the engine requires a top-level
`AlkType:Struct`; a `Union` is a field type within a struct").
This is a documentation fix, not an implementation change.
This is a documentation fix, not an implementation change. Tracked as
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
### 2. Builder field order is preserved as required
@@ -200,13 +201,16 @@ validator for JSON-only schemas".
fields including the discriminator. The byte-offset discriminator
pattern returns a `__discriminator`-tagged object. This asymmetry
needs resolution before shipping (likely both should return the same
shape — see Open Questions).
shape — tracked as
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md)).
3. **`Bytes` materialization** uses `String::from_utf8_lossy` to convert
raw bytes to a JSON string (the validator expects a string for
`AlkType:Bytes` — see schema-layer.md §TBytes). This is lossy for
non-UTF-8 bytes; strict byte-preserving validation would need a
different validator form. For the chunk header (no `Bytes` fields),
this doesn't matter, but it's a follow-up for the SFTP use case.
this doesn't matter, but it's a follow-up for the SFTP use case
(tracked as
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md)).
4. **`encoding` setter** in the builder only handles the
`OffsetIndirect` case (rewriting the boolean-true form to the object
form with the `encoding` annotation). The `LengthPrefixed` case is
@@ -217,19 +221,33 @@ validator for JSON-only schemas".
## Open Questions surfaced
- **OQ-005** (new): Should `Union` materialization return a consistent
shape for byte-offset and field-name discriminators? The POC's byte-
offset path returns `{ "__discriminator": <u32>, ...variant-fields }`;
the field-name path returns the struct fields directly (the
discriminator is just another field). For `validate_bytes` callers,
the shape matters for how the validator dispatches. Resolve before
shipping the next POC round (SFTP Packet `validate_bytes`).
- **OQ-006** (new): Should the builder spec's Example 3 (SFTP Packet)
wrap the `Union` in a `Struct`? Finding #1 above. Documentation fix,
not implementation.
- **OQ-007** (new): `Bytes` materialization — lossy UTF-8 conversion
vs. a stricter byte-preserving validation form. Affects the SFTP use
case (binary `handle` and `data` fields).
Three new open questions raised during the POC, now tracked in the
central OQ tracker ([open-questions.md](../../../architecture/open-questions.md)):
- **OQ-004** (open, low): `Discriminator::Field` name type — `&str` or
`String`? Raised in [builder.md](../../../architecture/builder.md)
§"Open Questions" during ADR-009 spec drafting. Doesn't block the
chunk header POC; must be resolved before the SFTP Packet POC's
field-name discriminator path. Full file:
[OQ-004](../../../architecture/questions/004-discriminator-field-name-type.md).
- **OQ-005** (open, medium): `Union` materialization shape —
consistency between byte-offset and field-name discriminators.
The POC's `materialize_union_packed` returns inconsistent shapes
for the two kinds. Blocks the SFTP Packet `validate_bytes` POC (the
natural next round). Full file:
[OQ-005](../../../architecture/questions/005-union-materialization-shape.md).
- **OQ-006** (open, low): Builder spec Example 3 — wrap the `Union`
in a `Struct`. The spec example shows a top-level `Schema::union_(...)`
which won't compile (`AlkTypeEngine::compile` requires
`AlkType:Struct` at the top level — Finding #1 above). Documentation
fix. Full file:
[OQ-006](../../../architecture/questions/006-builder-spec-example-3-wrap-union.md).
- **OQ-007** (open, medium, one-way door): `Bytes` materialization —
lossy UTF-8 conversion. The current materializer uses
`String::from_utf8_lossy`, which corrupts non-UTF-8 bytes. Blocks
the SFTP use case for `validate_bytes` (binary `handle`/`data`
fields). Full file:
[OQ-007](../../../architecture/questions/007-bytes-materialization-lossy-utf8.md).
## Recommendation