Sync architecture docs and ADRs to BAST pivot (steps 9-10)
Step 9 (convert tests to BAST format) was a no-op: steps 4-8 converted
the tests as they went. The only remaining reference in
src/tests was the intentional rejection test at
src/schema.rs:462 (asserting the old keyword form is rejected). Full
suite passes: 389 tests (312 lib + 77 integration).
Step 10 (sync architecture docs and ADRs):
Descriptive docs rewritten/updated for BAST:
- schema-layer.md: rewritten for the BAST parser (BastDoc/BastDef/
BastType typed tree, AlkTypeKind enum with to_bast_str/from_bast_str,
what was removed). Points at bast-format.md for the normative format.
- validation.md: rewritten for the two-validator model
(bast_validation for validate_bytes, standard jsonschema for
validate_json). Documents the repurposed build_validator, the
AlkTypeError::Validation uniform payload (D-BAST-009), and what is
removed.
- builder.md: updated all output examples to BAST JSON
(struct_() -> { kind: struct, fields: [...] }; object() -> standard
JSON Schema). Documents build_doc, count(), and the field-name union
fields requirement (D-BAST-005).
- overview.md: updated for BAST (what/why, schema-is-the-format table,
dependencies, architecture pointers, design decisions table).
- README.md (architecture index): updated document table, ADR table
(new ADR-BAST + ADR-VAL-SPLIT, superseded ADR-001), OQ table
(OQ-007/OQ-008 resolutions updated for BAST-native validator), and
key design principles (#1, #2, #7, #10 reworded for BAST).
- data-access.md: updated tunion function signatures to BastUnion and
the variant resolution to return BastType (resolve_typeref for refs).
- layout-engine.md: updated construct signatures
(LayoutBuilder::new(bast_doc, root_name), OffsetMap::compute(&doc),
SequentialReader::new(bast_doc, root_name)), the recursive-walk
description (BAST typed tree), and composite-kind headings
(TStruct/TUnion/TArray -> struct/union/array). Added D-BAST-004
note on array count requirement.
New ADRs:
- ADR-BAST (bast-bast-format.md): the BAST format, meta-schema,
//kind vocabulary, design principles, what is removed, the
enum index bounds bug fix. Supersedes ADR-001's format-specific
content; records D-BAST-001..009.
- ADR-VAL-SPLIT (val-split-two-validator-model.md): the two-validator
model (BAST-native for validate_bytes, standard jsonschema for
validate_json), the repurposed build_validator, the uniform
AlkTypeError::Validation payload. Refines ADR-004's validation
strategy and ADR-010's validation step; records D-BAST-006/007/009.
Amended ADRs (supersession/amendment notes added; original decision
text preserved as historical record):
- ADR-001: format-specific content superseded by ADR-BAST;
purpose/scope and schema-is-the-format principle retained.
- ADR-002: unchanged under the pivot; one-line note that the input
format changed but the modes didn't.
- ADR-003: annotation semantics retained; annotation location moved
to BAST type-level properties (amended by ADR-BAST).
- ADR-004: AlkTypeError enum retained (D-BAST-009); validation
strategy section refined by ADR-VAL-SPLIT.
- ADR-009: builder API surface retained; build() output format
amended to BAST / standard JSON Schema by ADR-BAST (D-BAST-008).
- ADR-010: validate_bytes two-step concept retained; validation step
amended to the BAST-native validator by ADR-VAL-SPLIT.
Other:
- Cargo.toml description: JSON Schema with AlkType:* custom keywords
-> BAST document.
- bast-pivot.md research record: status draft -> implemented, with a
pointer to the ADRs that superseded its decisions.
- bast-implementation.md plan: status draft -> complete, with a note
that step 9 was a no-op and step 10 is this commit.
- open-questions.md: OQ-006/OQ-007/OQ-008 resolutions updated for the
BAST-native validator.
- questions/008-unionvalidator-variant-dispatch.md: added a
post-BAST-pivot note pointing to the current bast_validation
implementation; v0.1.0 resolution text preserved as historical
record.
Verification:
- cargo test --release: 389 pass (312 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean
- cross-reference check: every relative link in the new/updated docs
resolves (verified by script).
This commit is contained in:
glm-5.2 committed
1 parent
54fd112fde
commit
62270b03ca
20 files changed
+1642
-1075
No files matched your search
+68
-60
@@ -1,12 +1,12 @@
|
||||
---
|
||||
status: draft
|
||||
last_updated: 2026-08-11
|
||||
status: accepted
|
||||
last_updated: 2026-08-15
|
||||
---
|
||||
|
||||
# alktype
|
||||
|
||||
The binary struct engine: a small Rust crate that takes a JSON Schema
|
||||
with `AlkType:*` custom keywords and produces an offset map, read/write
|
||||
The binary struct engine: a small Rust crate that takes a BAST (Binary
|
||||
Abstract Syntax Tree) document and produces an offset map, read/write
|
||||
functions, and validation — all driven by the schema. The schema is the
|
||||
format definition; the engine is generic.
|
||||
|
||||
@@ -14,68 +14,73 @@ format definition; the engine is generic.
|
||||
|
||||
| Document | Status | Description |
|
||||
|----------|--------|-------------|
|
||||
| [overview.md](overview.md) | draft | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
|
||||
| [schema-layer.md](schema-layer.md) | draft | The 19 `AlkType:*` kinds, jsonschema custom keyword integration, TypeBox interop, schema annotations. *(Current v0.1.0 format; will be rewritten for BAST — see [bast-format.md](bast-format.md).)* |
|
||||
| [bast-format.md](bast-format.md) | draft | **Target schema format.** BAST (Binary Abstract Syntax Tree): meta-schema, TypeRef, examples, the two-validator model. Supersedes the format-spec content of schema-layer.md when the [BAST pivot](../plans/bast-implementation.md) lands. |
|
||||
| [overview.md](overview.md) | accepted | Crate purpose, "schema is the format" principle, dependencies, consumers, scope boundaries |
|
||||
| [`bast-format.md`](bast-format.md) | accepted | **Normative BAST format specification.** Meta-schema, TypeRef, TypeDef shapes (Struct/Union/Enum/FieldDef), examples, validation model. The format the engine consumes. |
|
||||
| [schema-layer.md](schema-layer.md) | accepted | The BAST parser (`src/bast.rs`) — the typed tree (`BastDoc`/`BastDef`/`BastType`/…) every engine module walks, the 19 BAST kinds, the `AlkTypeKind` enum, and the foundational annotation types. |
|
||||
| [layout-engine.md](layout-engine.md) | draft | Offset computation, the two layout modes (packed sequential vs aligned static), alignment, endianness, variable-length handling |
|
||||
| [data-access.md](data-access.md) | draft | Read/write functions, TUnion dispatch, field paths, zero-copy access, length-prefix reading |
|
||||
| [validation.md](validation.md) | draft | Custom keyword validators for all 19 `AlkType:*` kinds, `AlkTypeError`, load-time vs access-time validation, `AlkTypeEngine`; `validate_bytes` for binary buffers (ADR-010). *(Current v0.1.0 validation; will be rewritten for the validator split — see [bast-format.md §Validation Model](bast-format.md#validation-model).)* |
|
||||
| [builder.md](builder.md) | draft | Fluent Rust API for constructing alktype JSON Schemas at runtime, producing `serde_json::Value`; covers AlkType kinds + standard JSON Schema (ADR-009) |
|
||||
| [validation.md](validation.md) | accepted | The two-validator model (BAST-native for `validate_bytes`, standard `jsonschema` for `validate_json`), `AlkTypeError`, load-time vs access-time validation, `AlkTypeEngine` as the compiled form of a BAST document (ADR-010, ADR-VAL-SPLIT). |
|
||||
| [builder.md](builder.md) | accepted | Fluent Rust API for constructing BAST documents (`struct_()`) and standard JSON Schemas (`object()`) at runtime, producing `serde_json::Value` (ADR-009, D-BAST-008). |
|
||||
|
||||
### In-progress work
|
||||
|
||||
| Document | Status | Description |
|
||||
|----------|--------|-------------|
|
||||
| [BAST pivot — research record](../research/bast-pivot.md) | draft | Motivation, POC scope and result, decisions D-BAST-001..009, risks for the BAST format pivot |
|
||||
| [BAST pivot — implementation plan](../plans/bast-implementation.md) | draft | Ordered implementation steps, the public-API semver contract, and the ADR-sync checklist for the BAST pivot |
|
||||
| [BAST pivot — research record](../research/bast-pivot.md) | accepted | Motivation, POC scope and result, decisions D-BAST-001..009, risks for the BAST format pivot. Implemented in steps 1–10. |
|
||||
| [BAST pivot — implementation plan](../plans/bast-implementation.md) | accepted | Ordered implementation steps, the public-API semver contract, and the ADR-sync checklist for the BAST pivot. Steps 1–10 complete. |
|
||||
|
||||
## Applicable ADRs
|
||||
|
||||
| ADR | Title | Relevance |
|
||||
|-----|-------|-----------|
|
||||
| [001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries |
|
||||
| [002](decisions/002-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` |
|
||||
| [003](decisions/003-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Concrete JSON shapes for all schema-level annotations |
|
||||
| [004](decisions/004-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `AlkTypeError` enum; load-time build, access-time check; field-path-carrying errors |
|
||||
| [001](decisions/001-alktype-purpose-scope-jsonschema-engine.md) | Purpose, Scope, and the jsonschema Engine | What the crate is/isn't; why jsonschema not a custom engine; "schema is the format" principle; scope boundaries. *Format-specific content superseded by ADR-BAST; purpose/scope retained.* |
|
||||
| [BAST](decisions/bast-bast-format.md) | BAST (Binary Abstract Syntax Tree) as the Schema Format | The BAST format, meta-schema, `$defs`/`$ref`/`kind` vocabulary. Supersedes ADR-001's format-specific content; records D-BAST-001..009. |
|
||||
| [VAL-SPLIT](decisions/val-split-two-validator-model.md) | Two-Validator Model — BAST-Native for Bytes, Standard jsonschema for JSON | `validate_bytes` uses the BAST-native validator; `validate_json` uses a standard `jsonschema::Validator` from a consumer-provided JSON Schema. Records D-BAST-006/007/009. |
|
||||
| [002](decisions/002-two-layout-modes-packed-vs-aligned.md) | Two Layout Modes — Packed Sequential vs Aligned Static | The most important architectural finding; when to use each mode; `LayoutBuilder`/`SequentialReader` vs `OffsetMap` (format-agnostic — input format changed, modes didn't) |
|
||||
| [003](decisions/003-schema-annotations.md) | Schema Annotations — Endianness, Alignment, Encoding, TUnion Discriminators | Annotation *semantics* (carry forward unchanged); annotation *location* moved to BAST type-level properties under the pivot |
|
||||
| [004](decisions/004-error-handling-validation-strategy.md) | Error Handling and Validation Strategy | `AlkTypeError` enum (shape unchanged, D-BAST-009); load-time build, access-time check; field-path-carrying errors. *Validation-strategy section refined by ADR-VAL-SPLIT.* |
|
||||
| [005](decisions/005-int64-uint64-first-class-kinds.md) | Int64/Uint64 as First-Class Kinds | 64-bit integers (SFTP offsets, metatensor data_offsets); JSON precision caveat |
|
||||
| [006](decisions/006-reject-non-final-inline-length-prefixed-in-aligned-mode.md) | Reject Non-Final Inline Length-Prefixed Variable Fields in Aligned Mode | Prevents silent data corruption (inline variable data clobbering subsequent fields) |
|
||||
| [007](decisions/007-packed-mode-read-factory.md) | Packed-Mode Read API — Engine as SequentialReader Factory | `engine.sequential_reader()` returns an owned reader, not a reference |
|
||||
| [008](decisions/008-reject-tunion-in-aligned-mode.md) | Reject TUnion in Aligned Mode for v1 | Unions are the protocol pattern; aligned-mode union semantics were broken |
|
||||
| [009](decisions/009-builder-api.md) | Builder API for Schema Construction | Fluent Rust API producing `serde_json::Value`; covers AlkType kinds + standard JSON Schema; resolves OQ-003 |
|
||||
| [010](decisions/010-generalized-validation-validate-bytes.md) | Generalized Validation — `validate_bytes` on `AlkTypeEngine` | Single-call binary-buffer validation; materialize `Value` from bytes, then validate; two methods on one struct, not a trait |
|
||||
| [009](decisions/009-builder-api.md) | Builder API for Schema Construction | Fluent Rust API producing `serde_json::Value`; covers BAST kinds + standard JSON Schema; resolves OQ-003. *Output format amended to BAST / standard JSON Schema by ADR-BAST.* |
|
||||
| [010](decisions/010-generalized-validation-validate-bytes.md) | Generalized Validation — `validate_bytes` on `AlkTypeEngine` | Single-call binary-buffer validation; materialize `Value` from bytes, then validate. *Validation step amended to the BAST-native validator by ADR-VAL-SPLIT.* |
|
||||
|
||||
## Relevant Open Questions
|
||||
|
||||
| OQ | Title | Status | Relevance |
|
||||
|----|-------|--------|-----------|
|
||||
| OQ-001 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it |
|
||||
| OQ-001 | Arrays of variable-length-element structs | deferred(scope) | Requires lazy walking logic; blocked on a concrete consumer that needs it. BAST arrays require `count` in v1 (D-BAST-004), aligning with this deferral. |
|
||||
| OQ-002 | `no_std` + `alloc` support | deferred(scope) | Target `std` for v1; blocked on an embedded use case |
|
||||
| OQ-003 | Builder API for schema construction | resolved (ADR-009) | Resolved in v0.1.0; alkcall is the concrete consumer; see [builder.md](builder.md) |
|
||||
| OQ-004 | `Discriminator::Field` name — `&str` or `String` | open | Builder API ownership question; resolve before the SFTP Packet POC's field-name discriminator path |
|
||||
| OQ-005 | `Union` materialization shape — byte-offset vs field-name consistency | open | Blocks the SFTP Packet `validate_bytes` POC (next round) |
|
||||
| OQ-006 | Builder spec Example 3 — wrap `Union` in a `Struct` | open | Documentation fix in builder.md; the engine requires `AlkType:Struct` at the top level |
|
||||
| OQ-007 | `Bytes` materialization — lossy UTF-8 conversion | open | Blocks the SFTP use case for `validate_bytes` (binary `handle`/`data` fields) |
|
||||
| OQ-003 | Builder API for schema construction | resolved (ADR-009) | Shipped in v0.1.0; alkcall is the concrete consumer; see [builder.md](builder.md) |
|
||||
| OQ-004 | `Discriminator::Field` name — `&str` or `String` | resolved | `String`, for ownership simplicity |
|
||||
| OQ-005 | `Union` materialization shape — byte-offset vs field-name consistency | resolved | Both kinds return `{ "__discriminator": <value>, ...variant-fields }` |
|
||||
| OQ-006 | Builder spec Example 3 — wrap `Union` in a `Struct` | resolved | [builder.md](builder.md) Example 3 wraps the union in a `Schema::struct_().field("payload", ...)` |
|
||||
| OQ-007 | `Bytes` materialization — lossy UTF-8 conversion | resolved | Array of u8: materializer produces `Value::Array` of `Value::Number`; BAST-native validator accepts both `Value::String` and `Value::Array` |
|
||||
| OQ-008 | `UnionValidator` variant dispatch | resolved | BAST-native validator recurses into the selected variant's BAST definition on `__discriminator` lookup — no custom keywords, no `inline_union_variant_refs` |
|
||||
|
||||
## Key Design Principles
|
||||
|
||||
1. **The schema is the format.** A JSON Schema with `AlkType:*` custom
|
||||
keywords is both the validation spec and the layout spec. No separate
|
||||
format definition, no separate parser, no separate validator. One
|
||||
schema, three uses: validate, compute offsets, access data. See
|
||||
[overview.md](overview.md) and [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md).
|
||||
1. **The schema is the format.** A BAST document is both the layout
|
||||
spec and the validation spec for bytes. No separate format
|
||||
definition, no separate parser, no separate validator. One schema,
|
||||
three uses: validate, compute offsets, access data. See
|
||||
[overview.md](overview.md), [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md),
|
||||
and [ADR-BAST](decisions/bast-bast-format.md).
|
||||
|
||||
2. **jsonschema is the validation engine, not a custom engine.** The
|
||||
`jsonschema` crate (v0.46.5, Draft 2020-12) handles validation with
|
||||
custom keyword support. The novel code is the offset computation, not
|
||||
the validation. This eliminates ~14,000 lines of hand-rolled schema
|
||||
engines (typebox-rs, the @alkdev/alktype prototype). See [schema-layer.md](schema-layer.md)
|
||||
and [ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md).
|
||||
2. **BAST is a JSON Schema dialect, not a custom format.** A BAST
|
||||
document is valid JSON conforming to the BAST meta-schema (a
|
||||
standard Draft 2020-12 JSON Schema). Any JSON Schema validator can
|
||||
check whether a BAST document is well-formed; editors with JSON
|
||||
Schema support provide autocomplete for free. See
|
||||
[`bast-format.md`](bast-format.md) and
|
||||
[ADR-BAST](decisions/bast-bast-format.md).
|
||||
|
||||
3. **Two layout modes for two use cases.** Packed sequential
|
||||
(`LayoutBuilder`/`SequentialReader`) for protocol wire formats (SFTP,
|
||||
channels, TTY). Aligned static (`OffsetMap`) for mmap-friendly formats
|
||||
(metatensor). The consumer selects the mode; the schema is the same.
|
||||
See [layout-engine.md](layout-engine.md) and
|
||||
(metatensor). The consumer selects the mode; the BAST document is the
|
||||
same. See [layout-engine.md](layout-engine.md) and
|
||||
[ADR-002](decisions/002-two-layout-modes-packed-vs-aligned.md).
|
||||
|
||||
4. **Variable-length types default to inline length-prefixing.**
|
||||
@@ -92,38 +97,41 @@ format definition; the engine is generic.
|
||||
[ADR-003](decisions/003-schema-annotations.md).
|
||||
|
||||
6. **Endianness is per-schema, default little-endian.** The engine reads
|
||||
the `"endian"` annotation and byte-swaps accordingly. SFTP consumers
|
||||
specify `"endian": "big"`. See [layout-engine.md](layout-engine.md)
|
||||
and [ADR-003](decisions/003-schema-annotations.md).
|
||||
the struct-level `"endian"` annotation and byte-swaps accordingly.
|
||||
SFTP consumers specify `"endian": "big"`. See
|
||||
[layout-engine.md](layout-engine.md) and
|
||||
[ADR-003](decisions/003-schema-annotations.md).
|
||||
|
||||
7. **Validation is opt-in, built once at load time.** The jsonschema
|
||||
validator is compiled once at schema load time. Access-time validation
|
||||
is a fast `is_valid()` check. High-throughput paths can skip
|
||||
validation; security-sensitive paths can validate every frame. See
|
||||
7. **Two validators for two input types.** `validate_bytes(&[u8])` uses
|
||||
the BAST-native validator (a recursive walker over the BAST type
|
||||
tree — no `jsonschema` involvement). `validate_json(&Value)` uses a
|
||||
standard `jsonschema::Validator` from a consumer-provided JSON Schema
|
||||
(BAST is not involved — BAST describes bytes, not JSON shape). One
|
||||
`AlkTypeError::Validation` variant covers both (D-BAST-009). See
|
||||
[validation.md](validation.md) and
|
||||
[ADR-004](decisions/004-error-handling-validation-strategy.md).
|
||||
[ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md).
|
||||
|
||||
8. **Not a serialization framework.** The alktype engine is not a
|
||||
general-purpose serde replacement. It operates on raw byte buffers at
|
||||
computed offsets — no intermediate `Value` tree, no reflection, no
|
||||
dynamic dispatch per field. For JSON data, use serde. For binary data
|
||||
with a known schema, use alktype. See [overview.md](overview.md) and
|
||||
computed offsets — no reflection, no dynamic dispatch per field. For
|
||||
JSON data, use serde. For binary data with a known BAST document, use
|
||||
alktype. See [overview.md](overview.md) and
|
||||
[ADR-001](decisions/001-alktype-purpose-scope-jsonschema-engine.md).
|
||||
|
||||
9. **Schemas can be built at runtime from Rust (v0.1.0).** A fluent
|
||||
builder API produces `serde_json::Value` for both AlkType-kind
|
||||
schemas and standard JSON Schema, covering alkcall's two roles
|
||||
9. **Schemas can be built at runtime from Rust.** A fluent builder API
|
||||
produces `serde_json::Value` for both BAST documents (`struct_()`) and
|
||||
standard JSON Schemas (`object()`), covering alkcall's two roles
|
||||
(binary layout + JSON payloads) from one module. The builder is
|
||||
additive — consumers with static schemas continue to load JSON.
|
||||
See [builder.md](builder.md) and [ADR-009](decisions/009-builder-api.md).
|
||||
additive — consumers with static BAST documents continue to load
|
||||
JSON. See [builder.md](builder.md) and
|
||||
[ADR-009](decisions/009-builder-api.md).
|
||||
|
||||
10. **Two validation entry points, one engine (v0.1.0).**
|
||||
`validate_json(&Value)` for already-parsed JSON (call's payloads);
|
||||
`validate_bytes(&[u8])` for binary buffers (channels' chunk header).
|
||||
Same underlying `jsonschema` validator; the bytes path materializes
|
||||
a `Value` tree via the layout engine, then validates. See
|
||||
[validation.md](validation.md) and
|
||||
[ADR-010](decisions/010-generalized-validation-validate-bytes.md).
|
||||
10. **Two validation entry points, one engine.** `validate_json(&Value)`
|
||||
for already-parsed JSON (call's payloads); `validate_bytes(&[u8])`
|
||||
for binary buffers (channels' chunk header). Different validators,
|
||||
one `AlkTypeError::Validation` variant. See [validation.md](validation.md),
|
||||
[ADR-010](decisions/010-generalized-validation-validate-bytes.md),
|
||||
and [ADR-VAL-SPLIT](decisions/val-split-two-validator-model.md).
|
||||
|
||||
## References
|
||||
|
||||
@@ -143,4 +151,4 @@ format definition; the engine is generic.
|
||||
> **Note**: The research findings, POC code, and prior-attempt paths above
|
||||
> refer to the parent `@alkdev/alknet` workspace where this crate originated.
|
||||
> They are preserved here as historical context for the architectural
|
||||
> decisions; the artifacts themselves are not part of this standalone repo.
|
||||
> decisions; the artifacts themselves are not part of this standalone repo.
|
||||
Reference in new issue
Block a user