Traces the ~400x read-path gap (alktty wire_vs_bast bench) to
SequentialReader::read_field_at re-parsing BastDoc::new on every field
read, with the self-referential lifetime constraint as the root cause.
Findings:
- H1: per-field BastDoc::new re-parse in sequential_reader.rs:262
- M1: same re-parse in four one-shot paths (LayoutBuilder::build,
validate_bytes, engine read_field/write_field)
- L1: dead _field_schema param + Value clones in SequentialReader
- L2: ADR-007 Cost section + engine doc comment understate the re-parse
- N1: carry-forward of review #003 N2 (no new action)
Lays out fix options: Option A (owned typed tree, recommended, closes
H1+M1, breaking), Option B (read-plan precompute, fallback, H1 only,
non-breaking), Option C (borrow-from-engine, rejected, contradicts
ADR-007).
Verification: docs-only review; alktype source unchanged.
cab4932
Bump version to 0.2.0, exclude AGENTS.md from the published crate, and
add a CHANGELOG.md covering the breaking BAST pivot (schema format,
compile signature, validation split) plus bug fixes vs v0.1.0.
Verification:
- cargo test --release: all tests pass
- cargo clippy --all-targets -- -D warnings: clean
- cargo publish --dry-run --allow-dirty: packages as v0.2.0, no collision
- AGENTS.md no longer in cargo package --list; CHANGELOG.md included
- README: 19 -> 18 kinds, drop the timestamp row from the kinds table,
drop 'timestamp shape' from the validate_bytes constraint list, remove
the residual 'upcoming alkcall crate' sentence from the crate
independence section (alkcall exists now), fix two '19 kinds' refs in
the documentation pointer list and schema-layer link.
- src/data_access.rs: module doc comment still said 'all 19 AlkType
kinds' -> 18.
- bast-format.md (normative): 19 -> 18 AlkTypeKind enum variants.
- layout-engine.md: cross-reference to 'the 19 AlkType kinds' -> 18.
- ADR-BAST (bast-bast-format.md): the Decision section claimed the post-
pivot enum has '19 unchanged' variants; now 18, with the wording
adjusted so it no longer says 'unchanged' across the pivot.
- ADR-VAL-SPLIT: drop 'timestamp shape' from the value-domain constraint
list and the validator-arm table row (validate_timestamp no longer
exists).
Left as historically accurate (describe the v0.1.0 pre-pivot state):
ADR-003/004/006 Context mentions of AlkType:Timestamp, ADR-005 'engine
now has 19', and the '19 jsonschema::Keyword factories' references in
the What-is-removed sections of ADR-BAST and ADR-VAL-SPLIT.
Verification: cargo test --release (407 pass), cargo clippy --all-targets
-- -D warnings (clean), cargo doc --no-deps (clean), cargo build --target
wasm32-unknown-unknown --release (clean).
The Timestamp kind was a residual from an early research reference. It
was byte-identical to String everywhere (length-prefixed UTF-8) and its
only distinguishing behavior was a hand-rolled non-strict RFC 3339 check
that the docs admitted was incomplete (Feb 31 passes, seconds range
unchecked, no leap seconds). JSON-level timestamp validation is
jsonschema's job (format: date-time on the validate_json path), not
alktype's.
Removes the AlkTypeKind::Timestamp variant, its to_bast_str/from_bast_str
mapping, the builder's Schema::timestamp() constructor, the
validate_timestamp/is_rfc3339_timestamp validator arms, and the
materializer/reader/engine timestamp arms. Updates the meta-schema
primitive enum (14 -> 13), the spec docs (bast-format.md, schema-layer.md,
builder.md, validation.md, overview.md, data-access.md, README.md), and
the kind-count references (19 -> 18).
Verification: cargo test --release (407 pass), cargo clippy --all-targets
-- -D warnings (clean), cargo doc --no-deps (clean), cargo build --target
wasm32-unknown-unknown --release (clean).
- N1: number_from_f64 now returns AlkTypeError::Access for non-finite
floats instead of silently materializing Value::Null. A NaN/Inf in the
buffer surfaces as a clear access error rather than a misleading
'expected a number' validation error.
- N3: drop the dead Value::String arm from check_bytes; the materializer
only ever emits bytes as an array of u8. Update the validation-model
docs to match.
- N4: fix stale ADR references in doc comments (ADR-096 -> ADR-002,
ADR-097 -> ADR-003, ADR-098 -> ADR-004, ADR-101 -> ADR-007).
N2 (BastType::alk_kind returning Struct for ) left as documented;
every current caller resolves the ref first, so forcing Option through
the call sites is churn without benefit.
Verification: cargo test --release (411 pass), cargo clippy --all-targets
-- -D warnings (clean), cargo doc --no-deps (clean).
Resolves review #003 finding M3.
- Add bast_meta::validate_bast_doc and call it from AlkTypeEngine::compile
before parsing. Malformed annotations (unknown endian/encoding strings,
non-integer align/maxLength, missing required properties) now surface as
AlkTypeError::Schema instead of being silently tolerated by the parser.
- Align the meta-schema TypeRef with the parser: allow inline struct/union/
enum as TypeRefs (the parser and builder already accepted them; the
meta-schema and spec did not). Update bast-format.md TypeRef table and
the BastType doc comment to the seven-form vocabulary.
Verification: cargo test --release (410 pass), cargo clippy --all-targets
-- -D warnings (clean), cargo doc --no-deps (clean), cargo build --target
wasm32-unknown-unknown --release (clean).
Resolves review #003 findings M1, M2, L1, and L2.
- offset-indirect (L1 + M2 arm): rework read_*_indirect to same-buffer
absolute offsets (safetensors-style, per the metatensor model) and add
write_*_indirect. Wire the encoding dispatch into engine.read_field/
write_field and the aligned materializer. Previously the encoding was
laid out but never read back.
- field-level endian override (M1): thread field.effective_endian through
sequential_reader, engine.read_field/write_field, and
materialize_struct_aligned. Previously a per-field endian override was
silently ignored, misreading multi-byte values.
- aligned-mode materialization (M2): materialize fixed-size arrays via
their vals[i] offset-map entries and maxLength reservations as
zero-padded fixed-size slices (trailing NULs trimmed). Previously both
read garbage or errored.
- dead endian param (L2): drop the ignored endian argument from
materialize_packed/materialize_aligned; endianness is read from the
root struct.
Verification: cargo test --release (404 pass), cargo clippy --all-targets
-- -D warnings (clean), cargo build --target wasm32-unknown-unknown
--release (clean), cargo llvm-cov --release (89.60% lines).
The README still described the v0.1.0 custom-keyword JSON Schema format
(`AlkType:*` kinds, `compile(&mut schema, mode)`, single jsonschema
validator). Rewrite it for the BAST-era shipped API:
- What-it-is table: two validation specs (bytes via BAST-native, JSON
via consumer-provided JSON Schema) instead of one.
- Usage example: `Definitions::new().build_doc(ChunkHeader, ...)` +
`AlkTypeEngine::compile(&doc, ChunkHeader, LayoutMode::Packed,
None)`; added a json!-literal variant showing the BAST document
shape.
- The 19 kinds table: lowercase BAST `kind` strings instead of
`AlkType:*` keywords; note the enum index bounds fix.
- Two layout modes: updated `compile` signature.
- Variable-length handling: BAST field-level `encoding` annotation.
- Union discriminators: `kind: union`, lazy variant $ref
resolution, D-BAST-005 fields requirement.
- Endianness: struct-level (was 'top-level schema').
- Validation: two validators (ADR-VAL-SPLIT), BAST-native for bytes,
standard jsonschema for JSON; uniform AlkTypeError::Validation
payload (D-BAST-009); enum bounds fix noted.
- New 'BAST document shape' section: $defs required, root_name
parameter, $ref restricted to #/$defs/<name>, BAST_META_SCHEMA
re-export.
- Untrusted input: BAST document (was 'schema'); BAST parser
preserves the no-panic invariant.
- Documentation index: updated for the new/rewritten docs and the
ADR-BAST / ADR-VAL-SPLIT additions.
Verified both code examples compile and pass against the shipped API
(via a throwaway integration test, since removed).
Step 9 (convert tests to BAST format) was a no-op: steps 4-8 converted
the tests as they went. The only remaining reference in
src/tests was the intentional rejection test at
src/schema.rs:462 (asserting the old keyword form is rejected). Full
suite passes: 389 tests (312 lib + 77 integration).
Step 10 (sync architecture docs and ADRs):
Descriptive docs rewritten/updated for BAST:
- schema-layer.md: rewritten for the BAST parser (BastDoc/BastDef/
BastType typed tree, AlkTypeKind enum with to_bast_str/from_bast_str,
what was removed). Points at bast-format.md for the normative format.
- validation.md: rewritten for the two-validator model
(bast_validation for validate_bytes, standard jsonschema for
validate_json). Documents the repurposed build_validator, the
AlkTypeError::Validation uniform payload (D-BAST-009), and what is
removed.
- builder.md: updated all output examples to BAST JSON
(struct_() -> { kind: struct, fields: [...] }; object() -> standard
JSON Schema). Documents build_doc, count(), and the field-name union
fields requirement (D-BAST-005).
- overview.md: updated for BAST (what/why, schema-is-the-format table,
dependencies, architecture pointers, design decisions table).
- README.md (architecture index): updated document table, ADR table
(new ADR-BAST + ADR-VAL-SPLIT, superseded ADR-001), OQ table
(OQ-007/OQ-008 resolutions updated for BAST-native validator), and
key design principles (#1, #2, #7, #10 reworded for BAST).
- data-access.md: updated tunion function signatures to BastUnion and
the variant resolution to return BastType (resolve_typeref for refs).
- layout-engine.md: updated construct signatures
(LayoutBuilder::new(bast_doc, root_name), OffsetMap::compute(&doc),
SequentialReader::new(bast_doc, root_name)), the recursive-walk
description (BAST typed tree), and composite-kind headings
(TStruct/TUnion/TArray -> struct/union/array). Added D-BAST-004
note on array count requirement.
New ADRs:
- ADR-BAST (bast-bast-format.md): the BAST format, meta-schema,
//kind vocabulary, design principles, what is removed, the
enum index bounds bug fix. Supersedes ADR-001's format-specific
content; records D-BAST-001..009.
- ADR-VAL-SPLIT (val-split-two-validator-model.md): the two-validator
model (BAST-native for validate_bytes, standard jsonschema for
validate_json), the repurposed build_validator, the uniform
AlkTypeError::Validation payload. Refines ADR-004's validation
strategy and ADR-010's validation step; records D-BAST-006/007/009.
Amended ADRs (supersession/amendment notes added; original decision
text preserved as historical record):
- ADR-001: format-specific content superseded by ADR-BAST;
purpose/scope and schema-is-the-format principle retained.
- ADR-002: unchanged under the pivot; one-line note that the input
format changed but the modes didn't.
- ADR-003: annotation semantics retained; annotation location moved
to BAST type-level properties (amended by ADR-BAST).
- ADR-004: AlkTypeError enum retained (D-BAST-009); validation
strategy section refined by ADR-VAL-SPLIT.
- ADR-009: builder API surface retained; build() output format
amended to BAST / standard JSON Schema by ADR-BAST (D-BAST-008).
- ADR-010: validate_bytes two-step concept retained; validation step
amended to the BAST-native validator by ADR-VAL-SPLIT.
Other:
- Cargo.toml description: JSON Schema with AlkType:* custom keywords
-> BAST document.
- bast-pivot.md research record: status draft -> implemented, with a
pointer to the ADRs that superseded its decisions.
- bast-implementation.md plan: status draft -> complete, with a note
that step 9 was a no-op and step 10 is this commit.
- open-questions.md: OQ-006/OQ-007/OQ-008 resolutions updated for the
BAST-native validator.
- questions/008-unionvalidator-variant-dispatch.md: added a
post-BAST-pivot note pointing to the current bast_validation
implementation; v0.1.0 resolution text preserved as historical
record.
Verification:
- cargo test --release: 389 pass (312 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean
- cross-reference check: every relative link in the new/updated docs
resolves (verified by script).
- Schema internals: flat Map<String, Value> → Repr enum distinguishing
BAST primitives (bare TypeRef strings), BAST composites (struct/
union/enum/array/record objects), $ref, standard JSON Schema
objects, and raw adopted values. Annotations (endian/align/encoding/
maxLength) stored on the Schema and placed correctly by build()
(struct-level) or field() (field-level).
- Primitive constructors (uint32, string, bytes, etc.) now produce
bare BAST kind strings ("uint32") instead of {"AlkType:Uint32":true}.
- struct_().field(...) produces {"kind":"struct","fields":[{name,kind,...annos}]}
with an ordered fields array (BAST design principle #4 — no reliance
on preserve_order for field order).
- union_() emits {"kind":"union","discriminator":{...},"mapping":{...}}
with BAST kind strings in the discriminator ("uint8" not
"AlkType:Uint8"). Field-name discriminator unions emit the fields
array; byte-offset unions omit it.
- enum_of() produces {"kind":"enum","values":[...]}.
- array_of(element).count(n) produces {"kind":"array","element":...,"count":n}.
.count() is an additive method (D-BAST-004 requires count; the
array_of signature is unchanged per the semver contract).
- record_of(values) produces {"kind":"record","values":...}.
- object()/string_()/etc. unchanged — standard JSON Schema output.
- encoding() now stores the value on the Schema; field() extracts it
as a field-level property. No more keyword-object duality.
- max_length() on standard types emits the standard keyword; on BAST
types it is extracted by field() as a field-level constraint.
- Definitions::build_doc(root_name, root) — new additive method
producing a complete BAST document with the root type inside $defs
(where BAST requires it). The old build()/merge_into() remain for
backward compatibility.
- All builder tests updated to expect BAST output shapes. New tests:
field annotations, field-name union with fields, array with count,
ref element, encoding emission, Definitions::build_doc round-trip
through compile(), SFTP-style union compile, array/record compile.
- Module doc comments updated (builder.rs, lib.rs).
Verification:
- cargo test --release: 427 pass (350 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean (no warnings)
- cargo build --target wasm32-unknown-unknown --release: clean
- AlkTypeEngine::compile gains a 4th param json_schema: Option<&Value>.
When Some, a standard jsonschema::Validator is built from the
consumer-provided JSON Schema and stored for the JSON-validation
path. When None, validate_json returns AlkTypeError::Schema and
is_valid_json returns false (D-BAST-007).
- validate_json / is_valid_json signatures unchanged (per semver
contract). Behavior: they now validate against the consumer JSON
Schema, not a custom-keyword validator built from the alktype
schema. The BAST document is not involved in this path.
- build_validator repurposed (deferred decision #2): same signature,
now builds a standard jsonschema::Validator with no custom keywords.
Behavioral break, not a type break. Re-export kept.
- Removed the 19 jsonschema::Keyword implementations and the 4
define_*_validator! macros (dead on the bytes path since step 5,
now dead on the JSON path too). The is_rfc3339_timestamp helper
lives on in bast_validation.rs (already copied there in step 5).
- All compile call sites updated to pass None for json_schema (the
layout/read/write/validate_bytes tests don't need JSON validation).
- New tests: validate_json accepts/rejects against consumer JSON
Schema, returns Schema error when no JSON Schema supplied,
is_valid_json false when no schema, independence from BAST doc,
malformed JSON Schema build error, nested object JSON Schema.
Verification:
- cargo test --release: 409 pass (332 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo build --target wasm32-unknown-unknown --release: clean
- cargo doc --no-deps: clean
Replace the jsonschema custom-keyword validator on the bytes path with
a recursive walker over the BAST typed tree. The materializer already
guarantees structural correctness; the validator enforces only the
value-domain constraints expressed in the BAST document.
- New `src/bast_validation.rs`: `validate_value` dispatches on
`BastType`, resolving `$ref`s lazily via `BastDoc::resolve_typeref`.
Constraint arms: integer ranges (Int8..Uint64), float finiteness,
string/bytes `maxLength` (from `BastField::max_length`), RFC 3339
timestamp shape, enum index bounds, union variant dispatch (recurses
into the variant, recovering OQ-008 per-variant constraints), struct
field presence, array count, record values.
- `engine::validate_bytes` now calls `bast_validation::validate_value`
instead of `self.validator.validate`. The `validator` field is still
built and used by `validate_json`/`is_valid_json` (step 6 reworks
those).
- Enum index bounds check (`idx < values.len()`) fixes the v0.1.0 dead
constraint: the built-in `enum` keyword checked string membership, but
the materializer emits a numeric index that never matched.
- Errors constructed via `jsonschema::ValidationError::custom` so
`AlkTypeError::Validation` keeps its payload type uniform with the
`validate_json` path (D-BAST-009).
- `lib.rs`: add `pub mod bast_validation;` (engine-internal, not
re-exported in the `pub use` block) and update the module doc.
Verification:
- cargo test --release: 446 pass (369 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo build --target wasm32-unknown-unknown --release: clean
- cargo doc --no-deps: clean
Step 4 of the BAST pivot: the layout engines, materializer, tunion
dispatch, and engine now consume the BAST typed tree (BastDoc/
BastStruct/BastField/BastType/...) instead of walking raw JSON with
get_alktype_kind*.
Breaking changes (per the pivot plan's semver contract):
- AlkTypeEngine::compile signature:
compile(schema: &mut Value, mode)
-> compile(bast_doc: &Value, root_name: &str, mode)
Drops &mut (BAST needs no in-place normalize_refs); adds required
root_name (D-BAST-001); input is a BAST document, not a custom-keyword
JSON Schema.
- OffsetMap::compute, LayoutBuilder::new, SequentialReader::new now take
a BAST document (&Value) + root_name (or &BastDoc) instead of a
v0.1.0 schema.
- tunion::read_byte_discriminator / read_field_discriminator /
resolve_variant / discriminator_size now take &BastUnion instead of
&Value.
- materialize::materialize_packed / materialize_aligned now take
&BastDoc instead of &Value.
Key design points:
- The engine stores a clone of the BAST Value + root_name so
sequential_reader() and read_field() can re-parse the typed tree on
demand without lifetime entanglement with the caller's Value.
- resolution is a single hash lookup via BastDoc::resolve_typeref;
no normalize_refs, no inline_union_variant_refs.
- bast.rs gains BastField::synthetic() (pub(crate)) for constructing
synthetic fields wrapping TypeRefs (array elements, record values,
union variants — these aren't fields and carry no field annotations).
- The v0.1.0 schema.rs helpers and validation.rs custom-keyword
validators remain defined (step 8 removes them). build_validator still
runs on the BAST doc — with no AlkType:* keywords present, the custom
factories don't trigger and jsonschema performs structural validation
only. The validate_bytes value-constraint enforcement (maxLength, enum
bounds) is step 5's concern (the BAST-native validator).
Tests:
- All engine, layout, materialize, tunion, and integration tests
converted to BAST format (kind/fields vocabulary, /
composition). Expected validation outcomes for the layout path are
identical; the maxLength/enum-bounds validate_bytes tests are step 5's
regression target.
- builder.rs::builder_chunk_header_compiles_in_packed_mode uses a
hand-written BAST doc (the builder still emits v0.1.0 format; step 7
converts it).
Verification:
- cargo test --release: 425 pass (348 lib + 77 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo build --target wasm32-unknown-unknown --release: clean
- cargo doc --no-deps: clean
- New src/bast.rs: typed surface over a BAST document — BastDoc,
BastDef, BastDefKind, BastStruct, BastField, BastUnion,
BastDiscriminator, BastEnum, BastType, BastRef, BastArray,
BastRecord. Borrows from the source Value (no clone of the tree).
- $ref resolution is a single hash lookup against $defs
(#/$defs/<name> only); union variant refs resolved lazily via
BastDoc::resolve_typeref / resolve_ref — replaces
normalize_refs + inline_union_variant_refs (those stay for now;
step 8 removes them).
- Untrusted-input safe: every walk returns AlkTypeError::Schema on
a malformed document, never panic/unwrap (AGENTS.md §3).
Overflow-safe usize parsing via try_from (AGENTS.md §4).
- D-BAST-005 enforced: fields array only valid with field-name
discriminators; required for them.
- Additive only: new pub mod bast + re-exports in lib.rs. No
existing re-exports removed (those go in step 8). 47 new tests.
Verification:
- cargo test --release: 465 pass (379 lib + 86 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean
- cargo build --target wasm32-unknown-unknown --release: clean
Step 2 of the BAST pivot. Adds the normative BAST meta-schema
(Draft 2020-12 JSON Schema) as a public serde_json::Value, embedded
at compile time and available for validating BAST document
well-formedness. Copied verbatim from docs/architecture/bast-format.md
§The Meta-Schema.
- New src/bast_meta.rs: BAST_META_SCHEMA static, built lazily via
LazyLock (the json! macro allocates, so it can't be a const;
parsed once, reused as &'static Value thereafter).
- src/lib.rs: pub mod bast_meta + re-export BAST_META_SCHEMA.
Additive public surface.
The meta-schema validates structure (correct $defs shape, known
kind strings, required properties, no additional properties,
$ref restricted to #/$defs/<name>). Value-domain constraints
(maxLength, enum index bounds) are enforced by the BAST-native
validator (step 5), not this meta-schema.
Verification:
- cargo test --release: 418 tests pass (332 lib + 86 integration);
16 new unit tests exercise the meta-schema against valid and
invalid BAST documents (missing $defs, unknown kind, additional
properties, byte-discriminator union, enum, array+count, record,
$ref, malformed ref, empty enum).
- cargo clippy --all-targets -- -D warnings: clean.
- cargo build --target wasm32-unknown-unknown --release: clean
(LazyLock + serde_json::json! macro are wasm-safe).
Step 1 of the BAST pivot. Adds the lowercase-string mapping
("uint32" <-> AlkTypeKind::Uint32) that the BAST parser and
validator dispatch on (D-BAST-002).
- to_bast_str(self) -> &'static str: returns the lowercase BAST
string for all 19 variants (14 primitives + struct/union/array/
record/enum). Boolean -> "bool", distinct from the PascalCase
variant name.
- from_bast_str(s) -> Result<AlkTypeKind, AlkTypeError>: inverse of
to_bast_str; returns AlkTypeError::Schema for unknown strings.
These are additive inherent methods on the already-re-exported enum.
The existing FromStr impl (parsing the v0.1.0 "AlkType:Uint32"
keyword form) is unchanged and removed in step 8. The two surfaces
are deliberately distinct: from_bast_str rejects "AlkType:Uint32"
and FromStr rejects "uint32".
Verification:
- cargo test --release: 402 tests pass (316 lib + 86 integration);
6 new unit tests cover both directions, round-trip, and the
from_bast_str/from_str distinctness.
- cargo clippy --all-targets -- -D warnings: clean.
The bast-pivot.md research doc had grown to 1477 lines (~58KB) through
iterative editing, pushing its most actionable content (D-BAST
decisions, POC result, migration steps) past the 50KB Read tool cap.
Agents peeking at the truncated file landed in duplicated/out-of-order
sections. Decompose into three readable-sized files with distinct roles:
- docs/architecture/bast-format.md (28KB, new): the normative BAST
format spec -- meta-schema, TypeRef, examples, validation model.
Grounded in the POC and D-BAST-001..009. Stable and safe to write
now; schema-layer.md/validation.md stay describing current code and
are rewritten post-implementation (per AGENTS.md ADR-grounding rule).
- docs/plans/bast-implementation.md (31KB, new): the execution entry
point -- ordered 10-step plan with per-step goal/files/spec-ref/
verification, the public-API semver contract table up front as a
scope-creep guardrail, and the ADR-sync checklist at the end. Each
step links to the specific bast-format.md section and D-BAST anchor.
- docs/research/bast-pivot.md (28KB, trimmed): now the research record
only -- Summary, Motivation, POC scope/result, Decisions, Risks,
References. The normative format spec, what-changes tables,
validator-split details, and migration steps moved to the two new
docs; pointers added. 1155 lines removed, 216 added.
- docs/architecture/README.md: index updated to list bast-format.md
and the two in-progress pivot docs, with notes on schema-layer.md
and validation.md being rewritten when the pivot lands.
All three files are under the 50KB Read cap, so an implementing agent
gets the whole document in one call. Cross-reference anchors verified
to resolve. No code changes; cargo test --release (396 tests) green.
Verification: cargo test --release (310 crate + 86 integration, all pass).
Scope-creep guardrail for the public API during implementation. Maps
every item re-exported from src/lib.rs to a class (breaking / additive /
unchanged) with the specific change, and every ADR (001-010) to an
action (supersede / amend / unchanged) with the reason.
Net breaking: compile (signature), validate_json/is_valid_json
(contract), Schema::build/Definitions::build (output format),
build_validator (signature or removal), and the ~13 schema::* helper
re-exports. Net additive: BAST parser, BAST-native validator,
AlkTypeKind::from_str/to_str. Net unchanged: the entire layout +
data-access + materialize + tunion layer, AlkTypeError (D-BAST-009),
the Discriminator builder, AlkTypeKind variants.
Flags three small decisions deferred to their implementation steps
(validate_json JSON Schema source, build_validator fate, schema::*
re-export retention) so they don't become drive-by semver changes.
This is a living guide — it may shift slightly during implementation,
but capturing the contract now prevents public-surface drift. Doc-only.
Converts the open question into a closed decision. Rationale: consumer
ergonomics on the combined validate_json + validate_bytes path — one
uniform payload type means one match arm downstream. Option 2
(Validation(String)) would force validate_json to flatten its
structured errors to a String, losing information on the richer path to
accommodate the less rich one. The no_std/minimal-build angle that
option 2 was meant to enable is moot: validate_json requires jsonschema
regardless, so a bytes-only no_std build already has to give up
validate_json as a separate larger decision; dropping the type from one
error variant doesn't unlock it.
Updates Phase 1 step 5 to reference D-BAST-009 for the error
construction pattern, and rewrites POC Result observation 4 from
'deferred decision' to 'decided — see D-BAST-009'.
No semver-relevant change to the Validation variant. Doc-only.
Adds the POC Result section (POC on branch bast-validator-poc, commit
f371fe4 — 20/20 tests, full 416-test suite green, clippy/wasm/doc clean).
Hypothesis confirmed: a recursive walker over the BAST type tree fully
replaces the 19 custom keyword validators on the validate_bytes path,
recovers OQ-008 union variant dispatch, and fixes the enum-membership
dead constraint. The POC code is reference scaffolding on the branch
and is not merged to main — it is superseded by Phase 1 step 5.
Marks Phase 1 step 2 and the POC Scope section as done with pointers to
the result section.
Elevates the deferred AlkTypeError::Validation payload-shape question
to OQ-BAST-001: keep jsonschema::ValidationError<'static> (POC choice,
simplest, dependency stays) vs introduce Validation(String) (drops
jsonschema from the error type; semver-relevant public-API change).
Decision belongs to the production refactor.
Verification: doc-only change, no code touched.
- Rewrite Validator Split around BAST-native validator for
validate_bytes (walks BAST, checks value-domain constraints, no
external JSON Schema needed); validate_json uses standard
jsonschema::Validator from consumer-provided JSON Schema
- Add targeted POC: BAST-native validator replacing 19 custom keyword
validators on the bytes path, verified via existing test suite
- Fix meta-schema: require count on arrays (variable-element arrays
deferred per OQ-001), add optional fields array to UnionDef for
field-name discriminators
- Remove lying no-count array example, replace with deferred note
- Document dead enum constraint (materialized index never matches
string-membered enum); BAST-native validator fixes it via index
bounds check
- Resolve all 7 OQs + 3 spec gaps as D-BAST-001 through D-BAST-008
- Update Migration Path, Risks table, Engine internals to reflect
BAST-native validator
- Clarify 'no pocs needed' was an overcorrection: layout swap needs
no POC, but the validation model does
Verification: docs-only change, no code affected
- Remove the four proposed POCs: they were implementation smoke tests,
not de-risking probes. The pivot is a backend swap on a proven layout
engine; byte-identity is already proven and the layout code is
unchanged, so there is nothing empirical left to de-risk.
- Remove the recursive TreeNode example and the recursion mention in
design principle 2: recursion is not a binary-layout concern and the
engine has no cycle detection.
- Add a Spec Gaps section: validate_bytes semantics after keyword
validator removal (UnionValidator variant dispatch regression),
arrays of variable-length elements (engine rejects them), and
field-name discriminator unions (meta-schema cannot express them).
- Reframe the engine change as an accessor-layer refactor: the
walkers' (kind, field list, annotations) reads change; everything
beneath them carries over unchanged.
Correct the validator split section to clarify that jsonschema is
not removed — it remains the JSON Schema validator for both paths
(BAST meta-schema validation and standard JSON payload validation).
Only the custom keyword registration path is removed. Also fix the
build_validator and validate_json entries in the engine changes
table to reflect that they are repurposed, not removed.
Proposes replacing custom JSON Schema keywords with a standalone
kind-based JSON format (BAST) using / for composition.
Covers format design, meta-schema, engine changes, validator split,
codegen future, ABI adapter potential, migration path, 7 open
questions, and 4 proposed POCs.
- Add README.md reflecting the v0.1.0 state: 19 AlkType kinds, two
layout modes, builder + AlkTypeEngine usage example (verified to
compile and run), validation entry points, crate independence,
untrusted-schemas guarantee, docs pointers. Mirrors the alkvault
README structure.
- Add AGENTS.md with alktype-specific git workflow, project
conventions (no comments, AlkTypeError, untrusted schemas, overflow
safety, no async, no feature flags, wasm-clean, preserve_order
load-bearing, no unsafe), verification commands, and ADR/OQ index.
Blocks auto-commit on semver-relevant public API changes per the
crates.io 0.1.0 contract.
- Add LICENSE-MIT and LICENSE-APACHE (dual MIT/Apache-2.0, matching
alkvault and the Cargo.toml license field).
- Cargo.toml: add readme, keywords, categories, rust-version = "1.85".
- Fix broken intra-doc link in builder.rs: DiscriminatorKind ->
crate::schema::DiscriminatorKind (cargo doc now warning-free).
- N1 (review #002): document is_rfc3339_timestamp as non-strict in the
function doc comment. Lists the specific gaps (day-of-month per
month, seconds range, leap seconds) and points consumers needing
strict validation to chrono/time.
- N2 (review #002): document the FieldValue::Bytes-for-Record API
asymmetry in the FieldValue enum doc and on read_record_value.
- .opencode/agents/implementation-specialist.md: point to AGENTS.md
for full convention details (matches the alkvault pattern).
- review #002: mark N1/N2 resolved; all 7 findings now closed.
Verification:
- cargo test --release: 396 tests pass (310 crate + 86 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo doc --no-deps: clean (no broken intra-doc link warnings)
- cargo build --target wasm32-unknown-unknown --release: clean
- cargo publish --dry-run --allow-dirty: clean
The immediate downstream consumer (alkcall) accepts schemas from
arbitrary internet peers in its hub/spoke topology. A panic is the
wrong failure mode for a malicious or unsupported schema - an Err
the caller can handle is correct.
Three sites converted:
- offset_map.rs: _ => Err(Offset { unsupported AlkType kind for
aligned offset computation })
- layout_builder.rs: _ => Err(Offset { unsupported AlkType kind
for packed layout computation })
- materialize.rs: _ => Err(Schema { internal: union discriminator
type N is not a supported byte discriminator })
The materialize.rs site uses Schema (not Access) because a wrong
disc_type is a schema-authoring bug (parse_discriminator should have
caught it), not a buffer-access error. The sibling sites in
sequential_reader.rs and tunion.rs already returned Err(Schema) -
only materialize.rs was the holdout.
Note: the k if k.is_fixed_size() guard in offset_map and
layout_builder means the compiler cannot enforce exhaustiveness at
compile time. Converting _ from unreachable! to Err is the runtime
mitigation. A future refactor could list all fixed-size kinds
explicitly to restore compile-time checking. Deferred to a separate
cleanup pass.
Verified: no unreachable! remains in production code (the one
remaining hit at offset_map.rs:688 is inside a #[test] fn).
cargo test --release: 396 tests pass, 0 failures
cargo clippy --all-targets -- -D warnings: clean
cargo build --target wasm32-unknown-unknown --release: clean (prior)
Four of seven review findings resolved. 5 new tests (391 -> 396 crate
tests; 438 -> 443 total). cargo test, clippy, wasm32 all green.
M2 (data_access.rs): write_bytes now validates data_len fits in u32
before the length-prefix cast. A >4GiB blob returns Access error
instead of silently writing a truncated length prefix (silent data
corruption on read-back).
M1 (builder.rs): Definitions::merge_into rewritten to access self.defs
directly instead of round-tripping through self.build() with a double-
cloned/unwrap_or_default chain that could silently drop definitions
on a shape mismatch. 4 new tests: insert-when-absent, merge-into-
existing, overwrite-duplicate-keys, no-op-on-non-object-top.
L1 (materialize.rs): byte-offset discriminator arm of
materialize_union_packed now uses checked_add for offset+disc_offset
and disc_abs_offset+disc_size, returning Access error on overflow.
Mirrors the existing sequential_reader.rs::read_union_value pattern.
L3 (error.rs): AlkTypeError::source() now returns Some(inner) for the
Validation variant (jsonschema::ValidationError implements
std::error::Error). Existing source_returns_none_for_all_variants
test split into source_returns_none_for_schema_offset_access and
source_returns_some_for_validation_variant.
Deferred: L2 (unreachable! -> Err, defense-in-depth), N1 (non-strict
RFC 3339 validator, docs-only), N2 (FieldValue::Bytes for Record,
API asymmetry). Review doc updated with resolution section.
Full source read of all 13 src/*.rs files for correctness, panic
safety, and API ergonomics ahead of v0.1.0 crates.io publish.
Findings:
- M1: Definitions::merge_into silently drops data via double-
unwrap_or_default chain; untested public API
- M2: write_bytes truncates u32 length prefix on >4GiB data
(data_len as u32 without bounds check)
- L1: materialize.rs union path uses unchecked offset arithmetic
(sequential_reader.rs sibling uses checked_add)
- L2: three unreachable!() in production code (offset_map, layout_
builder, materialize)
- L3: AlkTypeError::source() returns None for Validation variant
(ValidationError implements std::error::Error)
- N1: is_rfc3339_timestamp is non-strict (Feb 31 passes, seconds
unchecked)
- N2: SequentialReader returns FieldValue::Bytes for Record (API
asymmetry vs other composites)
Verification baseline (commit c0217d9):
- cargo test --release: 438 tests pass (391 crate + 47 integration)
- cargo clippy --all-targets -- -D warnings: clean
- cargo build --target wasm32-unknown-unknown --release: clean
- no unsafe, no TODO/FIXME, all unwrap/expect/panic in test modules
POC: /workspace/alktype-builder-poc/ (18/18 tests pass, findings in
docs/research/alktype-builder-poc/findings.md). Round 2 adds the SFTP
Packet validate_bytes tests (7 new: valid Init/Read/Write/Status,
short buffer, unknown discriminator, over-maxLength Bytes).
Open questions resolved (OQ-004 through OQ-008):
- OQ-004: Discriminator::Field name is String (already implemented;
docs updated to mark resolved)
- OQ-005: Both union discriminator kinds return the same shape:
{__discriminator, ...variant-fields}. Field-name path also had a
real offset bug (returned start, not end) - fixed.
- OQ-006: builder.md Example 3 now wraps the Union in a
Schema::struct_().field("payload", ...) and merges $defs via
Definitions::merge_into (matches the engine's AlkType:Struct-at-root
constraint and the SFTP wire shape)
- OQ-007: Bytes materialization is array-of-u8 (Value::Array of
Value::Number, one entry per byte 0..=255). BytesValidator accepts
both Value::String (validate_json) and Value::Array (validate_bytes).
maxLength = max byte count. Replaces the lossy from_utf8_lossy path
that corrupted non-UTF-8 bytes and broke maxLength semantics.
- OQ-008 (new): UnionValidator now dispatches to variant schemas via
sub-validators built at factory time. AlkTypeEngine::compile calls
schema::inline_union_variant_refs before build_validator to inline
$refs in union mapping entries (necessary because union_factory
receives the union node, but $defs live at the schema root).
Production-readiness fixes in src/ (no stubs/hedges in a published crate):
- materialize.rs: Record stub -> full count-prefixed key/value pair
implementation per schema-layer.md TRecord
- materialize.rs: root_of() was broken (returned the current node, not
the schema root) -> root schema threaded through every recursive call
so resolve_ref_or_inline can resolve $refs for nested composites
- builder.rs: LengthPrefixed encoding setter was a no-op when the
keyword was already in object form -> complete the branch (updates
the encoding entry in place for both LengthPrefixed and OffsetIndirect)
- builder.rs, engine.rs: POC-referencing comments cleaned up; the
round-trip test's or_else fallback (papering over write_field being
aligned-only) replaced with direct byte writes
Documentation:
- builder.md: Example 3 updated; Discriminator::Field spec shows String;
Open Questions section updated (OQ-004 resolved)
- validation.md: AlkType:Bytes and AlkType:Union validator descriptions
updated for array-of-u8 form and variant dispatch
- open-questions.md: OQ-004/005/006/007/008 marked resolved; new
Validation theme entries
- questions/004-008: individual OQ files updated with resolutions
- findings.md: round 2 results documented
Verification:
- cargo test: 369 -> 391 tests pass (22 new: 14 materialize, 5
inline_union_variant_refs, 3 validation/builder)
- cargo clippy --all-targets: clean
- POC: 11 -> 18 tests (7 new SFTP Packet tests); all pass
Four open questions were scattered inline in builder.md and the POC
findings doc. Moved them into the central OQ tracker under
docs/architecture/questions/ and updated the index:
- OQ-004: Discriminator::Field name type (&str vs String) — raised in
builder.md during ADR-009 spec drafting
- OQ-005: Union materialization shape (byte-offset vs field-name
consistency) — raised in the POC findings
- OQ-006: Builder spec Example 3 wrap Union in Struct — raised in the
POC findings (doc fix; engine requires AlkType:Struct at top level)
- OQ-007: Bytes materialization lossy UTF-8 — raised in the POC
findings (blocks SFTP use case for validate_bytes)
Index updates:
- open-questions.md: new 'Schema Construction' and 'Validation' theme
groups; new 'Open' section for active investigation targets (distinct
from 'Deferred / Blocked' which holds scope-parked OQs)
- README.md: OQ table extended with OQ-004 through OQ-007
- builder.md: inline OQ-004 replaced with a tracker reference
- findings.md: inline OQ-005/006/007 replaced with tracker references
Doc-only change; 369 tests pass.
POC: /workspace/alktype-builder-poc/ (11/11 tests pass, findings in
docs/research/alktype-builder-poc/findings.md). The POC code lives outside
the repo per the internal dev convention.
Implementation:
- src/builder.rs: Schema, Definitions, Discriminator types. Constructors
for all 19 AlkType kinds + standard JSON Schema types (object/array/
string/integer/number/boolean/null/any). Setters for ADR-003 annotations
(endian/align/encoding/max_length), composite builders (field/required/
items/mapping), and standard JSON Schema constraints (minimum/maximum/
minLength/minItems/maxItems/format/title/description). 16 unit tests.
- src/materialize.rs: materialize_packed and materialize_aligned functions
that walk a schema + buffer to produce a serde_json::Value tree. Recurses
into Struct, Array, Union (byte-offset discriminator). Record is stubbed
(deferred for the POC scope).
- src/engine.rs: AlkTypeEngine::validate_bytes(&[u8]) added (ADR-010).
Dispatches on layout mode, materializes Value, then validates against
the existing jsonschema validator. 7 unit tests.
- src/lib.rs: pub mod builder, pub mod materialize; re-exports Schema,
Definitions, Discriminator.
Verification:
- cargo test: 346 -> 369 tests pass (23 new: 16 builder, 7 validate_bytes)
- cargo clippy --all-targets -- -D warnings: clean
- POC (11 tests): builder round-trip + validate_bytes (packed + aligned) +
validate_json for call payloads; all pass
Findings:
- Top-level schema must be AlkType:Struct (existing constraint); unions are
field types within a struct. Builder spec Example 3 needs a doc fix.
- Builder field order preserved (preserve_order feature, load-bearing for
packed mode).
- validate_bytes correctly distinguishes Access (read phase) from
Validation (validate phase) errors, with field paths.
- validate_json path unchanged for call payloads.
Open questions surfaced (OQ-005, OQ-006, OQ-007) tracked in findings.md;
to resolve before the SFTP Packet POC round.
Rename all 19 JSON Schema custom keyword strings from "TypeDef:*"
to "AlkType:*" (e.g., "TypeDef:Struct" -> "AlkType:Struct")
across source, tests, and docs. This is a breaking change to the
schema format itself — existing schemas using the old keywords
must be updated.
Rename the Rust identifiers:
- TypedefEngine -> AlkTypeEngine
- TypedefError -> AlkTypeError
- TypeDefKind -> AlkTypeKind
- TYPEDEF_PREFIX -> ALKTYPE_PREFIX
- get_typedef_kind{,_loose,_loose_enum,_enum} ->
get_alktype_kind{,_loose,_loose_enum,_enum}
Update error message strings ("unknown TypeDef kind" ->
"unknown AlkType kind"), 11 test function names containing
typedef_kind/to_typedef_error, and doc-comment prose ("TypeDef
kind" -> "AlkType kind", "typedef engine" -> "alktype
engine", "typedef schema" -> "alktype schema"). Fix the broken
docs/architecture/crates/typedef/ path references in source doc
comments to point at docs/architecture/ directly. Rebrand the
typedef:annotation test fixture and the "not-a-typedef" test
string to their alktype equivalents.
Update ~20 generic "typedef" prose references in the architecture
docs ("typedef is the binary struct engine", "use typedef",
"typedef limitation", "replaced by typedef", etc.) to alktype.
Rename TypedefEngine in the ADR-007 code example to AlkTypeEngine.
Preserve as provenance per the prior prose-rebrand decision:
typedef.ts references (external TypeBox source file),
docs/research/alknet-typedef/findings.md research citations,
/workspace/alknet-typedef-poc/ POC path, and the
"alknet-typedef:" research section headers in findings.
Build, 295 tests, and clippy all pass clean.
Renumber ADRs 095-102 to 001-008 and OQs 069-071 to 001-003, and
update all cross-references (titles, body prose, file-path links,
tables) across the 5 spec docs, README, open-questions index, and
all 11 ADR/OQ files. Inline the ADR-009 door-type definition from
the parent alknet project (broken cross-project reference).
Rebrand prose: alknet-typedef -> alktype in headings, body text,
dependency diagrams, and "additions" notes. Disambiguate the prior
failed attempt at /workspace/@alkimiadev/alktype/ as "the
@alkimiadev/alktype prototype" to distinguish it from this crate.
Historical research citations (docs/research/*, /workspace/alknet-typedef-poc/)
are kept as-is for provenance.
Rename the crate in Cargo.toml ([package].name, [lib].name) and
update the 11 use alknet_typedef::* imports across the 4 test files.
Rebrand the crate-level doc comment in src/lib.rs.
The TypeDef:* keyword strings, TypedefError/TypedefEngine identifiers,
and other code-level references are unchanged — those are a separate
code rebrand pass.
Build, 295 tests, and clippy all pass clean.
Copy the binary struct engine (src/, tests/) verbatim from
alknet/crates/alknet-typedef and create a standalone Cargo.toml
(workspace-inherited fields inlined). Port the architecture docs
(specs, ADRs 095-102, OQs 069-071) from alknet's nested multi-crate
layout to a flat single-crate layout, fixing relative link paths.
Build, 295 tests, and clippy all pass clean.
glm-5.2
deepseek-v4-pro