alktype/docs/architecture/data-access.md

status, last_updated

status	last_updated
draft	2026-07-22

alktype — Data Access

The data access layer: read/write functions, TUnion dispatch, field paths, zero-copy access for fixed-size types, and length-prefix reading for variable-length types. This is the consumer-facing API — given a compiled AlkTypeEngine and a byte buffer, read and write fields at schema-computed offsets.

This document covers two layers:

• Primitive read/write functions in the data_access module — typed reads/writes at a caller-provided offset. These are the building blocks used by the layout types (OffsetMap, LayoutBuilder, SequentialReader) and the AlkTypeEngine. Each operates on a raw byte buffer at a known offset and returns a AlkTypeError::Access carrying the field path on bounds or encoding failures.
• The FieldValue enum and the higher-level APIs — AlkTypeEngine::read_field/write_field (aligned mode) and SequentialReader::read_next/read_field (packed mode) — which look up a field's offset via the layout and dispatch to the primitive functions, returning a unified FieldValue<'a>.

The FieldValue enum

The higher-level read APIs return a single unified type — FieldValue<'a> — so one method can read any field kind without the caller dispatching on schema kind first. The variant carries the typed value; the lifetime borrows from the input buffer for variable-length kinds (zero-copy).

pub enum FieldValue<'a> {
    I8(i8), I16(i16), I32(i32), I64(i64),
    U8(u8), U16(u16), U32(u32), U64(u64),
    F32(f32), F64(f64),
    Bool(bool),
    Enum(u32),                              // u32 index into the schema's "enum" array
    String(&'a str),                        // borrows from the buffer
    Bytes(&'a [u8]),                        // borrows from the buffer
    Struct { start: usize, end: usize },    // consumer recurses with a fresh reader
    Union { discriminator: String, variant_start: usize },
    Array { count: u32, element_start: usize, element_stride: usize },
}

For composite kinds (Struct, Union, Array), FieldValue returns a layout descriptor, not the decoded contents — the consumer recurses with a fresh SequentialReader (or a sub-range read) scoped to the reported byte range. Array's element_stride is 0 for variable-length element types, signalling the consumer must walk each element sequentially.

Read/Write Model

The alktype engine operates on raw byte buffers (&[u8] for reading, &mut [u8] for writing). There is no intermediate Value tree, no reflection, no dynamic dispatch per field. The engine uses the offset map (or LayoutBuilder/SequentialReader) to locate fields, then performs typed access at the computed positions.

Higher-level read/write

The AlkTypeEngine and SequentialReader provide the primary consumer-facing read/write APIs. They look up a field's offset via the layout and dispatch to the primitive data_access functions, returning FieldValue (read) or accepting &FieldValue (write).

impl AlkTypeEngine {
    // Aligned mode: looks up the field's ByteRange in the OffsetMap,
    // dispatches to the right data_access function by AlkTypeKind.
    // Returns AlkTypeError::Access if compiled in packed mode
    // (use sequential_reader() for packed mode).
    pub fn read_field<'a>(&self, buffer: &'a [u8], field_path: &str)
        -> Result<FieldValue<'a>, AlkTypeError>;
    pub fn write_field(&self, buffer: &mut [u8], field_path: &str,
        value: &FieldValue<'_>) -> Result<(), AlkTypeError>;

    // Packed mode: returns an owned fresh SequentialReader (ADR-007).
    // Each call returns a new reader with the cursor at position 0.
    // The consumer owns the reader and drives read_next/read_field/reset.
    pub fn sequential_reader(&self) -> Option<SequentialReader>;
}

impl SequentialReader {
    // Packed mode: walks the buffer field-by-field, reading length
    // prefixes to find each field's position. read_field walks all
    // preceding fields to reach the target.
    pub fn read_next<'a>(&mut self, buffer: &'a [u8])
        -> Result<Option<(String, FieldValue<'a>)>, AlkTypeError>;
    pub fn read_field<'a>(&mut self, buffer: &'a [u8], field_path: &str)
        -> Result<FieldValue<'a>, AlkTypeError>;
    pub fn reset(&mut self);
    pub fn position(&self) -> usize;
    pub fn endian(&self) -> Endian;
}

read_field/write_field on AlkTypeEngine work for the fixed-size primitive kinds and the length-prefixed String/Bytes/Timestamp fields. Composite kinds (Struct, Union, Array, Record) return a FieldValue carrying a layout descriptor (byte range, variant start, or array stride) for the consumer to recurse on — see §"FieldValue" above.

For writing in packed mode, the consumer uses LayoutBuilder::build to compute positions, then calls the primitive data_access::write_* functions at the computed offsets. There is no packed-mode engine.write_field — the layout depends on the actual data sizes, which the builder consumes at build time.

Primitive read/write functions

The data_access module exposes typed read/write functions for each primitive kind. Each takes field_path: &str for error attribution (produces a AlkTypeError::Access carrying the path on bounds or encoding failures) and, for multi-byte types, an Endian parameter.

Fixed-size types

Fixed-size types (TFloat32, TInt32, TUint8, TEnum, etc.) are accessed via zero-copy reads of N bytes at the offset:

// Read a u32 at a known offset, applying endianness. Bounds-checked.
fn read_u32(buffer: &[u8], offset: usize, field_path: &str, endian: Endian)
    -> Result<u32, AlkTypeError> {
    let bytes: [u8; 4] = read_array(buffer, offset, field_path)?;
    Ok(match endian {
        Endian::Little => u32::from_le_bytes(bytes),
        Endian::Big => u32::from_be_bytes(bytes),
    })
}

// Write a u32 at a known offset, applying endianness. Bounds-checked.
fn write_u32(buffer: &mut [u8], offset: usize, value: u32,
              field_path: &str, endian: Endian) -> Result<(), AlkTypeError> {
    let bytes = match endian {
        Endian::Little => value.to_le_bytes(),
        Endian::Big => value.to_be_bytes(),
    };
    write_array(buffer, offset, bytes, field_path)
}

The engine applies endianness at access time based on the schema's "endian" annotation (ADR-003). The offset computation is endian-agnostic. The read_array/write_array helpers perform the bounds check and produce AlkTypeError::Access with the field path on failure.

TEnum access

TEnum is a fixed-size type (4 bytes, u32 index). Read/write delegates to the u32 primitives, applying the schema's endianness:

pub fn read_enum(buffer: &[u8], offset: usize, field_path: &str, endian: Endian)
    -> Result<u32, AlkTypeError> {
    read_u32(buffer, offset, field_path, endian)
}

The consumer maps the u32 index back to the enum's string values using the schema's "enum" array (index 0 → first value, index 1 → second value, etc.). The engine does not perform this mapping — it operates on the raw u32 index. The jsonschema validator checks that the index corresponds to a valid enum value at the JSON level.

Variable-length types (inline length-prefixing)

For variable-length types with inline length-prefixing (the default), the data_access module provides read_string/write_string/ read_bytes/write_bytes. Each takes field_path: &str for error attribution and endian for the length prefix:

// Read a length-prefixed string, borrowing from the buffer.
fn read_string<'a>(buffer: &'a [u8], offset: usize,
                   field_path: &str, endian: Endian) -> Result<&'a str, AlkTypeError>;

// Write a length-prefixed string. Returns total bytes written (4 + data.len()).
fn write_string(buffer: &mut [u8], offset: usize, value: &str,
                field_path: &str, endian: Endian) -> Result<usize, AlkTypeError>;

// read_bytes / write_bytes have the same shape — raw bytes, no UTF-8 check.

The engine reads the 4-byte length prefix at the field's offset, then slices the data that follows. For writing, the engine writes the length prefix + data. read_string validates UTF-8 and returns a &str borrowing from the input buffer (zero-copy); read_bytes returns a &[u8] slice with no encoding check.

In packed sequential mode, the SequentialReader uses the length prefix to determine the position of the next field. In aligned static mode, the OffsetMap records the position of the length prefix; the variable data is accessed separately.

Variable-length types (offset indirection)

For variable-length types with offset indirection (opt-in), the data_access module provides read_string_indirect/read_bytes_indirect. The 8-byte struct at buffer[offset..offset+8] is { data_offset: u32, data_length: u32 } (endian-aware); the actual bytes live in a separate data_region:

fn read_string_indirect<'a>(buffer: &'a [u8], offset: usize,
                             data_region: &'a [u8], field_path: &str,
                             endian: Endian) -> Result<&'a str, AlkTypeError>;
fn read_bytes_indirect<'a>(buffer: &'a [u8], offset: usize,
                            data_region: &'a [u8], field_path: &str,
                            endian: Endian) -> Result<&'a [u8], AlkTypeError>;

The field is a struct {offset: u32, length: u32} at a known position in the OffsetMap. The consumer provides the data region separately; the engine reads the offset and length, then slices the data region.

TUnion Dispatch

The tunion module provides TUnion discriminator dispatch — reading the discriminator value from a byte buffer, looking up the variant schema in the union's mapping, and reporting the offset where the variant struct begins. All reads go through the data_access primitives so bounds checks and endianness handling are uniform with the rest of the engine.

The result of dispatch is a UnionDispatch struct:

pub struct UnionDispatch {
    pub key: String,              // mapping key (stringified disc value)
    pub variant_offset: usize,    // byte offset where the variant struct starts
    pub discriminator_size: usize, // discriminator's byte size
}

After dispatch, the consumer calls tunion::resolve_variant(union_node, &dispatch.key) to get the variant BastType, then reads the variant's fields at dispatch.variant_offset using the normal data_access functions (or a fresh SequentialReader scoped to the variant). $ref variant types are returned as BastType::Ref; the caller resolves them via BastDoc::resolve_typeref when a concrete definition is needed.

Byte-offset discriminator

/// Read the discriminator value from a byte-offset TUnion. The discriminator
/// is a fixed-size integer (uint8/uint16/uint32) at a known byte offset.
/// Returns the mapping key (stringified integer) and the variant struct
/// offset.
pub fn read_byte_discriminator(
    buffer: &[u8],
    union_node: &BastUnion<'_>,
    endian: Endian,
) -> Result<UnionDispatch, AlkTypeError>;

This is the SFTP Packet enum pattern — byte 0 is the type byte, bytes 1..N are the variant struct. The call protocol's 5 event types (call.requested → 0x01, etc.) use the same pattern. The variant struct starts at offset + discriminator_size.

Field-name discriminator

/// Read the discriminator value from a field-name TUnion. The
/// discriminator is a named field within the struct — the consumer
/// provides the field's computed offset (from the OffsetMap or
/// LayoutBuilder). Supports string, uint8, and enum discriminator
/// fields.
pub fn read_field_discriminator(
    buffer: &[u8],
    union_node: &BastUnion<'_>,
    disc_field_offset: usize,
    endian: Endian,
) -> Result<UnionDispatch, AlkTypeError>;

The discriminator is a named field within the struct. Its offset is computed like any other field (the consumer passes it in as disc_field_offset). The mapping keys are string values. After reading the discriminator, the consumer looks up the variant schema and reads the variant's fields starting at the end of the discriminator field.

Variant resolution

/// Look up a variant type from the union's mapping. Inline struct/
/// union/enum types are returned directly; `$ref` pointers are
/// returned as `BastType::Ref` — the caller resolves them via
/// `BastDoc::resolve_typeref` when a concrete definition is needed.
pub fn resolve_variant<'a>(
    union_node: &'a BastUnion<'a>,
    key: &str,
) -> Result<&'a BastType<'a>, AlkTypeError>;

/// Get the discriminator's byte size (1/2/4 for uint8/16/32) for a
/// byte-offset TUnion. Field-name discriminators have no fixed size
/// and produce an `AlkTypeError::Schema`.
pub fn discriminator_size(union_node: &BastUnion<'_>) -> Result<usize, AlkTypeError>;

TUnion in the layout engines

The LayoutBuilder and SequentialReader also handle TUnion fields inline during traversal (the consumer does not need to call the tunion functions for a union field reached during a sequential walk). For LayoutBuilder, the consumer supplies the discriminator value (byte-offset) or variant index (field-name) in var_sizes under the synthetic key "<union_path>.__discriminator" or "<union_path>.__variant". For SequentialReader, a union field yields FieldValue::Union { discriminator, variant_start }. The standalone tunion functions are for dispatch outside the layout walk — e.g., a consumer that receives a bare union buffer and needs to identify the variant before recursing.

Field Paths

Fields are addressed by dotted paths: "header.version", "payload.data". Both OffsetMap and PackedLayout store fully-qualified paths (nested struct fields appear under their parent's path prefix). The higher-level APIs (AlkTypeEngine::read_field/write_field, SequentialReader::read_field) accept a field path, look up the byte range/position in the layout, and dispatch to the primitive data_access function for the field's kind.

For aligned-mode access, AlkTypeEngine::read_field(&buffer, "header.version") returns FieldValue — it looks up the ByteRange in the OffsetMap, finds the field's AlkTypeKind in the BAST typed tree, and calls the matching data_access::read_* function. write_field is the mirror. Composite kinds (Struct, Union, Array, Record) return a FieldValue carrying a layout descriptor; the consumer recurses with a fresh reader or sub-range read.

For packed-mode access, SequentialReader::read_field(&buffer, "c") walks all preceding fields to reach the target (sequential access is inherent to packed layouts). read_next walks fields in declaration order.

Nested structs produce nested field paths. The offset computation propagates the field path prefix during recursion, so the OffsetMap and PackedLayout contain entries like "header.version" and "header.magic".

Zero-Copy Access

For fixed-size types, the engine provides zero-copy access — the consumer gets a reference to the bytes in the buffer, not a copy. This is important for performance-sensitive paths (metatensor tensor access, high-throughput protocol parsing).

For variable-length types with inline length-prefixing, the engine returns a slice of the buffer — the string or byte array data is not copied. The consumer gets a &str or &[u8] that borrows from the input buffer.

For offset-indirect types, the consumer provides the data region; the engine returns a slice of that region.

Error Handling

Read/write errors carry the field path for debugging. See ADR-004 and validation.md for the full error model.

Design Decisions

Decision	ADR	Summary
Two layout modes	ADR-002	Determines whether offsets are fixed (OffsetMap) or sequential (SequentialReader)
Schema annotations	ADR-003	Endianness, encoding, and TUnion discriminator shapes that control data access
Error handling	ADR-004	Field-path-carrying errors for read/write operations

Open Questions

See open-questions.md for full details.

• OQ-001 (deferred(scope)): Arrays of variable-length-element structs — affects the sequential walking logic for array access.

References

• @alkdev/alknet: docs/research/alknet-typedef/findings.md §"POC Results" — POC 1 (read/write round-trip) and POC 2 (SFTP byte-identical round-trip)
• layout-engine.md — offset computation that produces the positions this layer reads/writes at
• validation.md — validation that runs on the same buffers