# Compatibility

moon-hdf5 targets the HDF5 1.14.6 common-file profile. “Verified” means the
repository contains a fixture, test or h5py check for that direction. This is a
documented subset, not complete HDF5 support.

## v1.0 profile

The v1.0 profile is the verified rows in the reader and writer tables below,
including their boundaries. External raw-data and virtual-dataset support are
post-v1.0 extensions. HDF5 2.0 datatype fixtures are forward-compatibility
evidence, not part of the 1.14.6 baseline.

Outside the profile: SWMR, parallel HDF5, streaming output, in-file object
storage reclamation, metadata editing beyond the supported object-creation,
deletion, attribute and link paths; and netCDF-4/MATLAB high-level semantics.

The release gate is `scripts/release_check.sh`: formatting, documentation and
package checks; wasm/native tests; writer demo; and independent h5py
verification with h5py 3.13.0/HDF5 1.14.6.

## Reader

| Area | Verified | Boundary |
| --- | --- | --- |
| Framing | Superblocks v0–v3, user blocks, 4- and 8-byte addresses, and mixed 4-byte-address/8-byte-length symbol tables | Other address/length combinations are not fully tested |
| Groups | Symbol-table, compact and dense groups, creation-order indexes, soft links with relative and `.`/`..` targets, all named hard-link paths | Traversal stops at an address already in the current ancestor chain; dot-segment normalization is lexical |
| Cross-file links | `Hdf5Resolver` follows chained external links, relative targets and version-4 references into registered external HDF5 files | `Hdf5File` alone does not open external targets |
| Virtual datasets | Layout-and-global-heap mapping blocks; multiple mappings; same-file and registered external sources; full reads, hyperslabs, points, chunk tiles and selection pages; fill values for unmapped elements; 4- and 8-byte addresses; nested virtual datasets; direct `Hdf5File` and resolver reads | Source files must be registered, printf-style source names are not expanded, and source and virtual datatypes must match exactly. A selected result is memory resident; page and tile reads remain bounded for all and non-overlapping hyperslab mappings, while point and irregular block mappings are expanded per mapping |
| Attributes | Inline v1–v3, dense, SOHM wrappers, shared datatype/dataspace fields | Unsupported attribute datatypes fail the read instead of returning empty values |
| Layouts and indexes | Compact, contiguous, chunked; v1 B-tree, single-chunk, implicit, fixed-array, extensible-array and v2 B-tree indexes | Unsupported index variants are rejected |
| Filters | Deflate, shuffle, Fletcher32, integer and f32/f64 decimal scale-offset, masks and fill markers; caller-registered dataset chunk and fractal-heap metadata decoders through `Hdf5CustomFilter` | Other scale modes are rejected; overrides of built-in filter IDs are unsupported; unregistered required filters raise `UnsupportedFeature` |
| Numeric values | Arbitrary-width fixed-point with byte order, bit offset/precision and padding, returned as `BigInt` beyond 64 bits; IEEE float16/32/64; explicit- or implied-MSB extended binary floats such as x87 80-bit storage, converted to `Double` | Floating values beyond `Double` precision are rounded during conversion; malformed field layouts are rejected |
| Structured values | Compound, enum, array, vlen sequence, bitfield, opaque, time, reference, complex; datatype messages v1–v5 | Reference values decode as bytes; resolution is separate |
| References | Legacy object and dataset-region references under 4- and 8-byte addresses; version-4 object, dataset-region and attribute references; same-file resolution on `Hdf5File`; registered external HDF5 targets through `Hdf5Resolver`; compact, contiguous and filtered targets; point, hyperslab, block, all and empty selections | Writer and patch output require external-file targets to be registered with `Hdf5Writer::add_external_source` or `Hdf5File::with_external_source`; `Hdf5File` alone resolves same-file targets; unresolved object targets keep a null path |
| Fill values | Default, fixed-size, structured, custom vlen-string and vlen-sequence fills; undefined and `FILL_TIME_NEVER` policies; disabled or absent fills zero unallocated storage | Malformed fill messages and unresolvable global-heap targets are rejected |
| Selections | Hyperslabs, ordered points, bounded-memory tile iteration and bounded-memory selection pages | A single `read_dataset_selection` result is memory resident; use `selection_iterator` for large selections |
| Shared messages | SOHM table/indexes/heaps, reserved indexes without heaps, wrappers v1–v3, committed targets, fills, filter pipelines, field sharing and dense shared records | Other shared message types are not fixture-verified |
| Dense metadata | Managed, tiny, direct-huge and indirect-huge heap objects; metadata v2 B-trees | Broader non-matrix interoperability remains untested |
| Producer files | netCDF4 1.7.5/libnetcdf 4.10.1/HDF5 2.2.0 structures; hdf5storage 0.2.2 MATLAB metadata | Verifies encoded HDF5 structures, not complete netCDF-4 or MATLAB semantics |

Fixtures cover superblock v0/v2/v3 matrices, HDF5 1.14.6 default/`latest` and
HDF5 2.0 datatypes, scale-offset, hard links, dense/fractal heaps, same-file and
external version-4 references, virtual datasets, shared messages and producer
files.

## External raw data

This post-v1.0 reader extension is resolver-only.

| Area | Verified | Boundary |
| --- | --- | --- |
| External raw data | Registered 4-/8-byte-address files, multiple slots, offsets, element-unaligned boundaries, full/selection reads and chunk iteration | `Hdf5File` cannot dereference it; HDF5 external storage is contiguous and unfiltered by format, so chunked/filtered combinations are not representable; remote URLs are unsupported; short sources are zero-filled |

`Hdf5Writer::add_external_dataset` writes the same contiguous, unfiltered subset
for fixed-size values. Callers create the external files and exact slot sizes.

## Writer

`src/cmd/write_demo` and `scripts/verify_writer_output.py` check the output with
h5py 3.13.0/HDF5 1.14.6. `src/cmd/virtual_dataset_demo` and
`scripts/verify_virtual_dataset_write.py` cover new-file and existing-file
virtual-dataset output. Version-4 reference output is checked by MoonBit
round-trip tests; `scripts/verify_external_reference_output.py` additionally
reads external version-4 object, region and attribute references through the
HDF5 1.14.6 C API because h5py's NumPy dtype bridge does not expose them.

| Area | Verified | Boundary |
| --- | --- | --- |
| Framing | Superblock v3 and v2 object headers with OCHK continuations | Older output formats are not selected |
| Groups and attributes | Compact/dense collections, creation order, continued headers, inline/dense attributes, compact/dense link editing and symbol-table conversion | Dense limits below |
| Object creation | `add_group`, contiguous `add_dataset`, finite fixed-array or resizable `add_chunked_dataset` with built-in or caller-registered filters, `add_virtual_dataset`, and unfiltered `add_external_dataset` in bytes-backed files, including numeric, fixed-string, complex, fixed-size structured, variable-length UTF-8 string, variable-length sequence and legacy object/region-reference values; these values may be nested in compound, fixed-array and fixed-base sequence members; fill values; 4-/8-byte descriptors, reference slots, heap addresses, chunk-index fields, external-data heap fields and virtual-mapping heap fields; extensible-array indexes for one unlimited axis and v2 B-tree indexes for multiple unlimited axes; appended global heaps; v1/v2 parent headers and symbol-table parents | Filtered external storage is unsupported; virtual sources are not opened or validated during creation; no free-space reclamation; structured variable-length custom fills require 8-byte file addresses because HDF5 1.14.6 cannot represent them safely in 4-byte files |
| Dense metadata | 9–65,535 records; FRHP/FHDB direct blocks, FHIB indirect roots, managed and huge heap objects with type-1 or filtered type-2 huge-object v2 B-trees, inline metadata filter pipelines, multi-level BTHD/BTIN/BTLF type 6/9 indexes; optional SOHM table with per-message-type indexes/heaps | No free-space manager or tiny heap output; larger compact sets use OCHK; filtered rebuilds require an encoder for every pipeline entry |
| Datasets | Compact, contiguous, external raw data, virtual storage, paged fixed-array, extensible-array and v2 B-tree chunks; new contiguous, finite fixed-array chunked, virtual and unfiltered external datasets | Virtual mappings support all, empty and hyperslab selections; HDF5 1.14.6 rejects point and block virtual mappings, so the writer rejects them too; printf-style source names and datatype conversion are unsupported, and reading requires registered sources; compact payloads are limited to 65,535 bytes; external storage is unfiltered fixed-size only |
| Numeric values | int/uint 1–8,191 bytes through `BigIntegerValues` or structured `BigInteger`/`BigTime`/`BigBitfield` values, in either byte order; IEEE float16/32/64; x87 80-bit extended floats in 10- or 16-byte storage with either byte order | Extended values retain `Double` precision; arbitrary non-x87 floating field layouts are not emitted; scale-offset is limited to 32/64-bit floats |
| Strings and structures | Fixed/vlen UTF-8; compound, enum, array, time, bitfield, opaque, references, complex and sequences; `Datatype::write_vlen_string` and `Datatype::write_vlen_sequence_with_address_size` can be nested in structured values without losing the file's address width | Complex output is little-endian canonical `r`/`i`; written sequence bases are fixed-size but may contain region references |
| References | Legacy 8-byte objects and 12-byte regions plus opt-in version-4 object, region and attribute references in datasets, attributes, compounds, arrays and fixed-base sequences, including region references nested in sequence bases; legacy and version-4 object-, region- and attribute-reference datasets and attributes can also be patched | Targets must be same-file objects or external sources registered before encoding or patching; external target files must remain available under the embedded name; region targets must be existing datasets with a non-null dataspace; attribute targets must already exist; legacy remains the default |
| Filters | Shuffle, deflate, Fletcher32, integer and f32/f64 decimal scale-offset, per-chunk masks, and caller-registered `Hdf5CustomFilter` encoders through `Custom(id, parameters)` | Fixed-size values support the full built-in set; variable-length values support shuffle/deflate, not Fletcher32 or scale-offset; scale-offset must lead; floating scale-offset cannot use Fletcher32; custom filters must be registered before dataset creation and provide `encode` for output |
| Fills | Default, fixed-size/structured, vlen-string and vlen-sequence fills; optional SOHM | Variable-length fills use the global heap; structured variable-length custom fills require 8-byte file addresses because HDF5 1.14.6 fill-size validation is incompatible with 4-byte structured fields |
| Shared messages | One SOHM master table with independent fill, filter and attribute indexes/heaps; managed direct blocks extend through FHIB roots; equal messages within a type reuse one heap ID; committed datatypes use wrapper v3 type 2 | One superblock SOHM master table; other shared message types are not emitted |
| Extension | One or more unlimited axes with in-memory resize, append and arbitrary rectangular slab writes; chunk filter masks stay attached to chunk coordinates across shape changes | Multiple unlimited axes use a v2 B-tree; `append_dataset` writes at the trailing corner, `write_dataset_slab` writes at arbitrary coordinates and grows unlimited axes to the slab edge |
| Value patching | `patch_dataset_values` updates fixed-size, variable-length UTF-8 string, sequence, legacy object-reference, legacy region-reference, or version-4 object/region/attribute-reference compact, allocated or unallocated contiguous and filtered or unfiltered chunked values in place, including references and variable-length values nested in compound, fixed-array and sequence members, big-endian numeric data, big-endian scale-offset with stored fill markers, and registered custom filters with encoders; chunk padding is preserved. Variable-length strings and sequences deduplicate into one appended global heap collection whose size is a multiple of 4096 bytes, rewrite only fixed-size heap references, and use null references for empty values. Variable-length values in sequence bases and nested members use the same recursive heap encoding; reference-bearing bases share the same target rules as other nested references. Legacy object-reference values resolve each path to the existing target's object-header address, zero-fill nulls and require a slot width equal to the file address size. Legacy region-reference values require a slot width four bytes larger, resolve each target to an existing dataset with a non-null dataspace, and append deduplicated target-address plus empty/all/point/hyperslab/block selection payloads to a global heap. Same-file version-4 object references write direct address tokens; region and attribute references append their indirect payloads to a global heap and require existing non-null dataset or existing-attribute targets. External version-4 object, region and attribute references use the same indirect slot layout with flag 1 and a registered external source; their payloads carry the normalized target file name and each target file's address width. Unallocated contiguous payloads, non-fill chunks in existing single, fixed-array or extensible-array slots, missing fixed-array data blocks, missing extensible-array data blocks, missing EAIB/EASB structures or fixed-/extensible-array pages, and missing v1 or v2 B-tree chunk records are appended at aligned file offsets. Rebuilding an EAIB creates its direct/super/EADB hierarchy, derives or extends the EAHD maximum set, and updates EAHD block counts/sizes, realized elements, checksums, layout/slot/page-bitmap fields and the superblock EOF. Adding or relocating v1 B-tree records rebuilds the complete v1 tree using the superblock's indexed-storage K, including a non-default K from the superblock-extension message, with balanced same-level nodes, sibling links and datatype-sized termination keys. Adding or relocating v2 B-tree records rebuilds the complete BTHD/node tree with its original node size and split/merge thresholds. Both paths update the layout address, object-header checksum and superblock EOF. Filtered chunks that grow are relocated the same way. `patch_contiguous_dataset` is allocated-contiguous-only. | Bytes-backed `Hdf5File`; exact shape/type; missing chunks in indexes without a writable record path read as fill values; record creation rebuilds the index for the current in-memory chunk grid rather than supporting arbitrary incremental insertion; scale-offset requires a stored fill value; unregistered or decoder-only custom filters cannot be patched; external storage is read-only and unregistered external reference files are rejected; version-4 indirect targets must already exist; legacy object and region references accepted only when their slot width matches the file address layout and every non-null target already exists; variable-length chunk data supports unfiltered storage, Shuffle or deflate, and its patch cannot create a missing fixed-array data block or extensible-array metadata; no dataset resize or chunk-grid relayout; appended or relocated data and old global heaps leave unreferenced bytes because no free-space reclamation is performed |
| Attribute patching | `patch_attribute_values` updates existing fixed-size inline, dense and creation-order-tracked attributes on groups or datasets, including big-endian numeric values, plus variable-length UTF-8 string, sequence, legacy object-reference, legacy region-reference, version-4 object/region/attribute-reference attributes and references or variable-length values nested in compound, fixed-array or sequence members. Ordinary inline values refresh the containing object-header/OCHK checksum; ordinary dense values write through the name/order index and managed fractal-heap object and refresh its direct-block checksum. Shared messages, filtered fractal heaps and tiny dense records are decoded, the target value is replaced, and the complete attribute collection is rebuilt as local compact or dense storage; an existing metadata-heap filter pipeline is reused when the collection must be rebuilt. Variable-length, region-reference and indirect version-4 values append an aligned global heap collection, rewrite only the fixed-size heap references and update the superblock EOF and checksum; all-empty values need no collection. Legacy object references resolve paths to existing objects and zero-fill nulls. Legacy and version-4 region references resolve to existing non-null datasets and use empty/all/point/hyperslab/block selections, with null references zero-filled. Version-4 attribute references validate that the target attribute already exists, including in a registered external source. Attribute names, creation order and datatypes are preserved. Compact, dense, ordered-dense, large fixed-size, shared compact/dense, empty and non-empty vlen-string attributes are verified by MoonBit round-trip tests and h5py 3.13.0/HDF5 1.14.6; fixed-base vlen sequences, reference-bearing sequence bases, legacy references, version-4 references and compound/fixed-array/sequence nested references and variable-length values are covered by MoonBit round-trip tests, including 4- and 8-byte-address files; external-reference attribute output is independently read by h5py/HDF5 1.14.6. | Bytes-backed `Hdf5File`; existing attributes only; exact shape/type; unregistered external reference files are rejected; fallback rebuilding may change compact/dense placement and preserves metadata-heap filters only when their encoders are available; previous global heaps remain unreferenced because no free-space reclamation is performed |
| Attribute editing | `add_attribute`, `rename_attribute` and `delete_attribute` rebuild the complete attribute collection for v1 and v2 object headers on groups, datasets or committed datatypes. Compact storage is rebuilt for up to eight attributes; larger collections use dense storage, including multiple direct heap blocks and huge objects when needed. Existing creation order is preserved through compact and ordered-dense updates. Added fixed-size and variable-length UTF-8 string or sequence values are supported, including vlen strings and sequences nested in compound and array members, as are legacy and version-4 object, region and attribute references, including references nested in compound and fixed-array members. Reference targets must already exist in the same file; region targets must be datasets with a non-null dataspace, and attribute-reference names must exist. New vlen and indirect-reference payloads use appended global heaps, with reference slots honoring 4- and 8-byte address widths. Compact and dense shared attribute messages are resolved and rebuilt as local messages, while other SOHM objects remain unchanged. Dense heap and B-tree layouts use the file's address and length widths; 4-byte-address dense rebuilding is fixture-verified. The initial object-header chunk is rewritten to a continuation message, rebuilt messages are appended at an aligned EOF, v2 continuations carry OCHK checksums, and the superblock EOF/checksum are refreshed. MoonBit tests cover add/rename/delete, compact-to-dense-to-compact conversion, ordered-dense renames, vlen strings and sequences, nested vlen compound/array members, legacy and version-4 references, 4- and 8-byte reference slots, committed datatypes, v1 and v2 object headers, shared compact/dense attributes, 4-byte dense rebuilding, duplicate/missing names and invalid names; v1, 4-byte, shared and nested-vlen edits were independently re-read by h5py 3.13.0/HDF5 1.14.6. | Bytes-backed `Hdf5File`; reference targets must already exist and remain in the same file; external-file version-4 references are rejected; registered custom filters can decode existing dense metadata, while unregistered metadata-heap filters are rejected; no free-space reclamation, so old attribute metadata remains unreferenced |
| Link editing | `add_hard_link`, `add_soft_link`, `add_external_link`, `rename_link`, `delete_link` and `delete_object` rebuild complete link collections for v2 link-message groups and symbol-table groups on bytes-backed files. `delete_object` removes every hard link to the resolved object so aliases cannot keep it reachable, leaves soft/external links untouched, and rejects the root group. Symbol-table links are read from their B-tree/local-heap representation and converted to compact or dense link messages. Compact groups rebuild their link-info and v1 link messages; collections above eight links rebuild a fractal heap with name and creation-order v2 B-tree indexes. Existing compact and dense links, available creation order and 4-/8-byte address widths are preserved. Dense output supports multiple FHDB direct blocks, FHIB indirect roots and huge link messages indexed by a type-1 or filtered type-2 huge-object v2 B-tree. Inline heap filter pipelines are preserved and re-applied to rebuilt direct blocks and huge objects; deflate, shuffle, Fletcher32 and registered custom encoders are supported. Hard-link targets must already exist in the same file; soft targets may be unresolved; external links store a target file/path without opening it. Name, soft-target and external-info limits are validated, and invalid NUL/name forms are rejected. The initial object-header chunk is rewritten to an aligned continuation, version 1 headers are upgraded in place to version 2 when their continuation cannot be eight-byte aligned, v2 checksums and the superblock EOF are refreshed, and old link metadata is left unreferenced. MoonBit tests cover add/rename/delete, object deletion with aliases, compact-to-dense-to-compact conversion, ordered-dense creation order, duplicate/missing names, invalid names, 4-byte addresses, multi-block 60,000-byte soft links, a huge external link, symbol-table conversion for 8/8 and mixed 4/8 widths, and attribute editing after the header upgrade; deflate-plus-Fletcher32 filtered link-heap edits and the release demos are independently re-read by h5py 3.13.0/HDF5 1.14.6. | Bytes-backed `Hdf5File`; hard targets must already exist; external targets are not resolved or created; no free-space reclamation, so old symbol tables, dense heaps and indexes remain unreferenced |

## Reproduce

```sh
moon test --target wasm
moon test --target native
moon run src/cmd/write_demo --target native -- output.h5
python scripts/verify_writer_output.py output.h5
moon run src/cmd/virtual_dataset_demo --target native -- new.h5 edited.h5
python scripts/verify_virtual_dataset_write.py new.h5 edited.h5
```

Ordinary MoonBit tests need no Python or native HDF5. Fixture generation is in
[testdata/README.md](../testdata/README.md).
