# Zarr implementation roadmap

The name “Zarr” means the library targets both Zarr storage format 2 and 3. It does **not** imply that all format extensions/codecs or the entire zarr-python API are implemented. Every phase has a testable exit gate; README support claims must follow passing gates.

## Stage 0 — Project foundation (local and remote gates passed)

- MoonBit package, Apache-2.0 license, source attribution, Git history, CI, a separate off-repository contest proposal, and runnable example.
- Parse regular-grid metadata for v2/v3; generate safe logical paths and chunk keys; reject malformed/ambiguous metadata.
- In-memory byte store for metadata and already-encoded chunks.
- Gate: `moon check --deny-warn`, `moon build`, `moon test --deny-warn`, and `moon run cmd/main` pass locally and in GitHub Actions.

Native/wasm-gc checks, builds, tests and example runs pass locally and in [GitHub Actions](https://github.com/zlhahaha/zarr/actions/workflows/ci.yml). The release archive is validated in CI; Mooncakes publication is still pending.

## Stage 1 — Useful uncompressed numeric arrays

Current implementation: typed `bool`, signed/unsigned 8/16/32/64-bit integers, and `float32`/`float64` create/open/element and rectangular-slice I/O in memory and native filesystem, plus v2/v3 group/attribute operations and safe ancestor creation. The `int8` API uses range-checked `Int` values because MoonBit has no dedicated `Int8` type. Slices batch file I/O and codec work by touched chunk but still buffer the requested result. Thirty-nine independent zarr-python 3.4.0 fixtures are read by native tests, and zarr-python reads thirty-two MoonBit-generated stores in CI, including Blosc LZ4 writes, a multi-chunk compressed slice, non-finite floating fills, boolean masks, signed bytes, full-range 64-bit integers and nested groups. Extended dtypes remain open.

- Store abstraction and native filesystem backend; create/open arrays and groups, metadata/attributes, and chunks for v2/v3.
- Core numeric dtypes (`bool`, signed/unsigned 8/16/32/64-bit, float32/64) with specified endian handling; C-order v2 and v3 bytes codec.
- Read/write a full chunk and a bounded N-dimensional rectangular slice, including edge chunks and absent-chunk fill values.
- Gate: separately generated zarr-python v2 and v3 datasets round-trip with MoonBit; invalid metadata, paths, ranks, ranges and byte lengths fail explicitly.

## Stage 2 — Common real-world codecs and v2 compatibility (partial)

Implemented locally: gzip and zstd read/write for v2/v3, including optional v3 Zstd frame checksums, zlib read/write for v2, Blosc1 LZ4 read/write with no, byte or bit shuffle (other Blosc compressors read-only), bounded or preflight-checked decompression, rejection of unsupported v2 filters, v2 consolidated metadata read-only snapshots, and sixteen independent compressed Python read fixtures. Zstd frames without content size may be conservatively rejected.

- v3 codec pipeline: bytes plus zstd and gzip; v2 compressor/filter adapter for raw, gzip/zlib, zstd and Blosc. Non-LZ4 Blosc writes, BloscLZ/Snappy and v2 filters remain open.
- v2 `.` and `/` chunk separators, F-order, `.zattrs`, `.zgroup`, and consolidated `.zmetadata` reads (implemented as read-only snapshots).
- Gate: compare against zarr-python fixtures across dtypes, endianness, fill values, partial chunks and codec combinations; document unsupported codecs.

## Stage 3 — Hierarchies, scale and deployment

Partial implementation: `store/http` can hydrate only the v2/v3 metadata and chunks intersecting a requested rectangle from a static HTTP-served hierarchy. It distinguishes 404 missing chunks from transport/server failures and caps metadata bytes, chunk bytes and touched chunk count. It is native-only and read-only; successful responses now use a bounded per-instance in-memory LRU cache, but values are not streamed as typed arrays. Native `FileStore::open_consolidated_v3` reads the zarr-python inline root-index convention as a read-only snapshot; this convention is not a core v3 codec or writable index.

- Group traversal and metadata mutation, filesystem and HTTP read-only stores; configurable chunk cache (implemented in `HttpStore`) and bounded streaming (open).
- v3 sharding-indexed codec and writable consolidated metadata; optional cloud/object-store adapter after the base Store API is stable.
- Gate: read representative scientific datasets without loading the whole array; cross-language tests and performance/memory benchmarks.

## Stage 4 — Release and maintenance

Current CI passes native check/build/test and examples on Linux, macOS and Windows, plus wasm-gc check/build/test and the in-memory example on Linux. Release packaging and Python bidirectional interoperability run on Linux. A deterministic property test checks 120 N-dimensional chunk/region partitions on native and wasm-gc. A draft [0.1.0 changelog](../CHANGELOG.md) exists, while Mooncakes publication and metadata fuzz tests remain open.

- Public Mooncakes release with versioned API docs, installation snippet, examples, changelog and support matrix.
- Linux/macOS/Windows and supported MoonBit backends in CI; fuzz/property tests for metadata and chunk boundaries.
- Gate: external user can follow README to open, slice, modify and save at least one v2 and one v3 dataset; all claims are backed by tests.

Non-goals for the first release: implementing a general-purpose ndarray math library, every third-party codec, and full zarr-python API parity. Interop with existing MoonBit numeric array libraries is preferred over duplicating them.