# Zarr implementation roadmap ## 0.10.0 usefulness batch Native complete-shard creation/replacement for uint8/uint16/float32, pinned HTTP/S3 metadata sessions with explicit fresh refresh, and an independent crop/normalize/shard volume application are implemented. Python verifies every output value across local/HTTP v2/v3/sharded inputs and authenticated private MinIO input. Ten additional MoonBit shard stores round-trip through Python. See [SHARD_WRITES.md](SHARD_WRITES.md), [SCANNING.md](SCANNING.md) and [the application](../examples/volume_pipeline/README.md). Next work should follow concrete user workloads: partial-shard updates require a separately specified concurrency/safety policy; remote writes require authorization and conditional-object semantics. Other typed shard writers, ndarray adapters and broader deployment validation can be added when a consumer needs them. Historical phases below describe their original scope; references to unsupported sharded writes apply to versions before 0.10.0. The name “Zarr” means the library targets both Zarr storage format 2 and 3. It does **not** imply that all format extensions/codecs or the entire zarr-python API are implemented. Every phase has a testable exit gate; README support claims must follow passing gates. ## Stage 0 — Project foundation (local and remote gates passed) - MoonBit package, Apache-2.0 license, source attribution, Git history, CI, a separate off-repository contest proposal, and runnable example. - Parse regular-grid metadata for v2/v3; generate safe logical paths and chunk keys; reject malformed/ambiguous metadata. - In-memory byte store for metadata and already-encoded chunks. - Gate: `moon check --deny-warn`, `moon build`, `moon test --deny-warn`, and `moon run cmd/main` pass locally and in GitHub Actions. Native/wasm-gc checks, builds, tests and example runs pass locally and in [GitHub Actions](https://github.com/zlhahaha/zarr/actions/workflows/ci.yml). The release archive is validated in CI; Mooncakes `0.1.0` was published on 2026-09-25. ## Stage 1 — Useful uncompressed numeric arrays Current implementation: typed `bool`, signed/unsigned 8/16/32/64-bit integers, and `float32`/`float64` create/open/element and rectangular-slice I/O in memory and native filesystem, plus v2/v3 group/attribute operations and safe ancestor creation. The `int8` API uses range-checked `Int` values because MoonBit has no dedicated `Int8` type. Slices batch file I/O and codec work by touched chunk but still buffer the requested result. Fifty-one independent zarr-python 3.4.0 fixtures are read by native tests, and zarr-python reads forty MoonBit-generated stores in CI, including Blosc LZ4 writes, a multi-chunk compressed slice, non-finite floating fills, boolean masks, signed bytes, full-range 64-bit integers and nested groups. Explicit 3D image-volume and 4D feature-tensor checks include v2 C/F order, endian variants, missing/edge chunks, overwrite/reopen and compressed/sharded reads. See [INTEROPERABILITY.md](INTEROPERABILITY.md). Extended dtypes remain open. - Store abstraction and native filesystem backend; create/open arrays and groups, metadata/attributes, and chunks for v2/v3. - Core numeric dtypes (`bool`, signed/unsigned 8/16/32/64-bit, float32/64) with specified endian handling; C-order v2 and v3 bytes codec. - Read/write a full chunk and a bounded N-dimensional rectangular slice, including edge chunks and absent-chunk fill values. - Gate: separately generated zarr-python v2 and v3 datasets round-trip with MoonBit; invalid metadata, paths, ranks, ranges and byte lengths fail explicitly. ## Stage 2 — Common real-world codecs and v2 compatibility (partial) Implemented locally: gzip and zstd read/write for v2/v3, including optional v3 Zstd frame checksums, zlib read/write for v2, Blosc1 LZ4 read/write with no, byte or bit shuffle (other Blosc compressors read-only), bounded or preflight-checked decompression, rejection of unsupported v2 filters, v2 consolidated metadata read-only snapshots, and sixteen independent compressed Python read fixtures. Zstd frames without content size may be conservatively rejected. - v3 codec pipeline: bytes plus zstd and gzip; v2 compressor/filter adapter for raw, gzip/zlib, zstd and Blosc. Non-LZ4 Blosc writes, BloscLZ/Snappy and v2 filters remain open. - v2 `.` and `/` chunk separators, F-order, `.zattrs`, `.zgroup`, and consolidated `.zmetadata` reads (implemented as read-only snapshots). - Gate: compare against zarr-python fixtures across dtypes, endianness, fill values, partial chunks and codec combinations; document unsupported codecs. ## Stage 3 — Hierarchies, scale and deployment Partial implementation through `0.7.0`: `chunk.RegionTileCursor` incrementally plans bounded N-D rectangles, but typed reads still buffer each tile. `store/http` hydrates ordinary v2/v3 chunks and supported v3 shard indexes/selected inner payload ranges from read-only HTTP; `store/s3` reuses that remote reader for anonymous or explicitly signed path-style S3-compatible GETs. The shared remote path validates response ranges and distinguishes 404 missing data from failures, with per-request timeouts, one transient retry and resource caps. `HttpStore::open_array` and `S3Store::open_array` expose validated metadata before data transfer. Live public OME-Zarr `uint8`/Zstd data and a 256 MiB logical tiled scan are reproducible; neither is a broad production-cloud benchmark or measured aggregate-RAM guarantee. Native `FileStore::open_consolidated_v3` reads the zarr-python inline root-index convention as a read-only snapshot. Native filesystem sharded reads still use bounded index/inner-chunk file ranges; six Python fixtures cover layouts, missing chunks, edges and explicit 3D/4D data. Sharded writes, remote writes and writable consolidated metadata remain open. - Version 0.8.0 implements explicit group children and bounded breadth-first traversal in memory, native files (including consolidated snapshots) and S3. S3 uses signed delimiter pagination; ordinary HTTP can inspect known node paths but cannot enumerate arbitrary directories. Metadata resize/reshape and writable consolidated snapshots remain open. HTTP has an encoded full-object LRU cache; changing its header provider starts a fresh cache for the new authorization context. - V3 sharding-indexed reads work on native filesystem and through strict HTTP/S3 Range hydration; sharded writes, other object-store services, remote write support and writable consolidated metadata remain open. The HTTP/S3 range transport is shared, but a universal filesystem/network Store trait is not yet public or assumed stable. - Native multi-chunk/shard latency and OS peak-memory benchmark harness implemented with independent synthetic stores, fresh-process trials, cache-on/off comparisons and Linux/macOS/Windows native CI smoke reproduction. Typed region reads reuse validated indexes in bounded operation-local FIFO caches. See [BENCHMARKS.md](BENCHMARKS.md); this only measures the documented workloads, not general large-scale dataset performance or streaming whole-array memory bounds. - Read resource budgets implemented: shared immutable policies for metadata/chunk sizes, region bytes/elements/count, rank and JSON depth; same-handle native size checks, explicit over-budget exceptions, and cumulative HTTP hydration bytes. See [RESOURCE_LIMITS.md](RESOURCE_LIMITS.md). These are per-input/work protections, not streaming or aggregate RAM guarantees. - Single-key filesystem write safety implemented for `0.5.0`: exclusive same-directory staging, complete data-synchronized writes, handle closure, replacement rename and cancellation-safe owned-temp cleanup. Fault-injection/public regressions cover partial write/rename failures, collisions, cleanup failure, cooperative cancellation, open readers and concurrent complete-key writers. See [WRITE_SAFETY.md](WRITE_SAFETY.md). Multi-key transactions, concurrent read-modify-write locking, directory-sync crash durability and broader filesystem guarantees remain open. - Gate: read representative scientific datasets without loading the whole array; cross-language tests and performance/memory benchmarks. ## Stage 4 — Release and maintenance Version 0.9.0 adds typed pull readers in `scan`, `scan/async` and `scan/remote`. They retain no previous tile results, support early stop, reject concurrent async pulls and preserve explicit failures. Memory/filesystem callbacks support existing typed arrays; HTTP/S3 conveniences automatically hydrate and decode uint8, uint16 and float32, with a generic decoder for other types. Tiles still buffer individual results and do not establish an aggregate-RAM cap. Fixed-seed property/mutation tests cover independent ND geometry/dense models, codec truncations, dimension mutations, index CRC/interval bounds and S3 XML/UTF-8; local gates are 157 native and 78 wasm-gc tests. The real-data histogram harness pins eight public encoded objects and independently checks a 128³ selection using Python while measuring fresh-process RSS and requests through loopback HTTP. See [SCANNING.md](SCANNING.md) and [REAL_DATA_BENCHMARK.md](REAL_DATA_BENCHMARK.md). Sharded writes and remote writes remain P2, outside this batch. The combined 0.9.0 release is published. [Source CI](https://github.com/zlhahaha/zarr/actions/runs/42502343527) passed all four platforms, and an independent registry consumer passed strict check/build/run with the new typed APIs and Python verification. Exact publication and benchmark evidence is recorded in HANDOFF; future work is not part of this completed batch. CI covers native check/build/test, examples and benchmark smoke runs on Linux/macOS/Windows, plus wasm-gc on Linux. Source packaging, real private MinIO integration and the older Python reverse checks run on Linux; the N-D reverse checks and independent registry consumer run on every native platform. The 0.8.0 source gate is 142 native and 70 wasm-gc tests, including deterministic geometry/discovery and malformed-input regressions. Historical 0.1.0–0.7.0 releases established the v2/v3, resource-budget, safe-write, N-D interop and measured read-kernel baselines; see [CHANGELOG.md](../CHANGELOG.md) and [BENCHMARKS.md](BENCHMARKS.md) for release details and measurement records. Every new immutable release is followed by independent registry installation and public-API verification. Exhaustive fuzzing, broad real-data throughput and production-cloud deployment validation remain open. Published `0.7.0` adds typed tile traversal, HTTP/S3 sharded hydration, explicit SigV4 credentials, real public-data checksum verification and native CI execution of the 256 MiB logical tiled scan. Native 132/132 and wasm-gc 67/67 tests passed; all four GitHub CI jobs passed, and an independent project downloaded the Mooncakes package and ran a native tile/S3 public-API sample. The optional live example checked eight compressed objects in a 128³ region of a public OME-Zarr dataset; it did not validate the whole dataset. Native private AWS/MinIO interoperability, portable total-memory bounds, broad real-world throughput measurements and remote writes are not established by the local mock or public anonymous example. See [REMOTE.md](REMOTE.md) for the exact scope and the [0.7.0 CI run](https://github.com/zlhahaha/zarr/actions/runs/36663467656). - Public Mooncakes release with versioned API docs, installation snippet, examples, changelog and support matrix. Version **0.8.0** combines the previous maintenance preparation (never published separately as 0.7.1) with explicit discovery and structured per-call errors. It targets MoonBit 0.10.20, retains optional APIs, requires opt-in for signed plaintext endpoints and tests a real private MinIO server on Linux CI. Windows verification observed 18 real Range responses and 16 signed listing requests, including forced pagination, Unicode names, missing fills, 403 errors and deterministic retry/timeout faults. The registry-only consumer runs on all native CI platforms. Production AWS, STS/role authorization, TLS deployment and remote writes remain unverified. See [DISCOVERY.md](DISCOVERY.md), [MAINTENANCE.md](MAINTENANCE.md) and [REMOTE.md](REMOTE.md#reproduce-private-minio-integration). - Linux/macOS/Windows and supported MoonBit backends in CI; fuzz/property tests for metadata and chunk boundaries. - Gate: external user can follow README to open, slice, modify and save at least one v2 and one v3 dataset; all claims are backed by tests. Non-goals for the first release: implementing a general-purpose ndarray math library, every third-party codec, and full zarr-python API parity. Interop with existing MoonBit numeric array libraries is preferred over duplicating them.