# Full compatibility tracking

Completion target: the behavior of Differential Dataflow 0.25.1 at the pinned
commit, including the required Timely runtime. Idiomatic MoonBit APIs can differ
from Rust syntax and ownership APIs; semantic differences must remain explicit
until resolved. A small core port does not satisfy this target.

## Current evidence

| Area | Status | Evidence / remaining work |
| --- | --- | --- |
| Lattice joins/meets and advance-by | Common built-ins and products | Signed/unsigned 8/16/32/64/128-bit integers, Duration, Unit and generic products; Int8 pairs tested exhaustively |
| Minimal/counted antichains | Implemented | 10,240 Rust and independent-oracle rounds |
| Change batches | Implemented | Signed cancellation and frontier tests |
| Difference algebra | Upstream built-ins implemented with typed adapters | All integer widths, Wrapping signed adapters, Present, singleton/tuple arities through four and VectorDiff; explicit multiplication callbacks adapt associated output types; scale_unit/pair/triple/quad support heterogeneous associated outputs |
| Sealed traces | Built-in layouts and public storage adapters | Snapshots, compaction, closure, exact-cut cursors, empty-interval traversal and lazy entered/frontier-filtered trace views with independent reader pins; custom immutable BatchView/TraceReader/TraceBackend adapters feed shared arrangements, joins, nested reductions and live imports; 768 external-reader reduction epochs match Rust; specialized chunk layouts remain |
| Buffered input | Implemented callback and worker integration | Explicit close replaces Rust Drop |
| Worker scheduling | Cooperative workers | Counted scopes, local/timed activators with explicit monotonic host clock, activation deduplication, re-entry guards, bounded queues, disposal tests; 8,192 activation rounds match Rust; fresh scopes and live trace queries can be added between runs |
| Unary and binary collection operators | Implemented baseline | Map/filter/flat-map/explode/negate/concat/consolidate and weighted join family; message-preserving results_in, delay, join_function, inspect_batch/container and immediate consolidate_stream match 2,496 Rust/model operator snapshots |
| Live weighted joins | Shared indexed arrangements and collection joins | 1,024 Rust epochs in four runtime variants, including twice-entered shared indexes; shared-reader lifetime and self-join tests; bounded cursor/emission work with explicit output capabilities, including compaction while joins are unfinished |
| Reduce/count/threshold/distinct | Arbitrary-frontier compaction and explicit semigroup corrections | 768 streamed Rust antichain epochs for collections, arrangements and entered views; 20,000 total-order and 5,000 incomparable-frontier retention epochs; unchanged keys cache pending corrections; reduce_core supports collection/shared/nested arrangements and non-Abelian output weights, checked against Rust over 2,304 partial-order epochs and 3,200 presence snapshots |
| Shared arrangements/imports/lookups/upsert | Implemented baseline | 1,024 Rust reader-compaction epochs, 8,192 Rust upsert records; live imports, deferred lookups, shared indexed joins and reductions tested |
| General operator builders/notifications/capabilities | Heterogeneous builders implemented | Multiple input/output types, summary-aware connectivity, borrowed input capabilities, notifications, local activators, build_reschedule and automatic callback retirement; 6,144 lifecycle rounds match Rust resource-lifetime behavior; thread-safe activators and broader scheduling interfaces remain |
| Nested scopes and iteration | Product nesting, custom refinements and identity regions implemented | 256 Rust feedback epochs and 256 cyclic graph fixed-point epochs; nested feedback cycles tested locally and across processes; 256 Rust epochs with signed-summary edges in advancing cycles |
| Multi-worker/distributed runtime | Cooperative groups and TCP process clusters | Globally counted serialized exchange and nested iteration tested in two and three native processes; 144 repeated nested-dataflow epochs across two/three processes; host-polled nonblocking transport and cluster execution match 784 process epochs; shared-memory parallel threads and decentralized peer scheduling remain |
| Columnar/spilling/fueled trace variants | Fueled spine, columnar arrangements and pluggable spilling implemented | 1,024 partial-order batching epochs and 2,048 fueled spine epochs match Rust; columnar storage also checked against upstream columnar spines, 16,384 cursor steps, 1,024 nested join and 768 partial-order reduction epochs; 2,048 upstream spilled spine epochs, paged cursors/joins, corruption checks and native disk compaction tests; 11,264 trie chunk snapshots match upstream and independent models; 256 columnar byte fixtures match Rust exactly; specialized chunk-channel containers and zero-copy decoding remain |
| Capture/replay, logging, diagnostics | Partial | Capture protocol matches Rust in 256 cases; retryable capture writers and activated replay sources compare with Rust across 1,536 epochs; POSIX capture journals provide fsynced persistence and tailing with 268 truncation and 7 corruption checks; partitioned distributed replay matches upstream three-worker sources across 128 two/three-process epochs; 128 eager-source cases distinguish record availability from sealed progress; typed logger registry, batch/merge/drop events, pending-record deltas and reader-share diagnostics are implemented; 1,600 accounting snapshots match Rust; native managed allocator byte metrics and Timely scheduler event instrumentation remain |
| Runtime-defined iteration scopes | Implemented | Canonical PointStamp coordinates, partial-order lattice, truncating/composed summaries, dynamic enter/leave and distributed codecs; 4,096 Rust/model cases and 512 fixed-point epochs at runtime-selected depths; two/three-process sync and host-polled coverage |
| Graph and auxiliary algorithms | BFS, label propagation and SCC edges | 192 dynamic graph epochs against Rust and independent BFS/transitive closure, including duplicate edges and scheduled labels; generic Ring weights tested; arranged graph APIs reuse shared indexes across feedback; bidijkstra and sequential updates match 192 Rust epochs; prefix aggregate/broadcast match 192 Rust epochs; identifiers match 384 collision/retraction epochs |

Three upstream example families now have runnable MoonBit adaptations: degree
histograms (`hello.rs`/`degrees.rs`), integer PageRank (`pagerank.rs`), and item
recommendations (`itembased_cf.rs`). They match pinned Rust and independent full
recomputation over 384 epochs. PageRank requires a positive iteration bound;
recommendations expose exact Jaccard fractions instead of final floating scores.
The min-plus shortest-path example (`monoid-bfs.rs`) is also ported: 1,024 incremental epochs match Rust and independent Dijkstra, including zero-cost cycles, new roots and decreasing weights. Empty feedback variables accept semigroup weights; only variables initialized from a source require negation. This is partial example coverage; the remaining example inventory is not complete.

## Completion gates

- Port upstream examples and behavior tests, recording intentional MoonBit API
  adaptations instead of silently dropping functionality.
- Compare incremental results and progress against pinned Rust for every operator
  family, including negative differences, incomparable times and nested loops.
- Exercise at least two workers and separate processes, including uneven input
  progress, exchange, closure, and transport ordering.
- Test compaction and reader/capability lifetimes under sustained updates; verify
  memory retention for optimized reductions and shared traces.
- Run native, JavaScript, Wasm and Wasm GC tests where the backend supports the
  required facilities. Native transport/runtime-specific tests need explicit
  target configuration; a passing single-worker suite is insufficient.
- Resolve the pending rows before describing this as a complete compatible port
  or labeling a public repository or package as a completed port.

## Known implementation limits

Cooperative worker groups execute on one thread. ClusterWorker executes independent
processes with globally counted progress; TcpTransport provides framed, bounded
POSIX TCP I/O. Joins and nested fixed points pass against Rust and independent
oracles in two and three processes. The cluster uses synchronous all-gather rounds
and an independent wire protocol, not Timely wire interoperability. It does not
yet offer shared-memory parallel threads or decentralized peer-to-peer scheduling. AsyncExchange and ClusterWorker.poll allow nonblocking host event loops; TCP polls perform bounded I/O and yield when peers are unavailable, while progress still uses collective rounds.
Transport failures terminate the local runtime; reconnect/recovery is not implicit.
Fault injection covers truncation, oversized frames, timeouts, malformed text,
wrong peers and graph mismatches. Native workloads pass ASan and LSan locally, including 512 shared graph/import
epochs using row and columnar indexes, across normal closure and disposal with live input capabilities.

Reduction compacts history at arbitrary antichains but still enumerates a timestamp
join-closure. Trace cursors merge retained batch indexes lazily; lookups still scan snapshots.
16,384 cursor navigation steps match Rust, including seeks, rewinds and
per-batch timestamp enumeration. Composed frontier and data-dependent entry
views match Rust in 128 seeds; another 1,024 join epochs cover assigned inner
times across two nested scopes. Trace imports use the same retained views.
Arranged joins reuse shared sorted batch indexes with binary key lookup; collection
joins retain private keyed histories. Arranged reductions read shared key ranges and retain output corrections separately.
Arranged joins now expose a per-activation fuel budget and retain immutable views
for unfinished work. The four-level columnar trie, streaming construction, partial merge, suffix,
split/meld and frontier extraction operations are implemented. Specialized chunk-channel containers and bounded computation for the
remaining operator families remain unfinished. Capture/replay supports nonblocking writer retries, explicit commit completion, cancellation,
and bounded activated sources. POSIX capture journals persist versioned checksummed frames and can replay complete
prefixes after interruption; missing terminal progress keeps replay incomplete.
Creation is exclusive and append calls fsync; automatic crashed-writer resumption
remains pending. Distributed replay partitions record deduplication by data hash and count state
by timestamp hash, broadcasting compact frontier updates. Like upstream
capture::source, it emits new records eagerly and rejects conflicting weights
for one data/time pair. The standalone Replayer retains upstream iterator-style
sealed-interval delivery. Group count sharding is configurable (peer zero by
default); process count sharding defaults to a hash of the timestamp codec. Journal encoding is independent of Rust serde.
The pinned Rust progress writer only retries inside its frontier-advance branch;
the port also retries queued progress when the frontier is unchanged, avoiding
a stalled terminal progress message. Its regression test uses a delayed final
message; the Rust oracle retries update writes and accepts progress immediately.

Built-in String ordering follows MoonBit (length-first), so lexical Rust behavior
requires a custom Compare implementation. Explicit release/close/dispose replace
Rust Drop semantics; APIs document ownership transfers and borrowed capabilities.

The pinned Rust spine rejects noncontiguous and zero-width insertions, despite
the generic Trace trait documentation describing automatic gap filling. The port
follows the pinned implementation: callers explicitly seal empty intervals.

Dynamic scopes preserve existing graph immutability. Completed built-in operators and imports now release task closures and resource-owning
finalizers after their mailboxes and outputs finish. Custom operators retain their
explicit incomplete-work condition. Completed scopes reclaim reachability matrices, observer tables and boundary
closures; stable identity slots remain. Group scopes retire together, and process
scopes require unanimous completion votes before discarding their graphs. The
cluster envelope is version 4; completed manifest entries become null so fresh
dataflows can still validate consistently. Local root dataflow identity/enumeration and explicit cancellation are implemented, including all nested scopes and independent trace imports; 1,024 cancelled nested dataflows match Rust. WorkerGroup cancels corresponding roots atomically while idle. ClusterWorker propagates any peer’s cancellation request at the next collective round, before record delivery; 128 two/three-process epochs exercise rotating requesters, nested scopes and live siblings.

Message capability timestamps are represented separately from record timestamps,
including input buffering, feedback, scope boundaries, custom batch operators and
keyed exchange. The version-4 cluster payload carries both; 208 two/three-process
snapshots check delayed future-record messages against Rust in sync and host-polled
execution. Batch boundaries remain scheduling-dependent, as upstream.

Shared probes can attach to several streams and combine their antichains, including
incomparable times, staggered closure and reuse after completion. 896 progress
snapshots match Rust and an independent model. Collection utilities include
mutable record mapping, stream concatenation (including an empty source list),
and consolidated equality/emptiness assertions.

Trace diagnostics use `Logger[DifferentialEvent]` and `Worker.differential_logs()`.
Register `differential/arrange` before constructing arrangements, then flush at
host polling boundaries. Logger replacement leaves existing operators attached
to their original sink. Standalone traces, spines and batchers accept a
`TraceLogger` and provide explicit disposal. Batch/merge/drop totals describe
this implementation's storage operations; exact merge schedules differ from Rust.
Pending record counts and reader shares are exact. Optional byte, capacity and
allocation fields remain `None` on managed backends rather than reporting invented
measurements. Event variants use descriptive names (`BatchCreated`, `MergeProgress`,
`BatchDropped`, `BatcherSize`) to avoid MoonBit constructor namespace collisions.

Columnar chunks use immutable `ColumnLists` and `UpdatesTyped` snapshots. Raw
four-level layouts support repeated and empty groups; canonical construction
streams sorted updates into columns. `merge_pair` returns the consumed prefix and
at most one suffix, removing cancellations immediately (upstream `merge_pair`
followed by `filter_zero`). Splitting and extraction copy selected column ranges
without building row arrays; meld streams leaves without flattening whole chunks.
Columnar arrangements now seal through this builder directly. 11,264 snapshots
across 1,024 randomized chunk cases match upstream's experimental columnar code
and independent sorted-multiset recomputation, including incomparable frontiers.

`ColumnarCodec` implements the pinned Rust `Updates<U>` byte framing and indexed
`Stash` columns, including compressed stride offsets. Built-in schemas cover
8/16/32/64-bit integers, UTF-8 strings/bytes, packed booleans, unit, vectors,
pairs/products, duration and double bit patterns; custom value layouts use
`ColumnCodec` callbacks. 256 fixtures are byte-identical to Rust, with both
decoders round-tripping and independent framing checks. Decoding validates bounds,
child counts and UTF-8, and bounds column counts; it currently materializes owned
columns rather than Rust's zero-copy byte views. This format targets the pinned
little-endian Rust layout, not a version-independent interchange standard.

`ColumnarCodec.spill_policy` connects the Rust columnar format to real paged
traces and native temporary files. The generic `SpillPolicy.binary` adapter also
supports application-defined byte encodings. Paged readers retain the existing
checksum/length checks, high-water budget and one-page cache; historical cursors
survive subsequent compaction. The 2,048 spilled-spine oracle epochs now use this
binary policy, and the complete native ASan/LSan suite passes with binary disk
spilling, shared graph/import lifetimes and multi-process workloads.

## Paused checkpoint

Work on this port is temporarily paused at the user's request while the Yrs port proceeds. Shared-reader compaction requests now join with existing guarantees, matching Rust for incomparable and backward requests (2,048 Rust/model rounds). The runtime gaps above remain unresolved; this checkpoint is not full upstream compatibility.

The public GitHub repository and Mooncakes 0.1.0 package expose this paused
checkpoint for experimentation. Publication does not change the completion
target or resolve the runtime gaps above.