# TideWeave

**A generic, embeddable event-time windowing library — windows, watermarks, UDAFs, triggers and state cleanup, aligned with the Flink model — plus a correction layer most windowing libraries lack: retraction rebuild, finality protection and before-image validation. Events carry arbitrary structured fields; the numeric measure is field-configurable.**

A MoonBit event-time windowing and correction-log engine. Late events merge sessions; retracting a bridge event splits them again. Every atomic correction emits a before/after changelog, an explicit watermark protects finalized facts, and a downstream view checks version continuity and stale before-images.

[Live laboratory](https://lmk-ai-nb.github.io/TideWeave/) · [Chinese README](README.md) · [Flink mapping](docs/FLINK_MAPPING.md) · [Semantics](docs/SEMANTICS.md) · [Protocol](docs/FORMAT.md)

## A concrete problem

Two records at 0 and 16 with gap 10 form separate sessions. A late record at 8 merges them; retracting that record splits them again. Subtracting a number from a stale SUM leaves wrong boundaries, members and min/max. TideWeave treats every correction as an atomic state change: compute candidate facts, rebuild affected keys, validate finality, commit a version and emit the full before/after changelog. A failed transaction leaves windows, indexes, dedup state and versions untouched.

## Implemented

- Tumbling, phase-offset sliding and session windows; half-open intervals; mathematical floor alignment for negative starts.
- Insert, replace and retract by event ID; cross-key moves; multi-mutation atomic transactions.
- Session late-merge and retraction-split; count / Int64 sum / min / max and full member provenance.
- Explicit watermark advancement and allowed-lateness tolerance; `WatermarkTracker` for bounded-out-of-orderness watermark generation; `within_lateness` probe.
- Pluggable custom aggregation (UDAF): `Measure` + `measure_window`, with built-in count/sum/min/max/avg.
- Arbitrary structured events: events carry `fields` key/value attributes preserved through checkpoints and changelogs (flat key/value, at most 32 entries per event; serialize nested structures into a string); configure `value_field` to make any integer field the window measure (no fixed `value` required).
- Configurable trigger: `Watermark` or `Count(n)` early firing; before/after changelog output guarded by the downstream `View`.
- Explicit state cleanup: `Forget` freezes finalized facts, frees active capacity, rejects later corrections, and replays deterministically through checkpoints.
- open → closing → final; finalized windows and their member facts are protected from reopening.
- Success-command idempotency and conflict detection; replayable checkpoint recovery preserving dedup.
- Deterministically ordered before/after changelogs; a downstream `View` that validates version continuity, before-images and atomic application.
- Strict JSON: duplicate decoded keys, unknown fields, fractional-token rounding and inapplicable parameters are rejected.
- Node CLI, six reproducible examples and a six-scenario interactive browser laboratory, all sharing the MoonBit core.

See [Flink mapping](docs/FLINK_MAPPING.md) for a line-by-line mapping against Apache Flink's event-time windowing capabilities.

## Why not just use Flink

Flink is a distributed cluster runtime for large, long-running, multi-tenant streams. TideWeave addresses a different problem: **when an already-recorded fact is corrected, how do you produce a verifiable, replayable correction the downstream can check?**

| Scenario | Flink's usual approach | TideWeave |
| --- | --- | --- |
| Session boundary after retracting a bridge event | retract stream left to downstream; boundaries not re-split automatically | rebuilds connected components, emits full delete + upsert |
| Finalized window rewritten by late data | relies on allowed lateness and downstream idempotency | final windows and member facts explicitly protected, even for equal sums |
| Downstream misses or holds a stale value | build your own version check | `View` atomically validates version continuity, before-image and row identity |
| Deployment | JVM cluster, state backend, checkpoint storage | single-process embeddable, deterministic replay, no external dependency |

Every Flink windowing primitive here is implemented to parity; the correction/retraction rebuild, finality protection and downstream validation are the independent contribution. The two operate at different deployment layers rather than replacing each other.

## Using from MoonBit

Requires MoonBit compiler 0.10.16+.

```sh
moon add LMK-ai-nb/tideweave@0.2.4
```

```moonbit
import { "LMK-ai-nb/tideweave" @tide }

let cfg = @tide.config(@tide.Session(gap=10), lateness=20, value_field=Some("watts")) // measure comes from fields.watts
let engine = @tide.Engine::new(cfg)
let view = @tide.View::new()
let outcome = engine.apply(@tide.Transact(id="batch-1", changes=[
  @tide.Put({ id: "left", key: "device-A", at: 0, value: 20, fields: Some([("watts", "20")]) }),
  @tide.Put({ id: "right", key: "device-A", at: 16, value: 35, fields: Some([("watts", "35")]) }),
]))
view.apply(outcome)
let restored = @tide.restore(engine.checkpoint())
assert_true(restored.rows() == engine.rows())
```

The full public interface is generated by `moon info` (see [pkg.generated.mbti](pkg.generated.mbti)). `aggregate` batches; `Engine` accepts corrections within capacity; `replay_json` / `resume_json` provide strict text boundaries.

## When you actually need this: three concrete settings

First, what the library does not do: it does not "fix a row that is already in the database" — that is an UPDATE's job. It handles something else: **keeping windowed results consistent when the correction itself arrives as a stream, and being able to tell the reconciling party exactly what changed**. Three concrete settings:

| Setting | What the database or business system does | Why it falls short |
| --- | --- | --- |
| Device gateways, edge controllers, desktop tools, CI fixtures | There is no database to correct | A single process on site, no DB and no cluster, yet late merges, retraction splits and window boundaries still have to be right |
| Ad billing, refund clawbacks, order amendments | Periodic full recomputation, or UPDATE plus a separate report | Corrections are not one-off: invalid-traffic retractions, refunds and backfills each arrive as streams, out of order and duplicated; replaying the command log would double-deduct |
| Reconciliation, audit and risk work that must explain *what changed* | Row-level after-values, or digging through binlogs | No member-level before/after, and no topology conclusion such as "these two sessions should be split"; the downstream cannot reject stale values before applying them |

Conversely, several cases should **not** use this library: if you only need a daily report and a full recompute is acceptable, use a warehouse; if you need cluster throughput and multi-tenant isolation, use Flink; if you need shared state across machines and high availability, this project is a single-process embeddable engine and is the wrong shape.

One set of window, watermark and retraction semantics already covers three unrelated domains — device sessions (`session-correction`), orders (`order-amendment`) and ad billing (`ad-billing`). Only `fields` and `value_field` differ; the engine knows nothing about domains, only `(key, at, structured fields)`.

Doing this in a database means materialized views, triggers, a self-built retraction stream and version checks, with event-time semantics scattered through application code. And an UPDATE is not replayable on its own: idempotency needs a separate command-dedup layer. That dedup layer, the finality guards and the changelog are exactly what this library provides.


## Relationship to existing MoonBit packages

August reviews rejected several submissions for overlapping with existing projects without disclosing the relationship. This project discloses the search results up front — see [Prior art](docs/PRIOR_ART.md).

Related MoonBit packages each have a distinct role: `moonwatermarkkit` provides watermark decisions and window assignment primitives (no stateful aggregation, no retraction); `moon-stream-quality` is a streaming data-quality rule engine with late-event policies; `moon-distinct-sketch` provides approximate sketches over bounded windows; `mooncep` performs complex-event pattern detection and explicitly does not support event corrections or retractions.

TideWeave shares no code with them and is not a fork, rename or extension of any of them; window assignment, watermark advancement, aggregation, correction and checkpoints are implemented independently. The difference concentrates in the correction layer: session rebuild after retraction, atomic multi-event transactions, finality protection and before-image validation. If you only need watermark decisions or data-quality rules, those dedicated packages are a better fit; if you need "a corrected fact must leave the windows consistent", use TideWeave.

## Two integration paths, no rewrite required

1. **Embed as a library**: `moon add LMK-ai-nb/tideweave` and call `Engine::apply` from MoonBit.
2. **Drive it as a pipeline**: `stream` consumes newline-delimited JSON (first line config, one command per later line) and emits one outcome per line plus a final summary and checkpoint. Any line-oriented producer — shell, `jq`, a Kafka consumer script, Python — can drive it directly:

```sh
cat examples/session-stream.ndjson | node bin/tideweave.mjs stream
node bin/tideweave.mjs stream examples/session-stream.ndjson --checkpoint saved.json
```

The same core backs the typed MoonBit API, the strict JSON protocol, the CLI and the browser laboratory. See [HOST_GUIDE.md](docs/HOST_GUIDE.md) for host-side examples.

## Build and verify

```sh
moon check --target all --deny-warn
moon test --target wasm-gc,wasm,js --deny-warn
moon build --target wasm-gc,wasm,js --deny-warn
moon fmt --check
moon info
node scripts/check_identity.mjs
moon run cmd/demo
node scripts/build.mjs
node --test tests/cli.test.mjs
node scripts/serve.mjs
```

The browser runs at `http://127.0.0.1:4174`. Node 22+; no third-party npm dependencies.

These targets are deliberately restricted to wasm-gc / wasm / js so the block runs anywhere. `--target all` pulls in native, which needs a platform C toolchain (MSVC Build Tools plus the Windows SDK on Windows); without one, `moon test --target all` fails with `requires an MSVC compiler/linker driver`. Native is verified on all three operating systems by CI.

```sh
node bin/tideweave.mjs run examples/order-amendment.json --checkpoint checkpoint.json
node bin/tideweave.mjs run examples/session-correction.json
node scripts/benchmark.mjs

# Resume: a checkpoint is not a request; wrap it first
node -e "const c=require('./checkpoint.json');require('fs').writeFileSync('resume.json',JSON.stringify({schema_version:1,checkpoint:c,commands:[]}))"
node bin/tideweave.mjs resume resume.json
```

CLI exit codes: 0 for all-accepted/duplicate, 1 for rejected/conflict, 2 for input/IO/runtime errors. `resume` takes a request object `{schema_version:1,checkpoint:<saved object>,commands:[...]}`; passing the bare checkpoint file yields `unknown or inapplicable field`. The full format is in [FORMAT.md](docs/FORMAT.md).

## Quality and boundaries

70 MoonBit tests include an independent interval-enumeration oracle, an independent graph-connectivity oracle, mixed-key state diffs, atomic rollback, finality protection, checkpoint recovery, custom aggregation, watermark generation, trigger and state freezing. 10 Node integration tests cover the public protocol and real CLI; Chromium checks interaction, export, injected text and a 390px phone layout. GitHub CI checks wasm, wasm-gc, js and native on Linux, macOS and Windows.

This is a **bounded, synchronous, single-process** embeddable state machine for offline reconciliation, edge batches and stream-processing regression fixtures. At most 5000 active events, 10000 successful commands, 20000 windows, 2 Mi canonical journal units. State cleanup is explicit: `Forget` freezes finalized facts and frees active capacity; frozen facts become replayable immutable history — never silently evicted. Continuous unbounded streams require the host to design partition/batch lifecycles; the host also owns message IO, persistence, authorization and external consumption consistency. There is no distributed runtime and no external exactly-once guarantee.

Windows/Node 24 synthetic benchmark: 4096 events, 64 keys, 32 batches, including parsing, correction, logging, checkpoint and full JSON output — median ~84 ms over 10 samples. See [BENCHMARK.md](docs/BENCHMARK.md) for environment, scope and reproduction; do not extrapolate to production throughput.

## Verifiable evidence

Every claim maps to runnable code or a test, not just prose:

| Claim | Evidence |
| --- | --- |
| Session rebuild after retraction (not numeric subtraction) | `node scripts/compare_naive.mjs`; `examples/session-correction.json` |
| Finalized windows and member facts are protected | `boundary_wbtest.mbt`, `engine_wbtest.mbt` |
| User-defined aggregates (UDAF) | `aggregate.mbt`, `aggregate_wbtest.mbt` |
| Watermark generation and lateness probing | `watermark.mbt`, `watermark_wbtest.mbt` |
| Trigger (Watermark / Count) | `trigger_wbtest.mbt` |
| State freezing and deterministic replay | `forget_wbtest.mbt`; `moon run cmd/demo` ends with `frozen=2, active=0` |
| NDJSON streaming | `examples/session-stream.ndjson`, `tests/cli.test.mjs` |
| Structured events and configurable measure | `structured_wbtest.mbt`; `examples/structured-events.json`; `API.mbt.md` (compiled by tests) |
| Commit authorship | `node scripts/check_identity.mjs` |

Full suites: `moon test --target wasm-gc` (58 tests), `node --test tests/cli.test.mjs` (10 tests).

## Originality and declaration

Maintainer: Li Mingkun / `LMK-ai-nb`. An independent new repository and implementation that reuses no core from submit-guard, permscope or AgentCanary. Event time, windowing and changelogs are established concepts; the independent contribution is the combination of corrections, session splitting, transactional finality protection, downstream before-image validation and restart deduplication. No claim to have invented stream processing.

Developed with AI assistance; the maintainer understands the implementation and owns verification. Original code is MIT licensed, standard-library build artifacts retain their licenses and NOTICE, and all examples use synthetic data. See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md).

The October application form requires a one-page Markdown statement and explicitly says not to write it with AI. The factual documents in this repository are reference material; the official statement must be written by Li Mingkun after understanding the project. See [SUBMISSION.md](docs/SUBMISSION.md) and [DEMO.md](docs/DEMO.md).