# MoonSearch Browser Wasm Demo

Hosted demo: <https://lucius646.github.io/MoonSearch/>

MoonSearch 0.9.0 runs the public facade as a browser-facing WasmGC
foreign library inside a dedicated Web Worker. The main thread only collects
query options, reports progress, supports cancellation, and renders structured
results; offline corpus and analyzer-resource loading, Gzip decompression, analysis, indexing,
Term/Boolean/Phrase/Query-string construction, BM25 scoring, and UTF-8 highlighting
run away from the UI thread.

## Run locally

Requirements: Node.js 22, a current MoonBit toolchain, and a modern Chromium
browser.

```bash
npm run demo:build
npm run demo:serve
```

Open <http://127.0.0.1:4173>. The page loads `web/sample-documents.jsonl`
automatically. The dataset selector can replace it with Chinese Wikipedia
4,000, English Wikipedia 4,000, the combined 8,000 article corpus, pasted JSONL,
or a local `.jsonl` / `.jsonl.gz` file. Each non-empty line must be an object
whose `title` or `body` value is a string or an array of strings.

The Wikipedia choices only fetch static files from the same local Demo origin.
The browser does not call Wikimedia, Hugging Face, or another data API. Cancelling
a load terminates the Worker, discards the partial index, and starts a clean
runtime.

## Analyzer profiles

The Analyzer profile is fixed in the `title` and `body` field schema and is
reused for query parsing and highlighting. Dataset auto-selection resolves to:

| Profile | Default corpus | Pipeline |
| --- | --- | --- |
| `default_v1` | manual opt-in | Simple + lowercase |
| `en_stem_v1` | English 4,000 | Simple + lowercase + Porter2 |
| `zh_search_v1` | Chinese 4,000 | frequency-DAG + Chinese search-mode graph + lowercase |
| `multilingual_v1` | sample and combined 8,000 | Chinese DAG/search-mode plus Porter2 on supported Latin tokens |

Chinese profiles load `web/data/demo-zh-lexicon.txt` from the same origin. It
contains 137 project-authored demonstration terms and intentionally does not
copy an unverified third-party dictionary. Index-time search-mode subwords are
flattened at their containing word position, while query-time analysis uses the
precise DAG route. This preserves cross-word Phrase semantics and deliberately
does not treat two subwords inside one dictionary word as an adjacent phrase.
The page exposes both token streams through the Analyzer preview.

The library supports injected B/M/E/S HMM models, but the demo does not ship a fake
or test-only model as production language data. Activating HMM in this Demo
requires a separately reviewed trained resource with clear source and license.

## Offline Wikipedia snapshots

The large demo corpus is generated during development and served as a local
static asset, so visitors do not call Wikimedia APIs. From the repository root:

```bash
npm run data:wikipedia
```

This creates Chinese and English snapshots with 4,000 articles each under
`web/data/`, in JSONL and Gzip form, plus a manifest containing counts, byte
sizes, SHA-256 checksums, selection rules, and attribution details. The generator
uses HTTP Range requests against the Wikimedia `20231101` Parquet snapshot, so
it does not download the complete source shards. See
[`web/data/README.md`](web/data/README.md) before replacing or redistributing a
snapshot.

## Query modes

- `Term` requires exactly one analyzed term.
- `Boolean · OR` matches any analyzed term.
- `Boolean · AND` requires every analyzed term.
- `Phrase` performs a positional phrase query.
- `Query-string` strictly parses field scopes, double-quoted phrases,
  phrase slop, inclusive/exclusive/open ranges, wildcard/regex/fuzzy terms,
  parentheses, uppercase `AND`/`OR`/`NOT`, `+`/`-`, boosts, and backslash
  escapes. The Field selector supplies the default field.

All successful searches return ranked hits with BM25 scores. The four
structured modes also return UTF-8 byte ranges, which JavaScript maps to text
nodes and `<mark>` elements without inserting source text as HTML. Query-string
v1 renders safe stored text without marks because nested multi-field and
negative-clause highlighting needs a separate query-tree-aware contract.

## Performance baseline

Run the opt-in large-corpus suite with:

```bash
npm run benchmark:browser
```

The suite measures download, Gzip decompression, indexing, total loading, and a
representative query in a headless Chromium Worker. One local development run
after replacing the writer's linear term lookup with an internal hash table
produced the following indicative figures; these are not cross-machine limits:

| Corpus | Documents | Index | Total | Query |
| --- | ---: | ---: | ---: | ---: |
| English subset / `en_stem_v1` | 500 | 582 ms | 657 ms | 14.0 ms |
| 中文 Wikipedia / `zh_search_v1` | 4,000 | 2.86 s | 3.01 s | 6.2 ms |
| English Wikipedia / `en_stem_v1` | 4,000 | 3.41 s | 3.52 s | 11.7 ms |
| Combined / `multilingual_v1` | 8,000 | 6.83 s | 7.12 s | 10.2 ms |

The 500-document case still transfers the English 4,000-article Gzip asset and
then limits the indexed JSONL. It is an indexing-scale baseline, not a 500-row
network payload benchmark.

## Verify

```bash
npm install
npm run test:data
npm run test:wasm
npm run test:browser
npm run benchmark:browser
```

These are optional presentation-layer checks and are not part of the main CI
gate. `test:wasm` builds the release WasmGC artifact and checks its exported ABI and
JS-string imports. `test:browser` covers bilingual Analyzer quality, strict
Query-string execution and errors, sample search, the 4,000-document
same-origin offline path, UI responsiveness,
cancellation, invalid JSON, and a Wasm loading failure. `benchmark:browser` is kept separate from the default test
suite so CI can choose when to pay for all four corpus sizes. The normal MoonBit
checks remain:

```bash
moon check --target all --deny-warn --frozen
moon test --target all --deny-warn --frozen
moon bench benchmarks --build-only --target all --deny-warn --frozen
```

## Scope

The demo keeps one replaceable in-memory index with fixed `title` and `body`
fields and one explicit Analyzer profile. The library provides lenient parsing
and query-tree-aware highlighting, while this UI deliberately uses strict
Query-string parsing and renders those results without marks. The demo is not a persistent browser database, server
deployment, production-scale search service, or a claim that the compact
demonstration lexicon matches a full production Chinese dictionary.