# tools

Generators for the data files in the package root. Their output is
committed.

## gen_profiles.mjs

Writes `profiles.mbt`: the trigram profiles of franc-min 6.2.0 and the key
order of its `expressions.js`. Needs Node.js. franc-min is installed into a
temporary directory and not committed; the script refuses any other
version.

```
npm install --prefix "$TMP/franc" --no-save --ignore-scripts franc-min@6.2.0
node tools/gen_profiles.mjs "$TMP/franc/node_modules/franc-min" profiles.mbt
moon fmt
```

The `franc data` job in CI runs exactly these commands and fails if
the result differs from the committed `profiles.mbt`.

## gen_golden.mjs

Writes `golden_wbtest.mbt`: franc-min 6.2.0 `francAll` results for franc's
test fixtures in supported languages and for the cases in
`golden_samples.json`. Needs Node.js. The fixtures file is downloaded from
a pinned commit and checked against its SHA-256.

```
npm install --prefix "$TMP/franc" --no-save --ignore-scripts franc-min@6.2.0
curl -fsSL -o "$TMP/fixtures.js" https://raw.githubusercontent.com/wooorm/franc/3f9f0b51a96c5df32a407dad865a3011ac0fa2d1/test/fixtures.js
node tools/gen_golden.mjs "$TMP/franc/node_modules/franc-min" "$TMP/fixtures.js" golden_wbtest.mbt
moon fmt
```

A sample in `golden_samples.json` has `name` and `text`, and optionally
`repeat` (repeat `text` that many times), `suffix`, `allow`, `deny` and
`minLength`. The `franc data` CI job regenerates this file too, and
uploads the regenerated files as an artifact when they differ.

## unicode_case.py

Writes `unicode_case.mbt`: simple lowercase mappings and the `Cased` and
`Case_Ignorable` properties from the Unicode Character Database, which
`trigram.mbt` uses to reproduce JavaScript's `toLowerCase`. Needs Python 3
and network access; uses only the standard library.

```
python tools/unicode_case.py unicode_case.mbt
moon fmt
```

The Unicode version is pinned in the script (`UNICODE_VERSION`). The script
stops if that version adds an unconditional full lowercase mapping other
than U+0130, which `trigram.mbt` handles by hand.

## loc.py

Counts lines of MoonBit by kind and prints a table. Needs Python 3; uses
only the standard library. `-v` lists every file.

```
python tools/loc.py
```

Files whose first line says `Do not edit` (written by the generators
above) and `script_ranges.mbt` (transcribed from franc-min) count as
generated; `*_test.mbt` and `*_wbtest.mbt` as tests; the `cmd`,
`examples`, `demo` and `eval/run` packages as apps; everything else as
core library code. Non-blank lines and code lines (non-blank lines that
are not only a comment) are counted separately.

## gen_tatoeba.py

Writes `eval/tatoeba/corpus.mbt`, the benchmark corpus: Tatoeba sentences
(CC BY 2.0 FR, see NOTICE) for mbglot's languages in three lengths. Needs
Python 3 and network access; uses only the standard library. The
per-language exports are downloaded into the given cache directory, which
is not committed.

```
python tools/gen_tatoeba.py "$TMP/tatoeba" eval/tatoeba/corpus.mbt
moon fmt
moon run eval/run > eval/results.md
```

Sentences are picked in the order of SHA-256 of `<code>:<id>`, so the same
export gives the same corpus. Tatoeba updates its exports weekly, so a
later export can change the selection where sentences were removed; the
file header records the export date. The script's docstring lists the
length classes and the mapping from mbglot's language codes to Tatoeba's.

## gen_flores.py

Writes `eval/flores/corpus.mbt`: 40 FLORES-200 devtest sentences (CC
BY-SA 4.0, see NOTICE) for each of mbglot's languages that FLORES-200
has, the same sentence positions in every language. Needs Python 3 and
the unpacked dataset; uses only the standard library.

```
curl -fsSL -o "$TMP/flores200_dataset.tar.gz" https://dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz
sha256sum "$TMP/flores200_dataset.tar.gz"   # b8b0b767...44011f6
tar -xzf "$TMP/flores200_dataset.tar.gz" -C "$TMP"
python tools/gen_flores.py "$TMP/flores200_dataset" eval/flores/corpus.mbt
moon fmt
moon run eval/run > eval/results.md
```

The positions are the first 40 in the order of SHA-256 of
`flores:<position>`. The archive's full SHA-256 is
`b8b0b76783024b85797e5cc75064eb83fc5288b41e9654dabc7be6ae944011f6`.

## gen_full.mjs

Writes `full/data.mbt` (the Unicode ranges of franc 6.2.0's 30 script
expressions, decoded by testing every code point, and its 163 trigram
profiles) and `full/golden_test.mbt` (the ten best `francAll` entries and
their number for franc's fixtures in the supported languages and the
cases in `golden_samples.json`). Needs Node.js; the fixtures are the
same pinned file as for `gen_golden.mjs`.

```
npm install --prefix "$TMP/franc-full" --no-save --ignore-scripts franc@6.2.0
node tools/gen_full.mjs "$TMP/franc-full/node_modules/franc" "$TMP/fixtures.js" full
moon fmt
```

The `franc data` CI job regenerates both files and fails if they differ.

## bench_franc.mjs

Times franc-min 6.2.0's `francAll` on the texts of
`eval/tatoeba/corpus.mbt`, for comparison with
`moon run eval/speed --release --target js`. Needs Node.js.

```
npm install --prefix "$TMP/franc" --no-save --ignore-scripts franc-min@6.2.0
node tools/bench_franc.mjs "$TMP/franc/node_modules/franc-min"
```

## readme_tables.py

Checks that `README.mbt.md` quotes `eval/results.md` correctly: the
tables copied from it (the Tatoeba test half, FLORES-200, custom
profiles) and the summary under "At a glance". After rerunning the
benchmark, `python tools/readme_tables.py --write` updates the README.
The `franc data` CI job runs the check.