# Changelog

## 0.7.1 — 2026-11-18

Closes Issue #2. `detect` decoded all thirteen code pages in full and then
scored all thirteen texts, which is the reliable design and also the ceiling:
6.2–8.9 MB/s depending on backend. The speed-up here does not remove any of
that checking — it removes work that was being done twice.

### Changed

- **The four wide-encoding evidence scans are one scan.** UTF-16LE/BE and
  UTF-32LE/BE each walked the whole input asking the same question about the
  same bytes. One loop over four bytes now answers all four. It is the same
  arithmetic, including the 2- and 3-byte tails, and `detect_wbtest.mbt`
  asserts it against the single-purpose versions so a future edit cannot drift.
- **`text_score` no longer re-walks the text for `exclusive_script`.** It had
  already counted ideographs, kana and hangul for the script signal; the
  exclusivity signal now reuses those counts instead of scanning again.
- **GBK reuses nothing, but GB18030 reuses GBK.** Every mapping GBK defines is
  the same mapping in GB18030 — checked slot by slot in `detect_wbtest.mbt`
  (21 791 shared, 2 149 added) — and a GBK decode with zero errors cannot have
  touched a four-byte form, because that form needs a digit where GBK allows no
  trail byte. So when GBK is clean, the GB18030 decode and its score are
  carried over instead of recomputed.
- **ISO-8859-1 reuses windows-1252 when no byte is in 0x80..0x9F.** That range
  is the only place the two pages differ.
- **`detect` 32 KiB: 3.18 ms → 2.20 ms (wasm), 7.09 ms → 4.57 ms (js).**
  1 KiB: 100.31 µs → 77.76 µs (wasm). Exact numbers and the js table are in
  `docs/BENCHMARK.md`.

### Verification

- `detect --json` compared byte for byte against the 0.7.0 binary over 395
  inputs: identical on all 395. The inputs include random byte strings with
  every 0x80..0x9F byte replaced, which is the only way to reach the
  windows-1252 reuse path from a synthetic file.
- 84 tests × 4 targets; accuracy 31/31 top-1; e2e 303/303; byte space
  2 351 159/2 351 159.

## 0.7.0 — 2026-11-17

Closes Issue #1. A tie used to cost two rows in the candidate table, so a
short Western file spent two of its top three lines saying the same thing
twice — once as `windows-1252`, once as `latin1` — and the actual runner-up
fell off the table. The two labels are not two hypotheses: they are one reading
with two names.

### Changed

- **A candidate row is a reading, not a label.** When two code pages decode the
  bytes to byte-identical text at the same score and neither byte range
  contains the other, they now share a row: `charset` holds the representative
  and the new `Detection::equivalent()` holds the other names. Affects
  windows-1252/latin-1 on Western text, GBK/EUC-JP on kana, and ASCII (where
  all ASCII-compatible pages are one reading).
- **`"also": ["latin1"]` on affected `--json` candidates.** The key is absent,
  not empty, when a row has one name, so a consumer that reads `charset` sees
  the same object it saw in 0.6.x. Adding a field does not bump `schema`.
- **`confidence` is 0 when the label is a coin flip.** The text may be certain;
  the name is not, and a number that says "85%" about an arbitrary choice is
  the kind of wrong answer the project is supposed to avoid. `score` and
  `errors` still describe the reading itself.
- **`is_ambiguous` is true when the winner has `equivalent` labels**, not only
  when the runner-up ties it. `--fail-if-not` already accepted any tied name
  and now accepts `latin1` for a file whose `charset` is `windows-1252`.
- **Toolchain moved to `moon 0.1.20261117` / `moonc v0.10.21+06da1ed44`.**
  `StringBuilder(size_hint=...)` is deprecated in favour of `capacity=`. CI
  installs the latest toolchain, so the old spelling turned the zero-warning
  gate red on every push; the four call sites are updated. No behaviour change.

### Unchanged, deliberately

- GB18030 is still *dropped* rather than merged when a file never leaves GBK's
  range. Containment is not equivalence: the wider label has no evidence at
  all, and listing it as an alias would claim the bytes are indifferent to a
  distinction they never exercised.
- Two decoders that both failed and salvaged the same prefix are not merged.
  Same output from two failures is not evidence; folding them together would
  hide that both wide readers rejected the stream.

### Verification

- 79 tests pass on wasm/js/native; three of them are new (the merge, the case
  that must *not* merge, and the JSON field).
- `accuracy_check.py` stays at top-1 100% (31/31). The two Western cases now
  report as ties instead of guessing a label, which is the honest answer.
- `e2e_check.py` 303/303 and `space_check.py` 2 351 159/2 351 159 unchanged —
  this release touches ranking only, not decoding.

## 0.6.4 — 2026-11-16

Closes the third item of the "what is deliberately unfinished" list: the
`--json` output now says which version of the shape it is. Additive, no
behaviour change for a consumer that ignores unknown keys.

### Added

- **`"schema":1` as the first key of every `--json` object**, from `detect`,
  `repair`, `scan` and `fix`. The key is first rather than appended so a
  consumer can decide before buffering the document. The number comes from one
  exported constant (`json_schema_version` in `src/mojibake/json.mbt`), not a
  literal repeated in four writers.
- The contract is one-directional and is stated in `docs/USAGE.md`: adding a
  field leaves the number alone, removing one or changing its meaning bumps it.
  This is what lets a caller written against 0.6.x notice that the shape moved
  instead of silently misreading a renamed field.
- CI step `JSON carries its schema version` runs all four commands, parses the
  output, and asserts both the value and that `schema` is the first key.
- Test `every json object carries the schema version first`.

## 0.6.3 — 2026-11-15

Two things a reviewer asks for and the project could not answer with numbers
before: how fast it is, and what it adds over what the ecosystem already does.
Both now have a harness in the repository and a gate in CI. Measuring the
second one immediately found a wrong answer, which is the point of measuring.

### Fixed

- **BOM-less UTF-16 holding CJK text was answered as `latin1`.** The
  conservative rule halved any UTF-16 reading that could not show the
  alternating-zero pattern of Latin text, and CJK in UTF-16 never shows it —
  its high bytes are 0x4E..0x9F. Correct Chinese was scored 43 and lost to the
  Latin-1 reading of the same bytes at 45. The full score is now reinstated
  when, and only when, all three hold: the decode is clean, one script owns at
  least 80% of the characters, and the UTF-16 reading beats *every* 8-bit
  reading by at least 20 points. The last condition is what stops the mirror
  image of this bug: GBK bytes read as UTF-16 land in Hangul and look like
  text too.
  Covered by `bomless utf-16 chinese is read as utf-16, not as latin-1`,
  `gbk bytes are not claimed as utf-16 by the same rule`, and
  `utf-16 is not believed when the text is not one script`.
- Detection accuracy on the labelled corpus went from 90.3% to 100% top-1 as a
  result; the three misses were all BOM-less UTF-16 CJK.

### Added

- **`benches/`** — six benchmarks over the public entry points (`decode`,
  `detect` at 1 KiB and 32 KiB, `detect` on UTF-8, `repair`, `encode`), run
  with `moon bench benches --release`. Fixtures are built outside the timed
  closure.
- **`scripts/accuracy_check.py`** — a 31-case labelled corpus across five
  scripts and eleven code pages, with two honest baselines ("always GBK", "BOM
  else GBK"), acceptable-answer sets for genuinely indistinguishable byte
  strings, a separate ASCII floor, and a `--min-top1` gate.
- **`docs/BENCHMARK.md`** — environment, both tables, how to read them, the
  bug the benchmark found, and a section on what the numbers do not prove.
- **`scripted_share`** — public helper returning the percentage of characters
  that belong to a script, counting only characters able to carry one. It is
  what the UTF-16 rule above is gated on, and it is exported because it is a
  reasonable question to ask of any text.

### Changed

- CI gained `python scripts/accuracy_check.py --min-top1 0.95`, so a change
  that trades accuracy for speed fails the build instead of shipping.
- `ACCEPTANCE.md` now maps the four 25-point scoring dimensions as published
  for this season (engineering depth, technical contribution, ecosystem delta,
  development and verification), replacing the previous four headings.

## 0.6.2 — 2026-11-14

Review package for the hackathon: everything a reviewer needs to check the
project without reading the source, plus one runnable program that exercises
the whole surface. No library behavior changed; 0.6.2 is 0.6.1 plus documents,
an example, and a CI step that runs it.

### Added

- **`ACCEPTANCE.md`** — the acceptance criteria (MoonBit-first, public repo,
  runs, substantive work, license, AI explainability) mapped to commands and
  evidence; a five-minute reproduction; a wrong-answer audit listing every
  failure mode this project has fixed and the test that pins it.
- **`docs/DEVELOPMENT.md`** — development timeline, which decisions were hard
  and what was rejected, where AI helped and where it was not allowed to
  decide, and the mistakes that only testing found.
- **`docs/TEST_RECORD.md`** — `moon version --all` output, then each check with
  its observed result: 72 tests on four targets, three builds, 303 CPython
  cross-checks, 2 351 159 exhaustive checks, CLI smoke, mooncakes install.
- **`examples/detect_and_repair`** — a runnable tour with fixed inputs and no
  file IO: decode, ranked detection, one repair chain, and a deliberate tie
  reported as a tie. `moon run examples/detect_and_repair`.

### Changed

- CI runs the example and asserts three lines of its output, so the documented
  tour cannot rot silently.
- README links the new documents and lists the runnable example.

## 0.6.1 — 2026-11-13

Maintenance release. Two things happened this week: a review note attached to
this project described it as a COSE implementation whose acceptance range is
limited to deterministic CBOR messages (RFC 9052) — that describes a different
project entirely, MoonMojibake has never contained any COSE or CBOR code — and
the MoonBit toolchain deprecated two syntaxes this codebase still used.

### Added

- **`Scope` section in the README**: what the project is (character-encoding
  detection, repair, transcoding) and what it is not (no COSE, no CBOR, no
  RFC 9052, no message signing or encryption; inputs and outputs are plain
  text in the thirteen code pages it supports).
- **Scope statement at the top of `docs/USAGE.md`**, same boundary in one
  paragraph.

### Changed

- **Toolchain migration to `moon 0.1.20261104` / `moonc v0.10.20+65efd5b93`,
  verified with `moon check --deny-warn` (zero warnings).**
  - `} derive(Eq, Debug)` on `Charset` became `#derive(Eq, Debug)` before the
    declaration (E0027: the trailing `derive(...)` syntax is deprecated).
  - Trait implementations are no longer implicitly promoted to regular
    methods (E0079): three explicit `pub extend Charset with ...` declarations
    re-export `Eq::{equal, not_equal}`, `Debug::to_repr` and
    `Show::{to_string, output}`, so `==`, `!=`, `str(...)` and `to_repr` on
    `Charset` keep working exactly as before.
  - Blackbox tests no longer get an implicit import of the package under
    test: all 245 unqualified references in `mojibake_test.mbt` are now
    written `@mojibake.decode`, `@mojibake.Charset`, and so on.
- **CI tightened**: type check now runs with `--deny-warn` (a warning fails
  the build), and the wasm backend is built alongside js and wasm-gc.

### Verified

- `moon check --deny-warn`: clean, zero warnings.
- 72 unit tests pass on all four targets (default, js, wasm, wasm-gc).
- CLI smoke test on the js build: `detect --text "中文测试 😀"` reports utf-8,
  confidence 100, as before.

## 0.6.0 — 2026-11-11

`gb18030` was listed as an alias of `gbk`. It is not one: GB18030 defines a
four-byte form that reaches every code point in Unicode, and GBK has no idea
what those bytes are. The alias promised a standard the decoder did not
implement, and `--to gb18030` refused every character GBK lacks — an emoji, a
rare hanzi, a CJK extension character. That is the same class of problem as a
documented feature that returns the wrong answer, so it is fixed here rather
than documented as a limit.

### Added

- **GB18030's four-byte form, both directions.** 1 087 996 sequences: the BMP
  is a 206-run table generated from CPython by decoding, and the supplementary
  planes are the arithmetic the standard specifies. `--to gb18030` now encodes
  any text at all, including characters outside the BMP.
- **`Charset::Gb18030`**, a real code page rather than an alias: its own
  two-byte table (GBK's, plus 2 149 private-use slots), its own reverse index,
  and the four-byte branch in both directions.
- **Exhaustive check of the four-byte space.** `space_check.py` walks every
  pointer the standard maps, decodes it through MoonBit, encodes it back, and
  compares both directions with CPython. The suite went from 127 287 checks to
  2 351 159.
- **Six new unit tests**: the four-byte form, the shared two-byte form, a round
  trip of text GBK cannot encode, an unmapped pointer, and the detection rule
  below.

### Fixed

- **A private-use character no longer buys a clean decode.** GB18030's 2 149
  private-use slots let it consume a byte pair GBK rejects, which earned the
  full "no errors" bonus. A windows-1252 sentence then scored 53 as GB18030
  against 52 as windows-1252 — the wrong page won by one point. Such a
  character now counts as a near miss, the same as an undecodable sequence.
- **GBK is reported for files that never leave GBK.** Without a rule, every
  Chinese file would be `ambiguous — gbk and gb18030`, since GB18030 decodes all
  of it to the same text. A wider page is now dropped when a narrower one
  produces byte-identical text, and only then.

## 0.5.0 — 2026-11-09

This release is about the answers a tool gives when it cannot give a good one.
Three of them were wrong, and wrong in the worst way: plausible, quiet, and
exiting 0.

### Fixed

- **Every failure path now exits non-zero.** `detect`, `repair` and `convert`
  printed `error: …` and returned 0, so a missing file, a misspelled encoding
  or an unrepresentable target all looked like success to a shell or CI job.
  Exit codes are now 0 success, 1 usage/IO error, 2 assertion failed.
- **Reading a directory no longer leaks a Node stack trace.** The js FFI tested
  `exists` only, and `readFileSync` on a directory throws EISDIR, so
  `mm detect some-dir/` printed `node:fs:736` instead of a diagnostic.
  `read_file` now reports "not a file" and each command owns its message.
- **A missing option value is an error, not a silent fallback.** `--from` at
  the end of a command line used to parse as "no encoding given", so detection
  ran and printed a confident answer to a question the user never asked.
  `--text` combined with a non-UTF-8 `--from` is rejected for the same reason:
  `--text` bytes are UTF-8 by construction, and reading them as GBK can only
  produce mojibake.
- **Tied readings are reported as ties.** EUC-JP hiragana occupies the JIS X
  0208 rows GBK keeps for the same characters, so kana-only bytes decode to
  identical text under both names. Detection named `gbk` at 85% confidence —
  a coin flip dressed as a finding — because the `exclusive_script` floor
  fired without checking whether the tie had actually been broken. It now
  requires `top > second`; the confidence is 0, `detect` prints the tied
  candidates, `--json` carries `ambiguous` and `tied`, and `--fail-if-not`
  accepts either name.

### Added

- `is_ambiguous(rows)`, `ambiguous_candidates(rows)` and `is_ascii_text(text)`
  in the library, so callers can act on a tie instead of inheriting one label.
- [`scripts/space_check.py`](scripts/space_check.py) — an exhaustive
  differential test against CPython in both directions: every lead/trail pair
  for GBK, Big5, Shift_JIS and EUC-KR, every JIS X 0212 cell, every half-width
  katakana spelling, and every character encoded back. 63 677 decodes and
  63 610 encodes, 127 287 checks, no missing rows, no wrong code points.
- CLI contract checks in `e2e_check.py`: 257 checks now, including exit codes
  and the "a directory is not a file" case.
- Unit tests 61 → 67.

## 0.4.0 — 2026-11-09

### Added

- **`repair_auto(bytes)`** — reads bytes nobody labelled and returns the encoding
  it settled on plus the repair result: UTF-8 when it decodes cleanly, detection
  otherwise, then the same repair search as `repair`. The sequence previously
  lived inline in `cmd_repair`, where no test could reach it. `AutoFix::changed()`
  answers whether writing the result back would change anything.
- **`mm fix <dir>`** — repairs a whole tree, which is the half `scan` was
  missing. Dry run by default; `--write` applies. Each file is reported as
  `transcode`, `repair: A -> B`, or both, and binary files are counted and left
  alone. A second run finds nothing to change.
- **`--fail-if-not <encoding>`** on `detect` and `scan` — turns the walk into an
  assertion for CI. Exits `2` when a file reads as anything else, `1` when the
  command itself was called wrong. `--json` carries `fail_if_not` and
  `assert_ok`.
- `--json` support for `fix`, and `fail_if_not` / `assert_ok` fields on `scan`.
- CI steps for both: the dry-run → write → re-run sequence must be idempotent and
  leave every file readable as UTF-8, and the gate must return `2` for `samples`
  and `0` for a directory holding only UTF-8.

### Changed

- `cmd repair` now calls `repair_auto` instead of repeating the read-then-repair
  sequence.
- Unit tests 57 → 61.

## 0.3.0 — 2026-11-09

### Added

- **EUC-JP**, in both directions, so the candidate set is now twelve and
  Japanese is no longer reachable only through Shift_JIS. Three forms are
  handled: JIS X 0208 in the shared pair table (6 879 mappings), JIS X 0212
  behind the `0x8F` prefix in a separate 94×94 table (6 067 mappings), and
  half-width katakana as `0x8E` + `0xA1..0xDF`, computed rather than tabulated.
  Encoding consults the pair table first, so a character mapped in both levels
  keeps its standard two-byte spelling and round trips are stable. Aliases:
  `eucjp`, `ujis`, `hp-ux-euc`.
- **JIS X 0208 level-1 char set** (2 965 characters) in `tables_gen.mbt`, used
  as the frequency model for Japanese. Generated from CPython like the rest.

### Fixed

- **Japanese files were attributed to GBK.** EUC-JP's byte range nests inside
  GBK's, and GBK reserves the same `0xA4`/`0xA5` rows for kana, so a Japanese
  file decoded as GBK with zero errors *and kept its kana* — a 55-byte sample
  tied at 88/88 and lost only on candidate order. The first attempt at a fix,
  "are these kanji in the JIS repertoire at all", does not work: JIS holds about
  7 000 ideographs and therefore most Chinese hanzi too. The scorer now checks
  the kanji against JIS level 1 and subtracts up to 30 points, measured over the
  ideographs only, and only when the text contains kana. Text without kana is
  untouched, so GBK / Big5 / EUC-KR detection is unchanged.

### Changed

- `Charset::family()` is no longer one "CJK" bucket: Shift_JIS and EUC-JP report
  `Japanese`, EUC-KR reports `Korean`. The JSON output carries the new values.

## 0.2.0 — 2026-11-06

### Added

- **UTF-32LE / UTF-32BE** — decoding, encoding, detection and BOM handling, so
  the candidate set is now eleven. UTF-32 needs its own evidence rule (at least
  half of the code units must have a zero high half) and a BOM check that runs
  *before* UTF-16, because `FF FE 00 00` starts with the UTF-16LE BOM.
- **`scan`** — walks a directory, tallies every file by encoding, and reports
  non-text files as `binary` instead of force-decoding them into nonsense.
  Accepts `-r` to recurse. Unreadable and empty files are reported, not fatal.
- **`--json`** on `detect`, `repair` and `scan` — one compact object per run,
  stable field names, CJK emitted as-is. `repair.chain` is an array of
  `{wrong, right}` steps rather than a rendered string, so callers can inspect
  individual layers.
- **`looks_binary`** — public NUL-byte test, waived for UTF-16/32 shaped input.

### Fixed

- **Repair could translate Chinese into Korean.** The greedy search is allowed
  four rounds, and a later round could take already-correct 你好, re-encode it as
  EUC-KR, decode it back as Hangul, and call that an improvement — zero errors
  and a higher score, because Hangul has no frequency model to hold it back.
  Fixing the symptom by removing UTF-16 as a target was not enough. Each round
  now computes the dominant script of its input and refuses a step that swaps
  Han, Hangul or kana for a different one. Latin and script-less input may still
  go anywhere; that is the case repair exists for. Caught by the new JSON test,
  not by any of the existing ones.

- **CI was red from the first commit, and the cause was a reproducibility bug.**
  The "regenerated tables are up to date" job hashed differently on CI than on
  the author's machine, because `gen_tables.py` topped up the common-character
  set from a corpus file that sat outside the repository. That file existed
  locally and not in CI, so the committed table held one character CI could not
  reproduce. The generator now reads nothing but the codecs and the standards,
  the set is exactly 9 156 characters, and CI pins Python 3.12 because the codec
  tables themselves are part of the input.

### Changed

- 31 → 48 unit tests.
- CLI usage text lists `scan`, `-r` and `--json`.

## 0.1.0 — 2026-11-01

First release.

### Added

- **Decoders** for UTF-8, UTF-16LE/BE, GBK, Big5, Shift_JIS, EUC-KR, ISO-8859-1
  and windows-1252. Every decoder is total: malformed input becomes U+FFFD and
  is counted, never raised, so candidate readings can be compared.
- **`detect`** — ranks all nine candidates by a five-signal text-likeness score,
  with a confidence derived from the gap to the runner-up. Handles BOMs
  exactly, treats valid multi-byte UTF-8 as proof, and requires BOM-less UTF-16
  to show the alternating-zero pattern.
- **`repair`** — searches `(wrong, right)` re-encode/decode pairs, accepts only
  zero-error candidates with a real score gain, and chains up to four layers.
  Returns the chain, not just the result.
- **`encode`** — reverse lookup for all supported code pages; `None` when a
  character has no representation, which repair uses to prune its search.
- **CLI** with `detect`, `repair` and `convert`, plus `--text` so the tool still
  runs on backends without a filesystem.
- **Generated tables** — 50 605 two-byte mappings built from CPython codecs by
  `scripts/gen_tables.py`, committed for reproducibility.
- **Common-character set** — 9 157 characters derived from the GB2312 and Big5
  level-1 ranges, used as the frequency signal without shipping a corpus.
- 31 unit tests and a 192-check end-to-end comparison against CPython.

### Design notes worth keeping

- Sorting ties keep `Charset::all()` order, so ASCII resolves to UTF-8.
- Half-width katakana is excluded from script-neutral characters; otherwise a
  Chinese code page read as Shift_JIS looks like ordinary punctuation.
- UTF-16 is deliberately absent from repair targets; allowing it made repair
  turn Chinese into Hangul.
