# Independent corpus evaluation: `nst/JSONTestSuite`

Generated by `code/tests/eval/run_independent.py` on 2026-10-16 02:06 CST.

| field | value |
| --- | --- |
| corpus | `nst/JSONTestSuite/test_parsing` (third-party, not authored by us) |
| upstream | <https://github.com/nst/JSONTestSuite> |
| tarball (one-request alternative) | <https://codeload.github.com/nst/JSONTestSuite/tar.gz/refs/heads/master> |
| vendoring method | `git ls-remote` + `git clone --filter=blob:none --sparse`; see `PROVENANCE.md` |
| upstream commit | `1ef36fa01286573e846ac449e8683f8833c5b26a` |
| license | MIT License, Copyright (c) 2016 Nicolas Seriot |
| vendored files | 318 (`y_` 95, `n_` 188, `i_` 35) + `LICENSE` + `README.md` + `PROVENANCE.md` |
| vendored `test_parsing` bytes | 354024 |
| corpus fingerprint | `0eb026859149fec126f0d1e5312d696a9765164f9076025eeff554f4afec60f1` (sha256 over sorted `name\0bytes`) |
| strict oracle | `json.loads` with `NaN`/`Infinity` rejected via `parse_constant` |
| secondary strict oracle | Node `JSON.parse` |
| JSONMend | `./_build/native/release/build/cmd/jsonmend/jsonmend.exe`, sha256 `30ba263133108f6f9abac99ad4f8387784085ead630de1adb9ddb06c98c3bae8` |
| Python oracle | `json_repair` 0.64.0 |
| Node oracle | `jsonrepair` 3.15.0 (NODE_PATH=/private/tmp/node_modules) |
| network | not needed |

Every `test_parsing/*.json` file is fed to JSONMend on stdin, byte for byte, in
default repair mode with `--report`. The two repair oracles are asked the same
question the repository's other comparisons ask them: repair the text, then require
the result to be strict JSON, because the claim under test is that *a caller can now
parse it*, not that the library returned a string.

## Summary: family x implementation

| family | implementation | accept | repair | reject | error |
| --- | --- | --- | --- | --- | --- |
| **`y_`** (95 files) | strict `json.loads` (classifier) | 95 | 0 | 0 | 0 |
| | JSONMend (native) | 95 | 0 | 0 | 0 |
| | `json_repair` 0.64.0 (Python) | 94 | 0 | 1 | 0 |
| | `jsonrepair` 3.15.0 (Node) | 95 | 0 | 0 | 0 |
| **`n_`** (188 files) | strict `json.loads` (classifier) | 0 | 0 | 188 | 0 |
| | JSONMend (native) | 0 | 165 | 23 | 0 |
| | `json_repair` 0.64.0 (Python) | 0 | 166 | 20 | 2 |
| | `jsonrepair` 3.15.0 (Node) | 0 | 144 | 44 | 0 |
| **`i_`** (35 files) | strict `json.loads` (classifier) | 26 | 0 | 9 | 0 |
| | JSONMend (native) | 23 | 0 | 12 | 0 |
| | `json_repair` 0.64.0 (Python) | 21 | 9 | 5 | 0 |
| | `jsonrepair` 3.15.0 (Node) | 23 | 9 | 3 | 0 |

- **accept**: the file is strict JSON *and* the implementation returned a valid
  value, i.e. nothing had to be repaired.
- **repair**: the file is not strict JSON, but the implementation still returned a
  valid value.
- **reject**: the implementation refused, or returned text that is not JSON.
- **error**: the implementation crashed or timed out.

The strict row is the classifier, not a repairer, so it has no repair column by
construction. For `y_` and `n_` the corpus label and the strict oracle agree exactly
(0 disagreements), which is what makes the accept/repair split meaningful; the `i_`
family is free by definition, so any split there is ours and not the corpus's.

## Headline: repairs reported on valid JSON (`y_*`)

A document that is valid JSON needs no repair, so every repair record JSONMend
reports on a `y_` file is a **false repair**.

| metric | value |
| --- | --- |
| `y_` files | 95 |
| accepted by the strict oracle | 95 |
| produce a valid value under JSONMend | 95 |
| **files with at least one repair record** | **2** |
| **repair records in total** | **2** |
| `--check` exits 1 (repairs needed) on | 2 |
| refused by `--for-execution` (exit 5) | 1 |
| **refused with prose skipping on (`--skip-prose`, exit 3)** | **0** |

**2 of 95 `y_*` files get 2 repair record(s) in total.** Each one is listed here
with the record, its declared exactness, the `--check` status and the
`--for-execution` verdict:

| file | input | repair record | exactness | `--check` | `--for-execution` |
| --- | --- | --- | --- | --- | --- |
| `y_object_duplicated_key.json` | `{"a":"b","a":"c"}` | `R018 1:10 duplicate key merged` | lossy | 1 | refuse (5 E010) |
| `y_object_duplicated_key_and_value.json` | `{"a":"b","a":"b"}` | `R018 1:10 duplicate key merged` | cosmetic | 1 | accept (0) |

Every record is `R018 MergeDuplicateKey`. Both files are duplicate-key objects;
RFC 8259 permits duplicate names, so the input is valid JSON and JSONTestSuite
is right to list it under `y_`. JSONMend's default duplicate-key policy still
drops a member and records `R018`, and that record is what the false-repair count
measures. The distinction matters and is not hidden by the number: the record is
honest about the drop, and the exactness it declares separates the two files —
where the dropped and kept values differ it is `lossy` and the execution gate
refuses the document, where they are identical it is `cosmetic` and the gate
accepts it. So the repair machinery behaves as documented; the corpus simply
contains valid documents whose repair is optional rather than necessary.

No `y_*` file is rejected: JSONMend returns a valid value for all 95 of them.

### Valid `y_` files an oracle refuses

This is the direction worth recording: on this corpus JSONMend is the only
implementation that returns a valid value for every valid document.

| file | input | `json_repair` | `jsonrepair` | oracle note |
| --- | --- | --- | --- | --- |
| `y_structure_string_empty.json` | `""` | reject | accept | repaired text is not JSON |

The one case is `y_structure_string_empty.json`, whose whole content is `""`.
`json_repair.repair_json('""')` returns the empty string — not a JSON
document — so the text-in/text-out/parse round trip this harness uses (the
same one `oracle_node.mjs` and `run_ambiguity.py` use) fails. Its object API
`json_repair.loads('""')` returns the empty Python string, which is correct,
so this is an artifact of the text API and not a parsing failure. It is
recorded rather than patched, because patching it would mean scoring the
Python oracle by a different rule than the Node one.

## `n_*` files: who invents a valid value

These files must be rejected by a strict parser. A repairer that returns a valid
value for one of them has *invented* content; the corpus carries no ground truth for
what that content should be, so the honest reading is "silently accepted an invalid
document", not "repaired it correctly".

| implementation | produced a valid value | refused or non-JSON | crash/timeout |
| --- | --- | --- | --- |
| JSONMend (native) | 165 / 188 (87.8%) | 23 | 0 |
| `json_repair` 0.64.0 (Python) | 166 / 188 (88.3%) | 20 | 2 |
| `jsonrepair` 3.15.0 (Node) | 144 / 188 (76.6%) | 44 | 0 |

How the three agree on each `n_` file (`repair` = returned a valid value for a file
that is not valid JSON):

| JSONMend | `json_repair` | `jsonrepair` | files |
| --- | --- | --- | --- |
| repair | repair | repair | 120 |
| repair | repair | reject | 36 |
| repair | reject | repair | 9 |
| reject | repair | repair | 8 |
| reject | reject | repair | 7 |
| reject | reject | reject | 4 |
| reject | error | reject | 2 |
| reject | repair | reject | 2 |

Concrete examples, up to four per agreement pattern:

| file | input | JSONMend | `json_repair` | `jsonrepair` |
| --- | --- | --- | --- | --- |
| `n_array_1_true_without_comma.json` | `[1 true]` | repair | repair | repair |
| `n_array_colon_instead_of_comma.json` | `["": 1]` | repair | repair | repair |
| `n_array_comma_after_close.json` | `[""],` | repair | repair | repair |
| `n_array_comma_and_number.json` | `[,1]` | repair | repair | repair |
| `n_array_spaces_vertical_tab_formfeed.json` | `["\u000ba"\\f]` | repair | repair | reject |
| `n_number_++.json` | `[++1234]` | repair | repair | reject |
| `n_number_+1.json` | `[+1]` | repair | repair | reject |
| `n_number_+Inf.json` | `[+Inf]` | repair | repair | reject |
| `n_multidigit_number_then_00.json` | `123\u0000` | repair | reject | repair |
| `n_number_NaN.json` | `[NaN]` | repair | reject | repair |
| `n_number_infinity.json` | `[Infinity]` | repair | reject | repair |
| `n_number_minus_infinity.json` | `[-Infinity]` | repair | reject | repair |
| `n_array_a_invalid_utf8.json` | `[aå]` | reject | repair | repair |
| `n_array_invalid_utf8.json` | `[ÿ]` | reject | repair | repair |
| `n_number_invalid-utf-8-in-bigger-int.json` | `[123å]` | reject | repair | repair |
| `n_number_invalid-utf-8-in-exponent.json` | `[1e1å]` | reject | repair | repair |
| `n_string_single_string_no_double_quotes.json` | `abc` | reject | reject | repair |
| `n_structure_angle_bracket_..json` | `<.>` | reject | reject | repair |
| `n_structure_ascii-unicode-identifier.json` | `aå` | reject | reject | repair |
| `n_structure_lone-invalid-utf-8.json` | `å` | reject | reject | repair |
| `n_single_space.json` | ` ` | reject | reject | reject |
| `n_structure_UTF8_BOM_no_data.json` | `\ufeff` | reject | reject | reject |
| `n_structure_end_array.json` | `]` | reject | reject | reject |
| `n_structure_no_data.json` | `` | reject | reject | reject |
| `n_structure_100000_opening_arrays.json` | `[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[…` | reject | error | reject |
| `n_structure_open_array_object.json` | `[{"":[{"":[{"":[{"":[{"":[{"":[{"":[{"":[{"…` | reject | error | reject |
| `n_string_invalid-utf-8-in-escape.json` | `["\\uå"]` | reject | repair | reject |
| `n_structure_incomplete_UTF8_BOM.json` | `ï»{}` | reject | repair | reject |

Implementations that crashed (counted as `error`, not `reject`) on an `n_` file:

| file | implementation | note |
| --- | --- | --- |
| `n_structure_100000_opening_arrays.json` | `json_repair` 0.64.0 (Python) | RecursionError: maximum recursion depth exceeded while deco… |
| `n_structure_open_array_object.json` | `json_repair` 0.64.0 (Python) | RecursionError: maximum recursion depth exceeded while deco… |

Both are the arbitrarily large inputs (`100000` opening brackets; `[{"":` repeated
50 000 times). `json_repair` is recursive and hits Python's recursion limit; the
correct bucket for that is `error`, because it is neither a refusal nor a value.

### Files JSONMend rejects and both oracles accept

This is the safe direction for a repairer — refusing rather than inventing — but
it is a real behavioural difference on a third-party corpus, so it is listed in
full (8 files):

| file | input | JSONMend exit | note |
| --- | --- | --- | --- |
| `n_array_a_invalid_utf8.json` | `[aå]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 3 |
| `n_array_invalid_utf8.json` | `[ÿ]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 1 |
| `n_number_invalid-utf-8-in-bigger-int.json` | `[123å]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 5 |
| `n_number_invalid-utf-8-in-exponent.json` | `[1e1å]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 5 |
| `n_number_invalid-utf-8-in-int.json` | `[0å]\n` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 3 |
| `n_number_real_with_invalid_utf8_after_e.json` | `[1eå]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 4 |
| `n_object_lone_continuation_byte_in_key_and_trailing_comma.json` | `{"¹":"0",}` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 2 |
| `n_string_invalid_utf8_after_escape.json` | `["\\å"]` | 3 | jsonmend: E040 InvalidUtf8: invalid UTF-8 byte at offset 4 |

## Per-family breakdown

### `y_` — valid JSON: every strict parser must accept these

| field | value |
| --- | --- |
| files | 95 (1190 bytes) |
| strict oracle accepts | 95 |
| JSONMend produces a valid value | 95 |
| JSONMend repair records | 2, on 2 file(s) |
| repair codes | `R018` 2 |
| JSONMend (native) | accept 95, repair 0, reject 0, error 0 |
| `json_repair` 0.64.0 (Python) | accept 94, repair 0, reject 1, error 0 |
| `jsonrepair` 3.15.0 (Node) | accept 95, repair 0, reject 0, error 0 |

### `n_` — invalid JSON: every strict parser must reject these

| field | value |
| --- | --- |
| files | 188 (351272 bytes) |
| strict oracle accepts | 0 |
| JSONMend produces a valid value | 165 |
| JSONMend repair records | 253, on 165 file(s) |
| repair codes | `R002` 4, `R003` 5, `R004` 35, `R005` 24, `R006` 11, `R007` 3, `R008` 11, `R009` 41, `R010` 14, `R011` 26, `R012` 11, `R013` 4, `R014` 23, `R015` 1, `R016` 34, `R017` 4, `R018` 1, `R022` 1 |
| JSONMend (native) | accept 0, repair 165, reject 23, error 0 |
| `json_repair` 0.64.0 (Python) | accept 0, repair 166, reject 20, error 2 |
| `jsonrepair` 3.15.0 (Node) | accept 0, repair 144, reject 44, error 0 |

### `i_` — implementation-defined: a strict parser may accept or reject these

| field | value |
| --- | --- |
| files | 35 (1562 bytes) |
| strict oracle accepts | 26 |
| JSONMend produces a valid value | 23 |
| JSONMend repair records | 13, on 11 file(s) |
| repair codes | `R014` 13 |
| JSONMend (native) | accept 23, repair 0, reject 12, error 0 |
| `json_repair` 0.64.0 (Python) | accept 21, repair 9, reject 5, error 0 |
| `jsonrepair` 3.15.0 (Node) | accept 23, repair 9, reject 3, error 0 |

## Strict-oracle caveats

| caveat | value |
| --- | --- |
| files not decodable as UTF-8 | 25 — fed to the two oracles byte-per-code-point via latin-1, since both take text; JSONMend gets the real bytes |
| strict `json.loads` vs Node `JSON.parse` disagreements | 12 |
| files a *default* `json.loads` would wrongly accept | 3 (`n_number_NaN`, `n_number_infinity`, `n_number_minus_infinity`) |

| file | family | `json.loads` | `JSON.parse` |
| --- | --- | --- | --- |
| `i_string_UTF-16LE_with_BOM.json` | `i_` | True | False |
| `i_string_UTF-8_invalid_sequence.json` | `i_` | False | True |
| `i_string_invalid_utf-8.json` | `i_` | False | True |
| `i_string_iso_latin_1.json` | `i_` | False | True |
| `i_string_lone_utf8_continuation_byte.json` | `i_` | False | True |
| `i_string_not_in_unicode_range.json` | `i_` | False | True |
| `i_string_overlong_sequence_2_bytes.json` | `i_` | False | True |
| `i_string_overlong_sequence_6_bytes.json` | `i_` | False | True |
| `i_string_overlong_sequence_6_bytes_null.json` | `i_` | False | True |
| `i_string_truncated-utf-8.json` | `i_` | False | True |
| `i_string_utf16BE_no_BOM.json` | `i_` | True | False |
| `i_string_utf16LE_no_BOM.json` | `i_` | True | False |

All 12 are `i_` files, i.e. files the corpus explicitly leaves free.
They are also all non-UTF-8 files.
The two strict oracles disagree only because the character stream they see
differs, not because they read JSON differently. No `y_` or `n_` file is
affected, so the accept/repair split is unaffected.

The `NaN`/`Infinity` caveat is why `strict_json` passes `parse_constant`: without it
`json.loads` accepts three files the corpus marks `n_`, and those three would be
scored as "accept" for every implementation.

## What this does and does not prove

- **The corpus is independent; the harness and the reading are not.** The 318 test
  files were written by a third party (Nicolas Seriot) to test JSON parsers in
  general, and they are vendored verbatim at the commit above. The script that runs
  them, the choice of strict oracle, the accept/repair/reject classification and this
  analysis are ours, so a mistake in any of those is our mistake and the corpus being
  external would not catch it.
- **It tests syntax acceptance, not repair quality.** A `y_` file that needs no repair
  and an `n_` file that gets one are the only claims made. The corpus has no ground
  truth for what an invalid document *should* become, so "165 of 188 `n_` files
  repaired" measures how often JSONMend invents a value, not how often it invents
  the *right* value. It is not a substitute for `run_eval.py`, which scores recovered
  values against generated truth.
- **It is not a substitute for recorded real provider streams.** These are synthetic,
  hand-written edge cases. Real model output arrives as tool-call deltas with prose,
  truncation and retries; LLM_EVENTS.md and the recorded fixtures cover that, and this
  corpus says nothing about it.
- **The two oracles are repair libraries, not truth.** `json_repair` and `jsonrepair`
  appear here as *other implementations*. On `n_` files their willingness to return a
  value is evidence about them, and it cannot be used to argue that JSONMend is wrong
  when it accepts the same file.
- **What would move the headline.** If a later run reports 0 false repairs on `y_`,
  duplicate-key handling changed. The number is measured per run and the corpus
  fingerprint above fixes which files were measured.

## Reproduce

```bash
cd code
moon build cmd/jsonmend --target native --release
python3 tests/eval/run_independent.py
# The oracles are optional; without them those columns read as "n/a (library not installed)".
#   Node:   npm i jsonrepair@3.15.0   (NODE_PATH=/private/tmp/node_modules)
#   Python: python3 -m pip install "json-repair==0.64.0"
```