# Independent corpus evaluation: `nst/JSONTestSuite`

Generated by `code/tests/eval/run_independent.py` on 2026-10-16 17:16 CST.

| field | value |
| --- | --- |
| corpus | `nst/JSONTestSuite/test_parsing` (third-party, not authored by us) |
| upstream | <https://github.com/nst/JSONTestSuite> |
| tarball (one-request alternative) | <https://codeload.github.com/nst/JSONTestSuite/tar.gz/refs/heads/master> |
| vendoring method | `git ls-remote` + `git clone --filter=blob:none --sparse`; see `PROVENANCE.md` |
| upstream commit | `1ef36fa01286573e846ac449e8683f8833c5b26a` |
| license | MIT License, Copyright (c) 2016 Nicolas Seriot |
| vendored files | 318 (`y_` 95, `n_` 188, `i_` 35) + `LICENSE` + `README.md` + `PROVENANCE.md` |
| vendored `test_parsing` bytes | 354024 |
| corpus fingerprint | `0eb026859149fec126f0d1e5312d696a9765164f9076025eeff554f4afec60f1` (sha256 over sorted `name\0bytes`) |
| strict oracle | `json.loads` with `NaN`/`Infinity` rejected via `parse_constant` |
| secondary strict oracle | Node `JSON.parse` |
| JSONMend | `./_build/native/release/build/cmd/jsonmend/jsonmend.exe`, sha256 `cee21aa6994d68f6a7a602e77b1a230098a61cb76e080305c00eb24ebf037421` |
| Python oracle | `json_repair` 0.64.0 |
| Node oracle | n/a (library not installed) |
| network | not needed |

Every `test_parsing/*.json` file is fed to JSONMend on stdin, byte for byte, in
default repair mode with `--report`. The two repair oracles are asked the same
question the repository's other comparisons ask them: repair the text, then require
the result to be strict JSON, because the claim under test is that *a caller can now
parse it*, not that the library returned a string.

## Summary: family x implementation

| family | implementation | accept | repair | reject | error |
| --- | --- | --- | --- | --- | --- |
| **`y_`** (95 files) | strict `json.loads` (classifier) | 95 | 0 | 0 | 0 |
| | JSONMend (native) | 95 | 0 | 0 | 0 |
| | `json_repair` 0.64.0 (Python) | 94 | 0 | 1 | 0 |
| | `jsonrepair` 3.15.0 (Node) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) |
| **`n_`** (188 files) | strict `json.loads` (classifier) | 0 | 0 | 188 | 0 |
| | JSONMend (native) | 0 | 165 | 23 | 0 |
| | `json_repair` 0.64.0 (Python) | 0 | 166 | 20 | 2 |
| | `jsonrepair` 3.15.0 (Node) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) |
| **`i_`** (35 files) | strict `json.loads` (classifier) | 26 | 0 | 9 | 0 |
| | JSONMend (native) | 23 | 0 | 12 | 0 |
| | `json_repair` 0.64.0 (Python) | 21 | 9 | 5 | 0 |
| | `jsonrepair` 3.15.0 (Node) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) |

- **accept**: the file is strict JSON *and* the implementation returned a valid
  value, i.e. nothing had to be repaired.
- **repair**: the file is not strict JSON, but the implementation still returned a
  valid value.
- **reject**: the implementation refused, or returned text that is not JSON.
- **error**: the implementation crashed or timed out.

The strict row is the classifier, not a repairer, so it has no repair column by
construction. For `y_` and `n_` the corpus label and the strict oracle agree exactly
(0 disagreements), which is what makes the accept/repair split meaningful; the `i_`
family is free by definition, so any split there is ours and not the corpus's.

## Headline: repairs reported on valid JSON (`y_*`)

A document that is valid JSON needs no repair, so every repair record JSONMend
reports on a `y_` file is a **false repair**.

| metric | value |
| --- | --- |
| `y_` files | 95 |
| accepted by the strict oracle | 95 |
| produce a valid value under JSONMend | 95 |
| **files with at least one repair record** | **2** |
| **repair records in total** | **2** |
| `--check` exits 1 (repairs needed) on | 2 |
| refused by `--for-execution` (exit 5) | 1 |
| **refused with prose skipping on (`--skip-prose`, exit 3)** | **0** |

**2 of 95 `y_*` files get 2 repair record(s) in total.** Each one is listed here
with the record, its declared exactness, the `--check` status and the
`--for-execution` verdict:

| file | input | repair record | exactness | `--check` | `--for-execution` |
| --- | --- | --- | --- | --- | --- |
| `y_object_duplicated_key.json` | `{"a":"b","a":"c"}` | `R018 1:10 duplicate key merged` | lossy | 1 | refuse (5 E010) |
| `y_object_duplicated_key_and_value.json` | `{"a":"b","a":"b"}` | `R018 1:10 duplicate key merged` | cosmetic | 1 | accept (0) |

Every record is `R018 MergeDuplicateKey`. Both files are duplicate-key objects;
RFC 8259 permits duplicate names, so the input is valid JSON and JSONTestSuite
is right to list it under `y_`. JSONMend's default duplicate-key policy still
drops a member and records `R018`, and that record is what the false-repair count
measures. The distinction matters and is not hidden by the number: the record is
honest about the drop, and the exactness it declares separates the two files —
where the dropped and kept values differ it is `lossy` and the execution gate
refuses the document, where they are identical it is `cosmetic` and the gate
accepts it. So the repair machinery behaves as documented; the corpus simply
contains valid documents whose repair is optional rather than necessary.

### Hard criterion: which rules may fire on valid JSON (`y_*`)

This is a **gate**, not a measurement, and it is checked on every run: it does not
need `--strict`, because CI runs this harness without that flag.

| | |
| --- | --- |
| allowed on a valid (`y_*`) document | `R018` |
| rules actually seen on `y_*` | `R018` x2 |
| verdict | **PASS** |

**Named residual (not a failure, and not hidden):** 11 file(s) outside `y_*` are
accepted by the strict oracle *and* fire a rule outside `R018` — `R014` x13. This is the
measured counterexample that keeps the criterion scoped to `y_*`: `R014` replaces a
lone surrogate (`"\ud800"`, accepted by `json.loads` and `JSON.parse`) with
U+FFFD, so `RepairPlan::valid_json` is false for a document a strict parser accepts.
The first two, named so the boundary is checkable without reading the rule table:

- `i_object_key_lone_2nd_surrogate.json` — `R014`
- `i_string_1st_surrogate_but_2nd_missing.json` — `R014`

Every record on a valid document is `R018 MergeDuplicateKey`, the one rule that
can legitimately reach a valid input: RFC 8259 permits duplicate names, so the
document is valid and JSONMend's default policy still merges the duplicate and
records what it dropped. The document-level reason is in the section above.
No `y_*` file is rejected: JSONMend returns a valid value for all 95 of them.

### Valid `y_` files an oracle refuses

This is the direction worth recording: on this corpus JSONMend is the only
implementation that returns a valid value for every valid document.

| file | input | `json_repair` | `jsonrepair` | oracle note |
| --- | --- | --- | --- | --- |
| `y_structure_string_empty.json` | `""` | reject | n/a | repaired text is not JSON |

The one case is `y_structure_string_empty.json`, whose whole content is `""`.
`json_repair.repair_json('""')` returns the empty string — not a JSON
document — so the text-in/text-out/parse round trip this harness uses (the
same one `oracle_node.mjs` and `run_ambiguity.py` use) fails. Its object API
`json_repair.loads('""')` returns the empty Python string, which is correct,
so this is an artifact of the text API and not a parsing failure. It is
recorded rather than patched, because patching it would mean scoring the
Python oracle by a different rule than the Node one.

## `n_*` files: who invents a valid value

These files must be rejected by a strict parser. A repairer that returns a valid
value for one of them has *invented* content; the corpus carries no ground truth for
what that content should be, so the honest reading is "silently accepted an invalid
document", not "repaired it correctly".

| implementation | produced a valid value | refused or non-JSON | crash/timeout |
| --- | --- | --- | --- |
| JSONMend (native) | 165 / 188 (87.8%) | 23 | 0 |
| `json_repair` 0.64.0 (Python) | 166 / 188 (88.3%) | 20 | 2 |
| `jsonrepair` 3.15.0 (Node) | n/a (library not installed) | n/a (library not installed) | n/a (library not installed) |

How the three agree on each `n_` file (`repair` = returned a valid value for a file
that is not valid JSON):

| JSONMend | `json_repair` | `jsonrepair` | files |
| --- | --- | --- | --- |
| repair | repair | n/a (library not installed) | 156 |
| reject | reject | n/a (library not installed) | 11 |
| reject | repair | n/a (library not installed) | 10 |
| repair | reject | n/a (library not installed) | 9 |
| reject | error | n/a (library not installed) | 2 |

Concrete examples, up to four per agreement pattern:

| file | input | JSONMend | `json_repair` | `jsonrepair` |
| --- | --- | --- | --- | --- |
| `n_array_1_true_without_comma.json` | `[1 true]` | repair | repair | n/a |
| `n_array_colon_instead_of_comma.json` | `["": 1]` | repair | repair | n/a |
| `n_array_comma_after_close.json` | `[""],` | repair | repair | n/a |
| `n_array_comma_and_number.json` | `[,1]` | repair | repair | n/a |
| `n_single_space.json` | ` ` | reject | reject | n/a |
| `n_string_single_string_no_double_quotes.json` | `abc` | reject | reject | n/a |
| `n_structure_UTF8_BOM_no_data.json` | `\ufeff` | reject | reject | n/a |
| `n_structure_angle_bracket_..json` | `<.>` | reject | reject | n/a |
| `n_array_a_invalid_utf8.json` | `[aå]` | reject | repair | n/a |
| `n_array_invalid_utf8.json` | `[ÿ]` | reject | repair | n/a |
| `n_number_invalid-utf-8-in-bigger-int.json` | `[123å]` | reject | repair | n/a |
| `n_number_invalid-utf-8-in-exponent.json` | `[1e1å]` | reject | repair | n/a |
| `n_multidigit_number_then_00.json` | `123\u0000` | repair | reject | n/a |
| `n_number_NaN.json` | `[NaN]` | repair | reject | n/a |
| `n_number_infinity.json` | `[Infinity]` | repair | reject | n/a |
| `n_number_minus_infinity.json` | `[-Infinity]` | repair | reject | n/a |
| `n_structure_100000_opening_arrays.json` | `[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[…` | reject | error | n/a |
| `n_structure_open_array_object.json` | `[{"":[{"":[{"":[{"":[{"":[{"":[{"":[{"":[{"…` | reject | error | n/a |

Implementations that crashed (counted as `error`, not `reject`) on an `n_` file:

| file | implementation | note |
| --- | --- | --- |
| `n_structure_100000_opening_arrays.json` | `json_repair` 0.64.0 (Python) | RecursionError: maximum recursion depth exceeded while deco… |
| `n_structure_open_array_object.json` | `json_repair` 0.64.0 (Python) | RecursionError: maximum recursion depth exceeded while deco… |

Both are the arbitrarily large inputs (`100000` opening brackets; `[{"":` repeated
50 000 times). `json_repair` is recursive and hits Python's recursion limit; the
correct bucket for that is `error`, because it is neither a refusal nor a value.

No `n_` file is rejected by JSONMend while both oracles accept it.

## Per-family breakdown

### `y_` — valid JSON: every strict parser must accept these

| field | value |
| --- | --- |
| files | 95 (1190 bytes) |
| strict oracle accepts | 95 |
| JSONMend produces a valid value | 95 |
| JSONMend repair records | 2, on 2 file(s) |
| repair codes | `R018` 2 |
| JSONMend (native) | accept 95, repair 0, reject 0, error 0 |
| `json_repair` 0.64.0 (Python) | accept 94, repair 0, reject 1, error 0 |
| `jsonrepair` 3.15.0 (Node) | n/a (library not installed) |

### `n_` — invalid JSON: every strict parser must reject these

| field | value |
| --- | --- |
| files | 188 (351272 bytes) |
| strict oracle accepts | 0 |
| JSONMend produces a valid value | 165 |
| JSONMend repair records | 253, on 165 file(s) |
| repair codes | `R002` 4, `R003` 5, `R004` 35, `R005` 24, `R006` 11, `R007` 3, `R008` 11, `R009` 41, `R010` 14, `R011` 26, `R012` 11, `R013` 4, `R014` 23, `R015` 1, `R016` 34, `R017` 4, `R018` 1, `R022` 1 |
| JSONMend (native) | accept 0, repair 165, reject 23, error 0 |
| `json_repair` 0.64.0 (Python) | accept 0, repair 166, reject 20, error 2 |
| `jsonrepair` 3.15.0 (Node) | n/a (library not installed) |

### `i_` — implementation-defined: a strict parser may accept or reject these

| field | value |
| --- | --- |
| files | 35 (1562 bytes) |
| strict oracle accepts | 26 |
| JSONMend produces a valid value | 23 |
| JSONMend repair records | 13, on 11 file(s) |
| repair codes | `R014` 13 |
| JSONMend (native) | accept 23, repair 0, reject 12, error 0 |
| `json_repair` 0.64.0 (Python) | accept 21, repair 9, reject 5, error 0 |
| `jsonrepair` 3.15.0 (Node) | n/a (library not installed) |

## Strict-oracle caveats

| caveat | value |
| --- | --- |
| files not decodable as UTF-8 | 25 — fed to the two oracles byte-per-code-point via latin-1, since both take text; JSONMend gets the real bytes |
| strict `json.loads` vs Node `JSON.parse` disagreements | 0 |
| files a *default* `json.loads` would wrongly accept | 3 (`n_number_NaN`, `n_number_infinity`, `n_number_minus_infinity`) |

The `NaN`/`Infinity` caveat is why `strict_json` passes `parse_constant`: without it
`json.loads` accepts three files the corpus marks `n_`, and those three would be
scored as "accept" for every implementation.

## What this does and does not prove

- **The corpus is independent; the harness and the reading are not.** The 318 test
  files were written by a third party (Nicolas Seriot) to test JSON parsers in
  general, and they are vendored verbatim at the commit above. The script that runs
  them, the choice of strict oracle, the accept/repair/reject classification and this
  analysis are ours, so a mistake in any of those is our mistake and the corpus being
  external would not catch it.
- **It tests syntax acceptance, not repair quality.** A `y_` file that needs no repair
  and an `n_` file that gets one are the only claims made. The corpus has no ground
  truth for what an invalid document *should* become, so "165 of 188 `n_` files
  repaired" measures how often JSONMend invents a value, not how often it invents
  the *right* value. It is not a substitute for `run_eval.py`, which scores recovered
  values against generated truth.
- **It is not a substitute for recorded real provider streams.** These are synthetic,
  hand-written edge cases. Real model output arrives as tool-call deltas with prose,
  truncation and retries; LLM_EVENTS.md and the recorded fixtures cover that, and this
  corpus says nothing about it.
- **The two oracles are repair libraries, not truth.** `json_repair` and `jsonrepair`
  appear here as *other implementations*. On `n_` files their willingness to return a
  value is evidence about them, and it cannot be used to argue that JSONMend is wrong
  when it accepts the same file.
- **What would move the headline.** If a later run reports 0 false repairs on `y_`,
  duplicate-key handling changed. The number is measured per run and the corpus
  fingerprint above fixes which files were measured.

## Reproduce

```bash
cd code
moon build cmd/jsonmend --target native --release
python3 tests/eval/run_independent.py
# The oracles are optional; without them those columns read as "n/a (library not installed)".
#   Node:   npm i jsonrepair@3.15.0   (NODE_PATH=/private/tmp/node_modules)
#   Python: python3 -m pip install "json-repair==0.64.0"
```