# Reference library comparison on the registered corpus (I-4)

Generated by `code/tests/eval/refcompare.py` on 2026-10-15 21:39 CST.

| field | value |
| --- | --- |
| corpus | `eval-2`, 1200 cases, fingerprint `c528375a0d996a78c68cf54d541033e69f245adcafa5cbf8c36e24fc7ec40aac` |
| jsonmend | round 5 build |
| jsonrepair | 3.15.0 (npm) |
| json_repair | 0.64.0 (PyPI) |

## Recovery and safety

| repairer | recovered truth | silent wrong value | labelled wrong (`lossy`) | raised | produced non-JSON |
| --- | --- | --- | --- | --- | --- |
| jsonmend | 1200 / 1200 (100.0%) | 0 | 0 | 0 | 0 |
| jsonrepair | 951 / 1200 (79.2%) | 122 | 0 | 127 | 0 |
| json_repair | 899 / 1200 (74.9%) | 189 | 0 | 0 | 112 |

`silent wrong value` counts a parseable answer that differs from the truth and carries no warning. For the two reference libraries that is every wrong answer, because neither has a channel for it. For JSONMend it is 0 by construction: the wrong answers appear in the `labelled wrong` column, and the execution gate refuses them.

## The execution gate

| measure | value |
| --- | --- |
| cases refused by `--for-execution` | 254 / 1200 |
| values accepted by the gate that differ from the truth | 0 |
| refused with `E010` | 251 |
| refused with `E011` | 3 |

### Is the gate refusing more than it has to?

The only refusals a reader could call false are the ones inside a family that the corpus registers as notation only (the value is recoverable, so a refusal is not forced by information loss). They are:

| notation family | refused | cause |
| --- | --- | --- |
| `delete_comma` | 3 | the comma deletion leaves two adjacent strings, which really do read two ways (E011) |
| `literal_case` | 11 | `None` is a different word, not a case variant (R012, lossy) |

Every other refusal sits in one of the four lossy families, where the repair invents or drops information, so refusing is the registered behaviour rather than an accident.

## Per-family recovery

| family | cases | gate refused | jsonmend | jsonrepair | json_repair |
| --- | --- | --- | --- | --- | --- |
| clean | 60 | 0 | 60 | 60 | 59 |
| ws | 60 | 0 | 60 | 60 | 60 |
| quotes | 60 | 0 | 60 | 60 | 42 |
| trailing_comma | 60 | 0 | 60 | 60 | 49 |
| double_comma | 60 | 0 | 60 | 0 | 54 |
| delete_comma | 60 | 3 | 60 | 49 | 45 |
| delete_colon | 60 | 0 | 60 | 54 | 10 |
| fence | 60 | 0 | 60 | 60 | 34 |
| prose | 60 | 0 | 60 | 0 | 60 |
| comment | 60 | 0 | 60 | 60 | 41 |
| bom | 60 | 0 | 60 | 60 | 40 |
| dup_key | 60 | 0 | 60 | 60 | 60 |
| mismatched_closer | 60 | 0 | 60 | 35 | 44 |
| extra_closer | 60 | 0 | 60 | 60 | 60 |
| literal_case | 60 | 11 | 60 | 60 | 49 |
| control_char | 60 | 0 | 60 | 0 | 43 |
| missing_value | 60 | 60 | 60 | 60 | 0 |
| ellipsis | 60 | 60 | 60 | 34 | 34 |
| truncate_child | 60 | 60 | 60 | 60 | 56 |
| truncate_string | 60 | 60 | 60 | 59 | 59 |

Families where a reference library beats JSONMend are visible above and are the honest form of this comparison: a single number hides that the corpus is dominated by notation repairs.

## Configuration used for each case

The corpus has one family that cannot be read with the default options: `prose` puts a sentence in front of the value, and JSONMend only skips prose when `--skip-prose` is set (the default is to refuse, so that a scalar root such as `Sure!\n42` is never silently discarded). That family is therefore run with `--skip-prose` in every harness (`run_eval.py`, `run_eval_paths.py`, `refcompare.py`).

This is not a concession made to JSONMend: **both reference libraries strip leading prose by default**, so the flag puts JSONMend in the same position rather than a better one. The number of cases affected is 60 of 1200; every other case runs with default options on both sides.