# Benchmark

mbglot on Tatoeba sentences in 72 languages (see eval/tatoeba). Generated by `moon run eval/run`; do not edit.

`franc` is `Mode::Franc` (franc-min's ranking alone), `enhanced` is `Mode::Enhanced` (the default). Accuracy counts every detection. Group accuracy also counts a language of the expected language's group as right (bos/hrv/srp, ind/zlm, hin/bho/mai/mag, run/kin). Reliable precision is the share of reliable detections that are right, and coverage the share of samples with a reliable detection.

The samples of each language and length alternate between a dev half and a test half. Weights and thresholds are chosen on the dev half; the test half is held out, and its figures come first.

## Test half

| Length | Mode | Samples | Accuracy | Group accuracy | Macro F1 | Reliable precision | Coverage |
|---|---|---:|---:|---:|---:|---:|---:|
| short | franc | 1027 | 64.8% | 70.2% | 63.0% | 92.6% | 2.6% |
| short | enhanced | 1027 | 71.7% | 76.8% | 70.3% | 97.1% | 36.6% |
| medium | franc | 911 | 80.2% | 85.7% | 80.2% | 96.5% | 62.8% |
| medium | enhanced | 911 | 85.7% | 90.7% | 85.9% | 98.5% | 68.1% |
| long | franc | 823 | 89.2% | 95.7% | 89.8% | 99.4% | 58.1% |
| long | enhanced | 823 | 94.5% | 98.4% | 95.3% | 99.2% | 80.6% |
| all | franc | 2761 | 77.2% | 82.9% | 73.8% | 97.7% | 39.0% |
| all | enhanced | 2761 | 83.1% | 87.8% | 79.9% | 98.5% | 60.1% |

## Every length, enhanced mode, test half

Accuracy: 83.1% (2295 of 2761), unidentified: 1, macro F1: 79.9%

| Language | Samples | Precision | Recall | F1 |
|---|---:|---:|---:|---:|
| spa Spanish | 45 | 88.6% | 86.7% | 87.6% |
| eng English | 45 | 84.4% | 84.4% | 84.4% |
| por Portuguese | 45 | 69.0% | 88.9% | 77.7% |
| ind Indonesian | 45 | 62.7% | 71.1% | 66.7% |
| fra French | 45 | 86.3% | 97.8% | 91.7% |
| deu German | 45 | 86.3% | 97.8% | 91.7% |
| jav Javanese | 45 | 79.5% | 77.8% | 78.7% |
| vie Vietnamese | 45 | 100.0% | 97.8% | 98.9% |
| ita Italian | 45 | 94.6% | 77.8% | 85.4% |
| tur Turkish | 45 | 83.8% | 68.9% | 75.6% |
| pol Polish | 45 | 97.3% | 80.0% | 87.8% |
| swh Swahili | 45 | 63.2% | 80.0% | 70.6% |
| sun Sundanese | 45 | 76.9% | 66.7% | 71.4% |
| ron Romanian | 45 | 92.5% | 82.2% | 87.1% |
| hau Hausa | 45 | 70.9% | 86.7% | 78.0% |
| fuv Nigerian Fulfulde | 14 | 42.9% | 21.4% | 28.6% |
| bos Bosnian | 45 | 27.8% | 22.2% | 24.7% |
| hrv Croatian | 45 | 44.4% | 53.3% | 48.5% |
| nld Dutch | 45 | 88.6% | 86.7% | 87.6% |
| srp Serbian | 45 | 58.3% | 62.2% | 60.2% |
| ckb Central Kurdish | 45 | 93.2% | 91.1% | 92.1% |
| yor Yoruba | 19 | 100.0% | 52.6% | 69.0% |
| uzn Northern Uzbek | 45 | 79.5% | 68.9% | 73.8% |
| zlm Malay | 45 | 64.1% | 55.6% | 59.5% |
| ibo Igbo | 17 | 83.3% | 58.8% | 69.0% |
| ceb Cebuano | 45 | 73.5% | 55.6% | 63.3% |
| tgl Tagalog | 45 | 62.7% | 93.3% | 75.0% |
| hun Hungarian | 45 | 97.6% | 88.9% | 93.0% |
| azj North Azerbaijani | 45 | 91.9% | 75.6% | 82.9% |
| ces Czech | 45 | 94.1% | 71.1% | 81.0% |
| run Rundi | 45 | 82.4% | 62.2% | 70.9% |
| plt Plateau Malagasy | 21 | 80.8% | 100.0% | 89.4% |
| mad Madurese | 2 | 6.7% | 50.0% | 11.8% |
| nya Nyanja | 17 | 62.5% | 58.8% | 60.6% |
| kin Kinyarwanda | 13 | 22.7% | 38.5% | 28.6% |
| zul Zulu | 25 | 85.7% | 72.0% | 78.3% |
| swe Swedish | 45 | 95.0% | 84.4% | 89.4% |
| lin Lingala | 45 | 68.3% | 95.6% | 79.6% |
| som Somali | 34 | 93.5% | 85.3% | 89.2% |
| hnj Hmong Njua | 2 | 100.0% | 50.0% | 66.7% |
| ilo Iloko | 45 | 78.7% | 82.2% | 80.4% |
| rus Russian | 45 | 76.8% | 95.6% | 85.1% |
| ukr Ukrainian | 45 | 100.0% | 95.6% | 97.7% |
| koi Komi-Permyak | 15 | 100.0% | 60.0% | 75.0% |
| bel Belarusian | 45 | 95.2% | 88.9% | 92.0% |
| bul Bulgarian | 45 | 89.4% | 93.3% | 91.3% |
| kaz Kazakh | 45 | 97.5% | 86.7% | 91.8% |
| arb Standard Arabic | 45 | 93.5% | 95.6% | 94.5% |
| urd Urdu | 45 | 93.5% | 95.6% | 94.5% |
| pes Iranian Persian | 45 | 100.0% | 93.3% | 96.6% |
| skr Saraiki | 29 | 87.1% | 93.1% | 90.0% |
| pbu Northern Pashto | 29 | 96.0% | 82.8% | 88.9% |
| hin Hindi | 45 | 61.8% | 46.7% | 53.2% |
| mar Marathi | 45 | 97.4% | 82.2% | 89.2% |
| mai Maithili | 1 | 0.0% | 0.0% | 0.0% |
| bho Bhojpuri | 24 | 34.4% | 45.8% | 39.3% |
| npi Nepali | 45 | 97.4% | 84.4% | 90.5% |
| cmn Mandarin Chinese | 45 | 100.0% | 100.0% | 100.0% |
| ben Bengali | 45 | 100.0% | 100.0% | 100.0% |
| jpn Japanese | 45 | 100.0% | 100.0% | 100.0% |
| kor Korean | 45 | 100.0% | 100.0% | 100.0% |
| tel Telugu | 45 | 100.0% | 100.0% | 100.0% |
| tam Tamil | 45 | 100.0% | 100.0% | 100.0% |
| guj Gujarati | 34 | 100.0% | 100.0% | 100.0% |
| kan Kannada | 45 | 100.0% | 100.0% | 100.0% |
| mal Malayalam | 45 | 100.0% | 100.0% | 100.0% |
| mya Burmese | 45 | 100.0% | 100.0% | 100.0% |
| pan Panjabi | 42 | 100.0% | 97.6% | 98.8% |
| amh Amharic | 19 | 100.0% | 100.0% | 100.0% |
| tha Thai | 45 | 100.0% | 100.0% | 100.0% |
| sin Sinhala | 19 | 100.0% | 100.0% | 100.0% |
| ell Modern Greek | 45 | 100.0% | 100.0% | 100.0% |

| Expected | Predicted | Count |
|---|---|---:|
| bos | hrv | 21 |
| hin | bho | 15 |
| hrv | bos | 13 |
| ceb | tgl | 13 |
| zlm | ind | 12 |
| run | kin | 11 |
| bos | srp | 9 |
| hin | mag | 9 |
| bho | hin | 8 |
| srp | bos | 7 |
| ind | zlm | 5 |
| srp | hrv | 5 |
| azj | tur | 5 |
| kin | run | 5 |
| bel | rus | 5 |
| sun | tgl | 4 |
| hrv | srp | 4 |
| ckb | zlm | 4 |
| yor | swh | 4 |
| ibo | swh | 4 |

## FLORES-200

A second, out-of-domain corpus: the same 40 FLORES-200 devtest sentences (news and travel articles translated from English, CC BY-SA 4.0) in 68 of the languages (see eval/flores). Nothing was tuned on it.

| Corpus | Mode | Samples | Accuracy | Group accuracy | Macro F1 | Reliable precision | Coverage |
|---|---|---:|---:|---:|---:|---:|---:|
| flores | franc | 2720 | 89.2% | 95.7% | 88.6% | 99.4% | 63.2% |
| flores | enhanced | 2720 | 92.6% | 97.9% | 92.4% | 99.6% | 76.5% |

| Expected | Predicted (enhanced) | Count |
|---|---|---:|
| zlm | ind | 26 |
| bos | hrv | 15 |
| hrv | bos | 14 |
| bos | srp | 12 |
| hrv | srp | 12 |
| mai | mag | 11 |
| mag | hin | 10 |
| kin | run | 9 |
| bho | mag | 8 |
| mag | bho | 8 |

## Dev half

| Length | Mode | Samples | Accuracy | Group accuracy | Macro F1 | Reliable precision | Coverage |
|---|---|---:|---:|---:|---:|---:|---:|
| short | franc | 1032 | 66.4% | 70.7% | 65.8% | 84.8% | 3.2% |
| short | enhanced | 1032 | 72.0% | 76.3% | 71.5% | 98.2% | 36.7% |
| medium | franc | 918 | 80.5% | 86.5% | 79.0% | 96.6% | 64.1% |
| medium | enhanced | 918 | 86.6% | 91.2% | 85.1% | 97.9% | 71.1% |
| long | franc | 830 | 90.8% | 96.0% | 91.5% | 99.8% | 60.6% |
| long | enhanced | 830 | 95.2% | 98.6% | 96.0% | 99.7% | 80.7% |
| all | franc | 2780 | 78.3% | 83.5% | 76.2% | 97.7% | 40.4% |
| all | enhanced | 2780 | 83.7% | 87.8% | 81.5% | 98.6% | 61.2% |

## Custom profiles

A `Classifier` with franc-min's profiles of the Latin-script languages against one with 300-trigram profiles trained on the dev half (`Profile::train`), on the test half. Trigram distance only, without the passes of `Mode::Enhanced`.

| Profiles | Languages | Samples | Accuracy | Macro F1 |
|---|---:|---:|---:|---:|
| franc-min | 39 | 1469 | 70.1% | 66.4% |
| trained on the dev half | 39 | 1469 | 84.8% | 80.9% |