Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-007
easy en-frsign
The prompt
user
Translate the following text into French. Reply with only the translation. Closed for renovation. We reopen on Monday.
What we expected
{
"source_lang": "en",
"target_lang": "fr",
"register_note": null,
"source": "Closed for renovation. We reopen on Monday.",
"reference": "Fermé pour travaux. Nous rouvrons lundi.",
"alt_references": [
"Fermé pour rénovation. Réouverture lundi.",
"Fermé pour cause de rénovation. Nous rouvrirons lundi."
],
"must_keep": [
[
"lundi"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.50 s
Fermé pour travaux. Nous rouvrons lundi.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour travaux. Nous rouvrons lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 100,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.46 s
Fermé pour travaux. Nous rouvrons lundi.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour travaux. Nous rouvrons lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 100,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.54 s
Fermeture pour rénovation. Nous rouvrons lundi.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermeture pour rénovation. Nous rouvrons lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 65.75,
"best_ref": 2,
"pass_bar": 40,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}Spark-X2.5 4B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.35 s
Fermé pour travaux. Nous rouvrons lundi.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour travaux. Nous rouvrons lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 100,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 95 (scripted checks 94, judges 9.5/10 from 3) · 0.28 s
Fermeture pour rénovation. Nous réouvrons lundi.
Judge medians: Adequacy 10 · Fluency and register 9
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermeture pour rénovation. Nous réouvrons lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 60.33,
"best_ref": 2,
"pass_bar": 40,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 0.8833
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 85 (scripted checks 100, judges 7.0/10 from 3) · 0.22 s
Fermé pour rénovation. Nous rouvrons le lundi.
Judge medians: Adequacy 7 · Fluency and register 7
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour rénovation. Nous rouvrons le lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 68.72,
"best_ref": 2,
"pass_bar": 40,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}MiniCPM5 2B
✓ pass · score 79 (scripted checks 99, judges 6.0/10 from 3) · 0.27 s
Fermé pour rénovation. Nous ouvrons à lundi.
Judge medians: Adequacy 7 · Fluency and register 5
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour rénovation. Nous ouvrons à lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 63.81,
"best_ref": 1,
"pass_bar": 30,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 0.9703
}K2-Horizon 0.9B
✓ pass · score 85 (scripted checks 100, judges 7.0/10 from 3) · 0.13 s
Fermé pour rénovation. Nous rouvrons le lundi.
Judge medians: Adequacy 7 · Fluency and register 7
Scoring detail
{
"target_lang": "fr",
"cleaned": "Fermé pour rénovation. Nous rouvrons le lundi.",
"detected_lang": "fr",
"lang_ok": true,
"chrf": 68.72,
"best_ref": 2,
"pass_bar": 40,
"checks_passed": "3/3",
"failed_checks": [],
"chrf_scaled": 1
}