Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-037
easy fr-enchat
The prompt
user
Translate the following text into English. Reply with only the translation. Coucou ! Tu es libre samedi midi ? On pourrait déjeuner au petit bistrot près du marché.
What we expected
{
"source_lang": "fr",
"target_lang": "en",
"register_note": null,
"source": "Coucou ! Tu es libre samedi midi ? On pourrait déjeuner au petit bistrot près du marché.",
"reference": "Hi! Are you free on Saturday at lunchtime? We could have lunch at the little bistro near the market.",
"alt_references": [
"Hey! Are you free for lunch on Saturday? We could eat at the little bistro by the market.",
"Hi there! Are you free Saturday at noon? We could grab lunch at the small bistro near the market."
],
"must_keep": [
[
"Saturday"
],
[
"bistro",
"bistrot",
"café",
"cafe",
"restaurant"
],
[
"market"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.86 s
Hi! Are you free Saturday around noon? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hi! Are you free Saturday around noon? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 80.34,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.76 s
Hey! Are you free Saturday around noon? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hey! Are you free Saturday around noon? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 78.02,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.70 s
Hello! Are you free Saturday at noon? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hello! Are you free Saturday at noon? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 79.48,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}Spark-X2.5 4B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.51 s
Hi! Are you free this Saturday at noon? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hi! Are you free this Saturday at noon? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 81.81,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.34 s
Hi! Are you free at noon on Saturday? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hi! Are you free at noon on Saturday? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 81.61,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.29 s
Hi! Are you free at noon on Saturday? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hi! Are you free at noon on Saturday? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 81.61,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}MiniCPM5 2B
✗ fail · score 88 (scripted checks 100, judges 7.5/10 from 3) · 0.33 s
Hello! Are you free on Saturday afternoon? We could have lunch at the little café near the market.
Judge medians: Adequacy 6 · Fluency and register 9
Scoring detail
{
"target_lang": "en",
"cleaned": "Hello! Are you free on Saturday afternoon? We could have lunch at the little café near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 72.98,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}K2-Horizon 0.9B
✗ fail · score 85 (scripted checks 94, judges 7.5/10 from 3) · 0.15 s
Hello! Are you free this afternoon? We could have lunch at the little bistro near the market.
Judge medians: Adequacy 5 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Hello! Are you free this afternoon? We could have lunch at the little bistro near the market.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 71.41,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "4/5",
"failed_checks": [
{
"type": "keep",
"terms": [
"Saturday"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 1
}