Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-020
easy en-jachat
The prompt
user
Translate the following text into Japanese. Use casual Japanese, as between friends. Reply with only the translation. I'm at the station now. I'll be there in 5 minutes!
What we expected
{
"source_lang": "en",
"target_lang": "ja",
"register_note": "Use casual Japanese, as between friends.",
"source": "I'm at the station now. I'll be there in 5 minutes!",
"reference": "今、駅にいるよ。あと5分で着く！",
"alt_references": [
"今駅にいる。5分で着くよ！",
"今、駅に着いたとこ。あと5分で行くね！"
],
"must_keep": [
[
"駅"
],
[
"5分",
"五分"
]
],
"register_check": {
"require": [],
"forbid": [
"ます",
"ました",
"ません",
"ましょう",
"です",
"でした",
"でしょう",
"ございま"
]
},
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 62 (scripted checks 70, judges 5.5/10 from 3) · 0.50 s
駅に着きました。あと5分です！
Judge medians: Adequacy 8 · Fluency and register 3
Scoring detail
{
"target_lang": "ja",
"cleaned": "駅に着きました。あと5分です！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 30.41,
"best_ref": 0,
"pass_bar": 20,
"checks_passed": "4/5",
"failed_checks": [
{
"type": "register_forbid",
"ok": false,
"found": [
"ました",
"です"
]
}
],
"chrf_scaled": 0.5103
}Ornith-1.5 9B
✓ pass · score 87 (scripted checks 79, judges 9.5/10 from 3) · 0.45 s
駅に着いたよ。あと5分！
Judge medians: Adequacy 9 · Fluency and register 10
Scoring detail
{
"target_lang": "ja",
"cleaned": "駅に着いたよ。あと5分！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 32.87,
"best_ref": 0,
"pass_bar": 20,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 0.5718
}Granite 4.2 8B
✗ fail · score 63 (scripted checks 66, judges 6.0/10 from 3) · 0.63 s
今駅にいます。5分でそこに到着します！
Judge medians: Adequacy 10 · Fluency and register 2
Scoring detail
{
"target_lang": "ja",
"cleaned": "今駅にいます。5分でそこに到着します！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 27.76,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "4/5",
"failed_checks": [
{
"type": "register_forbid",
"ok": false,
"found": [
"ます"
]
}
],
"chrf_scaled": 0.444
}Spark-X2.5 4B
✓ pass · score 88 (scripted checks 86, judges 9.0/10 from 3) · 0.36 s
駅に着いてるよ。5分で着くから！
Judge medians: Adequacy 9 · Fluency and register 9
Scoring detail
{
"target_lang": "ja",
"cleaned": "駅に着いてるよ。5分で着くから！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 38.52,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 0.7131
}Mellum2.1 12B-A2.5B
✓ pass · score 78 (scripted checks 81, judges 7.5/10 from 3) · 0.31 s
電車の駅に着いてるんだ。5分で着くから！
Judge medians: Adequacy 8 · Fluency and register 7
Scoring detail
{
"target_lang": "ja",
"cleaned": "電車の駅に着いてるんだ。5分で着くから！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 34.96,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 0.624
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 0.19 s
今駅にいるよ！5分で着くよ！
Judge medians: Adequacy 10 · Fluency and register 9
Scoring detail
{
"target_lang": "ja",
"cleaned": "今駅にいるよ！5分で着くよ！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 61.83,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "5/5",
"failed_checks": [],
"chrf_scaled": 1
}MiniCPM5 2B
✗ fail · score 47 (scripted checks 60, judges 3.5/10 from 3) · 0.24 s
駅にいきました。5分でそこへ！
Judge medians: Adequacy 4 · Fluency and register 3
Scoring detail
{
"target_lang": "ja",
"cleaned": "駅にいきました。5分でそこへ！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 22.66,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "4/5",
"failed_checks": [
{
"type": "register_forbid",
"ok": false,
"found": [
"ました"
]
}
],
"chrf_scaled": 0.3165
}K2-Horizon 0.9B
✗ fail · score 40 (scripted checks 51, judges 3.0/10 from 3) · 0.12 s
私は駅にいます。5分後にいらっしゃいます！
Judge medians: Adequacy 5 · Fluency and register 1
Scoring detail
{
"target_lang": "ja",
"cleaned": "私は駅にいます。5分後にいらっしゃいます！",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 15.54,
"best_ref": 1,
"pass_bar": 20,
"checks_passed": "4/5",
"failed_checks": [
{
"type": "register_forbid",
"ok": false,
"found": [
"ます"
]
}
],
"chrf_scaled": 0.1385
}