Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-019
easy en-jasign
The prompt
user
Translate the following text into Japanese. Reply with only the translation. Please take off your shoes before entering. Slippers are provided.
What we expected
{
"source_lang": "en",
"target_lang": "ja",
"register_note": null,
"source": "Please take off your shoes before entering. Slippers are provided.",
"reference": "入室の際は靴をお脱ぎください。スリッパをご用意しております。",
"alt_references": [
"土足厳禁です。スリッパをご用意しております。",
"靴を脱いでお入りください。スリッパをご用意しています。"
],
"must_keep": [
[
"靴",
"土足"
],
[
"スリッパ"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 0.64 s
靴を脱いでからご入場ください。スリッパをご用意しております。
Judge medians: Adequacy 10 · Fluency and register 9
Scoring detail
{
"target_lang": "ja",
"cleaned": "靴を脱いでからご入場ください。スリッパをご用意しております。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 66.29,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.51 s
入室の際は靴を脱いでください。スリッパをご用意しております。
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "ja",
"cleaned": "入室の際は靴を脱いでください。スリッパをご用意しております。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 80.09,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 90 (scripted checks 100, judges 8.0/10 from 3) · 0.71 s
靴を脱いで入ってください。スリッパーは用意しております。
Judge medians: Adequacy 9 · Fluency and register 7
Scoring detail
{
"target_lang": "ja",
"cleaned": "靴を脱いで入ってください。スリッパーは用意しております。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 50.76,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Spark-X2.5 4B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 0.48 s
入る前に靴を脱いでください。スリッパをご用意しています。
Judge medians: Adequacy 10 · Fluency and register 9
Scoring detail
{
"target_lang": "ja",
"cleaned": "入る前に靴を脱いでください。スリッパをご用意しています。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 76.94,
"best_ref": 2,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 42 (scripted checks 54, judges 3.0/10 from 3) · 0.33 s
入る前に靴を脱ぎなさい。 slippersは用意されています。
Judge medians: Adequacy 4 · Fluency and register 2
Scoring detail
{
"target_lang": "ja",
"cleaned": "入る前に靴を脱ぎなさい。 slippersは用意されています。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 19.41,
"best_ref": 2,
"pass_bar": 30,
"checks_passed": "3/4",
"failed_checks": [
{
"type": "keep",
"terms": [
"スリッパ"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 0.2352
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 75 (scripted checks 70, judges 8.0/10 from 3) · 0.24 s
入る前に靴を脱いでください。履き物が用意されています。
Judge medians: Adequacy 8 · Fluency and register 8
Scoring detail
{
"target_lang": "ja",
"cleaned": "入る前に靴を脱いでください。履き物が用意されています。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 31.79,
"best_ref": 2,
"pass_bar": 30,
"checks_passed": "3/4",
"failed_checks": [
{
"type": "keep",
"terms": [
"スリッパ"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 0.5446
}MiniCPM5 2B
✓ pass · score 90 (scripted checks 100, judges 8.0/10 from 3) · 0.29 s
入室前に靴を脱いてください。スリッパをご用意しています。
Judge medians: Adequacy 10 · Fluency and register 6
Scoring detail
{
"target_lang": "ja",
"cleaned": "入室前に靴を脱いてください。スリッパをご用意しています。",
"detected_lang": "ja",
"lang_ok": true,
"chrf": 73.63,
"best_ref": 2,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}K2-Horizon 0.9B
✗ fail · score 0 (scripted checks 0, judges 0.0/10 from 3) · 0.10 s
Please take off your shoes before entering. Slippers are provided.
Judge medians: Adequacy 0 · Fluency and register 0
Scoring detail
{
"target_lang": "ja",
"cleaned": "Please take off your shoes before entering. Slippers are provided.",
"reason": "output is the untranslated source",
"detected_lang": "en",
"lang_ok": false,
"chrf": 0,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "0/4",
"failed_checks": [
{
"type": "keep",
"terms": [
"靴",
"土足"
],
"ok": false,
"found": null
},
{
"type": "keep",
"terms": [
"スリッパ"
],
"ok": false,
"found": null
},
{
"type": "length",
"ratio": 1.9,
"ok": false
},
{
"type": "no_untranslated_leftovers",
"ok": false,
"found": [
"are",
"before",
"entering",
"off",
"please",
"provided",
"shoes",
"slippers",
"take",
"your"
],
"cjk_chars": 0
}
],
"chrf_scaled": 0
}