Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-031
easy en-pt-brsign
The prompt
user
Translate the following text into Brazilian Portuguese. Reply with only the translation. Elevator out of service. Please use the stairs.
What we expected
{
"source_lang": "en",
"target_lang": "pt",
"register_note": null,
"source": "Elevator out of service. Please use the stairs.",
"reference": "Elevador fora de serviço. Por favor, use a escada.",
"alt_references": [
"Elevador fora de serviço. Favor usar a escada.",
"Elevador fora de funcionamento. Por favor, use as escadas."
],
"must_keep": [
[
"elevador"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.60 s
Elevador fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 90.42,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.49 s
Elevador fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 90.42,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 2.14 s
Elevador fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 90.42,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Spark-X2.5 4B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.37 s
Elevador fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 90.42,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.26 s
Elevador fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 90.42,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 80 (scripted checks 100, judges 6.0/10 from 3) · 0.20 s
Elevador fora do ar. Por favor, use os escadas.
Judge medians: Adequacy 8 · Fluency and register 4
Scoring detail
{
"target_lang": "pt",
"cleaned": "Elevador fora do ar. Por favor, use os escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 64.95,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 0.9987
}MiniCPM5 2B
✓ pass · score 80 (scripted checks 100, judges 6.0/10 from 3) · 0.27 s
Subir no elevador está fora de serviço. Por favor, use as escadas.
Judge medians: Adequacy 7 · Fluency and register 5
Scoring detail
{
"target_lang": "pt",
"cleaned": "Subir no elevador está fora de serviço. Por favor, use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 77.2,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 1
}K2-Horizon 0.9B
✓ pass · score 76 (scripted checks 86, judges 6.5/10 from 3) · 0.11 s
A elevador está fora de serviço. Use as escadas.
Judge medians: Adequacy 8 · Fluency and register 5
Scoring detail
{
"target_lang": "pt",
"cleaned": "A elevador está fora de serviço. Use as escadas.",
"detected_lang": "pt",
"lang_ok": true,
"chrf": 53.87,
"best_ref": 1,
"pass_bar": 30,
"checks_passed": "4/4",
"failed_checks": [],
"chrf_scaled": 0.7218
}