Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-005
hard en-esproduct
The prompt
user
Translate the following text into Spanish. Reply with only the translation. The TrailMate X2 headlamp delivers up to 450 lumens and runs for 6.5 hours on high (40 hours on low). It weighs just 78 g with the battery and is rated IPX4 — splash-proof, not waterproof. Charges via USB-C in about 2 hours.
What we expected
{
"source_lang": "en",
"target_lang": "es",
"register_note": null,
"source": "The TrailMate X2 headlamp delivers up to 450 lumens and runs for 6.5 hours on high (40 hours on low). It weighs just 78 g with the battery and is rated IPX4 — splash-proof, not waterproof. Charges via USB-C in about 2 hours.",
"reference": "La linterna frontal TrailMate X2 ofrece hasta 450 lúmenes y funciona durante 6,5 horas en modo alto (40 horas en modo bajo). Pesa solo 78 g con la batería y tiene clasificación IPX4: resistente a salpicaduras, no impermeable. Se carga por USB-C en unas 2 horas.",
"alt_references": [],
"must_keep": [
[
"TrailMate X2"
],
[
"450"
],
[
"6,5",
"6.5"
],
[
"40"
],
[
"78 g",
"78 gramos"
],
[
"IPX4"
],
[
"USB-C",
"USB C",
"USB tipo C"
],
[
"2 horas",
"dos horas",
"2 h"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 93 (scripted checks 100, judges 8.5/10 from 3) · 2.39 s
El faro TrailMate X2 ofrece hasta 450 lúmenes y funciona 6,5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene certificación IPX4: resistente a salpicaduras, no a la inmersión. Se carga por USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 9 · Fluency and register 8
Scoring detail
{
"target_lang": "es",
"cleaned": "El faro TrailMate X2 ofrece hasta 450 lúmenes y funciona 6,5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene certificación IPX4: resistente a salpicaduras, no a la inmersión. Se carga por USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 74.35,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 85 (scripted checks 100, judges 7.0/10 from 3) · 1.97 s
El linterna TrailMate X2 ofrece hasta 450 lúmenes y funciona durante 6,5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene clasificación IPX4: resistente a salpicaduras, no sumergible. Se carga mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 8 · Fluency and register 6
Scoring detail
{
"target_lang": "es",
"cleaned": "El linterna TrailMate X2 ofrece hasta 450 lúmenes y funciona durante 6,5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene clasificación IPX4: resistente a salpicaduras, no sumergible. Se carga mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 80.26,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✗ fail · score 8 (scripted checks 0, judges 1.5/10 from 3) · 52.8 s
El faro TrailMate X2 entrega hasta 450 luces y dura 6.5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y está calificado como IPX4 — impermeable a chorros, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas. Wait, I need to correct "luces" to "lumens" in Spanish? Actually, "lumens" is the unit, so in Spanish it's still "lumens" or "luces"? No, "lumens" is the unit, so it should be "hasta 450 luces"? No, "lumens" is the unit of light measurement, so in Spanish it's "hasta 450 luces"? Actually, "lumen" is the unit, so "450 luens"? No, "lumens" is English plural, in Spanish it's "luces" for lights? No, the correct term is "lumens" as a unit, so it should be "hasta 450 luces"? No, "lumens" is the unit, so it should be "hasta 450 luces" would be incorrect. In technical context, it's common to use "lumens" in Spanish as well, or "luces" might be ambiguous. Better to use "hasta 450 luces" would mean "up to 450 lights", which is wrong. So use "hasta 450 luces" no. The correct translation: "hasta 450 luces" is not correct; it should be "hasta 450 luces" no. Let's think: In Spanish, the unit "lumen" is used as is, so "hasta 450 luces" would be wrong; it should be "hasta 450 luces" no. Actually, "lumens" is the plural of lumen, and in Spanish often "lumens" is used directly or "luces" might be used colloquially but technically "lumens" is fine. However, the phrase "delivers up to 450 lumens" translates to "entrega hasta 450 luces"? No, "lumens" is a unit, so "hasta 450 luens" no. In Spanish, it's common to say "hasta 450 luces" for "up to 450 lumens"? No, "lumens" is the unit of illuminance, so you say "hasta 450 luces" would be incorrect; you say "hasta 450 luces" might be confused with "lights". Better to use "hasta 450 luces" no. Let's check: In Spanish technical writing, "lumens" is often used as is, e.g., "hasta 500 luces" would be wrong; it should be "hasta 500 luces" no. Actually, "lumen" is a noun, so "450 lúmenes" with accent on "lúmenes"? Yes, "lúmenes" with accent on the "u"? No, "lumen" in Spanish is often written "lumén" or "lúmen"? I think it's "lumén" (with accent on e) or "lumenes"? Standard: "lumén" (singular) and "lumenes" (plural)? I recall that in Spanish, the term "lumén" is used, e.g., "potencia luminosa de 100 lumenes". Yes, "lumenes" is common. So "hasta 450 lumenes" might be "hasta 450 lumenes"? But "lumenes" plural? Actually, "lumen" is a unit, and in Spanish it's often "lumén" (with accent) for singular, and "lumenes" for plural? I think it's "lumenes" as plural. Example: "El foco emite 200 lumenes". Yes, "lumenes" is used. So "hasta 450 lumenes" would be correct. But the original says "450 lumens", so translation should be "hasta 450 lumenes" or "hasta 450 lúmenes"? I think "lumenes" without accent is fine, but "lúmenes" with accent on u might be correct? I'm not 100% sure, but commonly it's "lumenes". Let's go with "hasta 450 lumenes" as it's standard. But the instruction says "Reply with only the translation." So I need to output the Spanish text. Let's translate carefully: "The TrailMate X2 headlamp delivers up to 450 lumens and runs for 6.5 hours on high (40 hours on low). It weighs just 78 g with the battery and is rated IPX4 — splash-proof, not waterproof. Charges via USB-C in about 2 hours." Translation: "El faro TrailMate X2 entrega hasta 450 lumenes y dura 6.5 horas en alto (40 horas en bajo). Pesas solo 78 g con la batería y está calificado como IPX4 — impermeable a chorros, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas." But "pesa solo" might be "pesa solo" or "pesa apenas"? "just" -> "solo" or "apenas". "just 78 g" -> "solo 78 g" or "apenas 78 g". "just" in this context means "only", so "solo" is fine. "rated IPX4 — splash-proof, not waterproof." In Spanish, "IPX4" is a standard, so "está calificado como IPX4 — impermeable a chorros, no impermeable." But "splash-proof" is often translated as "resistente a chorros" or "impermeable a chorr … [2,515 more characters]
Judge medians: Adequacy 2 · Fluency and register 1
Scoring detail
{
"target_lang": "es",
"cleaned": "El faro TrailMate X2 entrega hasta 450 luces y dura 6.5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y está calificado como IPX4 — impermeable a chorros, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas.\n\nWait, I need to correct \"luces\" to \"lumens\" in Spanish? Actually, \"lumens\" is the unit, so in Spanish it's still \"lumens\" or \"luces\"? No, \"lumens\" is the unit, so it should be \"hasta 450 luces\"? No, \"lumens\" is the unit of light measurement, so in Spanish it's \"hasta 450 luces\"? Actually, \"lumen\" is the unit, so \"450 luens\"? No, \"lumens\" is English plural, in Spanish it's \"luces\" for lights? No, the correct term is \"lumens\" as a unit, so it should be \"hasta 450 luces\"? No, \"lumens\" is the unit, so it should be \"hasta 450 luces\" would be incorrect. In technical context, it's common to use \"lumens\" in Spanish as well, or \"luces\" might be ambiguous. Better to use \"hasta 450 luces\" would mean \"up to 450 lights\", which is wrong. So use \"hasta 450 luces\" no. The correct translation: \"hasta 450 luces\" is not correct; it should be \"hasta 450 luces\" no. Let's think: In Spanish, the unit \"lumen\" is used as is, so \"hasta 450 luces\" would be wrong; it should be \"hasta 450 luces\" no. Actually, \"lumens\" is the plural of lumen, and in Spanish often \"lumens\" is used directly or \"luces\" might be used colloquially but technically \"lumens\" is fine. However, the phrase \"delivers up to 450 lumens\" translates to \"entrega hasta 450 luces\"? No, \"lumens\" is a unit, so \"hasta 450 luens\" no. In Spanish, it's common to say \"hasta 450 luces\" for \"up to 450 lumens\"? No, \"lumens\" is the unit of illuminance, so you say \"hasta 450 luces\" would be incorrect; you say \"hasta 450 luces\" might be confused with \"lights\". Better to use \"hasta 450 luces\" no. Let's check: In Spanish technical writing, \"lumens\" is often used as is, e.g., \"hasta 500 luces\" would be wrong; it should be \"hasta 500 luces\" no. Actually, \"lumen\" is a noun, so \"450 lúmenes\" with accent on \"lúme",
"detected_lang": "en",
"lang_ok": false,
"chrf": 12.43,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "8/10",
"failed_checks": [
{
"type": "length",
"ratio": 25,
"ok": false
},
{
"type": "no_untranslated_leftovers",
"ok": false,
"found": [
"about",
"and",
"battery",
"charges",
"delivers",
"for",
"headlamp",
"high",
"hours",
"just",
"low",
"lumens"
],
"cjk_chars": 0
}
],
"chrf_scaled": 0,
"reason": "wrong language (detected en, expected es)"
}Spark-X2.5 4B
✗ fail · score 65 (scripted checks 90, judges 4.0/10 from 3) · 1.62 s
La linterna TrailMate X2 proporciona hasta 450 lumens y se dura 6,5 horas a alta potencia (40 horas a baja). Pesa solo 78 g con la batería y tiene una clasificación IPX4 —impermeable a chorros, pero no a la agua. Se carga mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 4 · Fluency and register 4
Scoring detail
{
"target_lang": "es",
"cleaned": "La linterna TrailMate X2 proporciona hasta 450 lumens y se dura 6,5 horas a alta potencia (40 horas a baja). Pesa solo 78 g con la batería y tiene una clasificación IPX4 —impermeable a chorros, pero no a la agua. Se carga mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 57.01,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 0.8003
}Mellum2.1 12B-A2.5B
✗ fail · score 80 (scripted checks 100, judges 6.0/10 from 3) · 1.12 s
El candelabro TrailMate X2 proporciona hasta 450 lúmenes y dura 6,5 horas en modo alto (hasta 40 horas en modo bajo). Pesa solo 78 g con la batería y tiene una clasificación IPX4 — resistente a la salpicadura, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 5 · Fluency and register 7
Scoring detail
{
"target_lang": "es",
"cleaned": "El candelabro TrailMate X2 proporciona hasta 450 lúmenes y dura 6,5 horas en modo alto (hasta 40 horas en modo bajo). Pesa solo 78 g con la batería y tiene una clasificación IPX4 — resistente a la salpicadura, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 72.71,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 90 (scripted checks 100, judges 8.0/10 from 3) · 0.86 s
La linterna TrailMate X2 ofrece hasta 450 lúmenes y funciona durante 6.5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene la clasificación IPX4: resistente a salpicaduras, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 9 · Fluency and register 7
Scoring detail
{
"target_lang": "es",
"cleaned": "La linterna TrailMate X2 ofrece hasta 450 lúmenes y funciona durante 6.5 horas en alto (40 horas en bajo). Pesa solo 78 g con la batería y tiene la clasificación IPX4: resistente a salpicaduras, no impermeable. Se carga mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 82.07,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 1
}MiniCPM5 2B
✗ fail · score 73 (scripted checks 86, judges 6.0/10 from 3) · 0.99 s
El lámpara de cabeza TrailMate X2 ofrece hasta 450 lumens y dura 6,5 horas en alta (40 horas en baja). Pesa solo 78 g con la batería y está calificada IPX4 — resistente a la salpicadilla, no a la agua. Se carga mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 8 · Fluency and register 4
Scoring detail
{
"target_lang": "es",
"cleaned": "El lámpara de cabeza TrailMate X2 ofrece hasta 450 lumens y dura 6,5 horas en alta (40 horas en baja). Pesa solo 78 g con la batería y está calificada IPX4 — resistente a la salpicadilla, no a la agua. Se carga mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 53.83,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 0.7208
}K2-Horizon 0.9B
✗ fail · score 60 (scripted checks 80, judges 4.0/10 from 3) · 0.43 s
El TrailMate X2 entrega hasta 450 lúmenes y dura 6.5 horas en alta (40 horas en baja). Pesa solo 78 g con la batería y está calificado IPX4 — resistente a lluvia, no resistente al agua. Charges mediante USB-C en aproximadamente 2 horas.
Judge medians: Adequacy 4 · Fluency and register 4
Scoring detail
{
"target_lang": "es",
"cleaned": "El TrailMate X2 entrega hasta 450 lúmenes y dura 6.5 horas en alta (40 horas en baja). Pesa solo 78 g con la batería y está calificado IPX4 — resistente a lluvia, no resistente al agua. Charges mediante USB-C en aproximadamente 2 horas.",
"detected_lang": "es",
"lang_ok": true,
"chrf": 48.78,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "10/10",
"failed_checks": [],
"chrf_scaled": 0.5946
}