Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-027
medium en-zh-hansreminder
The prompt
user
Translate the following text into Simplified Chinese. Use a polite register (您). Reply with only the translation. Your table for 4 at Jade Garden is booked for Saturday at 7:00 p.m. We can hold the table for 15 minutes. To change your booking, reply to this message or call 020-5550-1987.
What we expected
{
"source_lang": "en",
"target_lang": "zh",
"register_note": "Use a polite register (您).",
"source": "Your table for 4 at Jade Garden is booked for Saturday at 7:00 p.m. We can hold the table for 15 minutes. To change your booking, reply to this message or call 020-5550-1987.",
"reference": "您在翠园预订的4人桌已确认，时间为周六晚上7:00。我们可为您保留餐位15分钟。如需更改预订，请回复此消息或致电020-5550-1987。",
"alt_references": [],
"must_keep": [
[
"Jade Garden",
"翠园",
"玉园",
"翡翠园",
"翠苑",
"玉苑",
"翡翠花园",
"玉花园",
"翠玉园",
"碧玉园"
],
[
"4人",
"四人",
"4位",
"四位",
"4名",
"四名"
],
[
"周六",
"星期六",
"礼拜六"
],
[
"7:00",
"7点",
"七点",
"19:00",
"19点",
"7时"
],
[
"15分钟",
"十五分钟"
],
[
"020-5550-1987"
]
],
"register_check": {
"require": [
[
"您"
]
],
"forbid": []
},
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 1.54 s
您的4位预订已安排在玉花园，时间为周六晚上7点。我们可为您保留座位15分钟。如需更改预订，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 10 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "您的4位预订已安排在玉花园，时间为周六晚上7点。我们可为您保留座位15分钟。如需更改预订，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 70.06,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✗ fail · score 81 (scripted checks 97, judges 6.5/10 from 3) · 1.34 s
您预订的4人桌已订于周六晚上7点，餐厅可为您保留15分钟。如需更改预订，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 5 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "您预订的4人桌已订于周六晚上7点，餐厅可为您保留15分钟。如需更改预订，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 64.83,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"Jade Garden",
"翠园",
"玉园",
"翡翠园",
"翠苑",
"玉苑",
"翡翠花园",
"玉花园",
"翠玉园",
"碧玉园"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 93 (scripted checks 100, judges 8.5/10 from 3) · 35.5 s
您在玉园的4人桌已预订星期六晚上7:00。我们可以为您保留15分钟。若需更改预订，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 10 · Fluency and register 7
Scoring detail
{
"target_lang": "zh",
"cleaned": "您在玉园的4人桌已预订星期六晚上7:00。我们可以为您保留15分钟。若需更改预订，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 64.27,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 1
}Spark-X2.5 4B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 1.09 s
您的4人雅园（Jade Garden）订座已确认，时间为周六晚上7:00，我们可为您的订座保留15分钟。如需更改订座，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 10 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "您的4人雅园（Jade Garden）订座已确认，时间为周六晚上7:00，我们可为您的订座保留15分钟。如需更改订座，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 64.51,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 76 (scripted checks 97, judges 5.5/10 from 3) · 0.71 s
您在吉花园为4人份餐桌预订了周六晚上7点。我们可以将该桌子保留15分钟。如需更改预订，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 6 · Fluency and register 5
Scoring detail
{
"target_lang": "zh",
"cleaned": "您在吉花园为4人份餐桌预订了周六晚上7点。我们可以将该桌子保留15分钟。如需更改预订，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 58.69,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"Jade Garden",
"翠园",
"玉园",
"翡翠园",
"翠苑",
"玉苑",
"翡翠花园",
"玉花园",
"翠玉园",
"碧玉园"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 0.57 s
您预订的 Jade Garden 4 人桌已定于周六晚上 7:00 保留。我们可以为您保留 15 分钟。如需更改预订，请回复此消息或致电 020-5550-1987。
Judge medians: Adequacy 10 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "您预订的 Jade Garden 4 人桌已定于周六晚上 7:00 保留。我们可以为您保留 15 分钟。如需更改预订，请回复此消息或致电 020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 64.37,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 1
}MiniCPM5 2B
✗ fail · score 71 (scripted checks 97, judges 4.5/10 from 3) · 0.57 s
翡翠花园的4张桌子在周六晚上7:00已预订。您可以提前15分钟预订该桌子。如需更改预订，请回复此消息或致电020-5550-1987。
Judge medians: Adequacy 3 · Fluency and register 6
Scoring detail
{
"target_lang": "zh",
"cleaned": "翡翠花园的4张桌子在周六晚上7:00已预订。您可以提前15分钟预订该桌子。如需更改预订，请回复此消息或致电020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 56.53,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"4人",
"四人",
"4位",
"四位",
"4名",
"四名"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 1
}K2-Horizon 0.9B
✗ fail · score 71 (scripted checks 97, judges 4.5/10 from 3) · 0.35 s
您的4位客人在杜兰花园的餐厅预订为周六上午7点。我们可以保留餐厅的座位15分钟。如需更改预订，请回复此消息或拨打020-5550-1987。
Judge medians: Adequacy 4 · Fluency and register 5
Scoring detail
{
"target_lang": "zh",
"cleaned": "您的4位客人在杜兰花园的餐厅预订为周六上午7点。我们可以保留餐厅的座位15分钟。如需更改预订，请回复此消息或拨打020-5550-1987。",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 51.03,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"Jade Garden",
"翠园",
"玉园",
"翡翠园",
"翠苑",
"玉苑",
"翡翠花园",
"玉花园",
"翠玉园",
"碧玉园"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 1
}