Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-030
hard en-zh-hansemail
The prompt
user
Translate the following text into Simplified Chinese. Use a formal, polite business register (您). Reply with only the translation. Dear Ms. Chen, Thank you for your patience. I'm afraid the delay is on our end — a supplier dropped the ball on the packaging order. We'll ship your order of 2,400 units by June 10 at no extra charge, and we'll knock 5% off your next invoice to make up for it. Best regards, Mark Ellison
What we expected
{
"source_lang": "en",
"target_lang": "zh",
"register_note": "Use a formal, polite business register (您).",
"source": "Dear Ms. Chen,\n\nThank you for your patience. I'm afraid the delay is on our end — a supplier dropped the ball on the packaging order. We'll ship your order of 2,400 units by June 10 at no extra charge, and we'll knock 5% off your next invoice to make up for it.\n\nBest regards,\nMark Ellison",
"reference": "尊敬的陈女士：\n\n感谢您的耐心等待。很抱歉，此次延误是我方的责任——一家供应商在包装订单上出了差错。我们将在6月10日前免费发出您订购的2,400件产品，并在您的下一张发票上减免5%，以作补偿。\n\n此致\n敬礼\n\nMark Ellison",
"alt_references": [],
"must_keep": [
[
"陈",
"Chen"
],
[
"2,400",
"2400",
"2 400",
"两千四百"
],
[
"6月10日",
"6月10号",
"六月十日",
"六月十号"
],
[
"5%",
"百分之五",
"九五折"
],
[
"Mark Ellison",
"马克·埃利森",
"马克·艾利森",
"马克埃利森",
"马克艾利森",
"马克·埃里森",
"埃利森",
"艾利森"
]
],
"register_check": {
"require": [
[
"您"
]
],
"forbid": []
},
"avoid": [
"球"
]
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 90 (scripted checks 95, judges 8.5/10 from 3) · 2.21 s
陈女士女士： 您好， 感谢您的耐心等待。此次延误的责任在我们一方——供应商在包装订单上出现了失误。我们将于6月10日前免费为您发货，共2,400件；为弥补此次不便，我们将在您下一张发票中为您减免5%。 此致 敬礼 Mark Ellison
Judge medians: Adequacy 9 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "陈女士女士：\n\n您好，\n\n感谢您的耐心等待。此次延误的责任在我们一方——供应商在包装订单上出现了失误。我们将于6月10日前免费为您发货，共2,400件；为弥补此次不便，我们将在您下一张发票中为您减免5%。\n\n此致\n敬礼\n\nMark Ellison",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 45.8,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 0.895
}Ornith-1.5 9B
✗ fail · score 81 (scripted checks 93, judges 7.0/10 from 3) · 1.83 s
尊敬的陈女士： 感谢您的耐心等待。很抱歉，此次延误出在我们这边——供应商在包装订单上出了差错。我们将免收额外费用，于6月10日为您发货，并将在您下一笔账单中减免5%，以作补偿。 顺颂商祺， 马克·埃利森
Judge medians: Adequacy 5 · Fluency and register 9
Scoring detail
{
"target_lang": "zh",
"cleaned": "尊敬的陈女士：\n\n感谢您的耐心等待。很抱歉，此次延误出在我们这边——供应商在包装订单上出了差错。我们将免收额外费用，于6月10日为您发货，并将在您下一笔账单中减免5%，以作补偿。\n\n顺颂商祺，\n马克·埃利森",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 46.93,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"2,400",
"2400",
"2 400",
"两千四百"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 0.9233
}Granite 4.2 8B
✓ pass · score 71 (scripted checks 81, judges 6.0/10 from 3) · 2.81 s
尊敬的陈女士： 感谢您的耐心。不幸的是，延误发生在我们这一端——供应商在包装订购中出现了问题。我们将于6月10日无需额外费用地发货2,400个单位，并将在您下一笔账单中减去5%的费用，以弥补此事。 此致， Mark Ellison
Judge medians: Adequacy 7 · Fluency and register 5
Scoring detail
{
"target_lang": "zh",
"cleaned": "尊敬的陈女士：\n\n感谢您的耐心。不幸的是，延误发生在我们这一端——供应商在包装订购中出现了问题。我们将于6月10日无需额外费用地发货2,400个单位，并将在您下一笔账单中减去5%的费用，以弥补此事。\n\n此致，\n\nMark Ellison",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 35,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 0.6249
}Spark-X2.5 4B
✓ pass · score 83 (scripted checks 82, judges 8.5/10 from 3) · 1.31 s
尊敬的陈女士： 感谢您耐心等待。此次延误由我方原因导致——供应商在包装订单方面出现了疏漏。我们将于6月10日前免费为您发货2,400件订单，并将在您下一笔账单中给予5%的折扣，以作补偿。 此致 马克·埃利森
Judge medians: Adequacy 9 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "尊敬的陈女士：\n\n感谢您耐心等待。此次延误由我方原因导致——供应商在包装订单方面出现了疏漏。我们将于6月10日前免费为您发货2,400件订单，并将在您下一笔账单中给予5%的折扣，以作补偿。\n\n此致\n\n马克·埃利森",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 35.27,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 0.6317
}Mellum2.1 12B-A2.5B
✗ fail · score 68 (scripted checks 86, judges 5.0/10 from 3) · 1.14 s
陈女士， 感谢您的耐心。很遗憾，延迟是在我们这端 — — 供应商在包装订单上出了问题。我们将在6月10日前按原样发货2,400件产品，且不收额外费用。同时，我们将在您下一张账单上给予5%的折扣，以compensate。 此致， Mark Ellison
Judge medians: Adequacy 6 · Fluency and register 4
Scoring detail
{
"target_lang": "zh",
"cleaned": "陈女士，\n\n感谢您的耐心。很遗憾，延迟是在我们这端 — — 供应商在包装订单上出了问题。我们将在6月10日前按原样发货2,400件产品，且不收额外费用。同时，我们将在您下一张账单上给予5%的折扣，以compensate。\n\n此致，\nMark Ellison",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 38.5,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 0.7125
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 68 (scripted checks 75, judges 6.0/10 from 3) · 0.71 s
尊敬的陈女士， 感谢您的耐心。我们对此次延迟负有责任——供应商在包装订单方面出现了失误。我们将为您订购的2,400件产品于6月10日发货，且不收取额外费用，并将在下一次发票中扣除5%以予以补偿。 此致， 马克·艾略特
Judge medians: Adequacy 6 · Fluency and register 6
Scoring detail
{
"target_lang": "zh",
"cleaned": "尊敬的陈女士，\n\n感谢您的耐心。我们对此次延迟负有责任——供应商在包装订单方面出现了失误。我们将为您订购的2,400件产品于6月10日发货，且不收取额外费用，并将在下一次发票中扣除5%以予以补偿。\n\n此致，\n马克·艾略特",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 33.04,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "8/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"Mark Ellison",
"马克·埃利森",
"马克·艾利森",
"马克埃利森",
"马克艾利森",
"马克·埃里森",
"埃利森",
"艾利森"
],
"ok": false,
"found": null
}
],
"chrf_scaled": 0.5761
}MiniCPM5 2B
✓ pass · score 88 (scripted checks 91, judges 8.5/10 from 3) · 0.81 s
尊敬的陈女士， 感谢您的耐心等待。恐怕延迟的问题在于我们这边——供应商在包装订单上出现了疏忽。我们将在6月10日前为您发货2,400个单位，且不收取任何额外费用。此外，我们将为您下一张发票减免5%，以弥补这一损失。 此致， Mark Ellison
Judge medians: Adequacy 9 · Fluency and register 8
Scoring detail
{
"target_lang": "zh",
"cleaned": "尊敬的陈女士，\n\n感谢您的耐心等待。恐怕延迟的问题在于我们这边——供应商在包装订单上出现了疏忽。我们将在6月10日前为您发货2,400个单位，且不收取任何额外费用。此外，我们将为您下一张发票减免5%，以弥补这一损失。\n\n此致，\nMark Ellison",
"detected_lang": "zh",
"lang_ok": true,
"chrf": 42.66,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "9/9",
"failed_checks": [],
"chrf_scaled": 0.8166
}K2-Horizon 0.9B
✗ fail · score 0 (scripted checks 0, judges 0.0/10 from 3) · 0.46 s
Dear Ms. Chen, Thank you for your patience. I am sorry to inform you that the delay is due to our side — a supplier failed to deliver the packaging for the order. We will ship your order of 2,400 units by June 10 at no additional charge, and we will reduce 5% on your next invoice to make up for the shortfall. Best regards, Mark Ellison
Judge medians: Adequacy 0 · Fluency and register 0
Scoring detail
{
"target_lang": "zh",
"cleaned": "Dear Ms. Chen,\n\nThank you for your patience. I am sorry to inform you that the delay is due to our side — a supplier failed to deliver the packaging for the order. We will ship your order of 2,400 units by June 10 at no additional charge, and we will reduce 5% on your next invoice to make up for the shortfall.\n\nBest regards, \nMark Ellison",
"detected_lang": "en",
"lang_ok": false,
"chrf": 9.02,
"best_ref": 0,
"pass_bar": 30,
"checks_passed": "5/9",
"failed_checks": [
{
"type": "keep",
"terms": [
"6月10日",
"6月10号",
"六月十日",
"六月十号"
],
"ok": false,
"found": null
},
{
"type": "register_require",
"terms": [
"您"
],
"ok": false,
"found": null
},
{
"type": "length",
"ratio": 2.464,
"ok": false
},
{
"type": "no_untranslated_leftovers",
"ok": false,
"found": [
"and",
"best",
"charge",
"dear",
"delay",
"for",
"invoice",
"june",
"make",
"next",
"order",
"our"
],
"cjk_chars": 0
}
],
"chrf_scaled": 0,
"reason": "wrong language (detected en, expected zh)"
}