Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Translate
translate-039
hard ja-enemail
The prompt
user
Translate the following text into English. Reply with only the translation. お世話になっております。先日ご注文いただいた商品（注文番号：A-30571）につきまして、メーカー欠品のため、発送が11月下旬になる見込みでございます。ご迷惑をおかけし、誠に申し訳ございません。キャンセルをご希望の場合は、お手数ですが今月25日までにご連絡くださいませ。
What we expected
{
"source_lang": "ja",
"target_lang": "en",
"register_note": null,
"source": "お世話になっております。先日ご注文いただいた商品（注文番号：A-30571）につきまして、メーカー欠品のため、発送が11月下旬になる見込みでございます。ご迷惑をおかけし、誠に申し訳ございません。キャンセルをご希望の場合は、お手数ですが今月25日までにご連絡くださいませ。",
"reference": "Thank you for your continued business. Regarding the item you recently ordered (order number: A-30571), it is out of stock at the manufacturer, so we expect it to ship in late November. We sincerely apologize for the inconvenience. If you would like to cancel, we apologize for the trouble, but please contact us by the 25th of this month.",
"alt_references": [],
"must_keep": [
[
"A-30571"
],
[
"November"
],
[
"late",
"end of",
"latter half",
"second half",
"latter part",
"later part",
"toward the end",
"towards the end"
],
[
"25"
],
[
"cancel"
]
],
"register_check": null,
"avoid": []
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 2.31 s
Thank you for your continued support. Regarding the item you ordered recently (Order No. A-30571), we regret to inform you that due to a manufacturer shortage, shipment is expected in the late November. We sincerely apologize for any inconvenience caused. If you would like to cancel, please contact us by the 25th of this month.
Judge medians: Adequacy 10 · Fluency and register 9
Scoring detail
{
"target_lang": "en",
"cleaned": "Thank you for your continued support. Regarding the item you ordered recently (Order No. A-30571), we regret to inform you that due to a manufacturer shortage, shipment is expected in the late November. We sincerely apologize for any inconvenience caused. If you would like to cancel, please contact us by the 25th of this month.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 66.93,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 2.26 s
Thank you for your continued support. Regarding the item you ordered recently (Order No.: A-30571), we regret to inform you that due to a shortage from the manufacturer, the shipment is now expected to be delayed until late November. We sincerely apologize for any inconvenience this may cause. If you would like to cancel your order, please contact us at your earliest convenience by the 25th of this month.
Judge medians: Adequacy 10 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "Thank you for your continued support. Regarding the item you ordered recently (Order No.: A-30571), we regret to inform you that due to a shortage from the manufacturer, the shipment is now expected to be delayed until late November. We sincerely apologize for any inconvenience this may cause. If you would like to cancel your order, please contact us at your earliest convenience by the 25th of this month.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 66.57,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 1
}Granite 4.2 8B
✓ pass · score 93 (scripted checks 96, judges 9.0/10 from 3) · 2.07 s
Thank you for your continued support. Regarding the product you ordered recently (order number: A-30571), due to a shortage from the manufacturer, we expect delivery in late November. We apologize for any inconvenience this may cause. If you wish to cancel, please contact us no later than the 25th of this month.
Judge medians: Adequacy 9 · Fluency and register 9
Scoring detail
{
"target_lang": "en",
"cleaned": "Thank you for your continued support. Regarding the product you ordered recently (order number: A-30571), due to a shortage from the manufacturer, we expect delivery in late November. We apologize for any inconvenience this may cause. If you wish to cancel, please contact us no later than the 25th of this month.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 62.16,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.929
}Spark-X2.5 4B
✓ pass · score 84 (scripted checks 87, judges 8.0/10 from 3) · 1.28 s
I hope this finds you well. Regarding the item you ordered (Order Number: A-30571), delivery is expected to be delayed to late November due to a manufacturer shortage. We sincerely apologize for the inconvenience this causes. If you wish to cancel, please contact us by this month's 25th.
Judge medians: Adequacy 8 · Fluency and register 8
Scoring detail
{
"target_lang": "en",
"cleaned": "I hope this finds you well. Regarding the item you ordered (Order Number: A-30571), delivery is expected to be delayed to late November due to a manufacturer shortage. We sincerely apologize for the inconvenience this causes. If you wish to cancel, please contact us by this month's 25th.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 54.8,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.7451
}Mellum2.1 12B-A2.5B
✗ fail · score 72 (scripted checks 75, judges 7.0/10 from 3) · 0.73 s
Thank you for your order (Order No. A-30571). Due to a shortage from the manufacturer, the shipment is expected in late November. We apologize for any inconvenience caused. If you wish to cancel, please contact us by October 25th.
Judge medians: Adequacy 5 · Fluency and register 9
Scoring detail
{
"target_lang": "en",
"cleaned": "Thank you for your order (Order No. A-30571). Due to a shortage from the manufacturer, the shipment is expected in late November. We apologize for any inconvenience caused. If you wish to cancel, please contact us by October 25th.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 44.91,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.4978
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 90 (scripted checks 91, judges 9.0/10 from 3) · 0.70 s
We are writing to inform you that, due to a manufacturer shortage, the item you ordered (Order Number: A-30571) will likely be shipped by late November. We sincerely apologize for any inconvenience this may cause. If you would like to cancel, please contact us by the 25th of this month.
Judge medians: Adequacy 8 · Fluency and register 10
Scoring detail
{
"target_lang": "en",
"cleaned": "We are writing to inform you that, due to a manufacturer shortage, the item you ordered (Order Number: A-30571) will likely be shipped by late November. We sincerely apologize for any inconvenience this may cause. If you would like to cancel, please contact us by the 25th of this month.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 57.47,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.8117
}MiniCPM5 2B
✓ pass · score 83 (scripted checks 96, judges 7.0/10 from 3) · 0.83 s
Thank you for your business. As regards the item ordered (order number: A-30571), we are anticipating delivery in late November due to a shortage of stock from the manufacturer. This may cause inconvenience, and we apologize for the inconvenience. If you wish to cancel, please kindly contact us by the 25th of this month.
Judge medians: Adequacy 7 · Fluency and register 7
Scoring detail
{
"target_lang": "en",
"cleaned": "Thank you for your business. As regards the item ordered (order number: A-30571), we are anticipating delivery in late November due to a shortage of stock from the manufacturer. This may cause inconvenience, and we apologize for the inconvenience. If you wish to cancel, please kindly contact us by the 25th of this month.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 61.96,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.9241
}K2-Horizon 0.9B
✗ fail · score 67 (scripted checks 89, judges 4.5/10 from 3) · 0.43 s
We are grateful for your assistance. Regarding the goods ordered yesterday (order number: A-30571), we are informing you that the manufacturer is out of stock, and delivery is expected to be around November. We apologize for any inconvenience. If you wish to cancel, please contact us as soon as possible by the end of this month, which is May 25th.
Judge medians: Adequacy 3 · Fluency and register 6
Scoring detail
{
"target_lang": "en",
"cleaned": "We are grateful for your assistance. Regarding the goods ordered yesterday (order number: A-30571), we are informing you that the manufacturer is out of stock, and delivery is expected to be around November. We apologize for any inconvenience. If you wish to cancel, please contact us as soon as possible by the end of this month, which is May 25th.",
"detected_lang": "en",
"lang_ok": true,
"chrf": 56.26,
"best_ref": 0,
"pass_bar": 40,
"checks_passed": "7/7",
"failed_checks": [],
"chrf_scaled": 0.7814
}