Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread
rewrite-039
hard shortenpolicy
The prompt
user
Shorten the text below to at most 35 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count. """ Effective November 1, employees may work remotely up to two days per week, provided that their manager approves a schedule in advance. Remote days cannot be Mondays, because that is when most team meetings happen. Employees must be reachable by phone or chat between 10 a.m. and 3 p.m. on remote days. Equipment such as monitors and keyboards may be borrowed from IT, but must be returned within five business days of leaving the company. The policy will be reviewed after six months. """
What we expected
{
"type": "shorten",
"source": "Effective November 1, employees may work remotely up to two days per week, provided that their manager approves a schedule in advance. Remote days cannot be Mondays, because that is when most team meetings happen. Employees must be reachable by phone or chat between 10 a.m. and 3 p.m. on remote days. Equipment such as monitors and keyboards may be borrowed from IT, but must be returned within five business days of leaving the company. The policy will be reviewed after six months.",
"max_words": 35,
"facts": [
{
"name": "November 1",
"any": [
"november 1",
"nov 1",
"nov. 1"
]
},
{
"name": "two days a week",
"any": [
"two days",
"2 days"
]
},
{
"name": "manager approval",
"any": [
"manager"
]
},
{
"name": "not Mondays",
"any": [
"monday",
"mondays"
]
},
{
"name": "reachable 10-3",
"any": [],
"re": "\\b10\\s*(?::00)?\\s*(?:a\\.?m\\.?)?\\s*(?:-|to|and|until|till|through|thru)\\s*3\\b"
},
{
"name": "return equipment in five business days",
"any": [
"five business days",
"5 business days",
"five days",
"5 days"
]
},
{
"name": "review after six months",
"any": [
"six months",
"6 months"
]
}
],
"banned": [],
"banned_re": [],
"extra_numbers_ok": [
"2",
"5",
"6"
],
"reference": "From November 1: remote work up to two days weekly with manager approval, never Mondays. Be reachable 10 a.m.-3 p.m. Return IT equipment within five business days of leaving. Reviewed after six months."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 4.49 s
Effective November 1, employees may work remotely up to two days per week with manager-approved schedules. Remote days cannot be Mondays. Employees must be reachable by phone or chat from 10 a.m. to 3 p.m. Equipment may be borrowed from IT but must be returned within five business days of leaving. The policy will be reviewed after six months.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 10
Scoring detail
{
"cleaned_words": 59,
"similarity_to_source": 0.704,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Ornith-1.5 9B
✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 2.02 s
Effective November 1, employees may work remotely up to two days per week, with manager-approved schedules. Remote days cannot be Mondays. Employees must be reachable by phone or chat between 10 a.m. and 3 p.m. Equipment may be borrowed from IT but must be returned within five business days of leaving. The policy will be reviewed after six months.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 10
Scoring detail
{
"cleaned_words": 59,
"similarity_to_source": 0.746,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Granite 4.2 8B
✗ fail · score 50 (scripted checks 33, judges 6.7/10 from 3) · 2.09 s
Effective November 1, employees may work remotely up to two days per week with manager approval. Remote days cannot be Mondays. Employees must be reachable by phone or chat 10 a.m.–3 p.m. on remote days. IT equipment may be borrowed but must be returned within five business days. Policy reviewed after six months.
Judge medians: Faithfulness 5 · Key information kept 6 · Readability 9
Scoring detail
{
"cleaned_words": 53,
"similarity_to_source": 0.676,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Spark-X2.5 4B
✗ fail · score 68 (scripted checks 76, judges 6.0/10 from 3) · 1.21 s
Effective November 1, employees may work remotely up to two days weekly, with manager approval. Remote days cannot be Mondays. Must be reachable 10 a.m. to 3 p.m. Equipment may be borrowed from IT and returned within five business days.
Judge medians: Faithfulness 5 · Key information kept 5 · Readability 8
Scoring detail
{
"cleaned_words": 40,
"similarity_to_source": 0.52,
"facts_missing": [
"review after six months"
],
"new_numbers": [],
"max_words": 35,
"gate": 1,
"checks": {
"facts": 0.8571,
"numbers": 1,
"length": 0.5
}
}Mellum2.1 12B-A2.5B
✗ fail · score 48 (scripted checks 33, judges 6.3/10 from 3) · 1.03 s
Effective November 1, employees may work remotely up to two days per week with manager-approved schedules; remote days cannot be Mondays due to meetings. Employees must be reachable by phone or chat between 10 a.m. and 3 p.m. on remote days. IT equipment (monitors, keyboards) may be borrowed and must be returned within five business days. Policy reviewed after six months.
Judge medians: Faithfulness 5 · Key information kept 5 · Readability 9
Scoring detail
{
"cleaned_words": 61,
"similarity_to_source": 0.667,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 48 (scripted checks 33, judges 6.3/10 from 3) · 0.71 s
Effective November 1, employees may work remotely up to two days per week, with manager approval. Remote days cannot be Mondays. Employees must be reachable between 10 a.m. and 3 p.m. on remote days. Equipment may be borrowed from IT and returned within five business days. The policy will be reviewed after six months.
Judge medians: Faithfulness 5 · Key information kept 5 · Readability 9
Scoring detail
{
"cleaned_words": 54,
"similarity_to_source": 0.701,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}MiniCPM5 2B
✗ fail · score 48 (scripted checks 33, judges 6.3/10 from 3) · 0.88 s
Effective November 1, remote work is allowed up to two days per week with manager approval. Remote days cannot be Mondays. Employees must be reachable 10 a.m. to 3 p.m. Equipment must be returned within five business days. The policy will be reviewed after six months.
Judge medians: Faithfulness 6 · Key information kept 5 · Readability 8
Scoring detail
{
"cleaned_words": 46,
"similarity_to_source": 0.574,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}K2-Horizon 0.9B
✗ fail · score 43 (scripted checks 33, judges 5.3/10 from 3) · 0.38 s
Effective Nov 1, remote work up to two days/week allowed with manager approval. Remote days cannot be Mondays. Employees reachable 10 a.m.–3 p.m. on remote days. Equipment borrowed from IT must be returned within five business days. Policy reviewed after six months.
Judge medians: Faithfulness 5 · Key information kept 4 · Readability 7
Scoring detail
{
"cleaned_words": 42,
"similarity_to_source": 0.512,
"facts_missing": [],
"new_numbers": [],
"max_words": 35,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}