Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread
rewrite-041
hard shorteninstructions
The prompt
user
Shorten the text below to at most 65 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count. """ Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at the Riverside Endoscopy Center, 400 Elm Street, Suite 2. Please arrive 45 minutes early to complete check-in. Starting three days before your procedure, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids, such as water, broth, apple juice, and plain gelatin; nothing red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, and drink the second half at 3 a.m. Do not eat or drink anything after 6:30 a.m. on the day of the procedure. You will receive sedation, so you must have an adult drive you home; you cannot take a taxi or rideshare alone. If you need to reschedule, call 555-0148 at least 72 hours in advance. """
What we expected
{
"type": "shorten",
"source": "Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at the Riverside Endoscopy Center, 400 Elm Street, Suite 2. Please arrive 45 minutes early to complete check-in. Starting three days before your procedure, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids, such as water, broth, apple juice, and plain gelatin; nothing red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, and drink the second half at 3 a.m. Do not eat or drink anything after 6:30 a.m. on the day of the procedure. You will receive sedation, so you must have an adult drive you home; you cannot take a taxi or rideshare alone. If you need to reschedule, call 555-0148 at least 72 hours in advance.",
"max_words": 65,
"facts": [
{
"name": "March 18",
"any": [
"march 18"
]
},
{
"name": "8:30",
"any": [
"8:30"
]
},
{
"name": "Riverside Endoscopy",
"any": [
"riverside"
]
},
{
"name": "400 Elm Street",
"any": [
"400 elm"
]
},
{
"name": "arrive 45 min early",
"any": [
"45 minutes",
"45 min"
]
},
{
"name": "three days before",
"any": [
"three days",
"3 days"
]
},
{
"name": "no nuts/seeds",
"any": [
"nuts"
]
},
{
"name": "clear liquids",
"any": [
"clear liquid",
"clear liquids"
]
},
{
"name": "nothing red or purple",
"any": [
"red"
]
},
{
"name": "prep at 6 pm",
"any": [],
"re": "\\b6\\s*(?::00)?\\s*(?:p\\.?m\\.?|pm)"
},
{
"name": "second half at 3 am",
"any": [],
"re": "\\b3\\s*(?::00)?\\s*(?:a\\.?m\\.?|am)\\b"
},
{
"name": "nothing after 6:30",
"any": [
"6:30"
]
},
{
"name": "adult driver",
"any": [
"drive",
"driver",
"driven"
]
},
{
"name": "reschedule 555-0148",
"any": [
"555-0148"
]
},
{
"name": "72 hours",
"any": [
"72 hours",
"72 hrs",
"72+ hours"
]
}
],
"banned": [],
"banned_re": [],
"extra_numbers_ok": [
"3"
],
"reference": "Colonoscopy: Tuesday, March 18, 8:30 a.m., Riverside Endoscopy Center, 400 Elm Street, Suite 2; arrive 45 minutes early. Three days before: no nuts, seeds, popcorn, or raw vegetables. Day before: clear liquids only, nothing red or purple. Drink prep at 6 p.m. and 3 a.m. Nothing after 6:30 a.m. An adult must drive you home. Reschedule: 555-0148, 72 hours ahead."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 60 (scripted checks 33, judges 8.7/10 from 3) · 5.08 s
Your colonoscopy is Tuesday, March 18, at 8:30 a.m. at Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early. Starting three days prior, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids—no red or purple. At 6 p.m., drink the first half of your prep; the second half at 3 a.m. Nothing after 6:30 a.m. Sedation means an adult must drive you home; taxis and rideshare are not allowed. To reschedule, call 555-0148 at least 72 hours ahead.
Judge medians: Faithfulness 8 · Key information kept 9 · Readability 9
Scoring detail
{
"cleaned_words": 86,
"similarity_to_source": 0.648,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Ornith-1.5 9B
✗ fail · score 66 (scripted checks 33, judges 10.0/10 from 3) · 4.56 s
Your colonoscopy is Tuesday, March 18, at 8:30 a.m. at the Riverside Endoscopy Center, 400 Elm Street, Suite 2. Please arrive 45 minutes early. Starting three days before, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids—water, broth, apple juice, plain gelatin; nothing red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, and the second half at 3 a.m. Do not eat or drink anything after 6:30 a.m. on the day of the procedure. You will receive sedation, so an adult must drive you home; you cannot take a taxi or rideshare alone. If you need to reschedule, call 555-0148 at least 72 hours in advance.
Judge medians: Faithfulness 10 · Key information kept 10 · Readability 10
Scoring detail
{
"cleaned_words": 119,
"similarity_to_source": 0.913,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Granite 4.2 8B
✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 5.10 s
Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early for check‑in. Starting three days before, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids (water, broth, apple juice, plain gelatin); no red or purple drinks. At 6 p.m. the evening before, drink the first half of the prep solution; drink the second half at 3 a.m. No eating or drinking after 6:30 a.m. on the procedure day. Sedation requires an adult driver; no taxi or rideshare alone. Reschedule by calling 555‑0148 at least 72 hours in advance.
Judge medians: Faithfulness 9 · Key information kept 10 · Readability 10
Scoring detail
{
"cleaned_words": 107,
"similarity_to_source": 0.708,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Spark-X2.5 4B
✗ fail · score 63 (scripted checks 33, judges 9.3/10 from 3) · 3.03 s
Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at the Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early. Starting three days prior, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids; nothing red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, and the second half at 3 a.m. Do not eat or drink after 6:30 a.m. on the day of. You will receive sedation, so an adult must drive you home. If you need to reschedule, call 555-0148 at least 72 hours in advance.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 9
Scoring detail
{
"cleaned_words": 104,
"similarity_to_source": 0.827,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Mellum2.1 12B-A2.5B
✗ fail · score 61 (scripted checks 33, judges 9.0/10 from 3) · 2.15 s
Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early for check‑in. Three days before, avoid nuts, seeds, popcorn, and raw vegetables. The day before, drink only clear liquids (water, broth, apple juice, plain gelatin); no red or purple drinks. At 6 p.m. the evening before, drink half the prep solution; at 3 a.m. drink the second half. No food or drink after 6:30 a.m. on procedure day. You will be sedated, so an adult must drive you home; you cannot take a taxi or rideshare alone. To reschedule, call 555-0148 at least 72 hours in advance.
Judge medians: Faithfulness 8 · Key information kept 9 · Readability 10
Scoring detail
{
"cleaned_words": 111,
"similarity_to_source": 0.738,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 58 (scripted checks 33, judges 8.3/10 from 3) · 1.69 s
Your colonoscopy is scheduled for Tuesday, March 18, at 8:30 a.m. at Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early. Avoid nuts, seeds, popcorn, and raw vegetables three days before. Drink clear liquids the day before, including water, broth, apple juice, and plain gelatin; no red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, and the second half at 3 a.m. Do not eat or drink after 6:30 a.m. on the day of the procedure. You will be sedated, so you must have an adult drive you home. If you need to reschedule, call 555-0148 at least 72 hours in advance.
Judge medians: Faithfulness 9 · Key information kept 6 · Readability 10
Scoring detail
{
"cleaned_words": 113,
"similarity_to_source": 0.813,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}MiniCPM5 2B
✗ fail · score 60 (scripted checks 33, judges 8.7/10 from 3) · 2.12 s
Your colonoscopy is on Tuesday, March 18, at 8:30 a.m. at the Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early. Avoid nuts, seeds, popcorn, and raw vegetables three days before. On the day before, drink only clear liquids like water, broth, apple juice, and plain gelatin; nothing red or purple. At 6 p.m. the evening before, drink the first half of your prep solution, then the second half at 3 a.m. Do not eat or drink after 6:30 a.m. on the procedure day. You will receive sedation, so have an adult drive you home; no taxi or rideshare. Reschedule by calling 555-0148 at least 72 hours in advance.
Judge medians: Faithfulness 8 · Key information kept 9 · Readability 9
Scoring detail
{
"cleaned_words": 112,
"similarity_to_source": 0.776,
"facts_missing": [],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}K2-Horizon 0.9B
✗ fail · score 50 (scripted checks 30, judges 7.0/10 from 3) · 0.73 s
Your colonoscopy is on Tuesday, March 18, at 8:30 a.m. at Riverside Endoscopy Center, 400 Elm Street, Suite 2. Arrive 45 minutes early. Avoid nuts, seeds, popcorn, and raw veggies three days before. Drink only clear liquids on the day before; no red or purple drinks. Do not eat or drink after 6:30 a.m. on the day of the procedure. Have an adult drive home; cannot take a taxi or rideshare alone. Call 555-0148 at least 72 hours ahead for rescheduling.
Judge medians: Faithfulness 9 · Key information kept 4 · Readability 8
Scoring detail
{
"cleaned_words": 81,
"similarity_to_source": 0.598,
"facts_missing": [
"prep at 6 pm",
"second half at 3 am"
],
"new_numbers": [],
"max_words": 65,
"gate": 0.5,
"checks": {
"facts": 0.8667,
"numbers": 1,
"length": 0
}
}