Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-039
easy summarizationemail
The prompt
user
Summarize the pet discharge instructions below in at most 70 words. Write it as a summary the owner can follow at home. Reply with only the summary. """ Cedar Lane Animal Hospital - Discharge Instructions Patient: Biscuit (canine, 7 yrs) Owner: Mr. Felix Arango Procedure (Oct 16): dental cleaning under anesthesia, two teeth extracted (upper left premolars) Biscuit did very well today. She may be sleepy tonight and a little wobbly; this is normal as the anesthesia wears off. Feeding: Offer a small meal tonight (about half her usual amount). For the next 7 days, feed only soft food - canned food or kibble soaked in warm water until mushy. Medication: Give one carprofen tablet (pain relief) twice daily with food for 5 days, starting tomorrow morning. Do not give any human pain medicine. Activity: No chew toys, bones, or rawhides for 2 weeks while the gums heal. Short leash walks are fine. Recheck: Friday, October 30 at 11:15 a.m. with Dr. Halvorsen to check the extraction sites. Call us right away at (503) 555-0177 if you notice bleeding that doesn't stop, swelling of the face, or if Biscuit refuses to eat for more than 24 hours. """
What we expected
{
"kind": "summarize",
"word_limit": 70,
"facts": [
{
"id": "food",
"desc": "soft food only for 7 days",
"any": [
[
"soft|soak|canned|mush",
"\\b7\\b|seven|week"
]
]
},
{
"id": "meds",
"desc": "pain tablet twice daily with food for 5 days",
"any": [
[
"carprofen|pain|medic|tablet|pill",
"twice|two times|2 times|2x|every 12|morning and (evening|night)|two a day|2 a day"
]
]
},
{
"id": "toys",
"desc": "no chew toys/bones for 2 weeks",
"any": [
"chew|bone|rawhide"
]
},
{
"id": "recheck",
"desc": "recheck Friday October 30",
"any": [
"(\\boct(ober|\\.)?\\s*30(st|nd|rd|th)?\\b|\\b30(st|nd|rd|th)?\\s+(of\\s+)?oct(ober|\\.)?\\b|\\b0?10/0?30\\b|\\b2026-10-30\\b|\\bthirtieth\\s+(of\\s+)?oct(ober|\\.)?\\b|\\boct(ober|\\.)?\\s+(the\\s+)?thirtieth\\b)"
]
},
{
"id": "call",
"desc": "call if bleeding, swelling or not eating 24 hours",
"any": [
"bleed|swell|refus|not eat|won'?t eat|stops eating"
]
}
],
"allowed_numbers": [],
"source_words": 169,
"reference_summary": "Biscuit had a dental cleaning and two extractions. Feed a half meal tonight, then only soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow; no human pain medicine. No chew toys or bones for 2 weeks. Recheck is Friday, October 30 at 11:15 a.m. Call if bleeding won't stop, the face swells, or she refuses food for over 24 hours."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 94 (scripted checks 93, judges 9.5/10 from 3) · 4.42 s
Biscuit may be sleepy or wobbly tonight; this is normal. Feed a small meal tonight, then soft food only for 7 days. Give one carprofen tablet twice daily with food for 5 days starting tomorrow; avoid human pain medicine. No chew toys, bones, or rawhides for 2 weeks; short leash walks are fine. Recheck Friday, Oct 30 at 11:15 a.m. Call (503) 555-0177 for uncontrolled bleeding, facial swelling, or refusal to eat over 24 hours.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 75,
"word_limit": 70,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 88 (scripted checks 85, judges 9.0/10 from 3) · 3.99 s
Biscuit did well after her dental cleaning and tooth extractions. She may be sleepy and wobbly tonight as anesthesia wears off. Feed a small meal tonight, then soft food only for 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow—no human pain medicine. Avoid chew toys, bones, and rawhides for 2 weeks. Recheck Friday, October 30 at 11:15 a.m. with Dr. Halvorsen. Call (503) 555-0177 for uncontrolled bleeding, facial swelling, or if Biscuit won't eat for over 24 hours.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 85,
"word_limit": 70,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 4.32 s
Feed Biscuit a small meal tonight (about half her usual amount). For the next 7 days, give only soft food—canned food or kibble soaked in warm water until mushy. Give one carprofen tablet twice daily with food for 5 days starting tomorrow morning; do not give any human pain medicine. No chew toys, bones, or rawhides for 2 weeks while gums heal; short leash walks are fine. Recheck on Friday, October 30 at 11:15 a.m. Call (503) 555‑0177 immediately if you see bleeding that won’t stop, facial swelling, or if Biscuit refuses to eat for more than 24 hours.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 99,
"word_limit": 70,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✓ pass · score 83 (scripted checks 100, judges 6.5/10 from 3) · 2.21 s
**Discharge Instructions for Biscuit** Biscuit may be sleepy and wobbly as anesthesia wears off. Feed a small meal tonight, then soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days. Avoid chew toys for 2 weeks. Keep short leash walks. Return Friday, Oct 30 at 11:15 a.m. Call (503) 555-0177 immediately for bleeding, swelling, or refusal to eat.
Judge medians: Faithfulness 7 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 64,
"word_limit": 70,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 1.38 s
Offer a small, soft meal tonight; feed soft food for 7 days. Give carprofen tablet twice daily with food for 5 days, starting tomorrow. No chew toys, bones, or rawhides for 2 weeks; use short leash walks. Expect sleepiness and wobbliness as anesthesia fades. Call immediately if bleeding persists, facial swelling, or refusal to eat >24 h. Recheck on October 30 at 11:15 a.m.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 64,
"word_limit": 70,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 86 (scripted checks 93, judges 8.0/10 from 3) · 1.19 s
Biscuit had a dental cleaning and two teeth removed. She may be sleepy and wobbly tonight. Feed her half her usual amount tonight, then soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days. No chew toys or bones for 2 weeks. She needs a recheck on October 30 at 11:15 a.m. Call (503) 555-0177 if she bleeds, swells, or won't eat for over 24 hours.
Judge medians: Faithfulness 8 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 72,
"word_limit": 70,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✓ pass · score 85 (scripted checks 100, judges 7.0/10 from 3) · 1.09 s
Biscuit may be sleepy as anesthesia wears off. Feed small meals tonight, then only soft food for 7 days. Give carprofen twice daily for 5 days. Avoid chew toys, bones, or rawhides for 2 weeks. Bring her in Friday, Oct 30, for a recheck. Call immediately if there's heavy bleeding, facial swelling, or she refuses to eat.
Judge medians: Faithfulness 7 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 57,
"word_limit": 70,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 0.95 s
Biscuit’s anesthesia wears off; she may be sleepy and wobbly. Feed a small meal tonight (≈½ usual amount) and keep soft food (canned or soaked kibble) for the next 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow morning—no human pain meds. No chew toys, bones, or rawhides for 2 weeks while gums heal. Short leash walks are fine. Recheck on Friday, Oct 30 at 11:15 a.m. with Dr. Halvorsen. Call us at (503) 555‑0177 if you see bleeding, facial swelling, or refusal to eat >24 hrs.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"food": true,
"meds": true,
"toys": true,
"recheck": true,
"call": true
},
"facts_frac": 1,
"word_count": 92,
"word_limit": 70,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-038All Summarize + extract itemssummarize-extract-040 →