Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-043
hard summarizationtranscript
The prompt
user
Summarize the voicemail transcript below in at most 50 words. Give the corrected facts only. Reply with only the summary. """ [Voicemail - received Mon 11/9/2026 4:12 PM - auto-transcribed] Hi, this is, um, Marguerite from Allen Street Auto Body, calling for Desmond Clarke about the gray hatchback. So, uh, I've got the updated estimate here. It's coming in at eighteen hundred and forty - oh, sorry, no, that's the old one, that was before we pulled the bumper. The new total is two thousand one hundred fifteen dollars, because the bumper bracket behind it is cracked too, so we have to replace that. The insurance adjuster is coming out Tuesday - no, sorry, Wednesday morning, he moved it. So Wednesday morning. Once he signs off, we'll order the bracket, and if it comes in when they say, you should have the car back by Friday the thirteenth. Friday, November thirteenth. Your rental through the insurance is covered until the car's done, so don't worry about that. Um, if you've got questions, call me back at the shop, that's 555-0164, before 5:30 today, or anytime tomorrow. Okay, thanks, bye. """
What we expected
{
"kind": "summarize",
"word_limit": 50,
"facts": [
{
"id": "estimate",
"desc": "new estimate $2,115",
"any": [
"2,?115|two thousand one hundred (and )?fifteen"
]
},
{
"id": "bracket",
"desc": "because the bumper bracket is cracked",
"any": [
"bracket|crack"
]
},
{
"id": "adjuster",
"desc": "insurance adjuster comes Wednesday morning",
"any": [
[
"adjuster|insurance|inspect",
"wed(nesday|\\.)?"
]
]
},
{
"id": "ready",
"desc": "car ready by Friday November 13",
"any": [
"(\\bnov(ember|\\.)?\\s*13(st|nd|rd|th)?\\b|\\b13(st|nd|rd|th)?\\s+(of\\s+)?nov(ember|\\.)?\\b|\\b0?11/0?13\\b|\\b2026-11-13\\b|\\bthirteenth\\s+(of\\s+)?nov(ember|\\.)?\\b|\\bnov(ember|\\.)?\\s+(the\\s+)?thirteenth\\b|\\bfri(day|\\.)?\\b)"
]
},
{
"id": "callback",
"desc": "call back at 555-0164",
"any": [
"555-?0164|call (her |marguerite )?back|callback|call the shop"
]
}
],
"allowed_numbers": [],
"source_words": 168,
"reference_summary": "Marguerite at Allen Street Auto Body says the new repair estimate is $2,115 because the bumper bracket is also cracked. The insurance adjuster comes Wednesday morning, and the car should be ready Friday, November 13. The rental is covered. Call her back at 555-0164."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 2.77 s
Marguerite from Allen Street Auto Body: updated gray hatchback estimate is $2,115 (bumper bracket cracked). Insurance adjuster visits Wednesday morning; car back by Friday, November 13. Rental covered. Call 555-0164 before 5:30 PM today or anytime tomorrow.
Judge medians: Faithfulness 9 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 1,
"word_count": 37,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 89 (scripted checks 93, judges 8.5/10 from 3) · 2.46 s
Marguerite from Allen Street Auto Body calls Desmond Clarke about a gray hatchback. The updated estimate is $2,115 after replacing a cracked bumper bracket. The insurance adjuster comes Wednesday morning. The car should be ready by Friday, November 13th. A rental is covered until then. Call 555-0164 before 5:30 PM today or anytime tomorrow.
Judge medians: Faithfulness 8 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 1,
"word_count": 54,
"word_limit": 50,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 80 (scripted checks 85, judges 7.5/10 from 3) · 2.97 s
Marguerite from Allen Street Auto Body called Desmond Clarke about the gray hatchback. The old estimate was $1,840; the new total is $2,115 due to a cracked bumper bracket. The insurance adjuster visits Wednesday morning; after approval, the bracket will be ordered and the car should be back by Friday, November 13. Rental through insurance is covered until repair completion. Call 555-0164 before 5:30 PM today or anytime tomorrow.
Judge medians: Faithfulness 8 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 1,
"word_count": 69,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 83 (scripted checks 80, judges 8.5/10 from 3) · 1.41 s
Marguerite from Allen Street Auto Body calls Desmond Clarke about a gray hatchback. The new estimate is $2,115 due to a cracked bumper bracket. The insurance adjuster will visit Wednesday morning, and the car should be back by Friday, November 13th.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": false
},
"facts_frac": 0.8,
"word_count": 41,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 78 (scripted checks 85, judges 7.0/10 from 3) · 1.29 s
Marguerite from Allen Street Auto Body called Desmond Clarke about a gray hatchback. The updated estimate is $2,115, including a cracked bumper bracket that was not in the previous $1,840 quote. The insurance adjuster will visit Wednesday morning; after signing, the bracket will be ordered and the car should be ready by Friday, November 13. Rental coverage continues until repairs are complete. Call 555-0164 before 5:30 today or anytime tomorrow.
Judge medians: Faithfulness 8 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 1,
"word_count": 70,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 85 (scripted checks 80, judges 9.0/10 from 3) · 0.76 s
A shop, Allen Street Auto Body, is calling for Desmond Clarke regarding a gray hatchback. The new estimate is $2,115, including a cracked bumper bracket replacement. The insurance adjuster will visit Wednesday morning, and the car is expected back by Friday, November 13th. The rental is covered by insurance.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": false
},
"facts_frac": 0.8,
"word_count": 49,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 0.81 s
Desmond Clarke is being called about a gray hatchback. Updated estimate is $2,115 due to a cracked bumper bracket. Insurance adjuster will be on Wednesday morning; car expected back by Friday, November 13. Contact 555-0164 with questions.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"estimate": true,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 1,
"word_count": 37,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 62 (scripted checks 74, judges 5.0/10 from 3) · 0.50 s
Marguerite from Allen Street Auto Body contacted Desmond Clarke regarding a gray hatchback. Updated estimate: $1,215, including bumper replacement due to cracked bracket. Insurance adjuster arrives Wednesday morning; car returns by Friday, November 13. Rental covered until completion. Contact shop at 555-0164 before 5:30 today or anytime tomorrow.
Judge medians: Faithfulness 3 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"estimate": false,
"bracket": true,
"adjuster": true,
"ready": true,
"callback": true
},
"facts_frac": 0.8,
"word_count": 48,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [
"1,215"
],
"number_score": 0.5
}← summarize-extract-042All Summarize + extract itemssummarize-extract-044 →