Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-035
easy summarizationemail
The prompt
user
Summarize the customer email below in at most 60 words. Reply with only the summary. """ Subject: Damaged sofa - order #CH-55219 Hello, I'm writing about my order #CH-55219, the Ashby three-seat sofa in slate gray. It was supposed to arrive on September 18 but didn't show up until September 27, nine days late, and I had to take a second day off work to be home for the delivery window. When the delivery team unwrapped it, we found a six-inch tear along the seam of the left seat cushion. The driver took photos and noted the damage on the delivery receipt, and I've attached my own photos as well. I've called customer service twice (on September 28 and October 1) and both times was told someone would call me back within 48 hours. Nobody has. I'd like either a replacement cushion shipped at no cost, or a $200 partial refund if a replacement isn't available. I'd prefer the cushion. I don't want to return the whole sofa - the frame and the other cushions are fine. Please reply by email; I'm hard to reach by phone during work hours. Thank you, Lena Marchetti """
What we expected
{
"kind": "summarize",
"word_limit": 60,
"facts": [
{
"id": "damage",
"desc": "sofa arrived with a torn seat cushion",
"any": [
"tear|torn|ripp|damage"
]
},
{
"id": "late",
"desc": "delivery was nine days late",
"any": [
"late|delay|nine days|9 days|sept(ember|\\.)?\\s*27"
]
},
{
"id": "ask",
"desc": "wants a free replacement cushion or a $200 partial refund",
"any": [
[
"cushion|replacement",
"200"
]
]
},
{
"id": "calls",
"desc": "called customer service twice, no callback",
"any": [
"twice|two (calls|times)|2 (calls|times)|no (one|callback|response|reply|follow)|nobody|never (called|heard)|no call ?back|not (been )?called back|called back|follow[- ]?up|unanswered|ignored|unreturned|not returned|hasn'?t (called|got|gotten|heard) back|never came|no one (has )?(called|gotten|got) back|still waiting|(not|n'?t) (yet )?(received|gotten|got|had|heard) (a |any )?(response|reply|call ?back|answer)"
]
}
],
"allowed_numbers": [],
"source_words": 178,
"reference_summary": "Lena Marchetti's Ashby sofa (order CH-55219) arrived nine days late with a six-inch tear in a seat cushion. She has called customer service twice with no callback. She wants a free replacement cushion, or a $200 partial refund if none is available, and asks for a reply by email."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 3.00 s
Lena Marchetti reports order #CH-55219 (Ashby slate gray three-seat sofa) arrived nine days late and a six-inch tear was found on the left seat cushion seam. She has photos and delivery receipt notes, twice called without a callback, and requests a free replacement cushion (preferred) or a $200 partial refund, without returning the sofa. She asks for an email reply.
Judge medians: Faithfulness 10 · Fluency and usefulness 10
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 60,
"word_limit": 60,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 86 (scripted checks 93, judges 8.0/10 from 3) · 2.82 s
Lena Marchetti reports that order #CH-55219, an Ashby three-seat sofa in slate gray, arrived nine days late (Sept 27 vs. Sept 18). The left seat cushion has a six-inch seam tear, documented with photos and on the delivery receipt. After two unanswered customer service calls, she requests a free replacement cushion or a $200 partial refund, preferring the cushion and asking to keep the sofa.
Judge medians: Faithfulness 9 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 65,
"word_limit": 60,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 83 (scripted checks 85, judges 8.0/10 from 3) · 3.99 s
Lena Marchetti reports that her Ashby three‑seat sofa (order #CH‑55219) arrived nine days late on September 27 after being promised September 18. Upon delivery, a six‑inch tear was found on the left seat cushion; photos were taken by the driver and by her. She contacted customer service twice (Sept 28 and Oct 1) and was promised a callback within 48 hours, which never occurred. She requests either a free replacement cushion or a $200 partial refund if replacement isn’t possible, preferring the cushion over returning the whole sofa. She asks for email response as she is hard to reach by phone during work hours.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 104,
"word_limit": 60,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 1.41 s
Lena Marchetti reports order #CH-55219, an Ashby sofa, arrived nine days late and had a six-inch seam tear. She called customer service twice but received no callback. She requests a free replacement cushion or a $200 partial refund, preferring the cushion.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 41,
"word_limit": 60,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 80 (scripted checks 85, judges 7.5/10 from 3) · 1.47 s
Customer Lena Marchetti reports a delayed delivery of order #CH-55219 (Ashby three-seat sofa in slate gray) arriving nine days late on September 27. She discovered a six-inch tear on the left seat cushion, documented by the driver and her own photos. Customer service contacted twice (Sept 28, Oct 1) promised a call within 48 hours but never received one. She requests either a replacement cushion shipped free or a $200 partial refund, preferring the cushion. She asks for email reply.
Judge medians: Faithfulness 9 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 80,
"word_limit": 60,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 0.62 s
Lena Marchetti reports her Ashby three-seat sofa arrived nine days late with a six-inch tear on the left seat cushion. She has contacted customer service twice without a response and requests a replacement cushion at no cost or a $200 partial refund.
Judge medians: Faithfulness 9 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 42,
"word_limit": 60,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✓ pass · score 93 (scripted checks 100, judges 8.5/10 from 3) · 0.92 s
Customer Lena Marchetti reports a late delivery and damage to the left seat cushion of her Ashby sofa (order #CH-55219). She requests a replacement cushion or a $200 refund, as she prefers not to return the sofa. She has not received a response to previous calls.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": true
},
"facts_frac": 1,
"word_count": 46,
"word_limit": 60,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 75 (scripted checks 75, judges 7.5/10 from 3) · 0.22 s
Customer reports sofa arrived late and damaged with a tear. Requested replacement cushion or $200 refund.
Judge medians: Faithfulness 10 · Fluency and usefulness 5
Scoring detail
{
"facts": {
"damage": true,
"late": true,
"ask": true,
"calls": false
},
"facts_frac": 0.75,
"word_count": 16,
"word_limit": 60,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-034All Summarize + extract itemssummarize-extract-036 →