Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-037
easy summarizationarticle
The prompt
user
Summarize the news brief below in at most 50 words. Reply with only the summary. """ Kettlebrook recalls 61,000 slow cookers over handle burn risk Kettlebrook Home Appliances is recalling about 61,000 of its 6-quart slow cookers because the side handles can overheat and burn users, the company said Thursday. The recall covers models KB-600 and KB-650 sold in black and stainless finishes between March and August 2026 at department stores and online. The model number is printed on a sticker on the bottom of the base. The company has received 42 reports of handles getting hot enough to cause burns, including 3 that needed medical treatment. No serious injuries have been reported. Owners should stop using the cookers immediately and contact Kettlebrook for a free replacement or a full refund. Customers can register for the recall at the company's website or call its recall line, 1-888-555-0143, weekdays from 8 a.m. to 8 p.m. Eastern. A receipt is not required. Kettlebrook said the problem was traced to a supplier that changed the plastic used in the handles partway through production. Cookers sold before March or after August use a different handle and are not part of the recall. Retailers have been told to pull remaining stock from shelves. """
What we expected
{
"kind": "summarize",
"word_limit": 50,
"facts": [
{
"id": "what",
"desc": "Kettlebrook slow cooker recall",
"any": [
[
"kettlebrook|slow cooker",
"recall"
]
]
},
{
"id": "models",
"desc": "models KB-600 and KB-650",
"any": [
"kb-?\\s?6[05]0"
]
},
{
"id": "hazard",
"desc": "handles can overheat and burn",
"any": [
"handle|overheat|burn"
]
},
{
"id": "action",
"desc": "stop using them",
"any": [
"stop us|do not use|don'?t use|discontinue|stop operating|no longer use|cease|unplug|quit using"
]
},
{
"id": "remedy",
"desc": "free replacement or full refund",
"any": [
"refund|replacement|replace"
]
}
],
"allowed_numbers": [],
"source_words": 192,
"reference_summary": "Kettlebrook is recalling about 61,000 six-quart slow cookers, models KB-600 and KB-650 sold March to August 2026, because the handles can overheat and burn users. Owners should stop using them and contact Kettlebrook for a free replacement or full refund; no receipt is needed."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 93 (scripted checks 100, judges 8.5/10 from 3) · 2.62 s
Kettlebrook recalls ~61,000 6-quart slow cookers (models KB-600/KB-650, March–August 2026) after 42 burn reports, 3 requiring medical treatment. Owners should stop use and register for a free replacement or refund; no serious injuries reported.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 34,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 2.31 s
Kettlebrook recalls ~61,000 6-quart slow cookers (models KB-600, KB-650) because side handles can overheat and burn users. About 42 reports were received, including 3 needing medical treatment. Owners should stop using them and contact the company for a free replacement or full refund.
Judge medians: Faithfulness 10 · Fluency and usefulness 10
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 43,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 2.40 s
Kettlebrook recalls about 61,000 6‑quart slow cookers (models KB‑600/KB‑650) because side handles can overheat and burn users; 42 reports of burns, 3 needing treatment. Owners must stop use, get free replacement or refund via website or 1‑888‑555‑0143.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 37,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 1.65 s
Kettlebrook recalls 61,000 slow cookers due to handle burn risk. The side handles can overheat, causing burns. Models KB-600 and KB-650 (black/stainless, March-August 2026) are affected. Owners should stop use and contact Kettlebrook for a replacement or refund.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 38,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 1.25 s
Kettlebrook recalls 61,000 6‑quart slow cookers (models KB‑600 and KB‑650) due to overheating handles that can cause burns. 42 incidents reported, including 3 requiring medical treatment. Stop use and contact Kettlebrook at 1‑888‑555‑0143 for replacement or refund. Handles issue traced to supplier plastic change; affected units sold March‑August 2026.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 49,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 68 (scripted checks 60, judges 7.5/10 from 3) · 0.75 s
Kettlebrook is recalling 61,000 slow cookers due to overheating side handles that can cause burns. The issue was traced to a supplier changing the plastic used in the handles. The recall affects models KB-600 and KB-650 sold between March and August 2026.
Judge medians: Faithfulness 10 · Fluency and usefulness 5
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": false,
"remedy": false
},
"facts_frac": 0.6,
"word_count": 42,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✗ fail · score 83 (scripted checks 80, judges 8.5/10 from 3) · 0.69 s
Kettlebrook is recalling 61,000 slow cookers due to a handle overheat risk. Owners should stop using them and contact the company for a free replacement or refund.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"what": true,
"models": false,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 0.8,
"word_count": 27,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 94 (scripted checks 93, judges 9.5/10 from 3) · 0.80 s
Kettlebrook Home Appliances is recalling roughly 61,000 of its 6‑quart slow cookers because side handles can overheat and burn users. The recall covers models KB‑600 and KB‑650 sold in black and stainless finishes from March through August 2026. Owners should stop using the units and contact the company at 1‑888‑555‑0143 for a free replacement or refund.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"what": true,
"models": true,
"hazard": true,
"action": true,
"remedy": true
},
"facts_frac": 1,
"word_count": 56,
"word_limit": 50,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-036All Summarize + extract itemssummarize-extract-038 →