Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-028
easy summarizationemail
The prompt
user
Summarize the school notice below in at most 50 words. Reply with only the summary. """ Dear Grade 4 Families, On Thursday, May 7, our fourth graders will visit the Riverbend Science Center for a hands-on day learning about rivers and wetlands. Buses leave school at 8:45 a.m. and return by 2:30 p.m., so students will be back before dismissal. The cost is $18 per student, which covers admission and the bus. Please send cash or a check made out to Hawthorne Elementary PTA, along with the signed permission slip, by Friday, May 1. No student can attend without a signed slip. Students should bring a packed lunch in a disposable bag, a water bottle, and wear closed-toe shoes they don't mind getting muddy. We still need four parent chaperones. If you can help, please email Ms. Okafor at okafor.t@hawthorne-elem.org. Financial help is available for the trip cost; just contact the school office confidentially. Students will be outdoors for part of the day, rain or shine, so a light rain jacket is a good idea. Please leave phones and electronics at home; teachers will have phones for emergencies. Thank you! Ms. Okafor and Mr. Leclerc """
What we expected
{
"kind": "summarize",
"word_limit": 50,
"facts": [
{
"id": "when_where",
"desc": "trip to Riverbend Science Center on Thursday May 7",
"any": [
[
"riverbend",
"(\\bmay\\s*7(st|nd|rd|th)?\\b|\\b7(st|nd|rd|th)?\\s+(of\\s+)?may\\b|\\b0?5/0?7\\b|\\b2026-05-07\\b|\\bseventh\\s+(of\\s+)?may\\b|\\bmay\\s+(the\\s+)?seventh\\b|\\bthu(rsday|\\.)?\\b)"
]
]
},
{
"id": "cost",
"desc": "$18 per student",
"any": [
"\\$\\s?18\\b|\\b18 (dollars|per|a student|each)|eighteen dollars"
]
},
{
"id": "slip",
"desc": "signed permission slip and payment due by Friday May 1",
"any": [
[
"(permission|slip|form)",
"(\\bmay\\s*1(st|nd|rd|th)?\\b|\\b1(st|nd|rd|th)?\\s+(of\\s+)?may\\b|\\b0?5/0?1\\b|\\b2026-05-01\\b|\\bfirst\\s+(of\\s+)?may\\b|\\bmay\\s+(the\\s+)?first\\b|\\bfri(day|\\.)?\\b)"
]
]
},
{
"id": "bring",
"desc": "pack a lunch / water / closed-toe shoes",
"any": [
"lunch|closed[- ]toe|shoes|water bottle"
]
},
{
"id": "chaperones",
"desc": "four parent chaperones needed",
"any": [
"chaperon"
]
}
],
"allowed_numbers": [],
"source_words": 179,
"reference_summary": "Fourth graders visit the Riverbend Science Center on Thursday, May 7 (buses 8:45 a.m. to 2:30 p.m.). Send $18 and the signed permission slip by Friday, May 1; no slip, no trip. Pack a lunch, water and closed-toe shoes. Four parent chaperones are still needed; financial help is available."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 2.75 s
Grade 4s visit Riverbend Science Center Thursday, May 7, 8:45 a.m.–2:30 p.m. Cost is $18 (admission and bus); return cash/check and signed slip by May 1. Bring lunch, water bottle, closed-toe shoes, and a rain jacket; no electronics. Four parent chaperones needed; financial aid available.
Judge medians: Faithfulness 10 · Fluency and usefulness 10
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": true
},
"facts_frac": 1,
"word_count": 45,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 70 (scripted checks 60, judges 8.0/10 from 3) · 2.37 s
On Thursday, May 7, Grade 4 students visit Riverbend Science Center (8:45 a.m.–2:30 p.m.) to learn about rivers and wetlands. Cost is $18 per student (admission and bus). Send cash or a check made out to Hawthorne Elementary PTA, along with the signed permission slip, by Friday, May 1.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": false,
"chaperones": false
},
"facts_frac": 0.6,
"word_count": 49,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 83 (scripted checks 85, judges 8.0/10 from 3) · 3.85 s
On Thursday, May 7, Grade 4 students will visit the Riverbend Science Center for a hands‑on rivers and wetlands day. Buses depart at 8:45 a.m. and return by 2:30 p.m. The cost is $18 per student (admission + bus); cash or a check to Hawthorne Elementary PTA with a signed permission slip must be received by May 1. Students should bring a packed lunch, water bottle, closed‑toe shoes, and a light rain jacket; leave phones at home. Four parent chaperones are still needed, and financial aid is available.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": true
},
"facts_frac": 1,
"word_count": 87,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 85 (scripted checks 80, judges 9.0/10 from 3) · 1.57 s
Grade 4 students visit Riverbend Science Center on May 7. Buses leave at 8:45 a.m., return by 2:30 p.m. Cost is $18. Bring packed lunch, water bottle, closed-toe shoes, and a rain jacket. Signed permission slips and payment by May 1 are required.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": false
},
"facts_frac": 0.8,
"word_count": 43,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 91 (scripted checks 93, judges 9.0/10 from 3) · 1.34 s
Fourth graders visit Riverbend Science Center on May 7; buses leave 8:45 a.m., return 2:30 p.m. Cost $18 covers admission and bus; cash or check to Hawthorne Elementary PTA due May 1 with signed permission slip. Bring disposable lunch, water bottle, closed‑toe shoes, rain jacket. Need four parent chaperones; email Ms. Okafor. Financial help available. Phones and electronics not allowed.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": true
},
"facts_frac": 1,
"word_count": 60,
"word_limit": 50,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 77 (scripted checks 74, judges 8.0/10 from 3) · 0.93 s
Fourth graders will visit the Riverbend Science Center on May 7. Buses leave at 8:45 a.m. and return by 2:30 p.m. The cost is $18 per student, covering admission and the bus. Permission slips must be signed and returned by May 1. Students should bring a packed lunch, water bottle, and closed-toe shoes. A light rain jacket is recommended.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": false
},
"facts_frac": 0.8,
"word_count": 59,
"word_limit": 50,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 1.50 s
Grade 4 students will visit the Riverbend Science Center on May 7 for a hands-on learning day. Buses leave at 8:45 a.m. and return by 2:30 p.m. The cost is $18 per student, payable by May 1 with a signed permission slip. Students should bring a packed lunch, water, and closed-toe shoes. Please provide four parent chaperones by emailing Ms. Okafor. A light rain jacket is recommended, and phones should be left at home.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"when_where": true,
"cost": true,
"slip": true,
"bring": true,
"chaperones": true
},
"facts_frac": 1,
"word_count": 74,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 0 (scripted checks 0, judges 0.0/10 from 0) · 5.13 s
(empty reply)
Judge medians:
Scoring detail
{
"error": "empty output"
}← summarize-extract-027All Summarize + extract itemssummarize-extract-029 →