Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-041
easy summarizationemail
The prompt
user
Summarize the event recap email below in at most 50 words. Reply with only the summary. """ Subject: Thank you, runners! Paws & Pavement 5K recap What a morning! Thanks to everyone who came out to the 4th annual Paws & Pavement 5K on Saturday. A record 312 runners and walkers (and 87 dogs!) crossed the finish line in Lindell Park, and together we raised $14,320 for the Pinecrest Animal Shelter. That's enough to cover vaccines and spay/neuter surgeries for roughly 140 animals. Congratulations to our overall winners, Tamsin Rhee (18:42) and Gideon Oyelaran (16:55), and to Pepper the border collie, who won the costume contest dressed as a hot dog. Huge thanks to our 40 volunteers and to Fernway Credit Union for sponsoring the water stations. We also want to thank the neighbors along Lindell Parkway for their patience with the road closure, and the Lindell Park staff, who had the grounds cleaned up by noon. If you signed up but couldn't make it, your race shirt is waiting for you at the shelter's front desk through the end of the month. Race photos are posted on our website. Save the date: next year's run is Saturday, September 11, 2027. - The Paws & Pavement team """
What we expected
{
"kind": "summarize",
"word_limit": 50,
"facts": [
{
"id": "raised",
"desc": "raised $14,320",
"any": [
"14,?320"
]
},
{
"id": "for",
"desc": "for Pinecrest Animal Shelter",
"any": [
"pinecrest|animal shelter|shelter"
]
},
{
"id": "runners",
"desc": "record 312 participants",
"any": [
"312"
]
},
{
"id": "next",
"desc": "next year's run September 11, 2027",
"any": [
"sept?(ember|\\.)?\\s*11(th)?\\b|11(th)?\\s+(of\\s+)?sept?|9/11|2027"
]
},
{
"id": "thanks",
"desc": "thanks to volunteers/sponsor/participants",
"any": [
"thank|volunteer|sponsor|fernway"
]
}
],
"allowed_numbers": [],
"source_words": 190,
"reference_summary": "The 4th Paws & Pavement 5K drew a record 312 runners and walkers plus 87 dogs and raised $14,320 for the Pinecrest Animal Shelter. Thanks go to 40 volunteers and sponsor Fernway Credit Union. Next year's run is Saturday, September 11, 2027."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 85 (scripted checks 80, judges 9.0/10 from 3) · 2.91 s
The 4th annual Paws & Pavement 5K drew 312 runners/walkers and 87 dogs in Lindell Park, raising $14,320 for the Pinecrest Animal Shelter. Winners were Tamsin Rhee and Gideon Oyelaran; Pepper won the costume contest. Next year's run is September 11, 2027.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": true,
"thanks": false
},
"facts_frac": 0.8,
"word_count": 41,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 83 (scripted checks 80, judges 8.5/10 from 3) · 2.63 s
The 4th annual Paws & Pavement 5K drew 312 runners, 87 dogs, and raised $14,320 for the Pinecrest Animal Shelter—funding vaccines and spay/neuter for ~140 animals. Winners: Tamsin Rhee and Gideon Oyelaran; Pepper won the costume contest. Next year's run: September 11, 2027.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": true,
"thanks": false
},
"facts_frac": 0.8,
"word_count": 42,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 83 (scripted checks 85, judges 8.0/10 from 3) · 3.96 s
The 4th annual Paws & Pavement 5K saw 312 runners, walkers, and 87 dogs cross the finish line, raising $14,320 for Pinecrest Animal Shelter—enough for vaccines and spay/neuter for ~140 animals. Winners: Tamsin Rhee (18:42) and Gideon Oyelaran (16:55); Pepper the border collie won the costume contest. Volunteers, Fernway Credit Union, neighbors, and park staff supported the event. Shirts for missed participants are at the shelter; photos online; next run scheduled for September 11 2027.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": true,
"thanks": true
},
"facts_frac": 1,
"word_count": 74,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 70 (scripted checks 60, judges 8.0/10 from 3) · 1.56 s
The 4th annual Paws & Pavement 5K on Saturday had 312 runners/walkers and 87 dogs, raising $14,320 for Pinecrest Animal Shelter. Tamsin Rhee and Gideon Oyelaran won, and Pepper won the costume contest.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": false,
"thanks": false
},
"facts_frac": 0.6,
"word_count": 32,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 80 (scripted checks 85, judges 7.5/10 from 3) · 1.52 s
Record 312 runners and 87 dogs raised $14,320 for Pinecrest Animal Shelter, covering vaccines and spay/neuter surgeries for ~140 animals. Winners: Tamsin Rhee (18:42) and Gideon Oyelaran (16:55); Pepper the border collie won costume contest as hot dog. Thanks to 40 volunteers, Fernway Credit Union, Lindell Park neighbors, and staff. Shirts available for absent registrants; photos on website. Next year's run: September 11, 2027.
Judge medians: Faithfulness 9 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": true,
"thanks": true
},
"facts_frac": 1,
"word_count": 64,
"word_limit": 50,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 85 (scripted checks 80, judges 9.0/10 from 3) · 0.84 s
The 4th annual Paws & Pavement 5K drew 312 runners and walkers, raising $14,320 for the Pinecrest Animal Shelter. Winners included Tamsin Rhee and Gideon Oyelaran, with Pepper the border collie winning the costume contest. The event was supported by 40 volunteers and Fernway Credit Union.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": false,
"thanks": true
},
"facts_frac": 0.8,
"word_count": 45,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✗ fail · score 89 (scripted checks 93, judges 8.5/10 from 3) · 1.19 s
The 4th annual Paws & Pavement 5K had 312 runners and 87 dogs, raising $14,320 for the Pinecrest Animal Shelter. Overall winners were Tamsin Rhee and Gideon Oyelaran, with Pepper the border collie winning the costume contest. Thanks to volunteers, sponsors, and neighbors. Race shirts are available until month's end; next year's run is September 11, 2027.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": true,
"thanks": true
},
"facts_frac": 1,
"word_count": 56,
"word_limit": 50,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 80 (scripted checks 80, judges 8.0/10 from 3) · 0.40 s
312 runners and walkers crossed the finish line, raising $14,320 for the shelter. Overall winners: Tamsin Rhee and Gideon Oyelaran. Pepper won the costume contest. Thanks to volunteers, Fernway Credit Union, and Lindell Park staff.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"raised": true,
"for": true,
"runners": true,
"next": false,
"thanks": true
},
"facts_frac": 0.8,
"word_count": 35,
"word_limit": 50,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-040All Summarize + extract itemssummarize-extract-042 →