Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-040
hard summarizationemail
The prompt
user
Summarize the building notice below in at most 80 words. Make sure each resident can tell when their own water is off. Reply with only the summary. """ NOTICE TO ALL RESIDENTS - PLANNED WATER SHUTOFF Ashgrove Terrace Apartments, Buildings A and B Posted: Monday, November 2, 2026 To replace the corroded main supply pipes, the water will be shut off in each building on the following dates. Please plan ahead. BUILDING A (units 101-148): Tuesday, November 10, 9:00 a.m. to 3:00 p.m. BUILDING B (units 201-248): Wednesday, November 11, 9:00 a.m. to 3:00 p.m. EXCEPTION: In Building B, units whose number ends in 4 (204, 214, 224, 234, 244) share an older riser that needs extra work. Water in those units will stay off until 5:00 p.m. The fitness room showers and the laundry room in Building A will be unavailable on both days. The east side of the parking lot (spaces 1-40) will be closed from Monday, November 9 through Thursday, November 12 so the plumbers can park their equipment. Residents with spaces there may use the overflow lot on Birch Street; please display your parking tag. Tips: Fill a few pitchers and a bathtub the night before for flushing. Do not run the dishwasher or washing machine the morning of your shutoff. When water is restored, the tap may sputter or run cloudy for a few minutes - run the cold tap until it clears. If your shutoff day doesn't work because of a medical need, contact the leasing office at least 3 business days ahead and we will try to help. Thank you, Ashgrove Terrace Management """
What we expected
{
"kind": "summarize",
"word_limit": 80,
"facts": [
{
"id": "a",
"desc": "Building A off Tuesday Nov 10, 9-3",
"any": [
"(building|bldg) a\\b(?:(?!building|bldg)[^.;]){0,60}(nov(ember)?\\s*10(th)?\\b|11/10\\b|\\b10(th)?\\s+(of\\s+)?nov|\\btue)",
"(\\btue(sday)?,?\\s*)?(nov(ember)?\\s*10(th)?\\b|11/10\\b|\\b10(th)?\\s+(of\\s+)?nov|\\btue)[\\s,:()-]{0,4}((for|in|at)\\s+)?(building|bldg) a\\b",
"buildings? a and b\\b[^.;]{0,60}(tue[^.;]{0,30}wed|10(th)?[^.;]{0,25}11)|(tue[^.;]{0,30}wed|10(th)?[^.;]{0,25}11)[^.;]{0,40}buildings? a and b"
]
},
{
"id": "b",
"desc": "Building B off Wednesday Nov 11, 9-3",
"any": [
"(building|bldg) b\\b(?:(?!building|bldg)[^.;]){0,60}(nov(ember)?\\s*11(th)?\\b|11/11\\b|\\b11(th)?\\s+(of\\s+)?nov|\\bwed)",
"(\\bwed(nesday)?,?\\s*)?(nov(ember)?\\s*11(th)?\\b|11/11\\b|\\b11(th)?\\s+(of\\s+)?nov|\\bwed)[\\s,:()-]{0,4}((for|in|at)\\s+)?(building|bldg) b\\b",
"buildings? a and b\\b[^.;]{0,60}(tue[^.;]{0,30}wed|10(th)?[^.;]{0,25}11)|(tue[^.;]{0,30}wed|10(th)?[^.;]{0,25}11)[^.;]{0,40}buildings? a and b"
]
},
{
"id": "hours",
"desc": "9 a.m. to 3 p.m.",
"any": [
"9(:00)?\\s*(a\\.?m\\.?)?\\s*(-|to|until|through)\\s*3|9\\s*a\\.?m|09:00|nine"
]
},
{
"id": "exception",
"desc": "Building B units ending in 4 stay off until 5 p.m.",
"any": [
[
"(end|ending) in 4|ends? (in|with) 4|ending with 4|204|units? .{0,20}4\\b",
"\\b5\\b|\\b5\\s*p|5:00|five|17:00"
]
]
},
{
"id": "parking",
"desc": "east parking lot (spaces 1-40) closed Nov 9-12; use Birch St overflow",
"any": [
"park|\\blot\\b"
]
},
{
"id": "prep",
"desc": "fill water ahead / avoid dishwasher & washer",
"any": [
"fill|pitcher|bathtub|store water|dishwasher|washing machine"
]
}
],
"allowed_numbers": [],
"source_words": 242,
"reference_summary": "Water will be off 9 a.m. to 3 p.m. in Building A on Tuesday, November 10 and in Building B on Wednesday, November 11. Building B units ending in 4 stay off until 5 p.m. Building A's laundry and fitness showers close both days. East parking spaces 1-40 close November 9-12; use the Birch Street overflow lot. Fill pitchers and the tub the night before."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 5.25 s
Water shutoffs at Ashgrove Terrace: Building A (units 101-148) Tuesday, Nov 10, 9 a.m.-3 p.m.; Building B (units 201-248) Wednesday, Nov 11, 9 a.m.-3 p.m. Building B units ending in 4 (204, 214, 224, 234, 244) stay off until 5 p.m. Fill pitchers/tub beforehand; don't run dishwasher or washing machine on shutoff morning. Fitness showers and Building A laundry unavailable both days. Parking spaces 1-40 closed Nov 9-12. Contact leasing office 3 business days ahead for medical needs.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 78,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 87 (scripted checks 83, judges 9.0/10 from 3) · 3.69 s
Water shutoffs at Ashgrove Terrace (Buildings A and B) run Nov 10 (A, units 101-148) and Nov 11 (B, units 201-248), 9 a.m. to 3 p.m. Building B units ending in 4 (204, 214, 224, 234, 244) stay off until 5 p.m. Fill pitchers and bathtub the night before; don't run dishwasher or washing machine in the morning.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": false,
"prep": true
},
"facts_frac": 0.8333,
"word_count": 58,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 6.70 s
Water will be shut off for replacement of corroded main pipes. Building A (units 101‑148) on Tuesday November 10 from 9:00 a.m. to 3:00 p.m.; Building B (units 201‑248) on Wednesday November 11 from 9:00 a.m. to 3:00 p.m., except units ending in 4 (204, 214, 224, 234, 244) which stay off until 5:00 p.m. Fitness‑room showers and laundry in Building A will be unavailable both days. Residents can use the overflow lot on Birch Street for parking spaces 1‑40 (closed Nov 9‑12). Fill pitchers and a bathtub the night before; avoid running dishwasher or washing machine on the morning of the shutoff. When water returns, run the cold tap until clear. Contact the leasing office at least three business days ahead for medical‑need exceptions.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 125,
"word_limit": 80,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 76 (scripted checks 67, judges 8.5/10 from 3) · 3.17 s
**Water Shutoff Schedule:** * **Building A (Units 101-148):** Tuesday, Nov. 10, 9:00 a.m. – 3:00 p.m. * **Building B (Units 201-248):** Wednesday, Nov. 11, 9:00 a.m. – 3:00 p.m. * **Exception:** Units ending in 4 in Building B (204, 214, 224, 234, 244) stay off until 5:00 p.m. **Building A Amenities:** Fitness room showers and laundry room closed on both days.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": false,
"prep": false
},
"facts_frac": 0.6667,
"word_count": 56,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 2.30 s
Building A (units 101‑148): water off Tuesday, Nov 10, 9 a.m.–3 p.m. Building B (units 201‑248): water off Wednesday, Nov 11, 9 a.m.–3 p.m., except units ending in 4 (204,214,224,234,244) stay off until 5 p.m. Fitness room showers and Building A laundry unavailable both days. Parking lot east side (spaces 1‑40) closed Nov 9‑12; use Birch Street overflow lot. Residents should fill pitchers and bathtub night before, avoid using appliances on shutoff day, and run cold tap until water clears after restoration. Medical need: contact leasing office at least 3 business days ahead.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 93,
"word_limit": 80,
"length_score": 0.5,
"unsupported_numbers": [
"204,214,224,234,244"
],
"number_score": 0.5
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 1.40 s
Planned water shutoff for Ashgrove Terrace Apartments, Buildings A and B. Building A: Nov 10, 9am-3pm. Building B: Nov 11, 9am-3pm, except units ending in 4 (204, 214, 224, 234, 244) off until 5pm. Fitness room showers and laundry in Building A unavailable both days. East parking lot closed Nov 9-12; use overflow lot on Birch Street. Fill pitchers and bathtub before, avoid running appliances. If medical need, contact leasing 3 business days ahead.
Judge medians: Faithfulness 9 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 74,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 1.73 s
Water shutoffs in Ashgrove Terrace: Building A (units 101-148) on Tuesday, Nov 10, 9am-3pm; Building B (units 201-248) on Wednesday, Nov 11, 9am-3pm. Exception: Building B units ending in 4 stay off until 5pm. Fitness room showers and laundry in Building A are unavailable. Parking lot east side (spaces 1-40) closed Nov 9-12; use overflow lot on Birch Street. Fill pitchers and tubs before, avoid running dishwasher or washing machine. If medical need, contact leasing office 3+ business days early.
Judge medians: Faithfulness 10 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 80,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✓ pass · score 93 (scripted checks 100, judges 8.5/10 from 3) · 0.72 s
Water shutoff scheduled for Building A on Tuesday, November 10, 9:00 a.m. to 3:00 p.m. and Building B on Wednesday, November 11, 9:00 a.m. to 3:00 p.m. (except units ending in 4, which remain shut until 5:00 p.m.). East side of parking lot closed Nov 9–12; overflow lot on Birch Street available. Avoid running dishwasher/washing machine on shutoff day. Contact leasing office 3 days early for medical needs.
Judge medians: Faithfulness 8 · Fluency and usefulness 9
Scoring detail
{
"facts": {
"a": true,
"b": true,
"hours": true,
"exception": true,
"parking": true,
"prep": true
},
"facts_frac": 1,
"word_count": 68,
"word_limit": 80,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-039All Summarize + extract itemssummarize-extract-041 →