Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread
rewrite-036
easy shortennewsletter
The prompt
user
Shorten the text below to at most 50 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count. """ Picture Day is coming up! Individual student photos will be taken on Thursday, September 25, in the school gym. Order forms were sent home in backpacks last Friday, and you can also order online at the photographer's website using our school code, WILLOW26. Please return paper forms with payment by September 23. Students who are absent on Picture Day can have their photos taken on the retake day, October 16. Students should wear their regular clothes; uniforms are not required for photos. If you have any questions, please contact the front office at extension 210. """
What we expected
{
"type": "shorten",
"source": "Picture Day is coming up! Individual student photos will be taken on Thursday, September 25, in the school gym. Order forms were sent home in backpacks last Friday, and you can also order online at the photographer's website using our school code, WILLOW26. Please return paper forms with payment by September 23. Students who are absent on Picture Day can have their photos taken on the retake day, October 16. Students should wear their regular clothes; uniforms are not required for photos. If you have any questions, please contact the front office at extension 210.",
"max_words": 50,
"facts": [
{
"name": "September 25",
"any": [
"september 25",
"sept 25",
"sept. 25",
"sep 25",
"sep. 25"
]
},
{
"name": "gym",
"any": [
"gym"
]
},
{
"name": "code WILLOW26",
"any": [
"willow26"
]
},
{
"name": "forms by September 23",
"any": [
"september 23",
"sept 23",
"sept. 23",
"sep 23",
"sep. 23"
]
},
{
"name": "retake October 16",
"any": [
"october 16",
"oct 16",
"oct. 16"
]
},
{
"name": "extension 210",
"any": [
"210"
]
}
],
"banned": [],
"banned_re": [],
"extra_numbers_ok": [],
"reference": "Picture Day is Thursday, September 25, in the gym. Order on paper forms or online with school code WILLOW26. Return paper forms with payment by September 23. Retakes are October 16. No uniforms needed. Questions: front office, extension 210."
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 97 (scripted checks 100, judges 9.3/10 from 3) · 2.59 s
Picture Day is Thursday, September 25, in the school gym. Order forms were sent home last Friday; order online with school code WILLOW26. Return paper forms with payment by September 23. Absent students may use the October 16 retake. Wear regular clothes; uniforms not required. Questions: front office, ext. 210.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 9
Scoring detail
{
"cleaned_words": 50,
"similarity_to_source": 0.524,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 1,
"checks": {
"facts": 1,
"numbers": 1,
"length": 1
}
}Ornith-1.5 9B
✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 2.45 s
Picture Day photos are on Thursday, September 25, in the school gym. Order forms were sent home last Friday; you can also order online using school code WILLOW26. Return paper forms with payment by September 23. Absent students can have retakes on October 16. Wear regular clothes; uniforms are not required. Contact the front office at extension 210 with questions.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 10
Scoring detail
{
"cleaned_words": 60,
"similarity_to_source": 0.658,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Granite 4.2 8B
✗ fail · score 90 (scripted checks 83, judges 9.7/10 from 3) · 2.28 s
Picture Day is on Thursday, September 25, in the school gym. Order forms were sent home last Friday; online orders use code WILLOW26. Return paper forms with payment by September 23. Absent students can be photographed on October 16. Wear regular clothes; uniforms are not required. Contact the front office at extension 210 with questions.
Judge medians: Faithfulness 10 · Key information kept 9 · Readability 10
Scoring detail
{
"cleaned_words": 55,
"similarity_to_source": 0.573,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 1,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0.5
}
}Spark-X2.5 4B
✗ fail · score 81 (scripted checks 83, judges 8.0/10 from 3) · 1.57 s
Picture Day is Thursday, September 25, in the school gym. Order forms were sent home last Friday and can be ordered online with code WILLOW26. Return paper forms with payment by September 23. Absent students can retake photos on October 16. Wear regular clothes; uniforms are not required. Contact the front office at extension 210 with questions.
Judge medians: Faithfulness 8 · Key information kept 8 · Readability 8
Scoring detail
{
"cleaned_words": 57,
"similarity_to_source": 0.579,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 1,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0.5
}
}Mellum2.1 12B-A2.5B
✗ fail · score 63 (scripted checks 33, judges 9.3/10 from 3) · 1.13 s
Picture Day is Thursday, September 25, in the gym. Order forms were sent home; you can also order online at the photographer's website using school code WILLOW26. Return paper forms with payment by September 23. Students absent on Picture Day can be retaken October 16. Wear regular clothes; uniforms not required. Contact front office at extension 210 for questions.
Judge medians: Faithfulness 10 · Key information kept 10 · Readability 8
Scoring detail
{
"cleaned_words": 59,
"similarity_to_source": 0.662,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 63 (scripted checks 33, judges 9.3/10 from 2) · 0.84 s
Picture Day is on Thursday, September 25, in the school gym. Order forms were sent home last Friday, and you can order online using school code WILLOW26. Please return forms with payment by September 23. Absent students can retake photos on October 16. Wear regular clothes; uniforms are not required. Contact the front office at extension 210 for questions.
Judge medians: Faithfulness 9.5 · Key information kept 9 · Readability 9.5
Scoring detail
{
"cleaned_words": 59,
"similarity_to_source": 0.649,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 0.5,
"checks": {
"facts": 1,
"numbers": 1,
"length": 0
}
}MiniCPM5 2B
✓ pass · score 87 (scripted checks 100, judges 7.3/10 from 3) · 1.10 s
Picture Day is Thursday, September 25, in the school gym. Order forms online at the photographer's website using school code WILLOW26, or return paper forms by September 23. Absent students can have photos on October 16. Wear regular clothes; uniforms are not required. Contact the front office at extension 210.
Judge medians: Faithfulness 7 · Key information kept 7 · Readability 8
Scoring detail
{
"cleaned_words": 50,
"similarity_to_source": 0.621,
"facts_missing": [],
"new_numbers": [],
"max_words": 50,
"gate": 1,
"checks": {
"facts": 1,
"numbers": 1,
"length": 1
}
}K2-Horizon 0.9B
✗ fail · score 78 (scripted checks 85, judges 7.0/10 from 3) · 0.32 s
Picture Day is Thursday, Sept 25. Order forms in backpacks last Friday. Return forms by Sept 23. Retake day is Oct 16. Wear regular clothes; uniforms not required. Contact front office ext 210.
Judge medians: Faithfulness 10 · Key information kept 4 · Readability 7
Scoring detail
{
"cleaned_words": 33,
"similarity_to_source": 0.375,
"facts_missing": [
"gym",
"code WILLOW26"
],
"new_numbers": [],
"max_words": 50,
"gate": 1,
"checks": {
"facts": 0.6667,
"numbers": 1,
"length": 1
}
}