Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-034
hard summarizationarticle
The prompt
user
Summarize the news article below in at most 100 words. Reply with only the summary. """ Council passes $412 million budget; bus fares rise, library Sundays spared By Devon Achebe, Carrow Valley Courier After a five-hour meeting that stretched past midnight, the Carrow Valley City Council on Tuesday approved a $412 million budget for the coming fiscal year, a 5.9% increase over this year's $389 million spending plan. The vote was 5-2, with council members Ilse Brandt and Omar Fakhoury voting no. The budget that passed is smaller than the $418 million version Mayor Teresa Valdivia proposed in May. To close the gap, the council trimmed a planned expansion of the city's tree-planting program and delayed the purchase of two street sweepers by a year. The property tax rate stays unchanged at 0.62 per $100 of assessed value, though rising home values mean many homeowners will still see larger bills. The city estimates the median homeowner's city tax bill will rise about $74 next year. The most contentious change is to bus fares. Starting January 1, a regular one-way fare will rise from $1.75 to $2.00, the first increase since 2019. Reduced fares for seniors, students and riders with disabilities will stay at 85 cents. Transit director Maya Lindholm told the council the increase would raise about $1.1 million a year and help cover a driver pay raise that the city agreed to in July. Several speakers during public comment asked the council to keep fares flat. "For people like me who ride four times a day, six days a week, that's another six dollars a week," said Rolando Vega, a hospital food-service worker. Council member Fakhoury proposed an amendment to delay the increase until July; it failed 3-4. Library supporters had a better night. An early draft of the budget would have closed all four library branches on Sundays to save about $640,000. After more than 200 residents emailed the council and dozens spoke at last week's hearing, the council restored the Sunday hours, paying for them by drawing down the city's equipment reserve fund. Public safety spending grows by about $6 million. The budget funds six new firefighter positions, which Fire Chief Ronan Mbeki said will let the department staff the new Station 9 on the city's south side around the clock when it opens in the spring. The police department's budget is essentially flat; a request for eight additional officers was not funded, though the council approved money for a study of the department's staffing. Street repair gets a $3.5 million boost, which Public Works Director Sun-Hee Park said would let crews repave about 22 more lane-miles than this year and fill potholes faster. The city's goal is to respond to pothole reports within 48 hours. Brandt, explaining her no vote, said she could not support a budget that raised fares while "spending reserves we may need in a downturn." Fakhoury said he voted no mainly because of the fare increase. Pension costs continue to squeeze the budget. The city's required contribution to the state retirement system rises by $4.3 million next year, and Finance Director Albert Szymanski warned that the increase is expected to continue for at least three more years. "Every dollar of growth we have is spoken for before we start," he said. The council asked his office to return in February with options for paying down the city's unfunded pension liability faster. The budget also sets aside $750,000 for a pilot program that would send mental-health responders, rather than police officers, to some 911 calls. The pilot will start in two neighborhoods and be evaluated after 12 months. Not every proposal survived the night. A plan to add a second shift of code-enforcement inspectors was dropped, as was a proposal to extend pool hours at the city's two outdoor pools into September. Council member Priya Venkataraman, who sponsored the pool proposal, said she would bring it back during next year's mid-year budget review. Residents who spoke during the nearly two hours of public comment were split on the overall plan. Several business owners praised the street repair money, while others urged the council to rebuild reserves instead of spending them on library hours. Valdivia, who does not vote except to break ties, praised the council's work. "No one gets everything they want in a budget, and that's a sign it's balanced," she said. The new fiscal year begins October 1. The council will hold a separate vote next month on fees for parks and recreation programs. """
What we expected
{
"kind": "summarize",
"word_limit": 100,
"facts": [
{
"id": "approved",
"desc": "council approved a $412 million budget",
"any": [
"412"
]
},
{
"id": "vote",
"desc": "vote was 5-2",
"any": [
"5\\s*-\\s*2|5 to 2|five to two|five-to-two|5–2"
]
},
{
"id": "fare",
"desc": "bus fare rises from $1.75 to $2.00 starting Jan 1",
"any": [
[
"fare",
"\\$2(\\.00)?\\b|2\\.00|two dollars|1\\.75"
]
]
},
{
"id": "library",
"desc": "library Sunday hours kept/restored",
"any": [
[
"librar",
"sunday"
]
]
},
{
"id": "tax",
"desc": "property tax rate unchanged",
"any": [
[
"property tax|tax rate",
"unchanged|flat|same|no (property )?(tax )?(increase|change|hike)|not (be )?(raise|rais|chang|increas)|steady|stay|remain|hold|won'?t (change|rise|go up|increase)|keeps?\\b|kept|untouched|frozen|freez"
]
]
},
{
"id": "fire",
"desc": "six new firefighter positions",
"any": [
"firefight|fire department|station 9"
]
},
{
"id": "streets",
"desc": "$3.5 million more for street repair",
"any": [
"3\\.5|street repair|pothole|repav|road repair"
]
}
],
"allowed_numbers": [],
"source_words": 737,
"reference_summary": "Carrow Valley's council approved a $412 million budget 5-2, below the mayor's $418 million proposal. The property tax rate stays unchanged, though the median bill rises about $74. Bus fares rise from $1.75 to $2.00 on January 1; reduced fares stay 85 cents. The council restored Sunday library hours using equipment reserves. The budget adds six firefighters to staff Station 9 and $3.5 million for street repair, but not the eight requested police officers. Brandt and Fakhoury voted no."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 85 (scripted checks 86, judges 8.5/10 from 3) · 4.91 s
The Carrow Valley City Council approved a $412 million budget, up 5.9% from $389 million, by a 5-2 vote. Bus fares rise to $2 from January 1, generating about $1.1 million annually, while library Sunday hours were restored after 200+ emails. Public safety spending grows about $6 million for six firefighter positions, street repair gets $3.5 million, and pension contributions rise $4.3 million. The new fiscal year begins October 1.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": false,
"fire": true,
"streets": true
},
"facts_frac": 0.8571,
"word_count": 70,
"word_limit": 100,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 73 (scripted checks 86, judges 6.0/10 from 3) · 5.59 s
The Carrow Valley City Council approved a $412 million budget, a 5.9% increase over this year's $389 million plan, by a 5-2 vote. To close the gap with Mayor Valdivia's $418 million proposal, the council trimmed the tree-planting program and delayed two street sweepers. The property tax rate stays at 0.62, though median homeowner bills rise about $74. Bus fares increase from $1.75 to $2.00 starting January 1, restoring Sunday library hours, and public safety spending grows by $6 million. Street repair receives a $3.5 million boost, and $750,000 funds a mental-health 911 pilot.
Judge medians: Faithfulness 5 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": true,
"fire": false,
"streets": true
},
"facts_frac": 0.8571,
"word_count": 94,
"word_limit": 100,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 83 (scripted checks 85, judges 8.0/10 from 3) · 9.54 s
The Carrow Valley City Council approved a $412 million fiscal‑year budget, a 5.9 % increase over this year’s $389 million plan, after a five‑hour meeting that ran past midnight. The vote was 5‑2, with Ilse Brandt and Omar Fakhoury voting no. Compared with Mayor Teresa Valdivia’s May proposal of $418 million, the council trimmed a tree‑planting expansion and delayed two street sweepers. Property tax rate stays at 0.62 per $100 assessed value; median homeowner tax bill rises about $74. Bus fares rise from $1.75 to $2.00 per one‑way trip on Jan 1 (first increase since 2019); reduced fares for seniors, students and disabled riders stay at 85 cents. Library Sunday hours were restored after resident pressure, funded by the equipment reserve. Public safety spending grows about $6 million, adding six firefighter positions and funding a study for police staffing. Street repair gets a $3.5 million boost for extra lane‑miles and faster pothole repair. Pension contributions rise $4.3 million next year, prompting a request for faster pension‑liability repayment. A $750,000 pilot sends mental‑health responders instead of police to some 911 calls. Some proposals were dropped, including a second code‑enforcement shift and extending pool hours into September. Valdivia praised the balanced outcome.
Judge medians: Faithfulness 10 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": true,
"fire": true,
"streets": true
},
"facts_frac": 1,
"word_count": 199,
"word_limit": 100,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 78 (scripted checks 71, judges 8.5/10 from 3) · 2.72 s
The Carrow Valley City Council approved a $412 million budget, a 5.9% increase, with a 5-2 vote. Key changes include a bus fare rise from $1.75 to $2.00, a maintained property tax rate, and restored library Sunday hours funded by reserves. Public safety and street repair spending increased, while pension costs and bus fare increases were major concerns.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": false,
"fire": false,
"streets": true
},
"facts_frac": 0.7143,
"word_count": 58,
"word_limit": 100,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 74 (scripted checks 78, judges 7.0/10 from 3) · 3.28 s
Carrow Valley City Council approved a $412 million budget, a 5.9% increase from $389 million, with a 5-2 vote. Property tax rate remains 0.62 per $100 of assessed value, but median homeowner tax bills will rise about $74. Bus fares increase from $1.75 to $2.00 starting January 1, while reduced fares stay at 85 cents. Library Sunday hours were restored after public pressure, funded by equipment reserve. Public safety spending grows $6 million for six new firefighter positions; police budget flat with no new officers. Street repair gets $3.5 million boost for more lane-miles. Pension contributions rise $4.3 million. A mental-health responder pilot program allocated $750,000. Other items like second code-enforcement shift and extended pool hours were dropped. Council member Ilse Brandt and Omar Fakhoury voted no, citing fare increase and potential need for reserves. The fiscal year begins October 1. (99 words)
Judge medians: Faithfulness 8 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": true,
"fire": true,
"streets": true
},
"facts_frac": 1,
"word_count": 143,
"word_limit": 100,
"length_score": 0,
"unsupported_numbers": [
"99"
],
"number_score": 0.5
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 71 (scripted checks 71, judges 7.0/10 from 3) · 1.47 s
The Carrow Valley City Council approved a $412 million budget for the coming fiscal year, a 5.9% increase over the current plan. The budget includes a bus fare increase from $1.75 to $2.00, starting January 1, and preserves Sunday library hours. Public safety spending rises by $6 million, and street repair receives a $3.5 million boost. Pension costs and rising home values continue to pressure the budget. The council voted 5-2, with two members opposing the budget.
Judge medians: Faithfulness 7 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": false,
"fire": false,
"streets": true
},
"facts_frac": 0.7143,
"word_count": 77,
"word_limit": 100,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✗ fail · score 77 (scripted checks 79, judges 7.5/10 from 3) · 2.32 s
The Carrow Valley City Council approved a $412 million budget, a 5.9% increase over the prior year. The vote was 5-2, with Brandt and Fakhoury dissenting. The budget includes cuts to a tree-planting program and delays for street sweepers. Bus fares rise from $1.75 to $2.00, a point of contention. Library Sunday hours were restored after public pressure. Public safety spending grows by $6 million, funding new firefighter positions. Street repair gets a $3.5 million boost. Pension costs are expected to rise for years. The budget also sets aside $750,000 for a mental health responder pilot program. Several proposals, like adding a second code-enforcement inspector shift, were dropped.
Judge medians: Faithfulness 9 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"approved": true,
"vote": true,
"fare": true,
"library": true,
"tax": false,
"fire": true,
"streets": true
},
"facts_frac": 0.8571,
"word_count": 108,
"word_limit": 100,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 54 (scripted checks 73, judges 3.5/10 from 3) · 1.51 s
Crow Valley City Council approved a $412 million budget, up 5.9% from last year’s $389 million plan. Bus fares rise to $2.00 per one‑way ride (seniors, students, and disabled riders keep $0.85), while library Sunday hours are restored after residents demanded them. Property tax rate unchanged at 0.62 % per $100 of value; median homeowner tax bill projected to increase $74. Public‑safety spending grows $6 million, adding six firefighter positions and a $3.5 million street‑repair boost. Pension costs rise $4.3 million, and a pilot mental‑health responder program is set aside $750 k. Council members Ilse Brandt and Omar Fakhoury voted no on the fare increase. The budget also includes a $750 k pilot for mental‑health responders, a $6 million firefighter expansion, and a $6 million police‑staffing study.
Judge medians: Faithfulness 3 · Fluency and usefulness 4
Scoring detail
{
"facts": {
"approved": true,
"vote": false,
"fare": true,
"library": true,
"tax": true,
"fire": true,
"streets": true
},
"facts_frac": 0.8571,
"word_count": 126,
"word_limit": 100,
"length_score": 0,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-033All Summarize + extract itemssummarize-extract-035 →