Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-038
medium summarizationtranscript
The prompt
user
Summarize the podcast transcript below in at most 90 words. Focus on the guest's advice for beginners. Reply with only the summary. """ [Backyard Matters podcast, episode 112 - partial transcript] HOST (Wren): My guest today is Abel Nakashima, who runs Rooftop Bloom, a small honey business with hives on rooftops all over the city. Abel, how did this start? ABEL: By accident, honestly. In 2018 a coworker was moving and gave me two hives. I put them on my garage roof, mostly lost them the first winter, and got hooked anyway. Now we manage about 40 hives on 11 rooftops - restaurants, a hotel, a couple of office buildings. WREN: And how much honey does that make? ABEL: In a good year, around 1,500 pounds. Last year was a bad year, a wet spring, so more like 900. WREN: People hear "bees" and they worry about stings. Is that the hard part? ABEL: No, stings are the easy part. The hard part is varroa mites. They're tiny parasites, and they're the number one reason colonies die. If you're not checking your mite levels every month in the warm season, you will lose hives. I'd say most beginners who quit, quit because of mites, even if they don't know it. WREN: So for someone listening who wants to start - what's your advice? ABEL: Three things. First, start with two hives, not one. If one gets weak, you can borrow brood from the other, and you can compare them so you learn what normal looks like. Second, join your local beekeeping club and find a mentor. Seriously, a mentor saves you a year of mistakes. Ours, the Tri-County Beekeepers, meets the second Tuesday of every month. Third, budget realistically. For two hives with all the equipment, a suit, and the bees themselves, you're looking at six to eight hundred dollars the first year. WREN: That's more than I expected. ABEL: It surprises people. And you won't get much honey that first year, maybe none, because the bees need it to get through winter. WREN: Any rules people need to know about? ABEL: Check your city. Here, you have to register your hives with the county agriculture office - it's free - and keep them at least 10 feet from the property line. Some HOAs ban them outright, so look before you buy anything. WREN: Last question. What's the best part? ABEL: Opening a hive on a quiet morning. It's the calmest I ever feel. That, and the look on a chef's face when you hand them honey from their own roof. """
What we expected
{
"kind": "summarize",
"word_limit": 90,
"facts": [
{
"id": "two",
"desc": "start with two hives",
"any": [
"(two|2) (hives|colonies)|pair of hives"
]
},
{
"id": "mentor",
"desc": "join a local club / find a mentor",
"any": [
"club|mentor|association"
]
},
{
"id": "cost",
"desc": "expect $600-800 first-year cost",
"any": [
"600|800|six to eight hundred|six hundred|eight hundred"
]
},
{
"id": "mites",
"desc": "varroa mites are the biggest threat; check monthly",
"any": [
"mite|varroa"
]
},
{
"id": "rules",
"desc": "check local rules: register hives, 10 ft setback, HOA bans",
"any": [
"regist|hoa|rule|ordinance|regulat|property line|permit|law"
]
},
{
"id": "honey_first",
"desc": "little/no honey the first year",
"any": [
"(no|little|not much|minimal|any|much) honey|honey[^.;]{0,50}first (year|season|winter)|first (year|season)[^.;]{0,40}honey"
]
}
],
"allowed_numbers": [],
"source_words": 408,
"reference_summary": "Beekeeper Abel Nakashima, who manages about 40 rooftop hives, advises beginners to start with two hives, join a local beekeeping club and find a mentor, and budget $600-800 for the first year. Expect little or no honey at first. The biggest threat is varroa mites, so check mite levels monthly. Check local rules: hives may need county registration and a 10-foot setback, and some HOAs ban them."
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 87 (scripted checks 83, judges 9.0/10 from 3) · 4.16 s
Abel advises beginners to start with two hives rather than one, so you can compare them and borrow brood if one weakens. Join a local club and find a mentor to avoid a year of mistakes. Budget $600–$800 for the first year, and expect little or no honey. Check local rules: register hives with the county, keep them 10 feet from the property line, and verify HOA restrictions before buying.
Judge medians: Faithfulness 10 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": true
},
"facts_frac": 0.8333,
"word_count": 70,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ornith-1.5 9B
✗ fail · score 74 (scripted checks 77, judges 7.0/10 from 3) · 3.88 s
Abel Nakashima of Rooftop Bloom advises beginners to start with two hives, not one, so you can borrow brood and compare them to learn what normal looks like. Join your local beekeeping club and find a mentor, since a mentor saves you a year of mistakes. Budget realistically, expecting six to eight hundred dollars for two hives with equipment, a suit, and bees, and little honey the first year. Also check your city's rules, register hives with the county agriculture office for free, and keep them at least 10 feet from the property line.
Judge medians: Faithfulness 8 · Fluency and usefulness 6
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": true
},
"facts_frac": 0.8333,
"word_count": 94,
"word_limit": 90,
"length_score": 0.5,
"unsupported_numbers": [],
"number_score": 1
}Granite 4.2 8B
✗ fail · score 84 (scripted checks 83, judges 8.5/10 from 3) · 4.07 s
Abel advises beginners to start with two hives instead of one so they can share brood and compare normal behavior. He recommends joining a local beekeeping club and finding a mentor to avoid costly mistakes. He also urges realistic budgeting—about $600–$800 for two hives, equipment, and bees in the first year—recognizing that little or no honey may be harvested the first year. Beginners should check local regulations: register hives with the county agriculture office, keep them at least 10 feet from property lines, and verify HOA rules before installing.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": true
},
"facts_frac": 0.8333,
"word_count": 89,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Spark-X2.5 4B
✗ fail · score 84 (scripted checks 83, judges 8.5/10 from 3) · 2.25 s
Abel advises beginners to start with two hives, not one, to compare health and borrow brood if needed. He urges joining a local beekeeping club and finding a mentor to avoid costly mistakes. Beginners should budget six to eight hundred dollars for initial equipment and bees, and expect little to no honey in the first year. Always check local regulations, such as registering hives and maintaining distance from property lines.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": true
},
"facts_frac": 0.8333,
"word_count": 70,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Mellum2.1 12B-A2.5B
✗ fail · score 76 (scripted checks 67, judges 8.5/10 from 3) · 1.68 s
Start with two hives to compare health and borrow brood if needed. Join a local beekeeping club and find a mentor to avoid early mistakes. Budget about $6‑8 hundred for equipment, suits, and bees; first‑year honey may be minimal. Check local regulations: register hives with the county agriculture office and maintain at least ten feet from property lines; be aware of HOA restrictions. Varroa mites are the main threat — monitor them monthly. The rewarding moment is opening a hive and seeing chefs enjoy honey from their own roof.
Judge medians: Faithfulness 9 · Fluency and usefulness 8
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": false,
"mites": true,
"rules": true,
"honey_first": false
},
"facts_frac": 0.6667,
"word_count": 88,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 76 (scripted checks 67, judges 8.5/10 from 3) · 0.96 s
For beginners, Abel Nakashima advises starting with two hives to compare them and have a backup. Join a local beekeeping club and find a mentor to avoid common mistakes. Budget realistically for equipment, bees, and a suit, expecting little honey in the first year. Always check local regulations, including hive registration and setback requirements, and be aware of HOA rules.
Judge medians: Faithfulness 10 · Fluency and usefulness 7
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": false,
"mites": false,
"rules": true,
"honey_first": true
},
"facts_frac": 0.6667,
"word_count": 60,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}MiniCPM5 2B
✗ fail · score 63 (scripted checks 67, judges 6.0/10 from 3) · 1.36 s
For beginners, start with two hives to learn normal behavior. Join a local beekeeping club to find a mentor who can save you years of mistakes. Budget realistically, as setting up two hives with equipment costs six to eight hundred dollars initially. Also, check local regulations, as county rules and HOAs can affect hive placement. The best part is opening a hive on a quiet morning and seeing a chef enjoy honey from their own roof.
Judge medians: Faithfulness 7 · Fluency and usefulness 5
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": false
},
"facts_frac": 0.6667,
"word_count": 76,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}K2-Horizon 0.9B
✗ fail · score 58 (scripted checks 67, judges 5.0/10 from 3) · 0.87 s
Begin with two hives, compare them, join a local beekeeping club for mentorship, and budget $600‑$800 for equipment and bees. Check city regulations (register hives, keep them ≥10 ft from property line) and avoid HOAs. Start small, learn normal colonies, and enjoy the calm of opening a hive on a quiet morning.
Judge medians: Faithfulness 5 · Fluency and usefulness 5
Scoring detail
{
"facts": {
"two": true,
"mentor": true,
"cost": true,
"mites": false,
"rules": true,
"honey_first": false
},
"facts_frac": 0.6667,
"word_count": 52,
"word_limit": 90,
"length_score": 1,
"unsupported_numbers": [],
"number_score": 1
}← summarize-extract-037All Summarize + extract itemssummarize-extract-039 →