Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-039

easy summarizationemail

The prompt

user

Summarize the pet discharge instructions below in at most 70 words. Write it as a summary the owner can follow at home. Reply with only the summary.

"""
Cedar Lane Animal Hospital - Discharge Instructions

Patient: Biscuit (canine, 7 yrs)    Owner: Mr. Felix Arango
Procedure (Oct 16): dental cleaning under anesthesia, two teeth extracted (upper left premolars)

Biscuit did very well today. She may be sleepy tonight and a little wobbly; this is normal as the anesthesia wears off.

Feeding: Offer a small meal tonight (about half her usual amount). For the next 7 days, feed only soft food - canned food or kibble soaked in warm water until mushy.

Medication: Give one carprofen tablet (pain relief) twice daily with food for 5 days, starting tomorrow morning. Do not give any human pain medicine.

Activity: No chew toys, bones, or rawhides for 2 weeks while the gums heal. Short leash walks are fine.

Recheck: Friday, October 30 at 11:15 a.m. with Dr. Halvorsen to check the extraction sites.

Call us right away at (503) 555-0177 if you notice bleeding that doesn't stop, swelling of the face, or if Biscuit refuses to eat for more than 24 hours.
"""
What we expected
{
  "kind": "summarize",
  "word_limit": 70,
  "facts": [
    {
      "id": "food",
      "desc": "soft food only for 7 days",
      "any": [
        [
          "soft|soak|canned|mush",
          "\\b7\\b|seven|week"
        ]
      ]
    },
    {
      "id": "meds",
      "desc": "pain tablet twice daily with food for 5 days",
      "any": [
        [
          "carprofen|pain|medic|tablet|pill",
          "twice|two times|2 times|2x|every 12|morning and (evening|night)|two a day|2 a day"
        ]
      ]
    },
    {
      "id": "toys",
      "desc": "no chew toys/bones for 2 weeks",
      "any": [
        "chew|bone|rawhide"
      ]
    },
    {
      "id": "recheck",
      "desc": "recheck Friday October 30",
      "any": [
        "(\\boct(ober|\\.)?\\s*30(st|nd|rd|th)?\\b|\\b30(st|nd|rd|th)?\\s+(of\\s+)?oct(ober|\\.)?\\b|\\b0?10/0?30\\b|\\b2026-10-30\\b|\\bthirtieth\\s+(of\\s+)?oct(ober|\\.)?\\b|\\boct(ober|\\.)?\\s+(the\\s+)?thirtieth\\b)"
      ]
    },
    {
      "id": "call",
      "desc": "call if bleeding, swelling or not eating 24 hours",
      "any": [
        "bleed|swell|refus|not eat|won'?t eat|stops eating"
      ]
    }
  ],
  "allowed_numbers": [],
  "source_words": 169,
  "reference_summary": "Biscuit had a dental cleaning and two extractions. Feed a half meal tonight, then only soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow; no human pain medicine. No chew toys or bones for 2 weeks. Recheck is Friday, October 30 at 11:15 a.m. Call if bleeding won't stop, the face swells, or she refuses food for over 24 hours."
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 94 (scripted checks 93, judges 9.5/10 from 3) · 4.42 s

Biscuit may be sleepy or wobbly tonight; this is normal. Feed a small meal tonight, then soft food only for 7 days. Give one carprofen tablet twice daily with food for 5 days starting tomorrow; avoid human pain medicine. No chew toys, bones, or rawhides for 2 weeks; short leash walks are fine. Recheck Friday, Oct 30 at 11:15 a.m. Call (503) 555-0177 for uncontrolled bleeding, facial swelling, or refusal to eat over 24 hours.

Judge medians: Faithfulness 10 · Fluency and usefulness 9

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 75,
  "word_limit": 70,
  "length_score": 0.5,
  "unsupported_numbers": [],
  "number_score": 1
}

Ornith-1.5 9B

✗ fail · score 88 (scripted checks 85, judges 9.0/10 from 3) · 3.99 s

Biscuit did well after her dental cleaning and tooth extractions. She may be sleepy and wobbly tonight as anesthesia wears off. Feed a small meal tonight, then soft food only for 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow—no human pain medicine. Avoid chew toys, bones, and rawhides for 2 weeks. Recheck Friday, October 30 at 11:15 a.m. with Dr. Halvorsen. Call (503) 555-0177 for uncontrolled bleeding, facial swelling, or if Biscuit won't eat for over 24 hours.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 85,
  "word_limit": 70,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

Granite 4.2 8B

✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 4.32 s

Feed Biscuit a small meal tonight (about half her usual amount). For the next 7 days, give only soft food—canned food or kibble soaked in warm water until mushy. Give one carprofen tablet twice daily with food for 5 days starting tomorrow morning; do not give any human pain medicine. No chew toys, bones, or rawhides for 2 weeks while gums heal; short leash walks are fine. Recheck on Friday, October 30 at 11:15 a.m. Call (503) 555‑0177 immediately if you see bleeding that won’t stop, facial swelling, or if Biscuit refuses to eat for more than 24 hours.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 99,
  "word_limit": 70,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

Spark-X2.5 4B

✓ pass · score 83 (scripted checks 100, judges 6.5/10 from 3) · 2.21 s

**Discharge Instructions for Biscuit**

Biscuit may be sleepy and wobbly as anesthesia wears off. Feed a small meal tonight, then soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days. Avoid chew toys for 2 weeks. Keep short leash walks. Return Friday, Oct 30 at 11:15 a.m. Call (503) 555-0177 immediately for bleeding, swelling, or refusal to eat.

Judge medians: Faithfulness 7 · Fluency and usefulness 6

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 64,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 98 (scripted checks 100, judges 9.5/10 from 3) · 1.38 s

Offer a small, soft meal tonight; feed soft food for 7 days. Give carprofen tablet twice daily with food for 5 days, starting tomorrow. No chew toys, bones, or rawhides for 2 weeks; use short leash walks. Expect sleepiness and wobbliness as anesthesia fades. Call immediately if bleeding persists, facial swelling, or refusal to eat >24 h. Recheck on October 30 at 11:15 a.m.

Judge medians: Faithfulness 10 · Fluency and usefulness 9

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 64,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 86 (scripted checks 93, judges 8.0/10 from 3) · 1.19 s

Biscuit had a dental cleaning and two teeth removed. She may be sleepy and wobbly tonight. Feed her half her usual amount tonight, then soft food for 7 days. Give one carprofen tablet twice daily with food for 5 days. No chew toys or bones for 2 weeks. She needs a recheck on October 30 at 11:15 a.m. Call (503) 555-0177 if she bleeds, swells, or won't eat for over 24 hours.

Judge medians: Faithfulness 8 · Fluency and usefulness 8

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 72,
  "word_limit": 70,
  "length_score": 0.5,
  "unsupported_numbers": [],
  "number_score": 1
}

MiniCPM5 2B

✓ pass · score 85 (scripted checks 100, judges 7.0/10 from 3) · 1.09 s

Biscuit may be sleepy as anesthesia wears off. Feed small meals tonight, then only soft food for 7 days. Give carprofen twice daily for 5 days. Avoid chew toys, bones, or rawhides for 2 weeks. Bring her in Friday, Oct 30, for a recheck. Call immediately if there's heavy bleeding, facial swelling, or she refuses to eat.

Judge medians: Faithfulness 7 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 57,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

K2-Horizon 0.9B

✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 0.95 s

Biscuit’s anesthesia wears off; she may be sleepy and wobbly. Feed a small meal tonight (≈½ usual amount) and keep soft food (canned or soaked kibble) for the next 7 days. Give one carprofen tablet twice daily with food for 5 days, starting tomorrow morning—no human pain meds. No chew toys, bones, or rawhides for 2 weeks while gums heal. Short leash walks are fine. Recheck on Friday, Oct 30 at 11:15 a.m. with Dr. Halvorsen. Call us at (503) 555‑0177 if you see bleeding, facial swelling, or refusal to eat >24 hrs.

Judge medians: Faithfulness 9 · Fluency and usefulness 8

Scoring detail
{
  "facts": {
    "food": true,
    "meds": true,
    "toys": true,
    "recheck": true,
    "call": true
  },
  "facts_frac": 1,
  "word_count": 92,
  "word_limit": 70,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

← summarize-extract-038All Summarize + extract itemssummarize-extract-040 →