Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-027

easy summarizationemail

The prompt

user

Summarize the email below in at most 40 words. Reply with only the summary.

"""
From: Dana Whitfield
To: Front Desk Team
Subject: Shift swap request for Saturday

Hi all,

I've had a family thing come up this weekend and need to swap my Saturday, April 18 morning shift (7:00 to 1:00) for any weekday shift next week. Kenji said he can probably cover, but he needs to check his daughter's soccer schedule first, so I don't want to count on it yet.

If anyone else can take it, please reply by Wednesday at noon so I can let Carla know before she posts the final schedule on Thursday. I'm happy to take a Tuesday or Thursday evening close in return, or two shorter shifts if that's easier for someone.

For anyone who hasn't done a Saturday morning before: it's usually quiet until about 9, then the weekend check-ins pick up. The opening checklist is in the blue binder under the desk, and I'll leave a note about the two late arrivals we're expecting from Friday night.

Thanks so much - I owe whoever helps a coffee.

Dana
"""

What we expected

{
  "kind": "summarize",
  "word_limit": 40,
  "facts": [
    {
      "id": "shift",
      "desc": "Dana needs her Saturday (April 18) morning shift covered",
      "any": [
        "(\\bapr(il|\\.)?\\s*18(st|nd|rd|th)?\\b|\\b18(st|nd|rd|th)?\\s+(of\\s+)?apr(il|\\.)?\\b|\\b0?4/0?18\\b|\\b2026-04-18\\b|\\beighteenth\\s+(of\\s+)?apr(il|\\.)?\\b|\\bapr(il|\\.)?\\s+(the\\s+)?eighteenth\\b|\\bsat(urday)?\\b)"
      ]
    },
    {
      "id": "return",
      "desc": "offers to take a weekday/evening shift in return",
      "any": [
        "in (return|exchange)|weekday|tuesday|evening|clos(e|ing)|trade|swap (it )?for|take (one of|a shift|your)|two shorter"
      ]
    },
    {
      "id": "deadline",
      "desc": "replies needed by Wednesday at noon",
      "any": [
        "\\bwed(nesday|\\.|\\b)"
      ]
    },
    {
      "id": "kenji",
      "desc": "Kenji might cover but isn't confirmed",
      "any": [
        "kenji"
      ]
    }
  ],
  "allowed_numbers": [],
  "source_words": 173,
  "reference_summary": "Dana needs someone to cover her Saturday, April 18 morning shift (7:00-1:00). Kenji might take it but isn't sure yet. Reply by Wednesday noon; Dana will take a weekday evening close or two shorter shifts in return."
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 83 (scripted checks 75, judges 9.0/10 from 3) · 1.96 s

Dana requests to swap her Saturday, April 18 morning shift (7:00–1:00) for any weekday shift next week, asking for replies by Wednesday noon. She offers Tuesday or Thursday evening close or two shorter shifts in return.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": false
  },
  "facts_frac": 0.75,
  "word_count": 36,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Ornith-1.5 9B

✗ fail · score 83 (scripted checks 75, judges 9.0/10 from 3) · 1.94 s

Dana Whitfield requests to swap her Saturday, April 18 morning shift (7:00–1:00) for any weekday shift next week. She asks for a reply by Wednesday noon and offers a Tuesday or Thursday evening close, or two shorter shifts in return.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": false
  },
  "facts_frac": 0.75,
  "word_count": 40,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Granite 4.2 8B

✗ fail · score 85 (scripted checks 85, judges 8.5/10 from 3) · 2.62 s

Dana requests to swap her Saturday, April 18 morning shift (7:00–1:00) for any weekday shift next week. Kenji may cover but needs to check his daughter’s soccer schedule. Dana asks for replies by Wednesday noon so she can inform Carla before Thursday’s schedule is posted. She offers to cover a Tuesday or Thursday evening close or two shorter shifts in return.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": true
  },
  "facts_frac": 1,
  "word_count": 61,
  "word_limit": 40,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

Spark-X2.5 4B

✗ fail · score 83 (scripted checks 75, judges 9.0/10 from 3) · 0.94 s

Dana requests a Saturday morning shift swap for April 18, offering Tuesday/Thursday evening or shorter shifts, needing replies by Wednesday noon.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": false
  },
  "facts_frac": 0.75,
  "word_count": 21,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 71 (scripted checks 93, judges 5.0/10 from 3) · 1.02 s

Dana needs to swap her Saturday, April 18 morning shift (7:00‑1:00) for a weekday next week. She’s open to Tuesday or Thursday evening or two shorter shifts, and asks for replies by Wednesday noon. She’ll cover Kenji’s check and leave a note about late arrivals.

Judge medians: Faithfulness 5 · Fluency and usefulness 5

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": true
  },
  "facts_frac": 1,
  "word_count": 45,
  "word_limit": 40,
  "length_score": 0.5,
  "unsupported_numbers": [],
  "number_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 83 (scripted checks 75, judges 9.0/10 from 3) · 0.51 s

Dana Whitfield requests a Saturday morning shift swap for April 18, offering a weekday evening close or two shorter shifts in return. She asks for a reply by Wednesday noon to inform Carla.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": false
  },
  "facts_frac": 0.75,
  "word_count": 33,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

MiniCPM5 2B

✓ pass · score 95 (scripted checks 100, judges 9.0/10 from 3) · 0.71 s

Dana needs to swap her Saturday morning shift for a weekday. Kenji may cover but needs to check his daughter's schedule first. Please reply by Wednesday noon so Carla can finalize the schedule.

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": true
  },
  "facts_frac": 1,
  "word_count": 33,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

K2-Horizon 0.9B

✗ fail · score 80 (scripted checks 75, judges 8.5/10 from 3) · 0.48 s

Dana asks to swap her Saturday morning shift for a weekday shift, needs a reply by Wednesday noon, and is open to Tuesday/Thursday evenings or shorter shifts.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail

{
  "facts": {
    "shift": true,
    "return": true,
    "deadline": true,
    "kenji": false
  },
  "facts_frac": 0.75,
  "word_count": 27,
  "word_limit": 40,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

← summarize-extract-026All Summarize + extract itemssummarize-extract-028 →