Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-036

hard summarizationemail

The prompt

user

Summarize the email thread (newest message first) below in at most 70 words. Give the final plan only: the move date, what staff must do and by when, and when work resumes. Reply with only the summary.

"""
From: Patrice Odum (Office Manager)
Sent: Wednesday, November 4, 2026 4:51 PM
To: All Staff - Dalton & Reyes
Subject: RE: RE: Office move - FINAL date

Hi everyone,

Last update on this, I promise. The building at 220 Commerce Way couldn't give us the freight elevator on the 13th after all - another tenant had it booked all day. We now have it confirmed, in writing, for Friday, November 20. That is our move date. The movers arrive at 7 a.m. on the 20th; please don't come to either office that day.

What you need to do:
- Pack your desk into the crates (they'll be delivered Monday, Nov 16) and label each crate with the colored stickers for your team.
- Crates must be packed and sealed by 5 p.m. on Thursday, November 19.
- Take home anything personal and valuable.

IT will set up computers over the weekend, and we'll be working in the new office (3rd floor, 220 Commerce Way) starting Monday, November 23. Parking passes for the new garage will be on your desks.

Thanks for your flexibility through all these changes!
Patrice

-----Original Message-----
From: Patrice Odum
Sent: Monday, October 26, 2026 9:10 AM
Subject: RE: Office move - date change

Quick change: because the carpet install is behind, we're pushing the move from November 6 to Friday, November 13. Crates will be delivered Nov 9. Pack by Thursday the 12th at 5 p.m.

-----Original Message-----
From: Patrice Odum
Sent: Tuesday, October 13, 2026 2:30 PM
Subject: Office move

Hi all, our move to the new office at 220 Commerce Way is scheduled for Friday, November 6. More details to come. We'll have 14 crates per team, and the movers have budgeted 10 hours for the job.
"""
What we expected
{
  "kind": "summarize",
  "word_limit": 70,
  "facts": [
    {
      "id": "date",
      "desc": "move is Friday, November 20",
      "any": [
        "(\\bnov(ember|\\.)?\\s*20(st|nd|rd|th)?\\b|\\b20(st|nd|rd|th)?\\s+(of\\s+)?nov(ember|\\.)?\\b|\\b0?11/0?20\\b|\\b2026-11-20\\b|\\btwentieth\\s+(of\\s+)?nov(ember|\\.)?\\b|\\bnov(ember|\\.)?\\s+(the\\s+)?twentieth\\b|\\bfri(day|\\.)?,?\\s+the\\s+20(st|nd|rd|th)\\b)"
      ]
    },
    {
      "id": "pack",
      "desc": "crates packed/sealed by 5 p.m. Thursday November 19",
      "any": [
        [
          "pack|crate|seal",
          "(\\bnov(ember|\\.)?\\s*19(st|nd|rd|th)?\\b|\\b19(st|nd|rd|th)?\\s+(of\\s+)?nov(ember|\\.)?\\b|\\b0?11/0?19\\b|\\b2026-11-19\\b|\\bnineteenth\\s+(of\\s+)?nov(ember|\\.)?\\b|\\bnov(ember|\\.)?\\s+(the\\s+)?nineteenth\\b|\\bthu(rsday|\\.)?,?\\s+the\\s+19(st|nd|rd|th)\\b)"
        ]
      ]
    },
    {
      "id": "where",
      "desc": "new office at 220 Commerce Way (3rd floor)",
      "any": [
        "220 commerce|commerce way"
      ]
    },
    {
      "id": "resume",
      "desc": "working in the new office Monday November 23",
      "any": [
        "(\\bnov(ember|\\.)?\\s*23(st|nd|rd|th)?\\b|\\b23(st|nd|rd|th)?\\s+(of\\s+)?nov(ember|\\.)?\\b|\\b0?11/0?23\\b|\\b2026-11-23\\b|\\btwenty[- ]third\\s+(of\\s+)?nov(ember|\\.)?\\b|\\bnov(ember|\\.)?\\s+(the\\s+)?twenty[- ]third\\b|\\bmon(day|\\.)?,?\\s+the\\s+23(st|nd|rd|th)\\b)"
      ]
    },
    {
      "id": "stay_away",
      "desc": "don't come in on move day (movers 7 a.m.)",
      "any": [
        "don'?t come|do not come|stay (home|away)|no one|not come|7\\s*(a\\.?m|am|:00)|movers arrive"
      ]
    }
  ],
  "allowed_numbers": [],
  "source_words": 293,
  "reference_summary": "The office move is final for Friday, November 20; movers arrive at 7 a.m., so don't come in that day. Crates arrive Monday, November 16 and must be packed, labeled and sealed by 5 p.m. Thursday, November 19. Work resumes Monday, November 23 at the new office, 3rd floor, 220 Commerce Way."
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 3.74 s

Final plan: Move to 3rd floor, 220 Commerce Way on Friday, Nov 20 (movers arrive 7 a.m.; no one at either office that day). Staff must pack desks into crates (delivered Mon, Nov 16), label with team stickers, seal by 5 p.m. Thu, Nov 19, and take personal items home. Work resumes Mon, Nov 23.

Judge medians: Faithfulness 10 · Fluency and usefulness 10

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": true,
    "resume": true,
    "stay_away": true
  },
  "facts_frac": 1,
  "word_count": 55,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Ornith-1.5 9B

✗ fail · score 73 (scripted checks 60, judges 8.5/10 from 3) · 2.50 s

Move date: Friday, November 20. Staff must pack desks into crates (delivered Monday, Nov 16), label with team stickers, seal crates by 5 p.m. Thursday, Nov 19, and take home personal/valuable items. Work resumes Monday, November 23 in the new office.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": false,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.6,
  "word_count": 41,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Granite 4.2 8B

✗ fail · score 73 (scripted checks 60, judges 8.5/10 from 3) · 2.89 s

Move date: Friday, November 20, 2026. Staff must pack their desks into crates, label each crate with team‑colored stickers, and seal all crates by 5 p.m. on Thursday, November 19. Take home personal and valuable items. Work resumes on Monday, November 23, 2026 in the new office.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": false,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.6,
  "word_count": 47,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Spark-X2.5 4B

✗ fail · score 78 (scripted checks 80, judges 7.5/10 from 3) · 2.79 s

**Summary:**

The office move to 220 Commerce Way is confirmed for Friday, November 20. Staff must pack and seal crates by 5:00 p.m. on Thursday, November 19, and take personal items home. Work resumes in the new office starting Monday, November 23.

**Final Plan:**
* **Move Date:** Friday, November 20
* **Staff Action:** Pack and seal crates by 5:00 p.m. on Thursday, November 19; take personal items home.
* **Work Resumes:** Monday, November 23

Judge medians: Faithfulness 10 · Fluency and usefulness 5

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": true,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.8,
  "word_count": 70,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 85 (scripted checks 80, judges 9.0/10 from 3) · 1.16 s

Move date: Friday, November 20. Staff must pack desks into crates delivered Monday Nov 16, label with colored stickers, seal by 5 p.m. Thursday Nov 19, and take personal items home. Work resumes Monday November 23 in the new office (3rd floor, 220 Commerce Way).

Judge medians: Faithfulness 10 · Fluency and usefulness 8

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": true,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.8,
  "word_count": 45,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 73 (scripted checks 60, judges 8.5/10 from 3) · 0.77 s

The move is confirmed for Friday, November 20. Pack desks into labeled crates by 5 p.m. on Thursday, November 19. Take home personal items. IT will set up computers over the weekend, and work resumes Monday, November 23.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": false,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.6,
  "word_count": 38,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

MiniCPM5 2B

✗ fail · score 73 (scripted checks 60, judges 8.5/10 from 3) · 0.90 s

Final move date: Friday, November 20. Pack desks into crates and label them by 5 p.m. Thursday, November 19. Work resumes Monday, November 23.

Judge medians: Faithfulness 10 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": false,
    "resume": true,
    "stay_away": false
  },
  "facts_frac": 0.6,
  "word_count": 24,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

K2-Horizon 0.9B

✗ fail · score 68 (scripted checks 80, judges 5.5/10 from 3) · 0.63 s

Move date: Friday, November 20.  
Staff must pack crates into labeled boxes by 5 p.m. on Thursday, November 19; movers arrive at 7 a.m. on that day.  
Work resumes Monday, November 23.

Judge medians: Faithfulness 6 · Fluency and usefulness 5

Scoring detail
{
  "facts": {
    "date": true,
    "pack": true,
    "where": false,
    "resume": true,
    "stay_away": true
  },
  "facts_frac": 0.8,
  "word_count": 32,
  "word_limit": 70,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

← summarize-extract-035All Summarize + extract itemssummarize-extract-037 →