Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-044

medium summarizationemail

The prompt

user

Summarize the incident report email below in at most 80 words. Reply with only the summary.

"""
From: Owen Fairbanks, Safety Coordinator
To: Plant Management; Warehouse Supervisors
Date: Wednesday, October 14, 2026
Subject: Incident report - partial rack collapse, Aisle 7

Summary of incident:
At approximately 2:15 p.m. on Tuesday, October 13, the top beam level of rack bay 7-C in Aisle 7 partially collapsed while an operator was retrieving a pallet with a reach truck. Several cases of product fell from the top shelf. No one was injured. The operator, who has eight years of experience, stopped and backed out immediately and followed the evacuation procedure correctly.

Damage:
Roughly $8,000 in finished-goods inventory was damaged, mostly glass jars. One upright and two beams on bay 7-C are bent and must be replaced. The reach truck was not damaged.

Cause (preliminary):
The top shelf was carrying an estimated 2,600 pounds, but the beams at that level are rated for 2,000 pounds. Two heavier pallets of the new 24-count jar cases were put there during last Friday's overflow, and the load signs on that bay were missing.

Corrective actions:
1. Aisle 7 is closed until the rack vendor inspects it on Friday, October 16. Use Aisles 8 and 9 for picks.
2. All forklift and reach-truck operators will complete load-capacity refresher training by October 30. Supervisors, please schedule your people.
3. Facilities will install new load-rating signs on every bay in the building by the end of next week.
4. Heavier SKUs will be limited to the bottom two levels until further notice.

I'll send the final report after the vendor inspection. Thanks to everyone who helped with the cleanup.

Owen
"""
What we expected
{
  "kind": "summarize",
  "word_limit": 80,
  "facts": [
    {
      "id": "what",
      "desc": "partial rack collapse in Aisle 7 on Oct 13",
      "any": [
        [
          "rack|shelf|beam",
          "collaps|fell|fail"
        ]
      ]
    },
    {
      "id": "injury",
      "desc": "no one was injured",
      "any": [
        "no (one|injur|body)|nobody|without injur|no-one|uninjured"
      ]
    },
    {
      "id": "damage",
      "desc": "about $8,000 in damaged inventory",
      "any": [
        "8,?000|8k|\\$8 thousand|eight thousand"
      ]
    },
    {
      "id": "cause",
      "desc": "shelf overloaded (2,600 lb vs 2,000 lb rating)",
      "any": [
        "overload|over (its |the )?(rated |weight )?(capacity|limit|rating)|2,?600|exceed|too heavy|over-?weight|more than [^.;]{0,30}(rating|rated|capacity|limit)|beyond|above (its|the) (rating|rated|capacity|limit)|too much weight|heavier pallets|heavy pallets"
      ]
    },
    {
      "id": "closed",
      "desc": "Aisle 7 closed until inspection Friday Oct 16",
      "any": [
        [
          "clos|inspect",
          "(\\boct(ober|\\.)?\\s*16(st|nd|rd|th)?\\b|\\b16(st|nd|rd|th)?\\s+(of\\s+)?oct(ober|\\.)?\\b|\\b0?10/0?16\\b|\\b2026-10-16\\b|\\bsixteenth\\s+(of\\s+)?oct(ober|\\.)?\\b|\\boct(ober|\\.)?\\s+(the\\s+)?sixteenth\\b|\\bfri(day|\\.)?\\b)"
        ]
      ]
    },
    {
      "id": "training",
      "desc": "operator refresher training by Oct 30",
      "any": [
        [
          "train",
          "(\\boct(ober|\\.)?\\s*30(st|nd|rd|th)?\\b|\\b30(st|nd|rd|th)?\\s+(of\\s+)?oct(ober|\\.)?\\b|\\b0?10/0?30\\b|\\b2026-10-30\\b|\\bthirtieth\\s+(of\\s+)?oct(ober|\\.)?\\b|\\boct(ober|\\.)?\\s+(the\\s+)?thirtieth\\b)"
        ]
      ]
    }
  ],
  "allowed_numbers": [
    2026
  ],
  "source_words": 263,
  "reference_summary": "On Tuesday, October 13, a top rack level in Aisle 7 partially collapsed; no one was injured. About $8,000 of inventory was damaged. The shelf held about 2,600 pounds against a 2,000-pound rating, and load signs were missing. Aisle 7 is closed until the vendor inspection on Friday, October 16. All operators must finish refresher training by October 30, new load signs go up, and heavy items stay on the bottom two levels."
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 89 (scripted checks 93, judges 8.5/10 from 3) · 5.16 s

On Oct 13 at 2:15 p.m., the top beam level of rack bay 7-C in Aisle 7 partially collapsed while an operator retrieved a pallet; no injuries occurred. Roughly $8,000 in glass-jar inventory was damaged, and one upright plus two beams need replacement. Preliminary cause: a 2,600-pound load exceeded the 2,000-pound beam rating, with missing load signs. Aisle 7 is closed pending vendor inspection Oct 16; operators complete load-capacity training by Oct 30, new load-rating signs are being installed, and heavier SKUs are limited to the bottom two levels.

Judge medians: Faithfulness 9 · Fluency and usefulness 8

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": true,
    "training": true
  },
  "facts_frac": 1,
  "word_count": 89,
  "word_limit": 80,
  "length_score": 0.5,
  "unsupported_numbers": [],
  "number_score": 1
}

Ornith-1.5 9B

✗ fail · score 49 (scripted checks 33, judges 6.5/10 from 3) · 2.79 s

On Tuesday, October 13, 2026, at 2:15 p.m., the top beam level of rack bay 7-C in Aisle 7 partially collapsed while an operator retrieved a pallet with a reach truck. Several product cases fell, but no one was injured. The operator, who has eight years of experience, stopped and backed out immediately, following the evacuation procedure correctly.

Judge medians: Faithfulness 10 · Fluency and usefulness 3

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": false,
    "cause": false,
    "closed": false,
    "training": false
  },
  "facts_frac": 0.3333,
  "word_count": 58,
  "word_limit": 80,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Granite 4.2 8B

✗ fail · score 80 (scripted checks 85, judges 7.5/10 from 3) · 6.26 s

At about 2:15 p.m. on 13 Oct 2026, the top beam level of rack bay 7‑C in Aisle 7 partially collapsed while a reach‑truck operator retrieved a pallet; no one was injured. About $8,000 of glass‑jar inventory was damaged, and one upright and two beams need replacement. The reach truck was undamaged. Preliminary cause: the shelf carried ~2,600 lb against a 2,000 lb rating; load signs were missing and two heavy pallets were placed during last Friday’s overflow. Corrective actions: Aisle 7 closed until vendor inspection on 16 Oct; use Aisles 8‑9 for picks. All forklift/reach‑truck operators must complete load‑capacity refresher training by 30 Oct. Facilities will install load‑rating signs on all bays by end of next week, and heavier SKUs limited to bottom two levels until further notice.

Judge medians: Faithfulness 9 · Fluency and usefulness 6

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": true,
    "training": true
  },
  "facts_frac": 1,
  "word_count": 129,
  "word_limit": 80,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

Spark-X2.5 4B

✗ fail · score 58 (scripted checks 67, judges 5.0/10 from 3) · 2.31 s

On Tuesday, October 13, at 2:15 p.m., a partial rack collapse occurred in Aisle 7, Bay 7-C. An operator retrieving a pallet caused the top beam to collapse, dropping product. No injuries occurred. $8,000 in inventory was damaged. The cause was an overloaded shelf exceeding its rating, due to missing load signs. Corrective actions include closing Aisle 7, training operators, installing load-rating signs, and limiting heavier SKUs to lower levels.

Judge medians: Faithfulness 4 · Fluency and usefulness 6

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": false,
    "training": false
  },
  "facts_frac": 0.6667,
  "word_count": 70,
  "word_limit": 80,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 78 (scripted checks 85, judges 7.0/10 from 3) · 2.29 s

At approximately 2:15 p.m. on October 13, the top beam level of rack bay 7-C in Aisle 7 partially collapsed while an operator retrieved a pallet with a reach truck. Several cases of product fell; no one was injured. The operator, with eight years experience, stopped and evacuated correctly. Damage: about $8,000 in finished goods, mostly glass jars; one upright and two beams bent, requiring replacement; reach truck unharmed. Cause: top shelf carried ~2,600 lbs, exceeding the 2,000-lb rating; two heavier pallets were placed there last Friday and signs were missing. Corrective actions: Aisle 7 closed until vendor inspection on October 16; forklift/reach‑truck operators to complete load‑capacity refresher training by October 30; new load‑rating signs to be installed by end of next week; heavier SKUs limited to bottom two levels.

Judge medians: Faithfulness 9 · Fluency and usefulness 5

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": true,
    "training": true
  },
  "facts_frac": 1,
  "word_count": 130,
  "word_limit": 80,
  "length_score": 0,
  "unsupported_numbers": [],
  "number_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 73 (scripted checks 67, judges 8.0/10 from 3) · 1.07 s

On October 13, a partial rack collapse occurred in Aisle 7, damaging $8,000 in inventory. The top shelf was overloaded with 2,600 pounds, exceeding its 2,000-pound rating. The operator was uninjured and followed evacuation procedures. Corrective actions include closing Aisle 7 for inspection, refresher training for operators, installing new load signs, and limiting heavier SKUs to the bottom two levels.

Judge medians: Faithfulness 9 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": false,
    "training": false
  },
  "facts_frac": 0.6667,
  "word_count": 60,
  "word_limit": 80,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

MiniCPM5 2B

✗ fail · score 71 (scripted checks 67, judges 7.5/10 from 3) · 1.33 s

On Oct 13, a partial rack collapse in Aisle 7 occurred during a pallet retrieval. No injuries. About $8,000 in glass jars was damaged, and two beams plus an upright needed replacement. The cause was overloading: the top shelf carried 2,600 pounds, but beams are rated for 2,000. Corrective actions include closing Aisle 7 until vendor inspection, training for load-capacity, new load-rating signs, and limiting heavier SKUs to lower levels.

Judge medians: Faithfulness 8 · Fluency and usefulness 7

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": false,
    "training": false
  },
  "facts_frac": 0.6667,
  "word_count": 70,
  "word_limit": 80,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

K2-Horizon 0.9B

✗ fail · score 69 (scripted checks 83, judges 5.5/10 from 3) · 0.71 s

Incident report: Partial rack collapse in Aisle 7-C on Oct 13, 2:15 p.m., operator retrieved a pallet with a reach truck, causing product fall and no injuries. Damage: ~$8,000 in glass jars, beams bent. Causes: Overloaded shelf (2,600 lbs vs. 2,000 lb rating) and missing load signs. Corrections: Bay closed for vendor inspection, training by Oct 30, new load signs installed, heavier SKUs limited to bottom two levels.

Judge medians: Faithfulness 5 · Fluency and usefulness 6

Scoring detail
{
  "facts": {
    "what": true,
    "injury": true,
    "damage": true,
    "cause": true,
    "closed": false,
    "training": true
  },
  "facts_frac": 0.8333,
  "word_count": 68,
  "word_limit": 80,
  "length_score": 1,
  "unsupported_numbers": [],
  "number_score": 1
}

← summarize-extract-043All Summarize + extract items