Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-040

hard shortenemail

The prompt

user

Shorten the text below to at most 40 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count.

"""
Hi everyone!!

I can't believe it's been three years since we were all together. Grandma Rose turns 90 this summer, and Aunt Carla and I have been talking about doing a big family reunion to celebrate. We found a lodge at Pine Hollow State Park that sleeps 24 people, and it's available the weekend of August 8-10. It costs $1,800 for the whole weekend, so if we get 12 families to chip in, it's $150 each. We'd need everyone's money by June 15 so we can pay the deposit. Please don't tell Grandma, we want it to be a surprise! Reply to me or call Carla by May 31 to let us know if you're in. Can't wait to see you all.

Love,
Megan
"""
What we expected
{
  "type": "shorten",
  "source": "Hi everyone!!\n\nI can't believe it's been three years since we were all together. Grandma Rose turns 90 this summer, and Aunt Carla and I have been talking about doing a big family reunion to celebrate. We found a lodge at Pine Hollow State Park that sleeps 24 people, and it's available the weekend of August 8-10. It costs $1,800 for the whole weekend, so if we get 12 families to chip in, it's $150 each. We'd need everyone's money by June 15 so we can pay the deposit. Please don't tell Grandma, we want it to be a surprise! Reply to me or call Carla by May 31 to let us know if you're in. Can't wait to see you all.\n\nLove,\nMegan",
  "max_words": 40,
  "facts": [
    {
      "name": "Grandma Rose",
      "any": [
        "rose"
      ]
    },
    {
      "name": "turns 90",
      "any": [
        "90",
        "90th"
      ]
    },
    {
      "name": "Pine Hollow",
      "any": [
        "pine hollow"
      ]
    },
    {
      "name": "August 8-10",
      "any": [],
      "re": "aug(?:ust|\\.)? 8(?:th)?\\s*(?:-|to|through|thru)\\s*(?:aug(?:ust|\\.)? )?10"
    },
    {
      "name": "$150 each",
      "any": [
        "$150"
      ]
    },
    {
      "name": "money by June 15",
      "any": [
        "june 15"
      ]
    },
    {
      "name": "surprise",
      "any": [
        "surprise",
        "secret",
        "don't tell"
      ]
    },
    {
      "name": "RSVP May 31",
      "any": [
        "may 31"
      ]
    },
    {
      "name": "Carla",
      "any": [
        "carla"
      ]
    }
  ],
  "banned": [],
  "banned_re": [],
  "extra_numbers_ok": [
    "3"
  ],
  "reference": "Surprise reunion for Grandma Rose's 90th: Pine Hollow State Park lodge, August 8-10. It's $150 per family, due June 15. Keep it secret! Tell Megan or Carla by May 31 if you're coming."
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 60 (scripted checks 33, judges 8.7/10 from 3) · 5.53 s

Megan: Grandma Rose turns 90 this summer. Aunt Carla and I are planning a surprise reunion at a Pine Hollow State Park lodge sleeping 24, available Aug 8-10 for $1,800, or $150 per family from 12. Please reply or call Carla by May 31; funds needed by June 15. Love, Megan.

Judge medians: Faithfulness 10 · Key information kept 9 · Readability 7

Scoring detail
{
  "cleaned_words": 51,
  "similarity_to_source": 0.309,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Ornith-1.5 9B

✗ fail · score 66 (scripted checks 33, judges 10.0/10 from 3) · 2.58 s

Hi everyone! Grandma Rose turns 90 this summer, and we're planning a surprise family reunion. We found a 24-person lodge at Pine Hollow State Park, available August 8-10, for $1,800 ($150 per family if 12 families join). Please reply to me or call Carla by May 31. We need everyone's money by June 15 for the deposit.

Love,
Megan

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 10

Scoring detail
{
  "cleaned_words": 59,
  "similarity_to_source": 0.383,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Granite 4.2 8B

✗ fail · score 60 (scripted checks 33, judges 8.7/10 from 3) · 2.80 s

Hi everyone!

Three years since we were all together. Grandma Rose turns 90 this summer. Aunt Carla and I plan a big family reunion. We found a lodge at Pine Hollow State Park that sleeps 24 people, available August 8-10. It costs $1,800, so $150 each for 12 families. Need money by June 15 for deposit. Please don't tell Grandma—surprise! Reply or call Carla by May 31 to confirm.

Judge medians: Faithfulness 10 · Key information kept 9 · Readability 7

Scoring detail
{
  "cleaned_words": 69,
  "similarity_to_source": 0.591,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Spark-X2.5 4B

✗ fail · score 85 (scripted checks 83, judges 8.7/10 from 3) · 1.59 s

Hi everyone! Grandma Rose turns 90 this summer. We're planning a family reunion at Pine Hollow State Park, August 8-10, costing $1,800. At $150 per family, we need 12 families. Pay by June 15. Keep it a surprise! Reply or call Carla by May 31.

Judge medians: Faithfulness 9 · Key information kept 8 · Readability 9

Scoring detail
{
  "cleaned_words": 45,
  "similarity_to_source": 0.32,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0.5
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 61 (scripted checks 33, judges 9.0/10 from 3) · 1.18 s

Megan: Three years since we were together. Grandma Rose turns 90 this summer. We plan a family reunion at Pine Hollow State Park lodge (capacity 24) weekend Aug 8‑10. Cost $1,800 total, $150 per family if 12 families join. Need payments by June 15. Don't tell Grandma. Reply to Megan or call Carla by May 31.

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 7

Scoring detail
{
  "cleaned_words": 56,
  "similarity_to_source": 0.411,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 57 (scripted checks 30, judges 8.3/10 from 3) · 0.83 s

Hi everyone! Grandma Rose turns 90 this summer, and we're planning a family reunion at Pine Hollow State Park for August 8-10. It costs $1,800 total, or $150 per family if 12 families chip in. We need everyone's money by June 15. Please reply or call Carla by May 31.

Judge medians: Faithfulness 10 · Key information kept 5 · Readability 10

Scoring detail
{
  "cleaned_words": 50,
  "similarity_to_source": 0.425,
  "facts_missing": [
    "surprise"
  ],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 0.8889,
    "numbers": 1,
    "length": 0
  }
}

MiniCPM5 2B

✗ fail · score 55 (scripted checks 33, judges 7.7/10 from 3) · 1.16 s

Three years passed since our last gathering. Grandma Rose turns 90 this summer. We're planning a reunion at Pine Hollow State Park, available August 8-10, for $1,800. If 12 families join, it's $150 each. Please send money by June 15 and keep the surprise. Reply to me or Carla by May 31. Love, Megan.

Judge medians: Faithfulness 9 · Key information kept 7 · Readability 7

Scoring detail
{
  "cleaned_words": 54,
  "similarity_to_source": 0.404,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 40,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

K2-Horizon 0.9B

✗ fail · score 71 (scripted checks 85, judges 5.7/10 from 3) · 0.33 s

Hi everyone! Grandma Rose turns 90 this summer. A family reunion at Pine Hollow State Park (24 guests) costs $1,800 for the weekend. Need contributions by June 15. Contact Carla by May 31.

Judge medians: Faithfulness 7 · Key information kept 3 · Readability 7

Scoring detail
{
  "cleaned_words": 33,
  "similarity_to_source": 0.306,
  "facts_missing": [
    "August 8-10",
    "$150 each",
    "surprise"
  ],
  "new_numbers": [],
  "max_words": 40,
  "gate": 1,
  "checks": {
    "facts": 0.6667,
    "numbers": 1,
    "length": 1
  }
}

← rewrite-039All Rewrite and proofread itemsrewrite-041 →