Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-042

hard shortenvoicemail

The prompt

user

Shorten the text below to at most 35 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count.

"""
Hey, it's Dr. Patel's office calling for Marisol Vega. Um, we need to move your follow-up appointment that was on Tuesday the 12th at 2 o'clock. Dr. Patel is going to be out that day. We can do Wednesday the 13th at 2, oh wait, sorry, no, Wednesday the 13th at 3:15 is what we have open. If that doesn't work, we also have Friday the 15th at 9 in the morning. Please call us back at 555-0172 to confirm. Also please remember to bring your updated insurance card, since we have the old one on file. Thanks, bye.
"""

What we expected

{
  "type": "shorten",
  "source": "Hey, it's Dr. Patel's office calling for Marisol Vega. Um, we need to move your follow-up appointment that was on Tuesday the 12th at 2 o'clock. Dr. Patel is going to be out that day. We can do Wednesday the 13th at 2, oh wait, sorry, no, Wednesday the 13th at 3:15 is what we have open. If that doesn't work, we also have Friday the 15th at 9 in the morning. Please call us back at 555-0172 to confirm. Also please remember to bring your updated insurance card, since we have the old one on file. Thanks, bye.",
  "max_words": 35,
  "facts": [
    {
      "name": "Dr. Patel",
      "any": [
        "patel"
      ]
    },
    {
      "name": "old appt 12th",
      "any": [
        "12th",
        "tuesday"
      ]
    },
    {
      "name": "Wednesday 13th",
      "any": [
        "13th",
        "wednesday"
      ]
    },
    {
      "name": "3:15",
      "any": [
        "3:15"
      ]
    },
    {
      "name": "Friday 15th",
      "any": [
        "15th",
        "friday"
      ]
    },
    {
      "name": "9 am",
      "any": [
        "9 a.m",
        "9 am",
        "9am",
        "9:00",
        "9 in the morning"
      ],
      "re": "(?:15th|fri(?:day)?)[^.;]{0,25}?(?:\\bat |, ?)(?:9|nine)\\b(?![:.]?\\d|\\s*p\\.?m)"
    },
    {
      "name": "callback 555-0172",
      "any": [
        "555-0172"
      ]
    },
    {
      "name": "insurance card",
      "any": [
        "insurance card"
      ]
    }
  ],
  "banned": [],
  "banned_re": [
    "13(?:th)?,? at 2(?![:\\d])"
  ],
  "extra_numbers_ok": [],
  "reference": "Dr. Patel's office: your Tuesday 12th appointment must move. Options: Wednesday the 13th at 3:15 or Friday the 15th at 9 a.m. Call 555-0172 to confirm. Bring your updated insurance card."
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 89 (scripted checks 84, judges 9.3/10 from 3) · 2.52 s

Dr. Patel's office calls Marisol Vega: Tuesday the 12th at 2 is cancelled. Please call 555-0172 to confirm Wednesday the 13th at 3:15, or Friday the 15th at 9 a.m., and bring your updated insurance card.

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 8

Scoring detail

{
  "cleaned_words": 36,
  "similarity_to_source": 0.356,
  "facts_missing": [],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "banned": 1,
    "numbers": 1,
    "length": 0.5
  }
}

Ornith-1.5 9B

✗ fail · score 59 (scripted checks 34, judges 8.3/10 from 3) · 2.40 s

Dr. Patel's office is calling for Marisol Vega. Your Tuesday, December 12, 2:00 PM appointment is moved because Dr. Patel is out. We have Wednesday, December 13, at 3:15 PM, or Friday, December 15, at 9:00 AM. Please call 555-0172 to confirm and bring your updated insurance card.

Judge medians: Faithfulness 5 · Key information kept 10 · Readability 10

Scoring detail

{
  "cleaned_words": 48,
  "similarity_to_source": 0.327,
  "facts_missing": [],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "banned": 1,
    "numbers": 1,
    "length": 0
  }
}

Granite 4.2 8B

✗ fail · score 60 (scripted checks 34, judges 8.7/10 from 3) · 2.21 s

Dr. Patel's office calls Marisol Vega to reschedule Tuesday the 12th 2 o'clock follow-up because Dr. Patel is out. New slot Wednesday the 13th 3:15 pm; alternative Friday the 15th 9 am. Call 555-0172 to confirm and bring updated insurance card.

Judge medians: Faithfulness 9 · Key information kept 10 · Readability 7

Scoring detail

{
  "cleaned_words": 41,
  "similarity_to_source": 0.371,
  "facts_missing": [],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "banned": 1,
    "numbers": 1,
    "length": 0
  }
}

Spark-X2.5 4B

✗ fail · score 80 (scripted checks 90, judges 7.0/10 from 3) · 1.36 s

Dr. Patel's office calls for Marisol Vega. Move your Tuesday 12th, 2 PM appointment. Wednesday 13th, 3:15 PM is open. Call 555-0172 to confirm and bring your updated insurance card.

Judge medians: Faithfulness 9 · Key information kept 5 · Readability 7

Scoring detail

{
  "cleaned_words": 30,
  "similarity_to_source": 0.295,
  "facts_missing": [
    "Friday 15th",
    "9 am"
  ],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 0.75,
    "banned": 1,
    "numbers": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 89 (scripted checks 84, judges 9.3/10 from 3) · 1.02 s

Dr. Patel's office calls Marisol Vega to reschedule her Tuesday, 12th, 2 pm appointment. New options: Wednesday, 13th at 3:15 pm or Friday, 15th at 9 am. Call 555-0172 to confirm and bring updated insurance card.

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 8

Scoring detail

{
  "cleaned_words": 36,
  "similarity_to_source": 0.267,
  "facts_missing": [],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "banned": 1,
    "numbers": 1,
    "length": 0.5
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 85 (scripted checks 84, judges 8.7/10 from 3) · 0.67 s

Dr. Patel's office calls to reschedule your 2 PM Tuesday appointment to Wednesday the 13th at 3:15 PM or Friday the 15th at 9 AM. Please call 555-0172 to confirm and bring your updated insurance card.

Judge medians: Faithfulness 10 · Key information kept 8 · Readability 8

Scoring detail

{
  "cleaned_words": 36,
  "similarity_to_source": 0.37,
  "facts_missing": [],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "banned": 1,
    "numbers": 1,
    "length": 0.5
  }
}

MiniCPM5 2B

✗ fail · score 79 (scripted checks 95, judges 6.3/10 from 3) · 0.70 s

Dr. Patel's office needs Marisol Vega's follow-up. Available: Wednesday the 13th at 3:15, or Friday the 15th at 9. Call 555-0172 to confirm. Bring updated insurance card.

Judge medians: Faithfulness 8 · Key information kept 5 · Readability 6

Scoring detail

{
  "cleaned_words": 27,
  "similarity_to_source": 0.302,
  "facts_missing": [
    "old appt 12th"
  ],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 0.875,
    "banned": 1,
    "numbers": 1,
    "length": 1
  }
}

K2-Horizon 0.9B

✗ fail · score 68 (scripted checks 90, judges 4.7/10 from 3) · 0.30 s

Dr. Patel's office calls Marisol Vega for Tuesday 12th at 2pm. Appointment moves to Wednesday 13th at 3:15. Back at 555-0172. Bring updated insurance card.

Judge medians: Faithfulness 5 · Key information kept 4 · Readability 5

Scoring detail

{
  "cleaned_words": 25,
  "similarity_to_source": 0.258,
  "facts_missing": [
    "Friday 15th",
    "9 am"
  ],
  "banned_hits": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 0.75,
    "banned": 1,
    "numbers": 1,
    "length": 1
  }
}

← rewrite-041All Rewrite and proofread items