Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-034

easy shortenemail

The prompt

user

Shorten the text below to at most 35 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count.

"""
Hi Jenna,

I hope your week is going well! I wanted to ask you a quick favor. I have a dentist appointment that I completely forgot about, and it's on Wednesday afternoon, which is when I'm scheduled to cover the front desk from 1 to 5. Would you be willing to swap shifts with me? I could take your Friday morning shift from 8 to 12 in exchange. No worries at all if it doesn't work for you, I can ask someone else.

Thanks so much,
Owen
"""

What we expected

{
  "type": "shorten",
  "source": "Hi Jenna,\n\nI hope your week is going well! I wanted to ask you a quick favor. I have a dentist appointment that I completely forgot about, and it's on Wednesday afternoon, which is when I'm scheduled to cover the front desk from 1 to 5. Would you be willing to swap shifts with me? I could take your Friday morning shift from 8 to 12 in exchange. No worries at all if it doesn't work for you, I can ask someone else.\n\nThanks so much,\nOwen",
  "max_words": 35,
  "facts": [
    {
      "name": "Jenna",
      "any": [
        "jenna"
      ]
    },
    {
      "name": "swap",
      "any": [
        "swap",
        "trade",
        "switch",
        "exchange"
      ]
    },
    {
      "name": "Wednesday",
      "any": [
        "wednesday",
        "wed"
      ]
    },
    {
      "name": "1 to 5",
      "any": [],
      "re": "\\b1\\s*(?::00)?\\s*(?:pm|p\\.m\\.)?\\s*(?:-|to|until|till|and|through)\\s*5\\b"
    },
    {
      "name": "Friday",
      "any": [
        "friday",
        "fri"
      ]
    },
    {
      "name": "8 to 12",
      "any": [],
      "re": "\\b8\\s*(?::00)?\\s*(?:am|a\\.m\\.)?\\s*(?:-|to|until|till|and|through)\\s*(?:12\\b|noon)"
    },
    {
      "name": "Owen",
      "any": [
        "owen"
      ]
    }
  ],
  "banned": [],
  "banned_re": [],
  "extra_numbers_ok": [],
  "reference": "Hi Jenna, could we swap shifts? I have a dentist appointment Wednesday during my 1 to 5 front desk shift. I'd take your Friday 8 to 12 shift instead. Thanks, Owen"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 98 (scripted checks 100, judges 9.7/10 from 3) · 2.06 s

Hi Jenna,

I forgot a dentist appointment Wednesday afternoon, conflicting with my 1–5 front desk shift. Could we swap? I'd cover your Friday 8–12 shift. No problem if it doesn't work.

Thanks,
Owen

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 9

Scoring detail

{
  "cleaned_words": 33,
  "similarity_to_source": 0.3,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 1
  }
}

Ornith-1.5 9B

✗ fail · score 66 (scripted checks 33, judges 10.0/10 from 3) · 2.04 s

Hi Jenna,

I have a dentist appointment on Wednesday afternoon, when I'm scheduled to cover the front desk from 1 to 5. Would you be willing to swap shifts with me? I could take your Friday morning shift from 8 to 12 in exchange. No worries if it doesn't work for you.

Thanks so much,
Owen

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 10

Scoring detail

{
  "cleaned_words": 56,
  "similarity_to_source": 0.769,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Granite 4.2 8B

✗ fail · score 77 (scripted checks 100, judges 5.3/10 from 3) · 1.84 s

Hi Jenna,

I forgot my Wednesday afternoon dentist appointment, which conflicts with my front‑desk coverage from 1–5 pm. Could you swap my Friday 8–12 am shift? No problem if not possible.

Thanks,
Owen

Judge medians: Faithfulness 4 · Key information kept 5 · Readability 7

Scoring detail

{
  "cleaned_words": 33,
  "similarity_to_source": 0.15,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 1
  }
}

Spark-X2.5 4B

✗ fail · score 90 (scripted checks 94, judges 8.7/10 from 3) · 0.99 s

Hi Jenna, I forgot my Wednesday afternoon dentist appointment, conflicting with my 1-5 front desk shift. Can we swap shifts? I'll cover your 8-12 Friday morning shift.

Judge medians: Faithfulness 10 · Key information kept 8 · Readability 8

Scoring detail

{
  "cleaned_words": 27,
  "similarity_to_source": 0.175,
  "facts_missing": [
    "Owen"
  ],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 0.8571,
    "numbers": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 (scripted checks 100, judges 10.0/10 from 3) · 0.82 s

Hi Jenna,

I have a dentist appointment Wednesday afternoon, conflicting with my front desk shift 1‑5. Could we swap? I’ll take your Friday morning shift 8‑12. Let me know.

Thanks,
Owen

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 10

Scoring detail

{
  "cleaned_words": 31,
  "similarity_to_source": 0.305,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 1
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 46 (scripted checks 33, judges 6.0/10 from 3) · 0.62 s

Hi Jenna, I have a dentist appointment on Wednesday afternoon, so I need to cover the front desk from 1 to 5. Could we swap shifts? I could take your Friday morning shift from 8 to 12. No worries if it doesn't work. Thanks, Owen.

Judge medians: Faithfulness 4 · Key information kept 7 · Readability 7

Scoring detail

{
  "cleaned_words": 45,
  "similarity_to_source": 0.53,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

MiniCPM5 2B

✗ fail · score 88 (scripted checks 94, judges 8.3/10 from 3) · 0.70 s

Hi Jenna, I have a dentist appointment on Wednesday afternoon. Can you swap shifts? I can take your Friday morning shift from 8 to 12. Thanks, Owen.

Judge medians: Faithfulness 10 · Key information kept 5 · Readability 10

Scoring detail

{
  "cleaned_words": 27,
  "similarity_to_source": 0.351,
  "facts_missing": [
    "1 to 5"
  ],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 0.8571,
    "numbers": 1,
    "length": 1
  }
}

K2-Horizon 0.9B

✗ fail · score 61 (scripted checks 83, judges 4.0/10 from 3) · 0.34 s

Hi Jenna, I hope your week is well. I need a dentist appointment on Wednesday afternoon. I can cover front desk 1-5 and swap with you for Friday 8-12. No problem if it doesn’t work. Thanks, Owen.

Judge medians: Faithfulness 3 · Key information kept 4 · Readability 5

Scoring detail

{
  "cleaned_words": 37,
  "similarity_to_source": 0.355,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 35,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0.5
  }
}

← rewrite-033All Rewrite and proofread itemsrewrite-035 →