Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-037

medium shortenemail

The prompt

user

Shorten the text below to at most 70 words. Keep the key facts (names, dates, times, amounts, numbers) accurate, and do not add anything new. Reply with only the shortened text, with no preamble and no word count.

"""
Hi Mr. and Mrs. Albright,

Thanks again for having us out last week to look at the deck. I know you mentioned you were hoping to get it done before your daughter's graduation party, so I wanted to get back to you as quickly as I could with some details. After looking at everything, our recommendation is to replace the deck boards and railings but keep the existing frame, which is in good shape except for two joists near the stairs that will need to be sistered. We'd use the composite boards you liked in the gray color, which come with a 25-year warranty. The total for materials and labor comes to $9,350, and we'd need a 30% deposit to order materials. The earliest we can start is May 12, and the job should take about six working days, weather permitting, so you'd be all set well before the party on June 7. If you'd like to go ahead, just sign the attached quote and send it back by April 30 so we can hold that slot. As always, feel free to call me with any questions.

Best,
Luis Ortega
Ortega Outdoor Builds
"""
What we expected
{
  "type": "shorten",
  "source": "Hi Mr. and Mrs. Albright,\n\nThanks again for having us out last week to look at the deck. I know you mentioned you were hoping to get it done before your daughter's graduation party, so I wanted to get back to you as quickly as I could with some details. After looking at everything, our recommendation is to replace the deck boards and railings but keep the existing frame, which is in good shape except for two joists near the stairs that will need to be sistered. We'd use the composite boards you liked in the gray color, which come with a 25-year warranty. The total for materials and labor comes to $9,350, and we'd need a 30% deposit to order materials. The earliest we can start is May 12, and the job should take about six working days, weather permitting, so you'd be all set well before the party on June 7. If you'd like to go ahead, just sign the attached quote and send it back by April 30 so we can hold that slot. As always, feel free to call me with any questions.\n\nBest,\nLuis Ortega\nOrtega Outdoor Builds",
  "max_words": 70,
  "facts": [
    {
      "name": "replace boards",
      "any": [
        "boards"
      ]
    },
    {
      "name": "railings",
      "any": [
        "railing",
        "railings"
      ]
    },
    {
      "name": "keep frame",
      "any": [
        "frame"
      ]
    },
    {
      "name": "composite",
      "any": [
        "composite"
      ]
    },
    {
      "name": "25-year warranty",
      "any": [
        "25-year",
        "25 year"
      ]
    },
    {
      "name": "$9,350",
      "any": [
        "$9350"
      ]
    },
    {
      "name": "30% deposit",
      "any": [
        "30%",
        "30 percent"
      ]
    },
    {
      "name": "start May 12",
      "any": [
        "may 12"
      ]
    },
    {
      "name": "six working days",
      "any": [
        "six working days",
        "6 working days",
        "six days",
        "6 days",
        "six business days",
        "6 business days"
      ]
    },
    {
      "name": "before June 7 party",
      "any": [
        "june 7"
      ]
    },
    {
      "name": "sign by April 30",
      "any": [
        "april 30"
      ]
    }
  ],
  "banned": [],
  "banned_re": [],
  "extra_numbers_ok": [
    "2",
    "6"
  ],
  "reference": "Hi Mr. and Mrs. Albright,\n\nWe recommend replacing the deck boards and railings and keeping the existing frame (two joists need sistering). Gray composite boards include a 25-year warranty. Total: $9,350, with a 30% deposit. We can start May 12; the work takes about six working days, so you'll be done before the June 7 party. Please sign and return the quote by April 30.\n\nLuis Ortega"
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 2) · 4.91 s

Hi Mr. and Mrs. Albright,

Thanks again for having us last week. Our recommendation is to replace the deck boards and railings, keeping the existing frame, which is sound except for two joists near the stairs needing sistering. We'd use the gray composite boards with a 25-year warranty. Total is $9,350, with a 30% deposit to order materials. Earliest start is May 12; the job takes about six working days, weather permitting, finishing well before your daughter's June 7 party. Please sign the attached quote and return it by April 30 to hold that slot.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 10 · Key information kept 9.5 · Readability 9.5

Scoring detail
{
  "cleaned_words": 101,
  "similarity_to_source": 0.558,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Ornith-1.5 9B

✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 5.44 s

Hi Mr. and Mrs. Albright,

Thanks again for having us out last week to look at the deck. Our recommendation is to replace the deck boards and railings but keep the existing frame, which is in good shape except for two joists near the stairs that will need to be sistered. We'd use the composite boards you liked in gray, which come with a 25-year warranty. The total for materials and labor comes to $9,350, and we'd need a 30% deposit to order materials. The earliest we can start is May 12, and the job should take about six working days, weather permitting, so you'd be all set well before the party on June 7. If you'd like to go ahead, just sign the attached quote and send it back by April 30 so we can hold that slot. As always, feel free to call me with any questions.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 9

Scoring detail
{
  "cleaned_words": 155,
  "similarity_to_source": 0.885,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Granite 4.2 8B

✗ fail · score 65 (scripted checks 30, judges 10.0/10 from 3) · 4.57 s

Hi Mr. and Mrs. Albright,

Thanks for having us out last week to look at the deck. Our recommendation is to replace the deck boards and railings but keep the existing frame, sistering two joists near the stairs. We’d use the composite boards you liked in gray, which carry a 25‑year warranty. Total cost is $9,350, with a 30 % deposit required to order materials. Work can start May 12 and should take about six working days, weather permitting, leaving time before your daughter’s graduation party on June 7. Please sign the attached quote and return it by April 30 to hold the slot. Feel free to call with any questions.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 10

Scoring detail
{
  "cleaned_words": 116,
  "similarity_to_source": 0.619,
  "facts_missing": [
    "30% deposit"
  ],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 0.9091,
    "numbers": 1,
    "length": 0
  }
}

Spark-X2.5 4B

✓ pass · score 88 (scripted checks 100, judges 7.7/10 from 3) · 2.17 s

Hi Mr. and Mrs. Albright,

We recommend replacing the deck boards and railings, keeping the frame. We will use gray composite boards with a 25-year warranty. The total is $9,350, requiring a 30% deposit. We can start May 12, taking six working days, completing before the June 7 party. Sign the attached quote by April 30 to hold the slot.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 7 · Key information kept 7 · Readability 9

Scoring detail
{
  "cleaned_words": 66,
  "similarity_to_source": 0.378,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 65 (scripted checks 33, judges 9.7/10 from 3) · 1.97 s

Hi Mr. and Mrs. Albright,

Thanks again for having us last week to look at the deck. We recommend replacing the deck boards and railings but keeping the existing frame; two joists near the stairs will need sistering. We will use the gray composite boards with a 25‑year warranty. Total cost: $9,350 (materials & labor). A 30% deposit is required to order materials. Earliest start: May 12. Project duration: about six working days, weather permitting, completing before your daughter's graduation party on June 7. Please sign the attached quote and return it by April 30 to secure the slot. Feel free to call me with any questions.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 10 · Key information kept 10 · Readability 9

Scoring detail
{
  "cleaned_words": 112,
  "similarity_to_source": 0.549,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 57 (scripted checks 30, judges 8.3/10 from 3) · 1.14 s

Hi Mr. and Mrs. Albright, thank you for last week's deck visit. We recommend replacing the boards and railings while keeping the frame, which is in good shape except for two joists near the stairs. We'll use composite boards in gray with a 25-year warranty. The total is $9,350, with a 30% deposit required to order materials. The earliest start is May 12, and the job should take about six working days. Please sign and return the quote by April 30.

Judge medians: Faithfulness 9 · Key information kept 6 · Readability 10

Scoring detail
{
  "cleaned_words": 81,
  "similarity_to_source": 0.438,
  "facts_missing": [
    "before June 7 party"
  ],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 0.9091,
    "numbers": 1,
    "length": 0
  }
}

MiniCPM5 2B

✗ fail · score 53 (scripted checks 33, judges 7.3/10 from 3) · 1.98 s

Hi Mr. and Mrs. Albright, thanks for last week's visit. Our recommendation: replace deck boards and railings, keep the frame (except two stair joists needing sintering), using the gray composite boards with a 25-year warranty. Total cost is $9,350, requiring a 30% deposit. Earliest start is May 12, taking about six working days, so you'd be ready before the June 7 party. Sign the attached quote and return by April 30 to hold the slot. Questions? Call me. Best, Luis Ortega, Ortega Outdoor Builds.

Judge medians: Faithfulness 6 · Key information kept 8 · Readability 8

Scoring detail
{
  "cleaned_words": 84,
  "similarity_to_source": 0.419,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 0.5,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0
  }
}

K2-Horizon 0.9B

✗ fail · score 73 (scripted checks 83, judges 6.3/10 from 3) · 0.71 s

Hi Mr. and Mrs. Albright,

Thanks for looking at the deck. Our recommendation: replace boards and railings, keep the frame, fix two joists near stairs. Use gray composite boards with 25-year warranty. Materials and labor: $9,350, 30% deposit needed. Start May 12, job takes ~6 days, weather permitting. Sign the attached quote and return by April 30 to hold the June 7 party slot. Feel free to call if you have questions.

Best,
Luis Ortega
Ortega Outdoor Builds

Judge medians: Faithfulness 5 · Key information kept 7 · Readability 7

Scoring detail
{
  "cleaned_words": 78,
  "similarity_to_source": 0.428,
  "facts_missing": [],
  "new_numbers": [],
  "max_words": 70,
  "gate": 1,
  "checks": {
    "facts": 1,
    "numbers": 1,
    "length": 0.5
  }
}

← rewrite-036All Rewrite and proofread itemsrewrite-038 →