Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-007

medium proofreadnewsletter

The prompt

user

Proofread the text below. Fix spelling, grammar, and punctuation mistakes only. Do not rephrase sentences that are already correct, and keep names, numbers, wording, and layout otherwise the same. Reply with only the corrected text, with no preamble and no list of changes.

"""
Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours is 7 am to 9 pm daily. Each of the residents are allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.
"""

What we expected

{
  "type": "proofread",
  "source": "Pool Update\n\nThe pool will reopen on June 1 after the resurfacing project. The new hours is 7 am to 9 pm daily. Each of the residents are allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.",
  "errors": [
    {
      "wrong": "hours is",
      "fixed": [
        "hours are"
      ]
    },
    {
      "wrong": "Each of the residents are",
      "fixed": [
        "Each of the residents is",
        "Each resident is"
      ]
    },
    {
      "wrong": "everyone who's patience",
      "fixed": [
        "everyone whose patience"
      ]
    }
  ],
  "preserve": [
    "June 1",
    "7 am",
    "9 pm",
    "two guests"
  ],
  "must_keep": [
    "may still be loose"
  ],
  "length_ratio": [
    0.8,
    1.25
  ],
  "reference": "Pool Update\n\nThe pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone whose patience made the project possible."
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.83 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone whose patience made the project possible.

Scoring detail

{
  "cleaned_words": 69,
  "similarity_to_source": 0.957,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 1,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 2.35 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each resident is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone whose patience made the project possible.

Scoring detail

{
  "cleaned_words": 67,
  "similarity_to_source": 0.926,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.971,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 0.971,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Granite 4.2 8B

✗ fail · score 42 · 2.56 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours is 7 am to 9 pm daily. Each of the residents are allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone whose patience made the project possible.

Scoring detail

{
  "cleaned_words": 69,
  "similarity_to_source": 0.986,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.971,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 0.6667,
  "checks": {
    "errors": 0.3333,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Spark-X2.5 4B

✗ fail · score 73 · 1.66 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents allows two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone's patience made the project possible.

Scoring detail

{
  "cleaned_words": 67,
  "similarity_to_source": 0.941,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": false,
      "score": 0.25
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": true,
      "fixed_present": false,
      "score": 0.25
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.956,
    "unknown_words": [
      "allows",
      "everyone's"
    ],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 0.971,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.5,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 82 · 1.14 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.

Scoring detail

{
  "cleaned_words": 69,
  "similarity_to_source": 0.971,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.986,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 82 · 0.84 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.

Scoring detail

{
  "cleaned_words": 69,
  "similarity_to_source": 0.971,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.986,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

MiniCPM5 2B

✗ fail · score 82 · 1.31 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each of the residents is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.

Scoring detail

{
  "cleaned_words": 69,
  "similarity_to_source": 0.971,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.986,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

K2-Horizon 0.9B

✗ fail · score 82 · 0.46 s

Pool Update

The pool will reopen on June 1 after the resurfacing project. The new hours are 7 am to 9 pm daily. Each resident is allowed two guests at a time, and guests must be accompanied by a resident. Please keep an eye on children near the deck, where a few tiles may still be loose. The board thanks everyone who's patience made the project possible.

Scoring detail

{
  "cleaned_words": 67,
  "similarity_to_source": 0.941,
  "errors": [
    {
      "wrong": "hours is",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "Each of the residents are",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "everyone who's patience",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.956,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 0.971,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

← rewrite-006All Rewrite and proofread itemsrewrite-008 →