Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-012

hard proofreadcover-letter

The prompt

user

Proofread the text below. Fix spelling, grammar, and punctuation mistakes only. Do not rephrase sentences that are already correct, and keep names, numbers, wording, and layout otherwise the same. Reply with only the corrected text, with no preamble and no list of changes.

"""
Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and it's waiting list has grown every semester. I have also trained four new staff members on the circulation system, which lead to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for you're time and consideration.

Sincerely,
Siobhan Kearney
"""

What we expected

{
  "type": "proofread",
  "source": "Dear Hiring Committee,\n\nI am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.\n\nNeither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and it's waiting list has grown every semester. I have also trained four new staff members on the circulation system, which lead to fewer checkout errors.\n\nI would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for you're time and consideration.\n\nSincerely,\nSiobhan Kearney",
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "fixed": [
        "colleagues were surprised"
      ]
    },
    {
      "wrong": "it's waiting list",
      "fixed": [
        "its waiting list"
      ]
    },
    {
      "wrong": "which lead to",
      "fixed": [
        "which led to",
        "which has led to"
      ]
    },
    {
      "wrong": "for you're time",
      "fixed": [
        "for your time"
      ]
    }
  ],
  "preserve": [
    "Fennimore Public Library",
    "Ashby Community College",
    "6,000",
    "45",
    "four",
    "Siobhan Kearney"
  ],
  "must_keep": [
    "Neither of our two campus libraries was open"
  ],
  "length_ratio": [
    0.8,
    1.25
  ],
  "reference": "Dear Hiring Committee,\n\nI am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.\n\nNeither my supervisor nor my colleagues were surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.\n\nI would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.\n\nSincerely,\nSiobhan Kearney"
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 86 · 5.95 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.979,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.993,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.75,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 4.89 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues were surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.972,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 1,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Granite 4.2 8B

✗ fail · score 86 · 4.91 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.979,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.993,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.75,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Spark-X2.5 4B

✗ fail · score 86 · 3.60 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.979,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.993,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.75,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 29 · 2.35 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and it's waiting list has grown every semester. I have also trained four new staff members on the circulation system, which lead to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.993,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "which lead to",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.979,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 0.5,
  "checks": {
    "errors": 0.25,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 86 · 1.69 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.979,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.993,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.75,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

MiniCPM5 2B

✗ fail · score 86 · 2.51 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and its waiting list has grown every semester. I have also trained four new staff members on the circulation system, which led to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.979,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "which lead to",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.993,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.75,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

K2-Horizon 0.9B

✗ fail · score 29 · 0.93 s

Dear Hiring Committee,

I am applying for the Assistant Librarian role at the Fennimore Public Library. For the past three years I have worked at Ashby Community College, where I catalogued more than 6,000 items and led a weekly story hour for toddlers. Neither of our two campus libraries was open on Sundays when I started, so I helped launch weekend hours.

Neither my supervisor nor my colleagues was surprised when I proposed a bilingual reading program; we had talked about the need for years. The program now serves 45 families, and it's waiting list has grown every semester. I have also trained four new staff members on the circulation system, which lead to fewer checkout errors.

I would be grateful for the chance to discuss how I could contribute to Fennimore. Thank you for your time and consideration.

Sincerely,
Siobhan Kearney

Scoring detail

{
  "cleaned_words": 142,
  "similarity_to_source": 0.993,
  "errors": [
    {
      "wrong": "colleagues was surprised",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "it's waiting list",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "which lead to",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "for you're time",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.979,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "must_keep_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 0.5,
  "checks": {
    "errors": 0.25,
    "fidelity": 1,
    "preserve": 1,
    "must_keep": 1,
    "length": 1
  }
}

← rewrite-011All Rewrite and proofread itemsrewrite-013 →