Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Rewrite and proofread

rewrite-001

easy proofreademail

The prompt

user

Proofread the text below. Fix spelling, grammar, and punctuation mistakes only. Do not rephrase sentences that are already correct, and keep names, numbers, wording, and layout otherwise the same. Reply with only the corrected text, with no preamble and no list of changes.

"""
Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm untill 6 pm. Please let me know when you recieve this.

Thank you,
Priya Raman
"""

What we expected

{
  "type": "proofread",
  "source": "Hi Mr. Delgado,\n\nI wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm untill 6 pm. Please let me know when you recieve this.\n\nThank you,\nPriya Raman",
  "errors": [
    {
      "wrong": "Its dripping",
      "fixed": [
        "It's dripping",
        "It is dripping"
      ]
    },
    {
      "wrong": "untill",
      "fixed": [
        "until"
      ]
    },
    {
      "wrong": "recieve",
      "fixed": [
        "receive"
      ]
    }
  ],
  "preserve": [
    "Delgado",
    "4B",
    "Tuesday",
    "Priya Raman"
  ],
  "must_keep": [],
  "length_ratio": [
    0.8,
    1.25
  ],
  "reference": "Hi Mr. Delgado,\n\nI wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. It's dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.\n\nThank you,\nPriya Raman"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.96 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. It is dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 66,
  "similarity_to_source": 0.947,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.977,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1.015,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 2.38 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. It's dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 65,
  "similarity_to_source": 0.954,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 1,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

Granite 4.2 8B

✓ pass · score 100 · 2.74 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. It is dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 66,
  "similarity_to_source": 0.947,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.977,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1.015,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

Spark-X2.5 4B

✗ fail · score 92 · 1.77 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. It is dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I am home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 67,
  "similarity_to_source": 0.924,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.955,
    "unknown_words": [
      "am"
    ],
    "reworded": false,
    "unrequested_edits": [
      {
        "op": "replace",
        "reference": "i'm",
        "answer": "i am"
      }
    ]
  },
  "preserve_missing": [],
  "length_ratio": 1.031,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 1,
    "fidelity": 0.5,
    "preserve": 1,
    "length": 1
  }
}

Mellum2.1 12B-A2.5B

✗ fail · score 80 · 1.13 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 65,
  "similarity_to_source": 0.969,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.985,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 80 · 0.84 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 65,
  "similarity_to_source": 0.969,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.985,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

MiniCPM5 2B

✗ fail · score 80 · 0.91 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 65,
  "similarity_to_source": 0.969,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.985,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

K2-Horizon 0.9B

✗ fail · score 80 · 0.48 s

Hi Mr. Delgado,

I wanted to let you know that the kitchen faucet in apartment 4B has been leaking since Tuesday. Its dripping constantly and the cabinet underneath is starting to get wet. Could you send someone to take a look this week? I'm home most afternoons from 3 pm until 6 pm. Please let me know when you receive this.

Thank you,
Priya Raman

Scoring detail

{
  "cleaned_words": 65,
  "similarity_to_source": 0.969,
  "errors": [
    {
      "wrong": "Its dripping",
      "wrong_gone": false,
      "fixed_present": false,
      "score": 0
    },
    {
      "wrong": "untill",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    },
    {
      "wrong": "recieve",
      "wrong_gone": true,
      "fixed_present": true,
      "score": 1
    }
  ],
  "fidelity": {
    "similarity_to_reference": 0.985,
    "unknown_words": [],
    "reworded": false,
    "unrequested_edits": []
  },
  "preserve_missing": [],
  "length_ratio": 1,
  "length_bounds": [
    0.8,
    1.25
  ],
  "gate": 1,
  "checks": {
    "errors": 0.6667,
    "fidelity": 1,
    "preserve": 1,
    "length": 1
  }
}

All Rewrite and proofread itemsrewrite-002 →