Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Jobs
chat · 40 items
Translate
Translate everyday texts (signs, chats, emails, reminders, recipes, product specs) between English and Spanish, French, German, Japanese, Chinese and Brazilian Portuguese.
Results
| Model | Accuracy | Pass | Easy | Medium | Hard | Median latency | Tok/s | Job time |
|---|---|---|---|---|---|---|---|---|
| Ornith-1.5 9B | 91.9 | 82.5% | 100.0% | 84.6% | 64.3% | 1.65 s | 45.8 | 1m 04s |
| MiMo-V2.6 Distill 9B | 89.5 | 72.5% | 84.6% | 69.2% | 64.3% | 2.05 s | 37.8 | 2m 09s |
| Spark-X2.5 4B | 84.7 | 70.0% | 92.3% | 76.9% | 42.9% | 1.29 s | 65.6 | 51.6 s |
| Ling-3.0-tiny (7.9B-A1.3B) | 84.5 | 70.0% | 76.9% | 76.9% | 57.1% | 0.68 s | 118.4 | 26.4 s |
| Granite 4.2 8B | 79.1 | 55.0% | 69.2% | 53.8% | 42.9% | 2.08 s | 41.9 | 3m 33s |
| Mellum2.1 12B-A2.5B | 76.7 | 40.0% | 69.2% | 38.5% | 14.3% | 0.85 s | 108.0 | 33.4 s |
| MiniCPM5 2B | 74.7 | 42.5% | 46.2% | 46.2% | 35.7% | 0.80 s | 99.3 | 30.0 s |
| K2-Horizon 0.9B | 51.9 | 7.5% | 15.4% | 7.7% | 0.0% | 0.36 s | 220.2 | 14.2 s |
How it is scored
Language gate (wrong language = 0), then 0.5 x scaled chrF vs reference(s) + 0.3 x checks (names, numbers, units, codes kept; register; no literal idioms; length; no untranslated leftovers) + 0.2. Pass = right language, every check, chrF above the script's bar. Fluency/adequacy judged with rubric.md.
Temperature 0 · max 2048 output tokens.
System prompt every model got
You are a professional translator.
Every item
Final score per item (0-100), which for this job blends the scripted checks with the judge panel's mark, as in the published results. Click any row to see the prompt, the expected answer and every model's reply. Columns sort.
✗ marks a fail (below the job's pass bar). A dash means no valid result.