Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Models
Mellum2.1 12B-A2.5B
JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M · llama.cpp · llama-server version: 0.7.0 (build 11669, commit 620d2225c)
#5 of 8
overall, text jobs
81.9
quality score
101.2
median tokens/s
8.34 GB
RAM with 8K context filled
7.52 GB
download size
Score per job
| Job | Accuracy | Pass | Easy | Medium | Hard | Median latency | Tok/s | Job time | Errors |
|---|---|---|---|---|---|---|---|---|---|
| Assistant actions | 75.7 | 62.0% | 93.3% | 52.9% | 44.4% | 0.57 s | 95.1 | 54.0 s | 0 |
| Ask my documents | 83.8 | 80.0% | 66.7% | 92.3% | 80.0% | 2.53 s | 97.3 | 2m 03s | 0 |
| Rewrite and proofread | 72.7 | 28.6% | 64.3% | 21.4% | 0.0% | 1.14 s | 96.2 | 51.4 s | 0 |
| Summarize + extract | 92.2 | 56.8% | 73.3% | 46.2% | 50.0% | 1.62 s | 101.2 | 1m 18s | 0 |
| Translate | 76.7 | 40.0% | 69.2% | 38.5% | 14.3% | 0.85 s | 108.0 | 33.4 s | 0 |
| Sort and triage | 90.2 | 87.5% | 100.0% | 94.7% | 70.0% | 0.31 s | 114.4 | 16.8 s | 0 |
Easy / medium / hard columns are pass rates within each difficulty tier.
Memory and loading
| Download size | 7.52 GB |
| Loaded, idle | 8.04 GB |
| In use (8K context) | 8.15 GB |
| 8K context filled | 8.34 GB |
| Server peak during the jobs (incl. reserved context) | 17.70 GB |
| Load time | 8.10 s |
Measured on Apple M4 Max, 36 GB, macOS 27.0.1. How memory was measured.