Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Analysis

Which Small AI Model Should You Run on a Mac? The Full Results

A job-by-job look at how eight small text models and three image readers did on 312 everyday tests on one M4 Max, with the speed and memory each one costs.

Published Oct 21, 2026 · All the data

Watch on YouTube

What we tested, and why

Most local-model benchmarks test things you’ll never ask a model on your Mac to do. We wanted the boring, useful jobs: setting a timer, finding a date in your lease, cleaning up an email, sorting an inbox, reading a receipt. So we wrote seven everyday jobs from scratch and ran current small models on them.

The lineup was eight text models, from about 1B to 12B parameters, all released on Hugging Face between August 7 and September 21, 2026: K2-Horizon 0.9B (IFM), MiniCPM5 2B (OpenBMB), Spark-X2.5 4B (XHToken), Granite 4.2 8B (IBM), Ling-3.0-tiny (inclusionAI, 7.9B total with 1.3B active), Ornith-1.5 9B (Ornith AI), MiMo-V2.6 Distill 9B (Xiaomi MiMo) and Mellum2.1 (JetBrains, 12.1B total with 2.5B active). On top of those, three models read images: MiniCPM-V 4.7 1B, LFM2.5-VL 3B (Liquid AI), and MiMo again, which does both.

The seven jobs, with their item counts:

  • Assistant actions (50): turn a spoken-style request into the right tool call, ask a question when something is missing, or just answer.
  • Ask my documents (40): answer questions from 22 fictional household documents, cite them, and say “I can’t find that” when the answer isn’t there.
  • Rewrite and proofread (42): fix errors without rewording, change tone without losing facts, shorten to a word limit.
  • Summarize and extract (44): pull exact JSON out of receipts, invoices and signatures (26 items), and summarize emails, transcripts and articles (18).
  • Translate (40): English into Spanish, French, German, Japanese, Simplified Chinese and Brazilian Portuguese (36 items), plus 4 going back into English from French, Spanish, Japanese and Chinese.
  • Sort and triage (56): label emails, route support tickets, read review sentiment.
  • Read images (40): receipts, screenshots, signs, labels, charts, calendars and handwriting.

That’s 312 items, 272 of them text. Every item is original, and the people, households and companies at their centre are made up (a few real brands like Visa or Venmo appear in passing), so no model can have seen the answer key in training.

The headline numbers

Overall quality is the plain average of the six text-job scores, each out of 100:

  1. MiMo-V2.6 Distill 9B: 91.0
  2. Ornith-1.5 9B: 88.4
  3. Granite 4.2 8B: 84.4
  4. Spark-X2.5 4B: 83.1
  5. Mellum2.1 12B-A2.5B: 81.9
  6. Ling-3.0-tiny: 80.5
  7. MiniCPM5 2B: 77.8
  8. K2-Horizon 0.9B: 55.9

MiMo had the top score in four of the six text jobs (assistant actions as a tie with Ornith, documents, rewrite, and summarize-extract). Ornith won translation and triage outright. K2-Horizon came last in all six jobs and passed 28.3% of items.

Three things stood out once we looked past the top of the table.

Size isn’t the order. Spark, at 4.1B, beat the 12.1B Mellum and the 7.9B Ling overall. MiniCPM5 at 2.5B came third at assistant actions (81.7) and fourth at answering from documents (81.7), ahead of the 9B Ornith on that job (78.9).

The middle of the table is close. Granite, Spark, Mellum and Ling sit within 3.9 points of each other. On sets of 40 to 56 items, we wouldn’t read much into a gap of a point or two. The cost of each model in speed and memory separates them far more than their quality does.

Hard items tell a different story for Ling. Ling is sixth overall, but on the 97 text items we tagged hard, it passed 55, third behind MiMo (65) and Ornith (58). On easy and medium items it ranks sixth, so it loses its ground on the simple stuff, not the tricky questions.

Job by job

A note on two numbers. Accuracy is the average item score, with partial credit. Pass rate is the share of items a model got completely right. For writing jobs, a pass needs every scripted check to pass and the judges’ marks to clear a minimum, so pass rates run much lower than accuracy.

Assistant actions

MiMo and Ornith tied at 86.0, with MiniCPM5 third on 81.7, Ling 80.4, Spark 79.3, Granite 76.7, Mellum 75.7 and K2 58.4.

The biggest split was on requests that need more than one action, like turning off the lights and setting a reminder in one breath. There were 12 of these. MiMo and Ornith got all 12. Granite got none: on every one of the 12 it made exactly one tool call and stopped. Mellum made a single call on 11 of the 12 and passed one. That matches what we saw while picking the lineup, where Granite with thinking off made one call per turn. So if your use is “do several things at once”, that alone rules out two of the larger models here.

Time arithmetic also caught models out. On “set a timer for an hour and a half”, only Ornith, Ling and Mellum sent 5,400 seconds. MiMo set 15 minutes, Granite 9 minutes, and three of the small models set 90 seconds. Two requests beat everyone: “Set it to 70” (set what?) and “Text Sam that I’ll call him back after lunch” (the contact list has two Sams). The right move on both is to ask, and no model did.

Ask my documents

MiMo led with 90.8, then Granite 86.3, Mellum 83.8, MiniCPM5 81.7, Ornith 78.9, Ling 76.6, Spark 70.9 and K2 41.6.

Eight of the 40 questions have no answer in the documents, and seven of those plant a related-but-wrong fact in the retrieved text. Spark was the only model to refuse all eight cleanly. But it got only 13 of the 24 single-document questions fully right, against MiMo’s 23, and the reason is odd: on 9 of the 32 answerable questions Spark replied with nothing but a citation tag, like [doc:lease], and no actual answer. It refused outright on another 4. MiMo did neither. So Spark is good at knowing when the answer isn’t there, but too often forgets to say the answer when it is.

The pool question from the video is the clearest trap. The family’s gym has no pool, but the homeowners’ association does. Spark was the only model that passed that one, and Ling was the only one to get the two-document laptop warranty date right.

Rewrite and proofread

MiMo 90.0, Ornith 88.2, Spark 88.0, Granite 82.9, Ling 77.7, Mellum 72.7, MiniCPM5 67.6, K2 49.1.

This job has three parts, and they ranked models differently:

  • Proofreading (16 items) was a 9B game. Ornith scored 97.5 and MiMo 95.8, while Mellum (66.1) and MiniCPM5 (61.7) trailed well behind.
  • Tone changes (16 items) were close among the top four: Ornith 92.5, Granite 92.4, MiMo 91.9, Spark 90.3.
  • Shortening to a word limit (10 items) was everyone’s weak spot. The best pass count was 3 of 10 (MiMo and Spark), and Ornith and Ling passed none. Spark had the best shortening score, 82.6.

Seven of the 42 rewrite items were passed by no model at all, more than in any other job. Hard rewrite items had an average pass rate of 14.3% across the eight models.

Summarize and extract

MiMo 96.1, Mellum 92.2, Ornith 89.9, Granite 89.5, MiniCPM5 88.8, Spark 88.0, Ling 87.1, K2 74.9.

This was the easiest job overall, with an average of 88.3 across the eight models, and the most even: without K2, first to last spans 9.0 points. For pulling JSON out of messy text, Mellum (98.3) almost matched MiMo (98.5), which is a decent showing for a model built mainly for coding. MiMo pulled clear on summaries, 92.6 against 84.8 for the next best (Ornith).

Watch for runaway answers here. Ornith returned an empty reply on one item, the book-club thread from the video’s opening, and Granite ran out of tokens on one while it argued with itself.

Translate

Ornith 91.9, MiMo 89.5, Spark 84.7, Ling 84.5, Granite 79.1, Mellum 76.7, MiniCPM5 74.7, K2 51.9.

English into Japanese was the hardest direction for most models. Ornith held up best there (88.8), while Mellum scored 58.9 and K2 23.2. For English into Chinese, Ling (91.6) and Spark (91.5) edged both 9B models, though on six items that is within the noise. Three items were passed by nobody, all hard: a hotel notice and a business email going into Japanese, and a long email into French. On the Japanese notice, MiMo got stuck repeating one phrase until it hit the 2,048-token limit.

Sort and triage

Ornith 95.5, MiMo 93.8, Granite 92.0, Mellum 90.2, Spark 87.5, Ling 76.8, MiniCPM5 72.3, K2 59.8.

With 18 of 56 items passed by all eight models, triage was the easiest job to pass. The models split on support tickets, which need both the right category and the right priority. MiMo, Ornith and Granite each got 16 of 17, and MiniCPM5 got 8. The newsletter with an urgent deadline buried inside fooled all eight. And on the “Ruth” gift-card scam, the top two overall marked it as needing a reply, while MiniCPM5, Spark and Mellum flagged it as phishing. So keep reading your own email.

Speed, memory and what to run

Speed and memory are reported beside quality, never mixed into it. The RAM figures here are each model’s footprint with an 8K-token conversation filling its context, which is the fairest comparison across the two runtimes we used.

Model Quality Tok/s Time, six text jobs RAM, 8K filled
MiMo 9B 91.0 37.8 12.2 min 8.03 GB
Ornith 9B 88.4 46.1 10.2 min 6.44 GB
Granite 8B 84.4 42.6 15.1 min 8.13 GB
Spark 4B 83.1 65.6 6.6 min 3.79 GB
Mellum 12B 81.9 101.2 5.9 min 8.34 GB
Ling 7.9B 80.5 117.9 3.3 min 5.17 GB
MiniCPM5 2B 77.8 99.3 4.1 min 2.36 GB
K2 0.9B 55.9 218.4 2.3 min 1.33 GB

A few trade-offs show up:

  • Ling and Mellum are fast because they’re mixtures of experts. Only 1.3B (Ling) or 2.5B (Mellum) parameters work on each token, so they write at around 100 to 118 tokens per second. All their weights still have to sit in memory, though: Mellum needs the most RAM of any model we tested.
  • Ling did all six text jobs in 3.3 minutes, while MiMo took 12.2. That’s about 3.7 times faster for 10.5 fewer points.
  • Granite was the slowest overall, at 15.1 minutes, even though its decode speed (42.6 tok/s) beats MiMo’s. It hit the token limit on 5 items, so it spent a lot of time writing text nobody needed.
  • Spark is the value pick. It’s 1.3 points behind Granite, needs less than half the RAM (3.79 vs 8.13 GB) and finished in under half the time (6.6 vs 15.1 min).

We only tested on a 36 GB machine, so the advice below is footprint arithmetic, not a test on smaller Macs. Remember that macOS and your open apps share the same memory.

  • 8 GB Mac: stay small. MiniCPM5 2B (2.36 GB) is the most capable model under 3 GB, and LFM2.5-VL (2.81 GB) is the image reader to pair with it. Spark (3.79 GB) is the largest we’d try, and we haven’t tried it on an 8 GB Mac.
  • 16 GB Mac: every model here fits on its own; the biggest needs 8.34 GB. Ornith (6.44 GB) gives you second-best quality with room to spare. Spark or Ling fit if you care more about speed than about the last few points.
  • 24 GB and up: run MiMo for text and keep LFM2.5-VL loaded for images, about 10.8 GB together.

Dev note: the context size you configure matters as much as the model. llama-server reserves memory for its whole context when it starts. In our benchmark runs, with 32,768 tokens reserved, MiMo’s server peaked at 17.53 GB on the text jobs, more than twice its 8K figure. If you don’t need long documents, set a smaller context.

Reading images

LFM2.5-VL 3B scored 85.0, MiMo 9B 77.5 and MiniCPM-V 4.7 1B 72.5.

On easy images, LFM and MiniCPM-V got all 12 right and MiMo got 11. The gap opened on medium and hard items, especially ones that need a step of reasoning on top of reading. On the 9 items tagged as reasoning, LFM got 6, while MiMo and MiniCPM-V got 4 each. Parking signs, where you check the day, time and limit against stacked rules, went 3 of 4 for LFM and 1 of 4 for MiMo.

Five questions beat all three models. Counting was a common thread: none counted Wednesday’s three calendar events correctly, or the six items not crossed off a handwritten list. All three also gave the per-serving sugar instead of the whole-container total, and all three picked the wrong shelf tag on a cost-per-ounce question. MiMo had one solo win: it was the only model to pick “Don’t Save” on the save dialog.

Speed is closer than the tokens-per-second figures suggest. LFM writes at 114.3 tok/s against MiMo’s 43.1, but image answers are short, and LFM took longer to start replying (2.43 s to the first token against 1.91 s). The whole 40-item job took 82.4 s on LFM and 86.1 s on MiMo. Memory is LFM’s real win: 2.81 GB against MiMo’s 8.03, about 35% as much.

How we scored it, and the limits

Scoring. Wherever there’s a single right answer (a tool call, a label, a JSON field, a date, a number read off an image), a script checks it against an answer key. Where there isn’t (tone, summaries, shortening, translation fluency), a panel of three AI judges also marks each answer out of 10 against a written rubric: Claude Code (claude-opus-5-5), Codex (gpt-6.1-sol) and Antigravity (gemini-4-argon-high). The judges are blind. Model and vendor names are scrubbed from the answers, and outputs from every model are shuffled together. The panel’s median per criterion is blended 50/50 with the script’s score.

Each model had 84 judged answers, 672 in all. All but ten got marks from all three judges; nine had two, and one empty reply was scored 0 without judging. The judges mostly agreed. The gap between the highest and lowest judge on a criterion averaged 0.99 points for translation, 1.26 for summaries and 1.39 for rewrites. Claude was the strictest judge in all three jobs, so using the panel’s median, rather than any single judge, matters.

Settings. Every model ran at 4-bit: Q4_K_M GGUF on llama.cpp for seven models, MLX 4-bit on mlx_lm for Granite, MiniCPM5 and Ornith. Temperature was 0. Thinking was switched off wherever a model has a switch. K2-Horizon has no off switch, so it ran at its lowest setting; it produced any reasoning on only 39 of its 272 text items.

Limits worth knowing:

  • One machine: an M4 Max with 36 GB. Speeds were measured with other apps running (one-minute load average between 4.4 and 10.3 when each model’s runs began), so a quiet Mac may go faster.
  • Two runtimes. Comparing speeds between an MLX model (Ornith) and a llama.cpp model (MiMo) mixes runtime and model, so treat the video’s “a fifth faster” (46.1 against 37.8 tokens per second) as rough.
  • Our tasks, our rules. Triage, for example, follows strict written label rules, which are tidier than real life. Per-job sets of 40 to 56 items separate strong from weak models but not neighbors a point apart.
  • Ornith and MiMo are both built on Qwen3.5-9B, and Mellum is built mainly for coding and agents, so the top two aren’t independent designs.
  • Not tested: thinking modes, long documents beyond what our items need, other quantizations, and smaller Macs. Thinking on is next on our list.

How to reproduce

The harness is um-jobs in our repo, and we’ll publish the results and the test data. With the lineup in jobs/lineup.json:

uv run um-jobs run        # every model x every job, resumable
uv run um-jobs judge      # blind judge panel for the writing items
uv run um-jobs memory     # comparable RAM at an 8K context
uv run um-jobs report     # rescore and write results.json / results.md

Each item’s record, including the full output, timing and score, lands in data/jobs/<model>/<job>.jsonl, so you can check any single answer we talked about.

← Back to the full results