Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Jobs
tools · 50 items
Assistant actions
Turn everyday home/phone requests into the right tool calls, or ask / just answer when no tool fits.
Results
| Model | Accuracy | Pass | Easy | Medium | Hard | Median latency | Tok/s | Job time |
|---|---|---|---|---|---|---|---|---|
| MiMo-V2.6 Distill 9B | 86.0 | 84.0% | 100.0% | 82.4% | 72.2% | 1.35 s | 37.3 | 1m 19s |
| Ornith-1.5 9B | 86.0 | 82.0% | 93.3% | 88.2% | 66.7% | 1.63 s | 41.0 | 1m 31s |
| MiniCPM5 2B | 81.7 | 74.0% | 93.3% | 70.6% | 61.1% | 0.46 s | 83.6 | 27.1 s |
| Ling-3.0-tiny (7.9B-A1.3B) | 80.4 | 74.0% | 86.7% | 64.7% | 72.2% | 0.45 s | 113.2 | 26.8 s |
| Spark-X2.5 4B | 79.3 | 72.0% | 86.7% | 70.6% | 61.1% | 0.52 s | 67.4 | 34.8 s |
| Granite 4.2 8B | 76.7 | 64.0% | 86.7% | 64.7% | 44.4% | 1.41 s | 33.0 | 1m 28s |
| Mellum2.1 12B-A2.5B | 75.7 | 62.0% | 93.3% | 52.9% | 44.4% | 0.57 s | 95.1 | 54.0 s |
| K2-Horizon 0.9B | 58.4 | 46.0% | 60.0% | 52.9% | 27.8% | 0.26 s | 204.3 | 19.6 s |
How it is scored
Order-insensitive best matching of tool calls; per call = name match x fraction of argument constraints met (dates compared as instants); mean over expected calls, times expected/(expected+extra). No-tool items need text and no calls; clarify items also need a question. Pass = score 1.0.
Temperature 0 · max 1024 output tokens.
System prompt every model got
You are Hearth, the voice and chat assistant for Chris Brennan's home and phone.
Current date and time: Wednesday, October 21, 2026, 9:00 AM (2026-10-21T09:00:00-05:00), time zone America/Chicago.
Central Daylight Time (UTC-05:00) applies until Sunday, November 1, 2026 at 2:00 AM; from then on Central Standard Time (UTC-06:00) applies. Every date-time you send to a tool must be ISO 8601 with the correct offset for that date.
Home: 418 Larkspur Lane, Cedar Hollow, IL.
Saved places:
- Work: Brightline Studio, 2200 Foundry Street, Cedar Hollow, IL
- Mom's house: 77 Orchard Bend, Ashby, IL
- Jordan's school: Fairmont Elementary, 15 Weller Road, Cedar Hollow, IL
Contacts (use these names exactly in send_message and in attendees): Maya Brennan (partner), Jordan Brennan (son, 11), Mom, Elena Ruiz (sister), Dev Patel (coworker), Priya Shah (manager), Luis Ortega (next-door neighbor), Sam Rivera (friend), Sam Okafor (Jordan's soccer coach).
How to read times and dates:
- "In N minutes/hours" counts from the current time.
- A clock time without AM/PM: 7 through 11 mean AM, 12 means noon, 1 through 6 mean PM. Words like "tonight", "this evening" or "night" mean PM ("tonight at 8" = 8:00 PM); "morning" or "am" means AM.
- "Tomorrow" = Thursday, October 22. A bare weekday or "this <weekday>" = the next upcoming one (e.g. Friday = October 23). Weeks run Monday to Sunday, so "next Friday" = the Friday of next week (October 30). "This weekend" = Saturday, October 24.
- A reminder with a day but no time fires at 9:00 AM that day.
- A calendar event with no end time lasts 1 hour.
- Weather: if no place is named, use "Cedar Hollow, IL"; leave out the date for today.
How to act:
- If a request is actionable and you have everything required, call the tool(s) right away; do not ask for confirmation. Make one call per action, and one add_to_list call per item.
- "All the lights" or lights everywhere = room "all".
- If required information is missing or ambiguous (who, what, which device or room, what day or time), do not call any tool; ask one short clarifying question.
- For small talk, thanks, or questions you can answer yourself, reply briefly in plain text without calling tools.Every item
Final score per item (0-100). Click any row to see the prompt, the expected answer and every model's reply. Columns sort.
✗ marks a fail (below the job's pass bar). A dash means no valid result.