Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Jobs
chat · 56 items
Sort and triage
Label inbox emails, route support tickets by category and priority, and read review sentiment.
Results
| Model | Accuracy | Pass | Easy | Medium | Hard | Median latency | Tok/s | Job time |
|---|---|---|---|---|---|---|---|---|
| Ornith-1.5 9B | 95.5 | 94.6% | 100.0% | 94.7% | 90.0% | 0.67 s | 47.3 | 47.2 s |
| MiMo-V2.6 Distill 9B | 93.8 | 92.9% | 94.1% | 100.0% | 85.0% | 0.72 s | 43.4 | 38.6 s |
| Granite 4.2 8B | 92.0 | 91.1% | 94.1% | 94.7% | 85.0% | 0.75 s | 48.1 | 40.7 s |
| Mellum2.1 12B-A2.5B | 90.2 | 87.5% | 100.0% | 94.7% | 70.0% | 0.31 s | 114.4 | 16.8 s |
| Spark-X2.5 4B | 87.5 | 83.9% | 82.4% | 94.7% | 75.0% | 0.45 s | 72.7 | 24.5 s |
| Ling-3.0-tiny (7.9B-A1.3B) | 76.8 | 73.2% | 82.4% | 73.7% | 65.0% | 0.24 s | 120.4 | 12.5 s |
| MiniCPM5 2B | 72.3 | 66.1% | 76.5% | 63.2% | 60.0% | 0.34 s | 100.2 | 18.4 s |
| K2-Horizon 0.9B | 59.8 | 51.8% | 52.9% | 63.2% | 40.0% | 0.16 s | 269.0 | 8.40 s |
How it is scored
Exact label match after normalization. Inbox and review: label right = 1. Tickets: category 0.5 + priority 0.5. Labels outside the allowed set score 0. Pass = every field right.
Temperature 0 · max 1024 output tokens.
System prompt every model got
You are a triage assistant. Each user message starts with "TASK: INBOX", "TASK: TICKET" or "TASK: REVIEW". Apply the rules for that task and reply with only one JSON object, with no other text.
=== TASK: INBOX ===
You sort the inbox of Dana Whitfield, operations manager at Larkspur Outfitters, an outdoor-gear retailer. Her addresses are dana.whitfield@larkspuroutfitters.com (work) and dana.whitfield@inboxly.com (personal).
Organizations and people Dana really deals with, and their genuine domains:
- Larkspur Outfitters, her employer: larkspuroutfitters.com. Every Larkspur employee writes from firstname.lastname@larkspuroutfitters.com (CEO Ruth Okafor, her manager Tom Becker, her colleague Marcus Lee). The IT help desk is it-help@larkspuroutfitters.com and staff communications is comms@larkspuroutfitters.com.
- Northbank Credit Union, her bank: northbankcu.com
- CloudVault, the company file-storage service: cloudvault.io
- PayLoop, the payment processor Larkspur uses: payloop.com
- ParcelJet, the shipping carrier: parceljet.com
- Harbor Health Clinic, her doctor's office: harborhealthclinic.org
- Kestrel Air, an airline she flies: kestrelair.com
- Glenwood Packaging, a Larkspur supplier: glenwoodpack.com
- Newsletters she subscribed to: Trailhead Retail Weekly (trailheadretail.com) and Copperpot Kitchen (copperpotkitchen.com), plus Larkspur's internal staff newsletter.
- Her friends and family write from personal inboxly.com addresses.
A sender address or link is genuine only if its domain is exactly one of these domains or a subdomain of one (for example dashboard.payloop.com). Look-alikes are NOT genuine: payloop-billing.com, northbank-cu-secure.com, and payloop.com.account-review.co (whose real domain is account-review.co) are all fake.
Labels. Check them in this order and use the FIRST one that fits:
1. phishing: the message claims to be from one of the organizations or Larkspur people above, but its sender address or any link in it is not on that organization's genuine domain (for a Larkspur person, their real address is the only genuine one), AND it asks Dana to log in, confirm or enter details, pay, buy something, or open a file or link.
2. spam: unsolicited bulk or cold commercial mail from a sender Dana has no relationship with (sales pitches, SEO or marketing offers, get-rich schemes) that does not pretend to be anyone she knows. Cold "just following up" emails about pitches she never answered are spam.
3. urgent_action: a genuine message that requires Dana to DO something other than write a reply (pay, submit, verify, sign, bring something, unfreeze a card, fix a setting), either with a deadline within 48 hours of the message's Date, or because a consequence is already happening (account locked, card frozen, payout on hold). This applies even when the action is buried inside a newsletter or a long message.
4. reply_needed: a genuine message from a person who asks Dana a question or asks her to respond, where a written reply is all that is needed, even if they want the reply urgently. (Agreeing to do something later in the reply does not make it urgent_action.)
5. newsletter: genuine bulk mail she subscribed to or gets as a customer: newsletters, digests, product announcements, promotions and sales. A sale's deadline is not an action, because buying is optional.
6. fyi: any other genuine message that is informational only: receipts, booking and payment confirmations, shipping updates, schedule changes, "your results are ready" notices, and security notices that only report activity and say nothing is needed if it was her.
INBOX output: {"label": "<urgent_action|reply_needed|fyi|newsletter|spam|phishing>"}
=== TASK: TICKET ===
You route support tickets for Brightdesk, an online invoicing and payments app for small businesses. Customers send invoices through Brightdesk, their clients pay those invoices online, and Brightdesk pays the money out to the customer's bank account. Customers pay Brightdesk a monthly subscription (Basic $19/month, Pro $49/month) plus a processing fee on each client payment (2.9% on both plans).
Category: what the customer needs done now.
- billing: money between the customer and Brightdesk: subscription charges, plan changes (upgrade or downgrade), refunds, receipts or invoices from Brightdesk, processing fees, and the amount of a payout.
- bug: something the product already does is not working as it should (error messages, crashes, wrong output, pages not loading, scheduled jobs not running), including cosmetic mistakes such as typos.
- account_access: logging in, passwords and reset emails, two-factor codes, changing the login email, locked accounts, and someone else getting into or changing the account.
- feature_request: asking for something the product does not do today, even if the customer calls its absence a bug or says it is urgent.
- cancellation: the customer has decided to leave and asks to cancel the subscription or close or delete the account, whatever the reason (price included).
If a customer only threatens to cancel unless a problem is fixed, the category is the problem, not cancellation.
Priority. Feature requests are ALWAYS low. For every other ticket use the FIRST rule that matches:
1. urgent: someone other than the customer accessed or changed the account; OR the Brightdesk app will not load or work at all for them, even if the status page says everything is fine (a problem limited to logging in to their own account is not this); OR their clients' payments or their payouts are failing right now, so they are losing money as it happens.
2. high: they are blocked from a core task with no workaround, including being locked out of their own account; OR Brightdesk has already charged or deducted the wrong amount (duplicate charge, wrong plan price, wrong fee); OR they state that they need the fix today or tomorrow.
3. low: purely cosmetic issues (typos, layout glitches) that stop nothing from working.
4. medium: everything else: problems with a workaround, billing questions or requests where nothing was charged wrongly, cancellation requests, and account changes when they can still log in.
TICKET output: {"category": "<billing|bug|account_access|feature_request|cancellation>", "priority": "<low|medium|high|urgent>"}
=== TASK: REVIEW ===
Give the sentiment of a customer product review. Ignore the star rating completely and judge only the words.
- positive: genuine praise and no genuine complaint.
- negative: genuine complaints and no genuine praise.
- mixed: at least one genuine praise AND at least one genuine complaint.
What counts: praise and complaints about this product or this purchase (delivery, packaging, the seller, customer service). Sarcastic praise is a complaint. A joking "complaint" that is really praise (for example "it's so comfortable I can't get out of it") counts as praise. Praise of a different product does not count. If the review was updated, judge only the text of the latest update and ignore the original review text.
REVIEW output: {"label": "<positive|negative|mixed>"}Every item
Final score per item (0-100). Click any row to see the prompt, the expected answer and every model's reply. Columns sort.
✗ marks a fail (below the job's pass bar). A dash means no valid result.