Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Sort and triage

triage-032

medium ticket

The prompt

user

TASK: TICKET

Ticket #48241
From: Victor Lam <victor@lamphotostudio.com>
Plan: Pro
Subject: Someone changed my payout bank account!!

I just got an email from Brightdesk saying "Your payout bank account was updated" and another saying my login email was changed. I did NOT do either of these. I can still get in with my password for now but the bank account on file is not mine. Please lock this down immediately.

Victor Lam

Reply with only a JSON object: {"category": "<billing|bug|account_access|feature_request|cancellation>", "priority": "<low|medium|high|urgent>"}

What we expected

{
  "category": "account_access",
  "priority": "urgent"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.78 s

{"category": "account_access", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 0.71 s

{"category": "account_access", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Granite 4.2 8B

✓ pass · score 100 · 0.81 s

{"category": "account_access", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Spark-X2.5 4B

✗ fail · score 50 · 0.50 s

{"category": "billing", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "billing",
    "expected": "account_access",
    "correct": false
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.33 s

{"category": "account_access", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 50 · 0.26 s

{"category": "account_access", "priority": "high"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "high",
    "expected": "urgent",
    "correct": false
  }
}

MiniCPM5 2B

✗ fail · score 50 · 0.40 s

{"category": "account_access", "priority": "high"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "high",
    "expected": "urgent",
    "correct": false
  }
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.17 s

{"category": "account_access", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "account_access",
    "expected": "account_access",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

← triage-031All Sort and triage itemstriage-033 →