Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Sort and triage

triage-034

hard ticket

The prompt

user

TASK: TICKET

Ticket #48250
From: Liam Brennan <liam@brennanplumbing.ie>
Plan: Pro
Subject: Close my account

Your processing fee and the Pro price have both gone up this year and I've had enough. I've already moved my invoicing to another provider. Please cancel my subscription, close my account and delete my data. Please don't offer me a discount, I've made up my mind.

Liam Brennan

Reply with only a JSON object: {"category": "<billing|bug|account_access|feature_request|cancellation>", "priority": "<low|medium|high|urgent>"}

What we expected

{
  "category": "cancellation",
  "priority": "medium"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.79 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 0.69 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Granite 4.2 8B

✓ pass · score 100 · 0.78 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Spark-X2.5 4B

✗ fail · score 50 · 0.47 s

{"category": "cancellation", "priority": "high"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "high",
    "expected": "medium",
    "correct": false
  }
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.32 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.25 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

MiniCPM5 2B

✗ fail · score 50 · 0.39 s

{"category": "cancellation", "priority": "low"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "low",
    "expected": "medium",
    "correct": false
  }
}

K2-Horizon 0.9B

✗ fail · score 50 · 0.16 s

{"category": "cancellation", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "medium",
    "correct": false
  }
}

← triage-033All Sort and triage itemstriage-035 →