Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Sort and triage

triage-028

easy ticket

The prompt

user

TASK: TICKET

Ticket #48219
From: Gus Whitaker <gus@whitakerbikes.com>
Plan: Basic
Subject: Please cancel my subscription

Hello, we're closing the shop at the end of the month, so please cancel our Brightdesk subscription at the end of the current billing period. Thanks for everything, it's been a great tool.

Gus

Reply with only a JSON object: {"category": "<billing|bug|account_access|feature_request|cancellation>", "priority": "<low|medium|high|urgent>"}

What we expected

{
  "category": "cancellation",
  "priority": "medium"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.72 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 0.70 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Granite 4.2 8B

✓ pass · score 100 · 0.69 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Spark-X2.5 4B

✓ pass · score 100 · 0.41 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.30 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.24 s

{"category": "cancellation", "priority": "medium"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "medium",
    "expected": "medium",
    "correct": true
  }
}

MiniCPM5 2B

✗ fail · score 50 · 0.37 s

{"category": "cancellation", "priority": "low"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "low",
    "expected": "medium",
    "correct": false
  }
}

K2-Horizon 0.9B

✗ fail · score 50 · 0.15 s

{"category": "cancellation", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "cancellation",
    "expected": "cancellation",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "medium",
    "correct": false
  }
}

← triage-027All Sort and triage itemstriage-029 →