Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Sort and triage

triage-029

medium ticket

The prompt

user

TASK: TICKET

Ticket #48223
From: Rosa Delgado <rosa@delgadocatering.com>
Plan: Pro
Subject: CLIENTS CANNOT PAY - Pay Now button broken

Since about 9 this morning every single client who clicks "Pay Now" on one of my invoices gets an error that says "Payment session error (code 502)". I've had four clients text me about it already. I have $6,000 in invoices due today and nobody can pay. Everything else in the app seems to load fine.

Rosa Delgado
Delgado Catering

Reply with only a JSON object: {"category": "<billing|bug|account_access|feature_request|cancellation>", "priority": "<low|medium|high|urgent>"}

What we expected

{
  "category": "bug",
  "priority": "urgent"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.82 s

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Ornith-1.5 9B

✓ pass · score 100 · 2.36 s

The message describes a broken "Pay Now" button causing a "Payment session error (code 502)" for clients, blocking payments on $6,000 in invoices due today. This is a bug (payment processing not working as it should). Clients' payments are failing right now, so they're losing money as it happens → urgent.

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Granite 4.2 8B

✓ pass · score 100 · 0.81 s

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Spark-X2.5 4B

✓ pass · score 100 · 0.50 s

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.33 s

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 50 · 0.25 s

{"category": "bug", "priority": "high"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "high",
    "expected": "urgent",
    "correct": false
  }
}

MiniCPM5 2B

✗ fail · score 0 · 0.40 s

{"category": "billing", "priority": "high"}

Scoring detail

{
  "category": {
    "got": "billing",
    "expected": "bug",
    "correct": false
  },
  "priority": {
    "got": "high",
    "expected": "urgent",
    "correct": false
  }
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.17 s

{"category": "bug", "priority": "urgent"}

Scoring detail

{
  "category": {
    "got": "bug",
    "expected": "bug",
    "correct": true
  },
  "priority": {
    "got": "urgent",
    "expected": "urgent",
    "correct": true
  }
}

← triage-028All Sort and triage itemstriage-030 →