Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-040

medium no_tool

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

What's 15% of $84?

What we expected

{
  "calls": [],
  "behavior": "no_tool",
  "text_contains_any": [
    "12.6"
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.55 s

15% of $84 is **$12.60**.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

Ornith-1.5 9B

✓ pass · score 100 · 0.53 s

15% of $84 is **$12.60**.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

Granite 4.2 8B

✓ pass · score 100 · 0.99 s

15% of $84 is $12.60. (Calculated as 0.15 × 84 = 12.60.)

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

Spark-X2.5 4B

✓ pass · score 100 · 0.32 s

15% of $84 is **$12.60**.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.53 s

15% of $84 is 0.15 * 84 = $12.60. I can answer this directly without a tool.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.32 s

set_timer({"seconds":15,"label":"15% of $84"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_timer",
      "args": {
        "seconds": 15,
        "label": "15% of $84"
      },
      "error": null
    }
  ],
  "behavior": "no_tool",
  "reason": "called a tool when none was wanted"
}

MiniCPM5 2B

✓ pass · score 100 · 0.23 s

15% of $84 is $12.60.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.14 s

$84 multiplied by 15% equals $12.60.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "no_tool",
  "answer_ok": true
}

← assistant-actions-039All Assistant actions itemsassistant-actions-041 →