Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-044

easy clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Set an alarm.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.34 s

What time should the alarm go off?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ornith-1.5 9B

✓ pass · score 100 · 0.88 s

I'll set an alarm for you. What time would you like to be woken up — and should it be this morning, or a specific time tomorrow?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Granite 4.2 8B

✓ pass · score 100 · 1.20 s

I need to know what time you'd like the alarm to ring. Please specify the clock time (e.g., "7:30 AM" or "14:00").

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Spark-X2.5 4B

✓ pass · score 100 · 0.47 s

To set the alarm, I need to know the time you'd like it to ring. What time should the alarm go off?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.23 s

What time for the alarm?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.31 s

I can help with that! Could you tell me what time you'd like the alarm set for, and what you'd like it labeled?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

MiniCPM5 2B

✓ pass · score 100 · 0.37 s

I need to know what time you'd like the alarm for. For example, "6am" or "7:30am."

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.23 s

set_alarm({"time":"06:30","label":"Morning routine"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_alarm",
      "args": {
        "time": "06:30",
        "label": "Morning routine"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

← assistant-actions-043All Assistant actions itemsassistant-actions-045 →