Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-043

easy clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Remind me later.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.48 s

What would you like me to remind you about, and when?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ornith-1.5 9B

✓ pass · score 100 · 2.04 s

I'd love to help with that reminder! But I need a little more detail:

1. **What** should I remind you about?
2. **When** would you like to be reminded (e.g., "in 30 minutes," "tomorrow at 8am," "Friday at 6pm")?

Once you tell me those, I'll set it up right away.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Granite 4.2 8B

✓ pass · score 100 · 2.94 s

I need a bit more detail to set up your reminder. Could you tell me:

1. What specifically would you like to be reminded about? (e.g., "buy groceries," "call Mom," "finish report")
2. When would you like to be reminded? (e.g., "in 30 minutes," "today at 3 PM," "tomorrow morning")

Once you provide these details, I'll create the reminder for you.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Spark-X2.5 4B

✗ fail · score 0 · 0.72 s

create_reminder({"text":"Reminder","when":"2026-10-21T09:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Reminder",
        "when": "2026-10-21T09:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.72 s

I need more information to create a reminder. The user said "Remind me later" but didn't specify what to remind them about or when. I should ask a clarifying question.

"Sure, what would you like me to remind you about and when?"

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.51 s

I can help with that! Could you tell me what you'd like to be reminded about, and when you'd like the reminder? For example:
- What should the reminder be about?
- What time or date should it go off?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

MiniCPM5 2B

✓ pass · score 100 · 0.36 s

I don't have enough details to set a reminder — could you tell me what you'd like to be reminded about and when?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.30 s

create_reminder({"text":"Remind me later","when":"2026-10-21T09:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Remind me later",
        "when": "2026-10-21T09:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

← assistant-actions-042All Assistant actions itemsassistant-actions-044 →