Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-050

hard clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Put the parent-teacher conference on my calendar for Thursday.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 0 · 2.67 s

add_calendar_event({"title":"Parent-teacher conference","start":"2026-10-22T09:00:00-05:00","end":"2026-10-22T10:00:00-05:00"})

I'll add it to your calendar for Thursday.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Parent-teacher conference",
        "start": "2026-10-22T09:00:00-05:00",
        "end": "2026-10-22T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Ornith-1.5 9B

✗ fail · score 0 · 2.74 s

add_calendar_event({"title":"Parent-Teacher Conference","start":"2026-10-22T09:00:00-05:00","end":"2026-10-22T10:00:00-05:00"})

I'll add the parent-teacher conference to your calendar for Thursday.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Parent-Teacher Conference",
        "start": "2026-10-22T09:00:00-05:00",
        "end": "2026-10-22T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Granite 4.2 8B

✗ fail · score 0 · 4.47 s

add_calendar_event({"title":"Parent-Teacher Conference","start":"2026-10-22T09:00:00-05:00","end":"2026-10-22T10:00:00-05:00"})

I'll add the parent-teacher conference to your calendar for Thursday, October 22, 2026. Since you didn't specify a time, I'll schedule it for the default 9:00 AM start time (9:00 AM to 10:00 AM).

Let me add this calendar event now.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Parent-Teacher Conference",
        "start": "2026-10-22T09:00:00-05:00",
        "end": "2026-10-22T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Spark-X2.5 4B

✗ fail · score 0 · 1.21 s

add_calendar_event({"title":"Parent-Teacher Conference","start":"2026-10-22T09:00:00-05:00","end":"2026-10-22T10:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Parent-Teacher Conference",
        "start": "2026-10-22T09:00:00-05:00",
        "end": "2026-10-22T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 2.46 s

What time Thursday?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.96 s

I need a bit more detail to set up the conference. Could you tell me:

- The exact date (you said Thursday — is that October 23, 2026?)
- The time
- The location (school, address, or room)
- Who you’re inviting (Maya, Jordan, Mom, Elena, Dev, Priya, Luis, Sam, or another name?)

Once I have that, I’ll add it to your calendar.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

MiniCPM5 2B

✓ pass · score 100 · 0.37 s

I need to know the specific time for the parent-teacher conference to schedule it properly. Could you tell me when it is scheduled?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.20 s

I'll add that parent-teacher conference to your calendar for Thursday. Let me create the event for you.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": false,
  "claimed_action": false
}

← assistant-actions-049All Assistant actions items