Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-049

hard clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Schedule a call with Priya next week.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.42 s

I can help with that, but I need a couple of details before scheduling:

1. **When** would you like the call? (e.g., "Thursday at 10:00 AM")
2. **How long** should it be? (e.g., 30 minutes)

Also, do you want me to send Priya an invite, or just put it on your calendar?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ornith-1.5 9B

✗ fail · score 0 · 2.78 s

add_calendar_event({"title":"Call with Priya","start":"2026-10-27T09:00:00-05:00","end":"2026-10-27T09:30:00-05:00"})

I'll schedule a call with Priya next week (Monday, October 27).

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Call with Priya",
        "start": "2026-10-27T09:00:00-05:00",
        "end": "2026-10-27T09:30:00-05:00"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Granite 4.2 8B

✓ pass · score 100 · 4.04 s

I'd be happy to schedule a call with Priya next week. To set it up accurately, I need a bit more information:

1. **Which day next week?** (e.g., "next Tuesday", "Thursday", "next week" could mean any day; please specify a day of the week or date)
2. **What time?** (e.g., "at 10 AM", "in the afternoon", etc.)
3. **How long?** (optional – if you know the duration)

Once you provide the day and time, I'll create a calendar event for the call. Let me know!

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Spark-X2.5 4B

✓ pass · score 100 · 0.59 s

To schedule a call with Priya next week, I need a bit more detail. What time would you like the call to happen, and how long should it last?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.96 s

I need to schedule a call with Priya next week. The user says "next week" but doesn't specify a day or time. I need to ask for clarification: which day next week and what time? Also, is it a calendar event or a reminder? Probably a calendar event. I should ask for the specific day and time. Let me respond.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.33 s

I can help with that! To schedule a call with Priya Shah next week, I need a few details:

1. **What day** next week would you like? (e.g., Monday Oct 26, Tuesday Oct 27, etc.)
2. **What time** should the call be? (e.g., 2:00 PM)
3. **How long** should the call last? (e.g., 30 minutes, 1 hour)
4. **What's the purpose** of the call? (e.g., project update, planning, etc.)

Once you give me those, I’ll add it to your calendar.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

MiniCPM5 2B

✓ pass · score 100 · 0.69 s

I need to clarify a few details before scheduling:
- What day next week works best for you (e.g., "next Friday," "this Saturday," etc.)?
- Do you have a preferred time of day?
- Would you like me to set a reminder for that time?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.33 s

I need to clarify what you mean by "Schedule a call with Priya next week." Are you looking to set up a calendar event for a meeting with Priya, or do you want to schedule a reminder for a call with Priya?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

← assistant-actions-048All Assistant actions itemsassistant-actions-050 →