Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-046

medium clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Add it to the list.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.39 s

Which list, and what item should I add?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ornith-1.5 9B

✓ pass · score 100 · 1.45 s

I'm happy to add it to the list, but you haven't told me what item to add. Could you let me know:

1. **The item** you'd like added, and
2. **Which list** — grocery, todo, packing, or hardware.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Granite 4.2 8B

✓ pass · score 100 · 3.35 s

I need a bit more information to add an item to the correct list. Could you please specify:

1. What item you'd like to add (e.g., "milk", "fix leaky faucet", "buy birthday cards")
2. Which list it belongs to (grocery, todo, packing, or hardware)?

For example: "Add 'bread' to the grocery list" or "Add 'change smoke detector batteries' to the todo list."

Once you provide these details, I'll add it right away.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Spark-X2.5 4B

✓ pass · score 100 · 0.55 s

You didn't specify what to add or which list. Could you tell me the item and whether it's for groceries, a todo, packing, or hardware?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.76 s

I need more information to add something to a list. The user said "Add it to the list" but didn't specify what item or which list. I should ask for clarification.

According to the instructions: "If required information is missing or ambiguous ... do not call any tool; ask one short clarifying question."

I need to ask what item and which list. The lists available are grocery, todo, packing, hardware.

I'll ask: "What item and which list?" But maybe I can be more specific: "Which list: grocery, todo, packing, or hardware? And what item?"

But the instruction says "ask one short clarifying question." So I'll keep it short.

"What item and which list?"

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.56 s

I need a bit more context to add something to a list. Could you tell me what item you'd like to add and which list it should go on? For example, "add milk to the grocery list" or "add a reminder to call Mom to the todo list."

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

MiniCPM5 2B

✓ pass · score 100 · 0.51 s

I need a bit more context to know what you'd like added to the list. Could you clarify what item you'd like to add and which list (grocery, todo, packing, or hardware)?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.22 s

add_to_list({"list":"todo","item":"buy groceries"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "buy groceries"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

← assistant-actions-045All Assistant actions itemsassistant-actions-047 →