Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-045

medium clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Text her that I'm on my way.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 0.47 s

Who are you texting — Maya, Mom, or someone else?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Ornith-1.5 9B

✗ fail · score 0 · 1.16 s

send_message({"to":"Maya Brennan","text":"I'm on my way."})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Maya Brennan",
        "text": "I'm on my way."
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Granite 4.2 8B

✓ pass · score 100 · 0.87 s

I'll send a message saying "I'm on my way." to Maya (your partner). Is that correct?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": true,
  "claimed_action": false
}

Spark-X2.5 4B

✗ fail · score 0 · 0.52 s

send_message({"to":"Maya Brennan","text":"I'm on my way."})

Maya Brennan!

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Maya Brennan",
        "text": "I'm on my way."
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Mellum2.1 12B-A2.5B

✗ fail · score 0 · 0.84 s

I need to send a text message to Maya Brennan (partner) saying "I'm on my way." The user said "Text her that I'm on my way." I need to use send_message tool with to: "Maya Brennan" and text: "I'm on my way." Let me do that.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": false,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.31 s

send_message({"to":"Maya Brennan","text":"I'm on my way!"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Maya Brennan",
        "text": "I'm on my way!"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

MiniCPM5 2B

✗ fail · score 0 · 0.47 s

I can't send a text message directly — I'm Hearth, your home and phone assistant. Would you like me to help with something related to your home or schedule instead?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": false,
  "claimed_action": false
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.24 s

send_message({"to":"Maya Brennan","text":"I'm on my way."})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Maya Brennan",
        "text": "I'm on my way."
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

← assistant-actions-044All Assistant actions itemsassistant-actions-046 →