Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-047

hard clarify

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Set it to 70.

What we expected

{
  "calls": [],
  "behavior": "clarify"
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 0 · 0.95 s

set_thermostat({"temp_f":70})

I'll set the thermostat to 70°F.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Ornith-1.5 9B

✗ fail · score 0 · 0.80 s

set_thermostat({"temp_f":70})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Granite 4.2 8B

✗ fail · score 0 · 1.45 s

set_thermostat({"temp_f":70,"mode":"auto"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70,
        "mode": "auto"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Spark-X2.5 4B

✗ fail · score 0 · 0.29 s

set_thermostat({"temp_f":70})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

Mellum2.1 12B-A2.5B

✗ fail · score 0 · 0.85 s

Set the thermostat to 70 degrees Fahrenheit. Use set_thermostat tool. Need to specify temp_f and optionally mode. User said "Set it to 70." Likely meaning home thermostat target temperature. So call set_thermostat with temp_f=70. No mode specified, so just temp_f.

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "behavior": "clarify",
  "asked_question": false,
  "claimed_action": false
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.22 s

set_thermostat({"temp_f":70})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

MiniCPM5 2B

✗ fail · score 0 · 0.28 s

set_thermostat({"temp_f":70})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.21 s

set_thermostat({"temp_f":70,"mode":"heat"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "set_thermostat",
      "args": {
        "temp_f": 70,
        "mode": "heat"
      },
      "error": null
    }
  ],
  "behavior": "clarify",
  "reason": "called a tool when none was wanted"
}

← assistant-actions-046All Assistant actions itemsassistant-actions-048 →