Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-031

hard multi

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

I'm heading out now. Lock the front door, turn off all the lights, and get me directions to Mom's.

What we expected

{
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": {
          "type": "norm",
          "value": "front"
        }
      },
      "allow_extra_args": []
    },
    {
      "name": "control_light",
      "args": {
        "room": {
          "type": "norm",
          "value": "all"
        },
        "state": {
          "type": "norm",
          "value": "off"
        }
      },
      "allow_extra_args": [
        "brightness"
      ]
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": {
          "type": "contains_any",
          "values": [
            "orchard bend",
            "mom"
          ]
        }
      },
      "allow_extra_args": []
    }
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.06 s

lock_door({"door":"front"})
control_light({"room":"all","state":"off"})
navigate_to({"destination":"77 Orchard Bend, Ashby, IL"})

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "control_light",
      "args": {
        "room": "all",
        "state": "off"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "77 Orchard Bend, Ashby, IL"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 1,
      "args": {
        "room": true,
        "state": true
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 2.91 s

lock_door({"door":"front"})
control_light({"room":"all","state":"off"})
navigate_to({"destination":"77 Orchard Bend, Ashby, IL"})

I'll lock the front door, turn off all the lights, and get you directions to Mom's.

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "control_light",
      "args": {
        "room": "all",
        "state": "off"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "77 Orchard Bend, Ashby, IL"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 1,
      "args": {
        "room": true,
        "state": true
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Granite 4.2 8B

✗ fail · score 33 · 1.02 s

lock_door({"door":"front"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    },
    {
      "expected": "navigate_to",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    }
  ],
  "extra_calls": 0,
  "base": 0.3333333333333333
}

Spark-X2.5 4B

✓ pass · score 100 · 0.84 s

lock_door({"door":"front"})
control_light({"room":"all","state":"off"})
navigate_to({"destination":"77 Orchard Bend, Ashby, IL"})

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "control_light",
      "args": {
        "room": "all",
        "state": "off"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "77 Orchard Bend, Ashby, IL"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 1,
      "args": {
        "room": true,
        "state": true
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 33 · 0.34 s

lock_door({"door":"front"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    },
    {
      "expected": "navigate_to",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    }
  ],
  "extra_calls": 0,
  "base": 0.3333333333333333
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.60 s

lock_door({"door":"front"})
control_light({"room":"all","state":"off"})
navigate_to({"destination":"77 Orchard Bend, Ashby, IL"})

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "control_light",
      "args": {
        "room": "all",
        "state": "off"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "77 Orchard Bend, Ashby, IL"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 1,
      "args": {
        "room": true,
        "state": true
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

MiniCPM5 2B

✓ pass · score 100 · 0.72 s

lock_door({"door":"front"})
control_light({"room":"all","state":"off"})
navigate_to({"destination":"77 Orchard Bend, Ashby, IL"})

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "control_light",
      "args": {
        "room": "all",
        "state": "off"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "77 Orchard Bend, Ashby, IL"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 1,
      "args": {
        "room": true,
        "state": true
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

K2-Horizon 0.9B

✗ fail · score 67 · 0.39 s

lock_door({"door":"front"})
set_alarm({"time":"06:30","label":"Mom's"})
navigate_to({"destination":"Mom's house"})

Scoring detail

{
  "n_calls": 3,
  "calls": [
    {
      "name": "lock_door",
      "args": {
        "door": "front"
      },
      "error": null
    },
    {
      "name": "set_alarm",
      "args": {
        "time": "06:30",
        "label": "Mom's"
      },
      "error": null
    },
    {
      "name": "navigate_to",
      "args": {
        "destination": "Mom's house"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "lock_door",
      "matched_index": 0,
      "score": 1,
      "args": {
        "door": true
      }
    },
    {
      "expected": "control_light",
      "matched_index": 1,
      "score": 0,
      "args": {
        "name": "mismatch"
      }
    },
    {
      "expected": "navigate_to",
      "matched_index": 2,
      "score": 1,
      "args": {
        "destination": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.6666666666666666
}

← assistant-actions-030All Assistant actions itemsassistant-actions-032 →