Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-028

medium multi

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Text Luis that his package is sitting on our porch, and add 'thank-you card for Luis' to my to-do list.

What we expected

{
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": {
          "type": "contains_all",
          "values": [
            "luis"
          ]
        },
        "text": {
          "type": "contains_all",
          "values": [
            "package",
            "porch"
          ]
        }
      },
      "allow_extra_args": []
    },
    {
      "name": "add_to_list",
      "args": {
        "list": {
          "type": "norm",
          "value": "todo"
        },
        "item": {
          "type": "contains_all",
          "values": [
            "thank"
          ]
        }
      },
      "allow_extra_args": []
    }
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.21 s

send_message({"to":"Luis Ortega","text":"Hi Luis, your package is sitting on our porch. Thanks!"})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

I'll text Luis and add the to-do item.

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Hi Luis, your package is sitting on our porch. Thanks!"
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 2.38 s

send_message({"to":"Luis Ortega","text":"Hi Luis — your package is sitting on our porch. Let me know when you pick it up!"})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Hi Luis — your package is sitting on our porch. Let me know when you pick it up!"
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Granite 4.2 8B

✗ fail · score 50 · 1.68 s

send_message({"to":"Luis Ortega","text":"Your package is sitting on our porch."})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Your package is sitting on our porch."
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

Spark-X2.5 4B

✓ pass · score 100 · 0.87 s

send_message({"to":"Luis Ortega","text":"Your package is sitting on our porch."})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Your package is sitting on our porch."
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 50 · 0.52 s

send_message({"to":"Luis Ortega","text":"Your package is sitting on our porch."})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Your package is sitting on our porch."
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": null,
      "score": 0,
      "args": "no call"
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.58 s

send_message({"to":"Luis Ortega","text":"Your package is sitting on our porch."})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Your package is sitting on our porch."
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

MiniCPM5 2B

✓ pass · score 100 · 0.67 s

send_message({"to":"Luis Ortega","text":"His package is sitting on our porch."})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "His package is sitting on our porch."
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.45 s

send_message({"to":"Luis Ortega","text":"Luis, I just checked and your package is sitting on our porch."})
add_to_list({"list":"todo","item":"thank-you card for Luis"})

Scoring detail

{
  "n_calls": 2,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Luis Ortega",
        "text": "Luis, I just checked and your package is sitting on our porch."
      },
      "error": null
    },
    {
      "name": "add_to_list",
      "args": {
        "list": "todo",
        "item": "thank-you card for Luis"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "send_message",
      "matched_index": 0,
      "score": 1,
      "args": {
        "to": true,
        "text": true
      }
    },
    {
      "expected": "add_to_list",
      "matched_index": 1,
      "score": 1,
      "args": {
        "list": true,
        "item": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

← assistant-actions-027All Assistant actions itemsassistant-actions-029 →