Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-020

hard single

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Book lunch with Dev at Rosalind's Cafe this Saturday, noon to 1:30.

What we expected

{
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": {
          "type": "contains_all",
          "values": [
            "lunch"
          ]
        },
        "start": {
          "type": "datetime",
          "value": "2026-10-24T12:00:00-05:00",
          "tolerance_s": 60
        },
        "end": {
          "type": "datetime",
          "value": "2026-10-24T13:30:00-05:00",
          "tolerance_s": 60
        },
        "location": {
          "type": "optional",
          "inner": {
            "type": "contains_all",
            "values": [
              "rosalind"
            ]
          }
        },
        "attendees": {
          "type": "optional",
          "inner": {
            "type": "list_contains_all",
            "values": [
              "dev"
            ]
          }
        }
      },
      "allow_extra_args": []
    }
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 0 · 1.19 s

I can help with that, but I don't have a booking tool for restaurants. Would you like me to text Dev to arrange it, or would you prefer to handle the reservation yourself?

Scoring detail

{
  "n_calls": 0,
  "calls": [],
  "reason": "no tool calls"
}

Ornith-1.5 9B

✓ pass · score 100 · 3.66 s

add_calendar_event({"title":"Lunch with Dev at Rosalind's Cafe","start":"2026-10-24T12:00:00-05:00","end":"2026-10-24T13:30:00-05:00","location":"Rosalind's Cafe, Cedar Hollow, IL","attendees":["Maya Brennan","Dev Patel"]})

I'll add that lunch to your calendar now.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Lunch with Dev at Rosalind's Cafe",
        "start": "2026-10-24T12:00:00-05:00",
        "end": "2026-10-24T13:30:00-05:00",
        "location": "Rosalind's Cafe, Cedar Hollow, IL",
        "attendees": [
          "Maya Brennan",
          "Dev Patel"
        ]
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 1,
      "args": {
        "title": true,
        "start": true,
        "end": true,
        "location": true,
        "attendees": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Granite 4.2 8B

✓ pass · score 100 · 3.44 s

add_calendar_event({"title":"Lunch with Dev","start":"2026-10-24T12:00:00-05:00","end":"2026-10-24T13:30:00-05:00","location":"Rosalind's Cafe","attendees":["Dev Patel"]})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Lunch with Dev",
        "start": "2026-10-24T12:00:00-05:00",
        "end": "2026-10-24T13:30:00-05:00",
        "location": "Rosalind's Cafe",
        "attendees": [
          "Dev Patel"
        ]
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 1,
      "args": {
        "title": true,
        "start": true,
        "end": true,
        "location": true,
        "attendees": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Spark-X2.5 4B

✓ pass · score 100 · 1.65 s

add_calendar_event({"title":"Lunch with Dev at Rosalind's Cafe","start":"2026-10-24T12:00:00-05:00","end":"2026-10-24T13:30:00-05:00","location":"Rosalind's Cafe","attendees":["Dev Patel"]})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Lunch with Dev at Rosalind's Cafe",
        "start": "2026-10-24T12:00:00-05:00",
        "end": "2026-10-24T13:30:00-05:00",
        "location": "Rosalind's Cafe",
        "attendees": [
          "Dev Patel"
        ]
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 1,
      "args": {
        "title": true,
        "start": true,
        "end": true,
        "location": true,
        "attendees": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 9.79 s

add_calendar_event({"title":"Lunch with Dev","start":"2026-10-24T12:00:00-05:00","end":"2026-10-24T13:30:00-05:00","location":"Rosalind's Cafe","attendees":["Dev Patel"]})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Lunch with Dev",
        "start": "2026-10-24T12:00:00-05:00",
        "end": "2026-10-24T13:30:00-05:00",
        "location": "Rosalind's Cafe",
        "attendees": [
          "Dev Patel"
        ]
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 1,
      "args": {
        "title": true,
        "start": true,
        "end": true,
        "location": true,
        "attendees": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 60 · 0.93 s

add_calendar_event({"title":"Lunch with Dev at Rosalind's Cafe","start":"2026-10-25T12:00:00-05:00","end":"2026-10-25T13:30:00-05:00","attendees":["Dev Patel"]})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "add_calendar_event",
      "args": {
        "title": "Lunch with Dev at Rosalind's Cafe",
        "start": "2026-10-25T12:00:00-05:00",
        "end": "2026-10-25T13:30:00-05:00",
        "attendees": [
          "Dev Patel"
        ]
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 0.6,
      "args": {
        "title": true,
        "start": false,
        "end": false,
        "location": true,
        "attendees": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.6
}

MiniCPM5 2B

✗ fail · score 0 · 0.63 s

send_message({"to":"Dev Patel (coworker)","text":"Hey, I'd like to book lunch with you at Rosalind's Cafe this Saturday, noon to 1:30. Can you confirm?"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "send_message",
      "args": {
        "to": "Dev Patel (coworker)",
        "text": "Hey, I'd like to book lunch with you at Rosalind's Cafe this Saturday, noon to 1:30. Can you confirm?"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 0,
      "args": {
        "name": "mismatch"
      }
    }
  ],
  "extra_calls": 0,
  "base": 0
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.43 s

get_weather({"location":"Cedar Hollow, IL","date":"2026-10-23"})

I'll help you book lunch with Dev at Rosalind's Cafe for this Saturday. Let me check the weather first to make sure it's a good day to go.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "get_weather",
      "args": {
        "location": "Cedar Hollow, IL",
        "date": "2026-10-23"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "add_calendar_event",
      "matched_index": 0,
      "score": 0,
      "args": {
        "name": "mismatch"
      }
    }
  ],
  "extra_calls": 0,
  "base": 0
}

← assistant-actions-019All Assistant actions itemsassistant-actions-021 →