Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Assistant actions

assistant-actions-019

hard single

The prompt

Tools offered: create_reminder, add_calendar_event, set_timer, set_alarm, control_light, set_thermostat, lock_door, send_message, play_music, get_weather, add_to_list, navigate_to

user

Remind me on November 3rd at 10 AM to renew the car registration.

What we expected

{
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": {
          "type": "contains_all",
          "values": [
            "registration"
          ]
        },
        "when": {
          "type": "datetime",
          "value": "2026-11-03T10:00:00-06:00",
          "tolerance_s": 60
        }
      },
      "allow_extra_args": []
    }
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 1.81 s

create_reminder({"text":"Renew the car registration","when":"2026-11-03T10:00:00-06:00"})

I'll set that reminder for you.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew the car registration",
        "when": "2026-11-03T10:00:00-06:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 1,
      "args": {
        "text": true,
        "when": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Ornith-1.5 9B

✗ fail · score 50 · 1.85 s

create_reminder({"text":"Renew the car registration.","when":"2026-11-03T10:00:00-05:00"})

I'll set that reminder for you.

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew the car registration.",
        "when": "2026-11-03T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 0.5,
      "args": {
        "text": true,
        "when": false
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

Granite 4.2 8B

✓ pass · score 100 · 1.76 s

create_reminder({"text":"Renew the car registration","when":"2026-11-03T10:00:00-06:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew the car registration",
        "when": "2026-11-03T10:00:00-06:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 1,
      "args": {
        "text": true,
        "when": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Spark-X2.5 4B

✗ fail · score 50 · 0.76 s

create_reminder({"text":"Renew the car registration","when":"2026-11-03T10:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew the car registration",
        "when": "2026-11-03T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 0.5,
      "args": {
        "text": true,
        "when": false
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 0.74 s

create_reminder({"text":"Renew car registration","when":"2026-11-03T10:00:00-06:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew car registration",
        "when": "2026-11-03T10:00:00-06:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 1,
      "args": {
        "text": true,
        "when": true
      }
    }
  ],
  "extra_calls": 0,
  "base": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 50 · 0.61 s

create_reminder({"text":"Renew your car registration on November 3rd at 10 AM.","when":"2026-11-03T10:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew your car registration on November 3rd at 10 AM.",
        "when": "2026-11-03T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 0.5,
      "args": {
        "text": true,
        "when": false
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

MiniCPM5 2B

✗ fail · score 50 · 0.50 s

create_reminder({"text":"Renew car registration","when":"2026-11-03T10:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew car registration",
        "when": "2026-11-03T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 0.5,
      "args": {
        "text": true,
        "when": false
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

K2-Horizon 0.9B

✗ fail · score 50 · 0.31 s

create_reminder({"text":"Renew the car registration","when":"2026-11-03T10:00:00-05:00"})

Scoring detail

{
  "n_calls": 1,
  "calls": [
    {
      "name": "create_reminder",
      "args": {
        "text": "Renew the car registration",
        "when": "2026-11-03T10:00:00-05:00"
      },
      "error": null
    }
  ],
  "per_call": [
    {
      "expected": "create_reminder",
      "matched_index": 0,
      "score": 0.5,
      "args": {
        "text": true,
        "when": false
      }
    }
  ],
  "extra_calls": 0,
  "base": 0.5
}

← assistant-actions-018All Assistant actions itemsassistant-actions-020 →