Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-014

medium extractioninvite

The prompt

user

Extract information from the wedding invitation below. Reply with only a JSON object with exactly these keys:
- "couple" (array of strings): array of the two people getting married (full names)
- "date" (string "YYYY-MM-DD"): wedding date
- "time" (string "HH:MM" (24-hour)): ceremony time
- "venue" (string): venue name
- "city" (string): city
- "state" (string): US state as written
- "rsvp_deadline" (string "YYYY-MM-DD"): reply-by date (same year as the wedding)
- "dress_code" (string): dress code

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Wedding invitation:
"""
Together with their families

AMARA LINDQVIST
and
DANIEL OKONJO

request the pleasure of your company
at the celebration of their marriage

Saturday, the fourteenth of November
two thousand twenty-six
at half past four in the afternoon

Rosewood Conservatory
88 Hillcrest Drive
Asheville, North Carolina

Dinner and dancing to follow

Kindly reply by the first of October
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "couple": [
      "Amara Lindqvist",
      "Daniel Okonjo"
    ],
    "date": "2026-11-14",
    "time": "16:30",
    "venue": "Rosewood Conservatory",
    "city": "Asheville",
    "state": "North Carolina",
    "rsvp_deadline": "2026-10-01",
    "dress_code": null
  },
  "fields": {
    "couple": {
      "type": "string_set",
      "item_type": "string"
    },
    "date": {
      "type": "date"
    },
    "time": {
      "type": "time"
    },
    "venue": {
      "type": "string",
      "accept": [
        "The Rosewood Conservatory"
      ]
    },
    "city": {
      "type": "string"
    },
    "state": {
      "type": "string",
      "accept": [
        "NC"
      ]
    },
    "rsvp_deadline": {
      "type": "date"
    },
    "dress_code": {
      "type": "string"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.15 s

{"couple": ["Amara Lindqvist", "Daniel Okonjo"], "date": "2026-11-14", "time": "16:30", "venue": "Rosewood Conservatory", "city": "Asheville", "state": "North Carolina", "rsvp_deadline": "2026-10-01", "dress_code": null}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 1,
      "expected": "16:30",
      "got": "16:30"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ornith-1.5 9B

✓ pass · score 100 · 3.20 s

```json
{
  "couple": [
    "Amara Lindqvist",
    "Daniel Okonjo"
  ],
  "date": "2026-11-14",
  "time": "16:30",
  "venue": "Rosewood Conservatory",
  "city": "Asheville",
  "state": "North Carolina",
  "rsvp_deadline": "2026-10-01",
  "dress_code": null
}
```

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 1,
      "expected": "16:30",
      "got": "16:30"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Granite 4.2 8B

✗ fail · score 88 · 3.40 s

{
  "couple": [
    "AMARA LINDQVIST",
    "DANIEL OKONJO"
  ],
  "date": "2026-11-14",
  "time": "14:30",
  "venue": "Rosewood Conservatory",
  "city": "Asheville",
  "state": "North Carolina",
  "rsvp_deadline": "2026-10-01",
  "dress_code": null
}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "AMARA LINDQVIST",
        "DANIEL OKONJO"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 0,
      "expected": "16:30",
      "got": "14:30"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

Spark-X2.5 4B

✓ pass · score 100 · 2.39 s

{
  "couple": [
    "AMARA LINDQVIST",
    "DANIEL OKONJO"
  ],
  "date": "2026-11-14",
  "time": "16:30",
  "venue": "Rosewood Conservatory",
  "city": "Asheville",
  "state": "North Carolina",
  "rsvp_deadline": "2026-10-01",
  "dress_code": null
}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "AMARA LINDQVIST",
        "DANIEL OKONJO"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 1,
      "expected": "16:30",
      "got": "16:30"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.41 s

{
  "couple": ["AMARA LINDQVIST", "DANIEL OKONJO"],
  "date": "2026-11-14",
  "time": "16:30",
  "venue": "Rosewood Conservatory",
  "city": "Asheville",
  "state": "North Carolina",
  "rsvp_deadline": "2026-10-01",
  "dress_code": null
}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "AMARA LINDQVIST",
        "DANIEL OKONJO"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 1,
      "expected": "16:30",
      "got": "16:30"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 88 · 0.97 s

{"couple": ["Amara Lindqvist", "Daniel Okonjo"], "date": "2026-11-14", "time": "16:00", "venue": "Rosewood Conservatory", "city": "Asheville", "state": "North Carolina", "rsvp_deadline": "2026-10-01", "dress_code": null}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 0,
      "expected": "16:30",
      "got": "16:00"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

MiniCPM5 2B

✗ fail · score 88 · 1.14 s

{"couple": ["AMARA LINDQVIST", "DANIEL OKONJO"], "date": "2026-11-14", "time": "16:00", "venue": "Rosewood Conservatory", "city": "Asheville", "state": "North Carolina", "rsvp_deadline": "2026-10-01", "dress_code": null}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "AMARA LINDQVIST",
        "DANIEL OKONJO"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 0,
      "expected": "16:30",
      "got": "16:00"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

K2-Horizon 0.9B

✗ fail · score 88 · 0.58 s

{
  "couple": ["AMARA LINDQVIST", "DANIEL OKONJO"],
  "date": "2026-11-14",
  "time": "16:00",
  "venue": "Rosewood Conservatory",
  "city": "Asheville",
  "state": "North Carolina",
  "rsvp_deadline": "2026-10-01",
  "dress_code": null
}

Scoring detail

{
  "fields": {
    "couple": {
      "score": 1,
      "expected": [
        "Amara Lindqvist",
        "Daniel Okonjo"
      ],
      "got": [
        "AMARA LINDQVIST",
        "DANIEL OKONJO"
      ]
    },
    "date": {
      "score": 1,
      "expected": "2026-11-14",
      "got": "2026-11-14"
    },
    "time": {
      "score": 0,
      "expected": "16:30",
      "got": "16:00"
    },
    "venue": {
      "score": 1,
      "expected": "Rosewood Conservatory",
      "got": "Rosewood Conservatory"
    },
    "city": {
      "score": 1,
      "expected": "Asheville",
      "got": "Asheville"
    },
    "state": {
      "score": 1,
      "expected": "North Carolina",
      "got": "North Carolina"
    },
    "rsvp_deadline": {
      "score": 1,
      "expected": "2026-10-01",
      "got": "2026-10-01"
    },
    "dress_code": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

← summarize-extract-013All Summarize + extract itemssummarize-extract-015 →