Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-011
easy extractioninvite
The prompt
user
Extract information from the party invitation below. Reply with only a JSON object with exactly these keys: - "honoree" (string): name of the birthday child - "age" (number): age they are turning - "date" (string "YYYY-MM-DD"): party date - "start_time" (string "HH:MM" (24-hour)): start time - "end_time" (string "HH:MM" (24-hour)): end time - "location" (string): place name, e.g. "Some Park, Pavilion X" (no street address) - "rsvp_deadline" (string "YYYY-MM-DD"): RSVP deadline - "rsvp_phone" (string): RSVP phone number - "rsvp_email" (string): RSVP email address Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44. Party invitation: """ You're invited!! 🎈 Leo is turning 7! 🎈 Join us for pizza, cake & bouncy castle fun Saturday, May 16, 2026 2:00 – 4:30 PM Maplewood Park, Pavilion B 1450 Sycamore Lane, Fairview, OH RSVP to Hannah by May 9: (614) 555-0176 or hannah.okafor@fairviewmail.com No gifts please - just bring your smile! """
What we expected
{
"kind": "extract",
"answer": {
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "+16145550176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
},
"fields": {
"honoree": {
"type": "string"
},
"age": {
"type": "number"
},
"date": {
"type": "date"
},
"start_time": {
"type": "time"
},
"end_time": {
"type": "time"
},
"location": {
"type": "string",
"accept": [
"Maplewood Park, Pavilion B, Fairview",
"Maplewood Park, Pavilion B, Fairview, OH",
"Pavilion B, Maplewood Park"
]
},
"rsvp_deadline": {
"type": "date"
},
"rsvp_phone": {
"type": "phone"
},
"rsvp_email": {
"type": "email"
}
}
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.11 s
{"honoree": "Leo", "age": 7, "date": "2026-05-16", "start_time": "14:00", "end_time": "16:30", "location": "Maplewood Park, Pavilion B", "rsvp_deadline": "2026-05-09", "rsvp_phone": "+1 614 555 0176", "rsvp_email": "hannah.okafor@fairviewmail.com"}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}Ornith-1.5 9B
✓ pass · score 100 · 3.92 s
```json
{
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "+1 614 555 0176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
}
```Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}Granite 4.2 8B
✓ pass · score 100 · 3.86 s
{
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "+1 614 555 0176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}Spark-X2.5 4B
✗ fail · score 89 · 2.84 s
{
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "+614 555 0176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 0,
"expected": "+16145550176",
"got": "+614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 8,
"fields_total": 9
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.87 s
{
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "+1 614 555 0176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.25 s
{"honoree": "Leo", "age": 7, "date": "2026-05-16", "start_time": "14:00", "end_time": "16:30", "location": "Maplewood Park, Pavilion B", "rsvp_deadline": "2026-05-09", "rsvp_phone": "+1 614 555 0176", "rsvp_email": "hannah.okafor@fairviewmail.com"}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}MiniCPM5 2B
✓ pass · score 100 · 1.48 s
{"honoree": "Leo", "age": 7, "date": "2026-05-16", "start_time": "14:00", "end_time": "16:30", "location": "Maplewood Park, Pavilion B", "rsvp_deadline": "2026-05-09", "rsvp_phone": "+1 614 555 0176", "rsvp_email": "hannah.okafor@fairviewmail.com"}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 1,
"expected": "+16145550176",
"got": "+1 614 555 0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 9,
"fields_total": 9
}K2-Horizon 0.9B
✗ fail · score 89 · 0.71 s
{
"honoree": "Leo",
"age": 7,
"date": "2026-05-16",
"start_time": "14:00",
"end_time": "16:30",
"location": "Maplewood Park, Pavilion B",
"rsvp_deadline": "2026-05-09",
"rsvp_phone": "(614) 555-0176",
"rsvp_email": "hannah.okafor@fairviewmail.com"
}Scoring detail
{
"fields": {
"honoree": {
"score": 1,
"expected": "Leo",
"got": "Leo"
},
"age": {
"score": 1,
"expected": 7,
"got": 7
},
"date": {
"score": 1,
"expected": "2026-05-16",
"got": "2026-05-16"
},
"start_time": {
"score": 1,
"expected": "14:00",
"got": "14:00"
},
"end_time": {
"score": 1,
"expected": "16:30",
"got": "16:30"
},
"location": {
"score": 1,
"expected": "Maplewood Park, Pavilion B",
"got": "Maplewood Park, Pavilion B"
},
"rsvp_deadline": {
"score": 1,
"expected": "2026-05-09",
"got": "2026-05-09"
},
"rsvp_phone": {
"score": 0,
"expected": "+16145550176",
"got": "(614) 555-0176"
},
"rsvp_email": {
"score": 1,
"expected": "hannah.okafor@fairviewmail.com",
"got": "hannah.okafor@fairviewmail.com"
}
},
"fields_correct": 8,
"fields_total": 9
}← summarize-extract-010All Summarize + extract itemssummarize-extract-012 →