Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-023
hard extractionemail
The prompt
user
Extract information from the travel itinerary below. Reply with only a JSON object with exactly these keys:
- "confirmation_code" (string): confirmation code
- "passenger_last_name" (string): passenger's last name
- "flights" (array): array of every flight, each {"flight_number": string, "from": 3-letter airport code, "to": 3-letter airport code, "depart_date": "YYYY-MM-DD", "depart_time": "HH:MM"}
- "checked_bags" (number): checked bags included per passenger
- "total_paid" (number): total paid
Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.
Travel itinerary:
"""
AEROLUX | Your trip confirmation
Confirmation code: QX7LMN
Passenger: HOLLOWAY/RENEE MS
OUTBOUND
Flight AX 482 Mon 16 Nov 2026
Depart 6:05 PM Denver (DEN)
Arrive 11:48 PM Newark (EWR)
-- Overnight connection in Newark (7 h 27 m) --
Flight AX 1290 Tue 17 Nov 2026
Depart 7:15 AM Newark (EWR)
Arrive 9:02 AM Burlington (BTV)
RETURN
Flight AX 1291 Sun 22 Nov 2026
Depart 10:40 AM Burlington (BTV)
Arrive 12:31 PM Newark (EWR)
Flight AX 377 Sun 22 Nov 2026
Depart 2:10 PM Newark (EWR)
Arrive 4:52 PM Denver (DEN)
Fare: Main Cabin | Bags: 1 checked bag included per passenger
Total paid: USD 486.20
"""What we expected
{
"kind": "extract",
"answer": {
"confirmation_code": "QX7LMN",
"passenger_last_name": "Holloway",
"flights": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486.2
},
"fields": {
"confirmation_code": {
"type": "code"
},
"passenger_last_name": {
"type": "string"
},
"flights": {
"type": "list",
"item_fields": {
"flight_number": {
"type": "code"
},
"from": {
"type": "code"
},
"to": {
"type": "code"
},
"depart_date": {
"type": "date"
},
"depart_time": {
"type": "time"
}
}
},
"checked_bags": {
"type": "number"
},
"total_paid": {
"type": "number"
}
}
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 8.61 s
{"confirmation_code": "QX7LMN", "passenger_last_name": "HOLLOWAY", "flights": [{"flight_number": "AX 482", "from": "DEN", "to": "EWR", "depart_date": "2026-11-16", "depart_time": "18:05"}, {"flight_number": "AX 1290", "from": "EWR", "to": "BTV", "depart_date": "2026-11-17", "depart_time": "07:15"}, {"flight_number": "AX 1291", "from": "BTV", "to": "EWR", "depart_date": "2026-11-22", "depart_time": "10:40"}, {"flight_number": "AX 377", "from": "EWR", "to": "DEN", "depart_date": "2026-11-22", "depart_time": "14:10"}], "checked_bags": 1, "total_paid": 486.2}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}Ornith-1.5 9B
✓ pass · score 100 · 8.19 s
{
"confirmation_code": "QX7LMN",
"passenger_last_name": "HOLLOWAY",
"flights": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486.2
}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}Granite 4.2 8B
✓ pass · score 100 · 8.44 s
{
"confirmation_code": "QX7LMN",
"passenger_last_name": "HOLLOWAY",
"flights": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486.2
}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}Spark-X2.5 4B
✓ pass · score 100 · 6.36 s
{
"confirmation_code": "QX7LMN",
"passenger_last_name": "HOLLOWAY",
"flights": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486.2
}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 4.33 s
{
"confirmation_code": "QX7LMN",
"passenger_last_name": "HOLLOWAY",
"flights": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486.20
}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 80 · 2.64 s
{"confirmation_code": "QX7LMN", "passenger_last_name": "MS", "flights": [{"flight_number": "AX 482", "from": "DEN", "to": "EWR", "depart_date": "2026-11-16", "depart_time": "18:05"}, {"flight_number": "AX 1290", "from": "EWR", "to": "BTV", "depart_date": "2026-11-17", "depart_time": "07:15"}, {"flight_number": "AX 1291", "from": "BTV", "to": "EWR", "depart_date": "2026-11-22", "depart_time": "10:40"}, {"flight_number": "AX 377", "from": "EWR", "to": "DEN", "depart_date": "2026-11-22", "depart_time": "14:10"}], "checked_bags": 1, "total_paid": 486.20}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 0,
"expected": "Holloway",
"got": "MS"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 4,
"fields_total": 5
}MiniCPM5 2B
✓ pass · score 100 · 2.74 s
{"confirmation_code": "QX7LMN", "passenger_last_name": "HOLLOWAY", "flights": [{"flight_number": "AX 482", "from": "DEN", "to": "EWR", "depart_date": "2026-11-16", "depart_time": "18:05"}, {"flight_number": "AX 1290", "from": "EWR", "to": "BTV", "depart_date": "2026-11-17", "depart_time": "07:15"}, {"flight_number": "AX 1291", "from": "BTV", "to": "EWR", "depart_date": "2026-11-22", "depart_time": "10:40"}, {"flight_number": "AX 377", "from": "EWR", "to": "DEN", "depart_date": "2026-11-22", "depart_time": "14:10"}], "checked_bags": 1, "total_paid": 486.2}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 1,
"expected": 486.2,
"got": 486.2
}
},
"fields_correct": 5,
"fields_total": 5
}K2-Horizon 0.9B
✗ fail · score 80 · 1.51 s
{
"confirmation_code": "QX7LMN",
"passenger_last_name": "HOLLOWAY",
"flights": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"checked_bags": 1,
"total_paid": 486
}Scoring detail
{
"fields": {
"confirmation_code": {
"score": 1,
"expected": "QX7LMN",
"got": "QX7LMN"
},
"passenger_last_name": {
"score": 1,
"expected": "Holloway",
"got": "HOLLOWAY"
},
"flights": {
"score": 1,
"expected": [
{
"flight_number": "AX482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
],
"got": [
{
"flight_number": "AX 482",
"from": "DEN",
"to": "EWR",
"depart_date": "2026-11-16",
"depart_time": "18:05"
},
{
"flight_number": "AX 1290",
"from": "EWR",
"to": "BTV",
"depart_date": "2026-11-17",
"depart_time": "07:15"
},
{
"flight_number": "AX 1291",
"from": "BTV",
"to": "EWR",
"depart_date": "2026-11-22",
"depart_time": "10:40"
},
{
"flight_number": "AX 377",
"from": "EWR",
"to": "DEN",
"depart_date": "2026-11-22",
"depart_time": "14:10"
}
]
},
"checked_bags": {
"score": 1,
"expected": 1,
"got": 1
},
"total_paid": {
"score": 0,
"expected": 486.2,
"got": 486
}
},
"fields_correct": 4,
"fields_total": 5
}← summarize-extract-022All Summarize + extract itemssummarize-extract-024 →