Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract
summarize-extract-017
hard extractionshipping
The prompt
user
Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "order_number" (string): order number
- "tracking_number" (string): tracking number of the shipped package
- "arrival_date" (string "YYYY-MM-DD"): expected arrival date of the shipped package
- "shipped_items" (array): array of {"name": string, "quantity": number} for items that shipped
- "backordered_items" (array): array of {"name": string, "quantity": number} for items not yet shipped
- "order_total" (number): order total
- "charged_today" (number): amount charged today
- "backorder_tracking_number" (string): tracking number for the backordered package
Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.
Email:
"""
From: Larkfield Home <hello@larkfieldhome.com>
Sent: Friday, December 4, 2026 11:52 AM
Subject: Part of your order is on its way (LH-30917)
Hi Dominique,
Good news - part of your order LH-30917 is on its way!
PACKAGE 1 of 2 - SHIPPED
Carrier: SwiftRoute
Tracking: SR 4410 9982 1137
Arriving: Tuesday
Contents: Linen Duvet Cover (Queen, Oat) x1, Linen Pillowcase Set (Oat) x2
PACKAGE 2 of 2 - BACKORDERED
Contents: Desk Lamp (Brass) x1
We expect this item to ship in about two weeks. We'll email tracking as soon as it leaves our warehouse.
Payment: Your order total is $214.85. We charged $189.90 today for the items that shipped. The remaining $24.95 will be charged when your lamp ships.
"""What we expected
{
"kind": "extract",
"answer": {
"order_number": "LH-30917",
"tracking_number": "SR441099821137",
"arrival_date": "2026-12-08",
"shipped_items": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"backordered_items": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"order_total": 214.85,
"charged_today": 189.9,
"backorder_tracking_number": null
},
"fields": {
"order_number": {
"type": "code"
},
"tracking_number": {
"type": "code"
},
"arrival_date": {
"type": "date"
},
"shipped_items": {
"type": "list",
"item_fields": {
"name": {
"type": "text",
"needles_map": {
"Linen Duvet Cover (Queen, Oat)": [
"duvet"
],
"Linen Pillowcase Set (Oat)": [
"pillowcase",
"pillow case"
]
}
},
"quantity": {
"type": "number"
}
}
},
"backordered_items": {
"type": "list",
"item_fields": {
"name": {
"type": "text",
"needles_map": {
"Desk Lamp (Brass)": [
"lamp"
]
}
},
"quantity": {
"type": "number"
}
}
},
"order_total": {
"type": "number"
},
"charged_today": {
"type": "number"
},
"backorder_tracking_number": {
"type": "code"
}
}
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 88 · 5.28 s
{"order_number": "LH-30917", "tracking_number": "SR 4410 9982 1137", "arrival_date": null, "shipped_items": [{"name": "Linen Duvet Cover (Queen, Oat)", "quantity": 1}, {"name": "Linen Pillowcase Set (Oat)", "quantity": 2}], "backordered_items": [{"name": "Desk Lamp (Brass)", "quantity": 1}], "order_total": 214.85, "charged_today": 189.9, "backorder_tracking_number": null}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": null
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}Ornith-1.5 9B
✗ fail · score 88 · 5.09 s
{
"order_number": "LH-30917",
"tracking_number": "SR 4410 9982 1137",
"arrival_date": null,
"shipped_items": [
{"name": "Linen Duvet Cover (Queen, Oat)", "quantity": 1},
{"name": "Linen Pillowcase Set (Oat)", "quantity": 2}
],
"backordered_items": [
{"name": "Desk Lamp (Brass)", "quantity": 1}
],
"order_total": 214.85,
"charged_today": 189.9,
"backorder_tracking_number": null
}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": null
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}Granite 4.2 8B
✗ fail · score 88 · 5.49 s
{
"order_number": "LH-30917",
"tracking_number": "SR 4410 9982 1137",
"arrival_date": null,
"shipped_items": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"backordered_items": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"order_total": 214.85,
"charged_today": 189.9,
"backorder_tracking_number": null
}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": null
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}Spark-X2.5 4B
✗ fail · score 88 · 3.95 s
{
"order_number": "LH-30917",
"tracking_number": "SR 4410 9982 1137",
"arrival_date": null,
"shipped_items": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"backordered_items": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"order_total": 214.85,
"charged_today": 189.9,
"backorder_tracking_number": null
}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": null
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}Mellum2.1 12B-A2.5B
✗ fail · score 88 · 2.69 s
{
"order_number": "LH-30917",
"tracking_number": "SR 4410 9982 1137",
"arrival_date": "2026-12-13",
"shipped_items": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"backordered_items": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"order_total": 214.85,
"charged_today": 189.9,
"backorder_tracking_number": null
}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": "2026-12-13"
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.56 s
{"order_number": "LH-30917", "tracking_number": "SR 4410 9982 1137", "arrival_date": "2026-12-08", "shipped_items": [{"name": "Linen Duvet Cover", "quantity": 1}, {"name": "Linen Pillowcase Set", "quantity": 2}], "backordered_items": [{"name": "Desk Lamp", "quantity": 1}], "order_total": 214.85, "charged_today": 189.90, "backorder_tracking_number": null}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 1,
"expected": "2026-12-08",
"got": "2026-12-08"
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover",
"quantity": 1
},
{
"name": "Linen Pillowcase Set",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 8,
"fields_total": 8
}MiniCPM5 2B
✓ pass · score 100 · 1.76 s
{"order_number": "LH-30917", "tracking_number": "SR 4410 9982 1137", "arrival_date": "2026-12-08", "shipped_items": [{"name": "Linen Duvet Cover", "quantity": 1}, {"name": "Linen Pillowcase Set", "quantity": 2}], "backordered_items": [{"name": "Desk Lamp", "quantity": 1}], "order_total": 214.85, "charged_today": 189.90, "backorder_tracking_number": null}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 1,
"expected": "2026-12-08",
"got": "2026-12-08"
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover",
"quantity": 1
},
{
"name": "Linen Pillowcase Set",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 8,
"fields_total": 8
}K2-Horizon 0.9B
✗ fail · score 88 · 0.98 s
{
"order_number": "LH-30917",
"tracking_number": "SR 4410 9982 1137",
"arrival_date": "2026-12-11",
"shipped_items": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"backordered_items": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"order_total": 214.85,
"charged_today": 189.90,
"backorder_tracking_number": null
}Scoring detail
{
"fields": {
"order_number": {
"score": 1,
"expected": "LH-30917",
"got": "LH-30917"
},
"tracking_number": {
"score": 1,
"expected": "SR441099821137",
"got": "SR 4410 9982 1137"
},
"arrival_date": {
"score": 0,
"expected": "2026-12-08",
"got": "2026-12-11"
},
"shipped_items": {
"score": 1,
"expected": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
],
"got": [
{
"name": "Linen Duvet Cover (Queen, Oat)",
"quantity": 1
},
{
"name": "Linen Pillowcase Set (Oat)",
"quantity": 2
}
]
},
"backordered_items": {
"score": 1,
"expected": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
],
"got": [
{
"name": "Desk Lamp (Brass)",
"quantity": 1
}
]
},
"order_total": {
"score": 1,
"expected": 214.85,
"got": 214.85
},
"charged_today": {
"score": 1,
"expected": 189.9,
"got": 189.9
},
"backorder_tracking_number": {
"score": 1,
"expected": null,
"got": null
}
},
"fields_correct": 7,
"fields_total": 8
}← summarize-extract-016All Summarize + extract itemssummarize-extract-018 →