Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-015

easy extractionshipping

The prompt

user

Extract information from the shipping notification below. Reply with only a JSON object with exactly these keys:
- "retailer" (string): store name
- "order_number" (string): order number
- "carrier" (string): shipping carrier
- "tracking_number" (string): tracking number
- "estimated_delivery" (string "YYYY-MM-DD"): estimated delivery date
- "item_count" (number): total number of items in the shipment

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Shipping notification:
"""
Subject: Your Hollow Pine Goods order has shipped!

Hi Sam,

Good news - your order is on its way.

Order number: 114-2290
Shipped via: Parcelway Ground
Tracking number: PW7781234409US
Estimated delivery: Thursday, June 11, 2026

Items in this shipment:
  1 x Ceramic Pour-Over Set (Sand)

Track your package anytime from your account page.
Hollow Pine Goods
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "retailer": "Hollow Pine Goods",
    "order_number": "114-2290",
    "carrier": "Parcelway Ground",
    "tracking_number": "PW7781234409US",
    "estimated_delivery": "2026-06-11",
    "item_count": 1
  },
  "fields": {
    "retailer": {
      "type": "string"
    },
    "order_number": {
      "type": "code"
    },
    "carrier": {
      "type": "string",
      "accept": [
        "Parcelway"
      ]
    },
    "tracking_number": {
      "type": "code"
    },
    "estimated_delivery": {
      "type": "date"
    },
    "item_count": {
      "type": "number"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.90 s

{"retailer": "Hollow Pine Goods", "order_number": "114-2290", "carrier": "Parcelway Ground", "tracking_number": "PW7781234409US", "estimated_delivery": "2026-06-11", "item_count": 1}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Ornith-1.5 9B

✓ pass · score 100 · 2.73 s

{
  "retailer": "Hollow Pine Goods",
  "order_number": "114-2290",
  "carrier": "Parcelway Ground",
  "tracking_number": "PW7781234409US",
  "estimated_delivery": "2026-06-11",
  "item_count": 1
}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Granite 4.2 8B

✓ pass · score 100 · 2.54 s

{
  "retailer": "Hollow Pine Goods",
  "order_number": "114-2290",
  "carrier": "Parcelway Ground",
  "tracking_number": "PW7781234409US",
  "estimated_delivery": "2026-06-11",
  "item_count": 1
}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Spark-X2.5 4B

✓ pass · score 100 · 1.98 s

{
  "retailer": "Hollow Pine Goods",
  "order_number": "114-2290",
  "carrier": "Parcelway Ground",
  "tracking_number": "PW7781234409US",
  "estimated_delivery": "2026-06-11",
  "item_count": 1
}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.25 s

{
  "retailer": "Hollow Pine Goods",
  "order_number": "114-2290",
  "carrier": "Parcelway Ground",
  "tracking_number": "PW7781234409US",
  "estimated_delivery": "2026-06-11",
  "item_count": 1
}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.85 s

{"retailer": "Hollow Pine Goods", "order_number": "114-2290", "carrier": "Parcelway Ground", "tracking_number": "PW7781234409US", "estimated_delivery": "2026-06-11", "item_count": 1}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

MiniCPM5 2B

✓ pass · score 100 · 0.94 s

{"retailer": "Hollow Pine Goods", "order_number": "114-2290", "carrier": "Parcelway Ground", "tracking_number": "PW7781234409US", "estimated_delivery": "2026-06-11", "item_count": 1}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.47 s

{
  "retailer": "Hollow Pine Goods",
  "order_number": "114-2290",
  "carrier": "Parcelway Ground",
  "tracking_number": "PW7781234409US",
  "estimated_delivery": "2026-06-11",
  "item_count": 1
}

Scoring detail

{
  "fields": {
    "retailer": {
      "score": 1,
      "expected": "Hollow Pine Goods",
      "got": "Hollow Pine Goods"
    },
    "order_number": {
      "score": 1,
      "expected": "114-2290",
      "got": "114-2290"
    },
    "carrier": {
      "score": 1,
      "expected": "Parcelway Ground",
      "got": "Parcelway Ground"
    },
    "tracking_number": {
      "score": 1,
      "expected": "PW7781234409US",
      "got": "PW7781234409US"
    },
    "estimated_delivery": {
      "score": 1,
      "expected": "2026-06-11",
      "got": "2026-06-11"
    },
    "item_count": {
      "score": 1,
      "expected": 1,
      "got": 1
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

← summarize-extract-014All Summarize + extract itemssummarize-extract-016 →