Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-021

hard extractionsignature

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "name" (string): sender's name
- "title" (string): job title
- "company" (string): company
- "street" (string): street address line including unit
- "city" (string): city/suburb
- "state" (string): state abbreviation as written
- "postal_code" (string): postcode
- "country" (string): country
- "office_phone" (string): landline
- "mobile_phone" (string): mobile
- "email" (string): email

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
G'day Priya,

Kegs for the festival will go out on the Monday truck. Ring me if anything changes.

Cheers,
Liam O'Rourke
Head Brewer - Saltbush Brewing Co.
Unit 3, 27 Ferrier Street
Fremantle WA 6160
Australia
Ph: (08) 5550 4172  ·  Mob: 0491 570 110
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "name": "Liam O'Rourke",
    "title": "Head Brewer",
    "company": "Saltbush Brewing Co.",
    "street": "Unit 3, 27 Ferrier Street",
    "city": "Fremantle",
    "state": "WA",
    "postal_code": "6160",
    "country": "Australia",
    "office_phone": "+61855504172",
    "mobile_phone": "+61491570110",
    "email": null
  },
  "fields": {
    "name": {
      "type": "string"
    },
    "title": {
      "type": "string"
    },
    "company": {
      "type": "string",
      "accept": [
        "Saltbush Brewing Company",
        "Saltbush Brewing"
      ]
    },
    "street": {
      "type": "string",
      "accept": [
        "3/27 Ferrier Street",
        "Unit 3, 27 Ferrier St"
      ]
    },
    "city": {
      "type": "string"
    },
    "state": {
      "type": "string",
      "accept": [
        "Western Australia"
      ]
    },
    "postal_code": {
      "type": "code"
    },
    "country": {
      "type": "string"
    },
    "office_phone": {
      "type": "phone"
    },
    "mobile_phone": {
      "type": "phone"
    },
    "email": {
      "type": "email"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.31 s

{"name": "Liam O'Rourke", "title": "Head Brewer", "company": "Saltbush Brewing Co.", "street": "Unit 3, 27 Ferrier Street", "city": "Fremantle", "state": "WA", "postal_code": "6160", "country": "Australia", "office_phone": "+61 8 5550 4172", "mobile_phone": "+61 491 570 110", "email": null}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

Ornith-1.5 9B

✓ pass · score 100 · 4.07 s

{
  "name": "Liam O'Rourke",
  "title": "Head Brewer",
  "company": "Saltbush Brewing Co.",
  "street": "Unit 3, 27 Ferrier Street",
  "city": "Fremantle",
  "state": "WA",
  "postal_code": "6160",
  "country": "Australia",
  "office_phone": "+61 8 5550 4172",
  "mobile_phone": "+61 491 570 110",
  "email": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

Granite 4.2 8B

✓ pass · score 100 · 3.78 s

{
  "name": "Liam O'Rourke",
  "title": "Head Brewer",
  "company": "Saltbush Brewing Co.",
  "street": "Unit 3, 27 Ferrier Street",
  "city": "Fremantle",
  "state": "WA",
  "postal_code": "6160",
  "country": "Australia",
  "office_phone": "+61 8 5550 4172",
  "mobile_phone": "+61 491 570 110",
  "email": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

Spark-X2.5 4B

✓ pass · score 100 · 2.92 s

{
  "name": "Liam O'Rourke",
  "title": "Head Brewer",
  "company": "Saltbush Brewing Co.",
  "street": "Unit 3, 27 Ferrier Street",
  "city": "Fremantle",
  "state": "WA",
  "postal_code": "6160",
  "country": "Australia",
  "office_phone": "+61 8 5550 4172",
  "mobile_phone": "+61 491 570 110",
  "email": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.74 s

{
  "name": "Liam O'Rourke",
  "title": "Head Brewer",
  "company": "Saltbush Brewing Co.",
  "street": "Unit 3, 27 Ferrier Street",
  "city": "Fremantle",
  "state": "WA",
  "postal_code": "6160",
  "country": "Australia",
  "office_phone": "+61 8 5550 4172",
  "mobile_phone": "+61 491 570 110",
  "email": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.24 s

{"name": "Liam O'Rourke", "title": "Head Brewer", "company": "Saltbush Brewing Co.", "street": "Unit 3, 27 Ferrier Street", "city": "Fremantle", "state": "WA", "postal_code": "6160", "country": "Australia", "office_phone": "+61 8 5550 4172", "mobile_phone": "+61 491 570 110", "email": null}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 1,
      "expected": "+61855504172",
      "got": "+61 8 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 11,
  "fields_total": 11
}

MiniCPM5 2B

✗ fail · score 82 · 1.34 s

{"name": "Liam O'Rourke", "title": "Head Brewer", "company": "Saltbush Brewing Co.", "street": "Unit 3, 27 Ferrier Street", "city": "Fremantle", "state": "WA", "postal_code": "6160", "country": "Australia", "office_phone": "+61 5550 4172", "mobile_phone": "+61 491 570 110", "email": ""}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 1,
      "expected": "Head Brewer",
      "got": "Head Brewer"
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 1,
      "expected": "6160",
      "got": "6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 0,
      "expected": "+61855504172",
      "got": "+61 5550 4172"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+61491570110",
      "got": "+61 491 570 110"
    },
    "email": {
      "score": 0,
      "expected": null,
      "got": ""
    }
  },
  "fields_correct": 9,
  "fields_total": 11
}

K2-Horizon 0.9B

✗ fail · score 55 · 2.75 s

{
  "name": "Liam O'Rourke",
  "title": "Head Brewer - Saltbush Brewing Co.",
  "company": "Saltbush Brewing Co.",
  "street": "Unit 3, 27 Ferrier Street",
  "city": "Fremantle",
  "state": "WA",
  "postal_code": "WA 6160",
  "country": "Australia",
  "office_phone": "(08) 5550 4172",
  "mobile_phone": "0491 570 110",
  "email": "Liam O'Rourke"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Liam O'Rourke",
      "got": "Liam O'Rourke"
    },
    "title": {
      "score": 0,
      "expected": "Head Brewer",
      "got": "Head Brewer - Saltbush Brewing Co."
    },
    "company": {
      "score": 1,
      "expected": "Saltbush Brewing Co.",
      "got": "Saltbush Brewing Co."
    },
    "street": {
      "score": 1,
      "expected": "Unit 3, 27 Ferrier Street",
      "got": "Unit 3, 27 Ferrier Street"
    },
    "city": {
      "score": 1,
      "expected": "Fremantle",
      "got": "Fremantle"
    },
    "state": {
      "score": 1,
      "expected": "WA",
      "got": "WA"
    },
    "postal_code": {
      "score": 0,
      "expected": "6160",
      "got": "WA 6160"
    },
    "country": {
      "score": 1,
      "expected": "Australia",
      "got": "Australia"
    },
    "office_phone": {
      "score": 0,
      "expected": "+61855504172",
      "got": "(08) 5550 4172"
    },
    "mobile_phone": {
      "score": 0,
      "expected": "+61491570110",
      "got": "0491 570 110"
    },
    "email": {
      "score": 0,
      "expected": null,
      "got": "Liam O'Rourke"
    }
  },
  "fields_correct": 6,
  "fields_total": 11
}

← summarize-extract-020All Summarize + extract itemssummarize-extract-022 →