Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-019

medium extractionsignature

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "name" (string): sender's name without credentials
- "title" (string): job title
- "company" (string): company
- "office_phone" (string): office phone without extension
- "office_extension" (string): office extension digits as a string
- "mobile_phone" (string): mobile phone
- "fax" (string): fax number
- "email" (string): sender's email address
- "website" (string): website
- "postal_code" (string): ZIP code

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
Ruth - attached are the stamped drawings for the mezzanine. Call my cell if the inspector has questions.

--
Marcus T. Bellweather, PE
Principal Structural Engineer
Cobalt Ridge Engineering Group
1501 Larkspur Ave, Suite 300 | Boise, ID 83702
O: (208) 555-0110 x214  |  M: 208-555-0192  |  F: (208) 555-0111
www.cobaltridgeeng.com

CONFIDENTIALITY NOTICE: This message and any attachments are intended only for the named recipient.
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "name": "Marcus T. Bellweather",
    "title": "Principal Structural Engineer",
    "company": "Cobalt Ridge Engineering Group",
    "office_phone": "+12085550110",
    "office_extension": "214",
    "mobile_phone": "+12085550192",
    "fax": "+12085550111",
    "email": null,
    "website": "www.cobaltridgeeng.com",
    "postal_code": "83702"
  },
  "fields": {
    "name": {
      "type": "string",
      "accept": [
        "Marcus T. Bellweather, PE",
        "Marcus Bellweather"
      ]
    },
    "title": {
      "type": "string"
    },
    "company": {
      "type": "string"
    },
    "office_phone": {
      "type": "phone"
    },
    "office_extension": {
      "type": "code",
      "accept": [
        "x214",
        "ext 214",
        "ext. 214"
      ]
    },
    "mobile_phone": {
      "type": "phone"
    },
    "fax": {
      "type": "phone"
    },
    "email": {
      "type": "email"
    },
    "website": {
      "type": "url"
    },
    "postal_code": {
      "type": "code"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.53 s

{"name": "Marcus T. Bellweather", "title": "Principal Structural Engineer", "company": "Cobalt Ridge Engineering Group", "office_phone": "+1 208 555 0110", "office_extension": "214", "mobile_phone": "+1 208 555 0192", "fax": "+1 208 555 0111", "email": null, "website": "www.cobaltridgeeng.com", "postal_code": "83702"}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+12085550192",
      "got": "+1 208 555 0192"
    },
    "fax": {
      "score": 1,
      "expected": "+12085550111",
      "got": "+1 208 555 0111"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Ornith-1.5 9B

✓ pass · score 100 · 4.31 s

{
  "name": "Marcus T. Bellweather",
  "title": "Principal Structural Engineer",
  "company": "Cobalt Ridge Engineering Group",
  "office_phone": "+1 208 555 0110",
  "office_extension": "214",
  "mobile_phone": "+1 208 555 0192",
  "fax": "+1 208 555 0111",
  "email": null,
  "website": "www.cobaltridgeeng.com",
  "postal_code": "83702"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+12085550192",
      "got": "+1 208 555 0192"
    },
    "fax": {
      "score": 1,
      "expected": "+12085550111",
      "got": "+1 208 555 0111"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Granite 4.2 8B

✗ fail · score 70 · 3.54 s

{
  "name": "Marcus T. Bellweather",
  "title": "Principal Structural Engineer",
  "company": "Cobalt Ridge Engineering Group",
  "office_phone": "208555010",
  "office_extension": "214",
  "mobile_phone": "2085550192",
  "fax": "2085550111",
  "email": null,
  "website": "www.cobaltridgeeng.com",
  "postal_code": "83702"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 0,
      "expected": "+12085550110",
      "got": "208555010"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 0,
      "expected": "+12085550192",
      "got": "2085550192"
    },
    "fax": {
      "score": 0,
      "expected": "+12085550111",
      "got": "2085550111"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 7,
  "fields_total": 10
}

Spark-X2.5 4B

✓ pass · score 100 · 2.99 s

{
  "name": "Marcus T. Bellweather",
  "title": "Principal Structural Engineer",
  "company": "Cobalt Ridge Engineering Group",
  "office_phone": "+1 208 555 0110",
  "office_extension": "214",
  "mobile_phone": "+1 208 555 0192",
  "fax": "+1 208 555 0111",
  "email": null,
  "website": "www.cobaltridgeeng.com",
  "postal_code": "83702"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+12085550192",
      "got": "+1 208 555 0192"
    },
    "fax": {
      "score": 1,
      "expected": "+12085550111",
      "got": "+1 208 555 0111"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Mellum2.1 12B-A2.5B

✗ fail · score 90 · 1.81 s

{
  "name": "Marcus T. Bellweather",
  "title": "Principal Structural Engineer",
  "company": "Cobalt Ridge Engineering Group",
  "office_phone": "+1 208 555 0110",
  "office_extension": "214",
  "mobile_phone": "+1 208 555 0192",
  "fax": null,
  "email": null,
  "website": "www.cobaltridgeeng.com",
  "postal_code": "83702"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+12085550192",
      "got": "+1 208 555 0192"
    },
    "fax": {
      "score": 0,
      "expected": "+12085550111",
      "got": null
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 9,
  "fields_total": 10
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.32 s

{"name": "Marcus T. Bellweather", "title": "Principal Structural Engineer", "company": "Cobalt Ridge Engineering Group", "office_phone": "+1 208 555 0110", "office_extension": "214", "mobile_phone": "+1 208 555 0192", "fax": "+1 208 555 0111", "email": null, "website": "www.cobaltridgeeng.com", "postal_code": "83702"}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 1,
      "expected": "+12085550192",
      "got": "+1 208 555 0192"
    },
    "fax": {
      "score": 1,
      "expected": "+12085550111",
      "got": "+1 208 555 0111"
    },
    "email": {
      "score": 1,
      "expected": null,
      "got": null
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

MiniCPM5 2B

✗ fail · score 80 · 1.52 s

{"name": "Marcus T. Bellweather", "title": "Principal Structural Engineer", "company": "Cobalt Ridge Engineering Group", "office_phone": "+1 208 555 0110", "office_extension": "214", "mobile_phone": "208-555-0192", "fax": "+1 208 555 0111", "email": "not_stated", "website": "www.cobaltridgeeng.com", "postal_code": "83702"}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 1,
      "expected": "+12085550110",
      "got": "+1 208 555 0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 0,
      "expected": "+12085550192",
      "got": "208-555-0192"
    },
    "fax": {
      "score": 1,
      "expected": "+12085550111",
      "got": "+1 208 555 0111"
    },
    "email": {
      "score": 0,
      "expected": null,
      "got": "not_stated"
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 8,
  "fields_total": 10
}

K2-Horizon 0.9B

✗ fail · score 60 · 1.70 s

{
  "name": "Marcus T. Bellweather",
  "title": "Principal Structural Engineer",
  "company": "Cobalt Ridge Engineering Group",
  "office_phone": "(208) 555-0110",
  "office_extension": "214",
  "mobile_phone": "208-555-0192",
  "fax": null,
  "email": "Marcus T. Bellweather",
  "website": "www.cobaltridgeeng.com",
  "postal_code": "83702"
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Marcus T. Bellweather",
      "got": "Marcus T. Bellweather"
    },
    "title": {
      "score": 1,
      "expected": "Principal Structural Engineer",
      "got": "Principal Structural Engineer"
    },
    "company": {
      "score": 1,
      "expected": "Cobalt Ridge Engineering Group",
      "got": "Cobalt Ridge Engineering Group"
    },
    "office_phone": {
      "score": 0,
      "expected": "+12085550110",
      "got": "(208) 555-0110"
    },
    "office_extension": {
      "score": 1,
      "expected": "214",
      "got": "214"
    },
    "mobile_phone": {
      "score": 0,
      "expected": "+12085550192",
      "got": "208-555-0192"
    },
    "fax": {
      "score": 0,
      "expected": "+12085550111",
      "got": null
    },
    "email": {
      "score": 0,
      "expected": null,
      "got": "Marcus T. Bellweather"
    },
    "website": {
      "score": 1,
      "expected": "www.cobaltridgeeng.com",
      "got": "www.cobaltridgeeng.com"
    },
    "postal_code": {
      "score": 1,
      "expected": "83702",
      "got": "83702"
    }
  },
  "fields_correct": 6,
  "fields_total": 10
}

← summarize-extract-018All Summarize + extract itemssummarize-extract-020 →