Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-018

easy extractionsignature

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "name" (string): sender's full name
- "title" (string): job title
- "company" (string): company
- "email" (string): email address
- "phone" (string): phone number
- "website" (string): website

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
Hi Theo,

Confirming we can pick up the two pallets on Thursday morning. I'll send the BOL tonight.

Thanks,
Gabriela Ferreira-Santos
Senior Account Manager | Tallgrass Logistics
gabriela.fs@tallgrasslogistics.com
Direct: 402.555.0163
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "name": "Gabriela Ferreira-Santos",
    "title": "Senior Account Manager",
    "company": "Tallgrass Logistics",
    "email": "gabriela.fs@tallgrasslogistics.com",
    "phone": "+14025550163",
    "website": null
  },
  "fields": {
    "name": {
      "type": "string"
    },
    "title": {
      "type": "string"
    },
    "company": {
      "type": "string"
    },
    "email": {
      "type": "email"
    },
    "phone": {
      "type": "phone"
    },
    "website": {
      "type": "url"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.43 s

{"name": "Gabriela Ferreira-Santos", "title": "Senior Account Manager", "company": "Tallgrass Logistics", "email": "gabriela.fs@tallgrasslogistics.com", "phone": "+1 402 555 0163", "website": null}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+1 402 555 0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Ornith-1.5 9B

✓ pass · score 100 · 2.57 s

{
  "name": "Gabriela Ferreira-Santos",
  "title": "Senior Account Manager",
  "company": "Tallgrass Logistics",
  "email": "gabriela.fs@tallgrasslogistics.com",
  "phone": "+1 402 555 0163",
  "website": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+1 402 555 0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Granite 4.2 8B

✓ pass · score 100 · 2.45 s

{
  "name": "Gabriela Ferreira-Santos",
  "title": "Senior Account Manager",
  "company": "Tallgrass Logistics",
  "email": "gabriela.fs@tallgrasslogistics.com",
  "phone": "+14025550163",
  "website": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+14025550163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Spark-X2.5 4B

✗ fail · score 83 · 1.66 s

{
  "name": "Gabriela Ferreira-Santos",
  "title": "Senior Account Manager",
  "company": "Tallgrass Logistics",
  "email": "gabriela.fs@tallgrasslogistics.com",
  "phone": null,
  "website": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 0,
      "expected": "+14025550163",
      "got": null
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 5,
  "fields_total": 6
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.13 s

{
  "name": "Gabriela Ferreira-Santos",
  "title": "Senior Account Manager",
  "company": "Tallgrass Logistics",
  "email": "gabriela.fs@tallgrasslogistics.com",
  "phone": "+1 402 555 0163",
  "website": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+1 402 555 0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.74 s

{"name": "Gabriela Ferreira-Santos", "title": "Senior Account Manager", "company": "Tallgrass Logistics", "email": "gabriela.fs@tallgrasslogistics.com", "phone": "+1 402 555 0163", "website": null}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+1 402 555 0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

MiniCPM5 2B

✓ pass · score 100 · 0.92 s

{"name":"Gabriela Ferreira-Santos","title":"Senior Account Manager","company":"Tallgrass Logistics","email":"gabriela.fs@tallgrasslogistics.com","phone":"+1 402 555 0163","website":null}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 1,
      "expected": "+14025550163",
      "got": "+1 402 555 0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 6,
  "fields_total": 6
}

K2-Horizon 0.9B

✗ fail · score 83 · 1.16 s

{
  "name": "Gabriela Ferreira-Santos",
  "title": "Senior Account Manager",
  "company": "Tallgrass Logistics",
  "email": "gabriela.fs@tallgrasslogistics.com",
  "phone": "402.555.0163",
  "website": null
}

Scoring detail

{
  "fields": {
    "name": {
      "score": 1,
      "expected": "Gabriela Ferreira-Santos",
      "got": "Gabriela Ferreira-Santos"
    },
    "title": {
      "score": 1,
      "expected": "Senior Account Manager",
      "got": "Senior Account Manager"
    },
    "company": {
      "score": 1,
      "expected": "Tallgrass Logistics",
      "got": "Tallgrass Logistics"
    },
    "email": {
      "score": 1,
      "expected": "gabriela.fs@tallgrasslogistics.com",
      "got": "gabriela.fs@tallgrasslogistics.com"
    },
    "phone": {
      "score": 0,
      "expected": "+14025550163",
      "got": "402.555.0163"
    },
    "website": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 5,
  "fields_total": 6
}

← summarize-extract-017All Summarize + extract itemssummarize-extract-019 →