Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-008

hard extractioninvoice

The prompt

user

Extract information from the billing email below. Reply with only a JSON object with exactly these keys:
- "provider" (string): company name
- "account_number" (string): account number
- "statement_date" (string "YYYY-MM-DD"): statement date
- "previous_balance" (number): previous balance
- "payment_received" (number): payment received, as a positive number
- "past_due" (number): past due amount
- "new_charges" (number): total new charges
- "total_due" (number): total amount due
- "due_date" (string "YYYY-MM-DD"): date the new charges are due
- "autopay_enabled" (boolean): true or false

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Billing email:
"""
From: Northgate Fiber <statements@northgatefiber.net>
Subject: Your Northgate Fiber statement is ready

Hi Carmen, your monthly statement is ready.

Account: 8830-21197
Statement date: 08/05/2026

Previous balance ................ $142.60
Payment received 07/22 .......... -$100.00   Thank you!
Past due balance ................ $42.60   (due immediately)

New charges
  Gig Internet (Aug 5 - Sep 4) .... $75.00
  Equipment rental (router) ....... $12.00
  Promotional credit .............. -$15.00
  Taxes & fees .................... $6.38
Total new charges ................. $78.38

TOTAL AMOUNT DUE .................. $120.98
New charges due by ................ 08/26/2026

AutoPay is OFF. Turn on AutoPay and save $5/mo.
Questions? Call 1-800-555-0199.
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "provider": "Northgate Fiber",
    "account_number": "8830-21197",
    "statement_date": "2026-08-05",
    "previous_balance": 142.6,
    "payment_received": 100,
    "past_due": 42.6,
    "new_charges": 78.38,
    "total_due": 120.98,
    "due_date": "2026-08-26",
    "autopay_enabled": false
  },
  "fields": {
    "provider": {
      "type": "string"
    },
    "account_number": {
      "type": "code"
    },
    "statement_date": {
      "type": "date"
    },
    "previous_balance": {
      "type": "number"
    },
    "payment_received": {
      "type": "number"
    },
    "past_due": {
      "type": "number"
    },
    "new_charges": {
      "type": "number"
    },
    "total_due": {
      "type": "number"
    },
    "due_date": {
      "type": "date"
    },
    "autopay_enabled": {
      "type": "bool"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.67 s

{"provider": "Northgate Fiber", "account_number": "8830-21197", "statement_date": "2026-08-05", "previous_balance": 142.6, "payment_received": 100.0, "past_due": 42.6, "new_charges": 78.38, "total_due": 120.98, "due_date": "2026-08-26", "autopay_enabled": false}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Ornith-1.5 9B

✓ pass · score 100 · 3.97 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-08-05",
  "previous_balance": 142.6,
  "payment_received": 100.0,
  "past_due": 42.6,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Granite 4.2 8B

✓ pass · score 100 · 3.96 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-08-05",
  "previous_balance": 142.6,
  "payment_received": 100.0,
  "past_due": 42.6,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Spark-X2.5 4B

✗ fail · score 90 · 3.13 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-08-05",
  "previous_balance": 142.6,
  "payment_received": -100.0,
  "past_due": 42.6,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 0,
      "expected": 100,
      "got": -100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 9,
  "fields_total": 10
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 2.00 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-08-05",
  "previous_balance": 142.60,
  "payment_received": 100.00,
  "past_due": 42.60,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.37 s

{"provider": "Northgate Fiber", "account_number": "8830-21197", "statement_date": "2026-08-05", "previous_balance": 142.6, "payment_received": 100.0, "past_due": 42.6, "new_charges": 78.38, "total_due": 120.98, "due_date": "2026-08-26", "autopay_enabled": false}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

MiniCPM5 2B

✓ pass · score 100 · 1.68 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-08-05",
  "previous_balance": 142.60,
  "payment_received": 100.00,
  "past_due": 42.60,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 1,
      "expected": "2026-08-05",
      "got": "2026-08-05"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 1,
      "expected": 100,
      "got": 100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 10,
  "fields_total": 10
}

K2-Horizon 0.9B

✗ fail · score 80 · 0.72 s

{
  "provider": "Northgate Fiber",
  "account_number": "8830-21197",
  "statement_date": "2026-05-08",
  "previous_balance": 142.60,
  "payment_received": -100.00,
  "past_due": 42.60,
  "new_charges": 78.38,
  "total_due": 120.98,
  "due_date": "2026-08-26",
  "autopay_enabled": false
}

Scoring detail

{
  "fields": {
    "provider": {
      "score": 1,
      "expected": "Northgate Fiber",
      "got": "Northgate Fiber"
    },
    "account_number": {
      "score": 1,
      "expected": "8830-21197",
      "got": "8830-21197"
    },
    "statement_date": {
      "score": 0,
      "expected": "2026-08-05",
      "got": "2026-05-08"
    },
    "previous_balance": {
      "score": 1,
      "expected": 142.6,
      "got": 142.6
    },
    "payment_received": {
      "score": 0,
      "expected": 100,
      "got": -100
    },
    "past_due": {
      "score": 1,
      "expected": 42.6,
      "got": 42.6
    },
    "new_charges": {
      "score": 1,
      "expected": 78.38,
      "got": 78.38
    },
    "total_due": {
      "score": 1,
      "expected": 120.98,
      "got": 120.98
    },
    "due_date": {
      "score": 1,
      "expected": "2026-08-26",
      "got": "2026-08-26"
    },
    "autopay_enabled": {
      "score": 1,
      "expected": false,
      "got": false
    }
  },
  "fields_correct": 8,
  "fields_total": 10
}

← summarize-extract-007All Summarize + extract itemssummarize-extract-009 →