Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-010

medium extractioninvoice

The prompt

user

Extract information from the utility bill (text version) below. Reply with only a JSON object with exactly these keys:
- "utility" (string): utility company name
- "account_number" (string): account number
- "billing_start" (string "YYYY-MM-DD"): first day of the billing period
- "billing_end" (string "YYYY-MM-DD"): last day of the billing period
- "kwh_used" (number): kWh used this period
- "amount_due" (number): amount due
- "payment_date" (string "YYYY-MM-DD"): date the payment is due or will be drafted
- "autopay" (boolean): true if the bill is paid automatically, else false

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Utility bill (text version):
"""
PRAIRIE VALLEY ELECTRIC COOPERATIVE
Member: DOUGLAS & RUTH ABERNATHY
Service address: 4471 County Road 12, Hesston KS
Account No. 20-55871-03

Billing period: 09/14/2026 - 10/13/2026  (30 days)
Meter readings:  previous 48,212   current 48,913
Energy used this period: 701 kWh
Same period last year: 655 kWh

Energy charge  701 kWh x $0.1185 ....... $83.07
Facility charge ........................ $28.00
Power cost adjustment .................. $4.21
Total current charges .................. $115.28

AMOUNT DUE ............................. $115.28
This amount will be drafted automatically from your bank account on 11/02/2026.
No payment is needed.
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "utility": "Prairie Valley Electric Cooperative",
    "account_number": "20-55871-03",
    "billing_start": "2026-09-14",
    "billing_end": "2026-10-13",
    "kwh_used": 701,
    "amount_due": 115.28,
    "payment_date": "2026-11-02",
    "autopay": true
  },
  "fields": {
    "utility": {
      "type": "string"
    },
    "account_number": {
      "type": "code"
    },
    "billing_start": {
      "type": "date"
    },
    "billing_end": {
      "type": "date"
    },
    "kwh_used": {
      "type": "number"
    },
    "amount_due": {
      "type": "number"
    },
    "payment_date": {
      "type": "date"
    },
    "autopay": {
      "type": "bool"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.21 s

{"utility": "Prairie Valley Electric Cooperative", "account_number": "20-55871-03", "billing_start": "2026-09-14", "billing_end": "2026-10-13", "kwh_used": 701, "amount_due": 115.28, "payment_date": "2026-11-02", "autopay": true}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "Prairie Valley Electric Cooperative"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ornith-1.5 9B

✗ fail · score 88 · 3.74 s

```json
{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}
```

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 0,
      "expected": 701,
      "got": "<missing>"
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

Granite 4.2 8B

✓ pass · score 100 · 3.66 s

{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh_used": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Spark-X2.5 4B

✓ pass · score 100 · 2.86 s

{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh_used": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.88 s

{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh_used": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.20 s

{"utility": "Prairie Valley Electric Cooperative", "account_number": "20-55871-03", "billing_start": "2026-09-14", "billing_end": "2026-10-13", "kwh_used": 701, "amount_due": 115.28, "payment_date": "2026-11-02", "autopay": true}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "Prairie Valley Electric Cooperative"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

MiniCPM5 2B

✓ pass · score 100 · 1.48 s

{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh_used": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

K2-Horizon 0.9B

✓ pass · score 100 · 1.78 s

{
  "utility": "PRAIRIE VALLEY ELECTRIC COOPERATIVE",
  "account_number": "20-55871-03",
  "billing_start": "2026-09-14",
  "billing_end": "2026-10-13",
  "kwh_used": 701,
  "amount_due": 115.28,
  "payment_date": "2026-11-02",
  "autopay": true
}

Scoring detail

{
  "fields": {
    "utility": {
      "score": 1,
      "expected": "Prairie Valley Electric Cooperative",
      "got": "PRAIRIE VALLEY ELECTRIC COOPERATIVE"
    },
    "account_number": {
      "score": 1,
      "expected": "20-55871-03",
      "got": "20-55871-03"
    },
    "billing_start": {
      "score": 1,
      "expected": "2026-09-14",
      "got": "2026-09-14"
    },
    "billing_end": {
      "score": 1,
      "expected": "2026-10-13",
      "got": "2026-10-13"
    },
    "kwh_used": {
      "score": 1,
      "expected": 701,
      "got": 701
    },
    "amount_due": {
      "score": 1,
      "expected": 115.28,
      "got": 115.28
    },
    "payment_date": {
      "score": 1,
      "expected": "2026-11-02",
      "got": "2026-11-02"
    },
    "autopay": {
      "score": 1,
      "expected": true,
      "got": true
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

← summarize-extract-009All Summarize + extract itemssummarize-extract-011 →