Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-009

hard extractioninvoice

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "invoice_number" (string): invoice number
- "invoice_date" (string "YYYY-MM-DD"): invoice date
- "due_date" (string "YYYY-MM-DD"): payment due date
- "labor_hours" (number): hours of labor
- "hourly_rate" (number): hourly rate
- "deposit_applied" (number): deposit already paid
- "balance_due" (number): balance still owed
- "late_fee_percent" (number): monthly late fee percentage as a number (e.g. 2 for 2%)

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
From: Rafael Quintero <rafael@quinterocarpentry.com>
To: Helen Brandt
Date: Tue, 6 Oct 2026 16:40:11 -0600
Subject: Invoice for the built-in bookshelves

Hi Helen,

Hope you're enjoying the new shelves! Here's the breakdown for the job. The invoice (#A-7731) is dated today.

Labor: 12.5 hours at $96/hr = $1,200.00
Materials (walnut veneer ply, hardware, finish): $85.40
Subtotal: $1,285.40
Less deposit paid Sept 2: -$300.00
Balance due: $985.40

Terms are net 30 from the invoice date. A 1.5% monthly late fee applies to balances not paid by then. Check or Zelle both work.

Thanks again,
Rafael
Quintero Carpentry
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "invoice_number": "A-7731",
    "invoice_date": "2026-10-06",
    "due_date": "2026-11-05",
    "labor_hours": 12.5,
    "hourly_rate": 96,
    "deposit_applied": 300,
    "balance_due": 985.4,
    "late_fee_percent": 1.5
  },
  "fields": {
    "invoice_number": {
      "type": "code",
      "accept": [
        "#A-7731"
      ]
    },
    "invoice_date": {
      "type": "date"
    },
    "due_date": {
      "type": "date"
    },
    "labor_hours": {
      "type": "number"
    },
    "hourly_rate": {
      "type": "number"
    },
    "deposit_applied": {
      "type": "number"
    },
    "balance_due": {
      "type": "number"
    },
    "late_fee_percent": {
      "type": "number"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.80 s

{"invoice_number": "A-7731", "invoice_date": "2026-10-06", "due_date": "2026-11-05", "labor_hours": 12.5, "hourly_rate": 96, "deposit_applied": 300, "balance_due": 985.4, "late_fee_percent": 1.5}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-11-05",
      "got": "2026-11-05"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ornith-1.5 9B

✗ fail · score 88 · 3.34 s

{
  "invoice_number": "A-7731",
  "invoice_date": "2026-10-06",
  "due_date": "2026-11-06",
  "labor_hours": 12.5,
  "hourly_rate": 96,
  "deposit_applied": 300,
  "balance_due": 985.4,
  "late_fee_percent": 1.5
}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 0,
      "expected": "2026-11-05",
      "got": "2026-11-06"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

Granite 4.2 8B

✗ fail · score 88 · 3.36 s

{
  "invoice_number": "A-7731",
  "invoice_date": "2026-10-06",
  "due_date": null,
  "labor_hours": 12.5,
  "hourly_rate": 96,
  "deposit_applied": 300,
  "balance_due": 985.4,
  "late_fee_percent": 1.5
}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 0,
      "expected": "2026-11-05",
      "got": null
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 7,
  "fields_total": 8
}

Spark-X2.5 4B

✓ pass · score 100 · 2.63 s

{
  "invoice_number": "A-7731",
  "invoice_date": "2026-10-06",
  "due_date": "2026-11-05",
  "labor_hours": 12.5,
  "hourly_rate": 96,
  "deposit_applied": 300,
  "balance_due": 985.4,
  "late_fee_percent": 1.5
}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-11-05",
      "got": "2026-11-05"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.63 s

{
  "invoice_number": "A-7731",
  "invoice_date": "2026-10-06",
  "due_date": "2026-11-05",
  "labor_hours": 12.5,
  "hourly_rate": 96,
  "deposit_applied": 300,
  "balance_due": 985.4,
  "late_fee_percent": 1.5
}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-11-05",
      "got": "2026-11-05"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.13 s

{"invoice_number": "A-7731", "invoice_date": "2026-10-06", "due_date": "2026-11-05", "labor_hours": 12.5, "hourly_rate": 96, "deposit_applied": 300, "balance_due": 985.4, "late_fee_percent": 1.5}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-11-05",
      "got": "2026-11-05"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

MiniCPM5 2B

✓ pass · score 100 · 1.34 s

{"invoice_number": "A-7731", "invoice_date": "2026-10-06", "due_date": "2026-11-05", "labor_hours": 12.5, "hourly_rate": 96, "deposit_applied": 300, "balance_due": 985.40, "late_fee_percent": 1.5}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-11-05",
      "got": "2026-11-05"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 1,
      "expected": 300,
      "got": 300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 8,
  "fields_total": 8
}

K2-Horizon 0.9B

✗ fail · score 75 · 0.62 s

{
  "invoice_number": "A-7731",
  "invoice_date": "2026-10-06",
  "due_date": "2026-10-26",
  "labor_hours": 12.5,
  "hourly_rate": 96,
  "deposit_applied": -300,
  "balance_due": 985.40,
  "late_fee_percent": 1.5
}

Scoring detail

{
  "fields": {
    "invoice_number": {
      "score": 1,
      "expected": "A-7731",
      "got": "A-7731"
    },
    "invoice_date": {
      "score": 1,
      "expected": "2026-10-06",
      "got": "2026-10-06"
    },
    "due_date": {
      "score": 0,
      "expected": "2026-11-05",
      "got": "2026-10-26"
    },
    "labor_hours": {
      "score": 1,
      "expected": 12.5,
      "got": 12.5
    },
    "hourly_rate": {
      "score": 1,
      "expected": 96,
      "got": 96
    },
    "deposit_applied": {
      "score": 0,
      "expected": 300,
      "got": -300
    },
    "balance_due": {
      "score": 1,
      "expected": 985.4,
      "got": 985.4
    },
    "late_fee_percent": {
      "score": 1,
      "expected": 1.5,
      "got": 1.5
    }
  },
  "fields_correct": 6,
  "fields_total": 8
}

← summarize-extract-008All Summarize + extract itemssummarize-extract-010 →