Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-007

easy extractioninvoice

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "vendor" (string): company that issued the invoice
- "invoice_number" (string): invoice number
- "amount_due" (number): amount due
- "currency" (string): three-letter currency code
- "due_date" (string "YYYY-MM-DD"): payment due date

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
From: Priya Raman <billing@brightleafstudio.co>
To: Jordan Mattis
Date: Thu, 1 Oct 2026 09:14:02 -0500
Subject: Invoice INV-20418 from Brightleaf Studio

Hi Jordan,

Thanks again for the logo refresh project. Attached is invoice INV-20418 for the work completed in September.

Amount due: $1,850.00
Due date: October 31, 2026

You can pay by ACH or card using the link in the PDF. Let me know if you have any questions!

Best,
Priya Raman
Brightleaf Studio
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "vendor": "Brightleaf Studio",
    "invoice_number": "INV-20418",
    "amount_due": 1850,
    "currency": "USD",
    "due_date": "2026-10-31"
  },
  "fields": {
    "vendor": {
      "type": "string"
    },
    "invoice_number": {
      "type": "code"
    },
    "amount_due": {
      "type": "number"
    },
    "currency": {
      "type": "string"
    },
    "due_date": {
      "type": "date"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.46 s

{"vendor": "Brightleaf Studio", "invoice_number": "INV-20418", "amount_due": 1850, "currency": "USD", "due_date": "2026-10-31"}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

Ornith-1.5 9B

✓ pass · score 100 · 2.34 s

{
  "vendor": "Brightleaf Studio",
  "invoice_number": "INV-20418",
  "amount_due": 1850,
  "currency": "USD",
  "due_date": "2026-10-31"
}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

Granite 4.2 8B

✓ pass · score 100 · 2.21 s

{
  "vendor": "Brightleaf Studio",
  "invoice_number": "INV-20418",
  "amount_due": 1850,
  "currency": "USD",
  "due_date": "2026-10-31"
}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

Spark-X2.5 4B

✓ pass · score 100 · 1.64 s

{
  "vendor": "Brightleaf Studio",
  "invoice_number": "INV-20418",
  "amount_due": 1850.0,
  "currency": "USD",
  "due_date": "2026-10-31"
}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.06 s

{
  "vendor": "Brightleaf Studio",
  "invoice_number": "INV-20418",
  "amount_due": 1850.00,
  "currency": "USD",
  "due_date": "2026-10-31"
}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.69 s

{"vendor": "Brightleaf Studio", "invoice_number": "INV-20418", "amount_due": 1850.00, "currency": "USD", "due_date": "2026-10-31"}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

MiniCPM5 2B

✓ pass · score 100 · 0.88 s

{"vendor": "Brightleaf Studio", "invoice_number": "INV-20418", "amount_due": 1850.00, "currency": "USD", "due_date": "2026-10-31"}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.40 s

{
  "vendor": "Brightleaf Studio",
  "invoice_number": "INV-20418",
  "amount_due": 1850,
  "currency": "USD",
  "due_date": "2026-10-31"
}

Scoring detail

{
  "fields": {
    "vendor": {
      "score": 1,
      "expected": "Brightleaf Studio",
      "got": "Brightleaf Studio"
    },
    "invoice_number": {
      "score": 1,
      "expected": "INV-20418",
      "got": "INV-20418"
    },
    "amount_due": {
      "score": 1,
      "expected": 1850,
      "got": 1850
    },
    "currency": {
      "score": 1,
      "expected": "USD",
      "got": "USD"
    },
    "due_date": {
      "score": 1,
      "expected": "2026-10-31",
      "got": "2026-10-31"
    }
  },
  "fields_correct": 5,
  "fields_total": 5
}

← summarize-extract-006All Summarize + extract itemssummarize-extract-008 →