Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-025

easy extractionemail

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "service" (string): service name
- "plan" (string): plan name
- "billing_cycle" (string): one of "monthly", "annual"
- "renewal_date" (string "YYYY-MM-DD"): renewal date
- "renewal_price" (number): amount that will be charged before tax
- "card_last4" (string): last four digits of the card, as a string
- "cancel_notice_hours" (number): how many hours before renewal you must cancel

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
Subject: Your StreamNest Premium plan renews soon

Hi Kenji,

Just a reminder: your StreamNest Premium (Annual) plan will renew automatically on December 2, 2026.

Renewal price: $119.99 per year (that's just $9.99/month!), plus applicable tax
Payment method: Visa ending in 3321

Don't want to renew? Cancel at least 24 hours before your renewal date in Account > Plans.

Happy streaming,
The StreamNest Team
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "service": "StreamNest",
    "plan": "Premium",
    "billing_cycle": "annual",
    "renewal_date": "2026-12-02",
    "renewal_price": 119.99,
    "card_last4": "3321",
    "cancel_notice_hours": 24
  },
  "fields": {
    "service": {
      "type": "string",
      "accept": [
        "StreamNest Premium"
      ]
    },
    "plan": {
      "type": "string",
      "accept": [
        "Premium (Annual)",
        "Premium Annual",
        "StreamNest Premium"
      ]
    },
    "billing_cycle": {
      "type": "enum"
    },
    "renewal_date": {
      "type": "date"
    },
    "renewal_price": {
      "type": "number"
    },
    "card_last4": {
      "type": "last4"
    },
    "cancel_notice_hours": {
      "type": "number"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 2.91 s

{"service": "StreamNest", "plan": "Premium", "billing_cycle": "annual", "renewal_date": "2026-12-02", "renewal_price": 119.99, "card_last4": "3321", "cancel_notice_hours": 24}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest"
    },
    "plan": {
      "score": 1,
      "expected": "Premium",
      "got": "Premium"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 7,
  "fields_total": 7
}

Ornith-1.5 9B

✓ pass · score 100 · 2.67 s

{
  "service": "StreamNest",
  "plan": "Premium (Annual)",
  "billing_cycle": "annual",
  "renewal_date": "2026-12-02",
  "renewal_price": 119.99,
  "card_last4": "3321",
  "cancel_notice_hours": 24
}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest"
    },
    "plan": {
      "score": 1,
      "expected": "Premium",
      "got": "Premium (Annual)"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 7,
  "fields_total": 7
}

Granite 4.2 8B

✓ pass · score 100 · 2.80 s

{
  "service": "StreamNest",
  "plan": "Premium",
  "billing_cycle": "annual",
  "renewal_date": "2026-12-02",
  "renewal_price": 119.99,
  "card_last4": "3321",
  "cancel_notice_hours": 24
}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest"
    },
    "plan": {
      "score": 1,
      "expected": "Premium",
      "got": "Premium"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 7,
  "fields_total": 7
}

Spark-X2.5 4B

✓ pass · score 100 · 2.05 s

{
  "service": "StreamNest",
  "plan": "Premium",
  "billing_cycle": "annual",
  "renewal_date": "2026-12-02",
  "renewal_price": 119.99,
  "card_last4": "3321",
  "cancel_notice_hours": 24
}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest"
    },
    "plan": {
      "score": 1,
      "expected": "Premium",
      "got": "Premium"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 7,
  "fields_total": 7
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.31 s

{
  "service": "StreamNest Premium",
  "plan": "Premium",
  "billing_cycle": "annual",
  "renewal_date": "2026-12-02",
  "renewal_price": 119.99,
  "card_last4": "3321",
  "cancel_notice_hours": 24
}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest Premium"
    },
    "plan": {
      "score": 1,
      "expected": "Premium",
      "got": "Premium"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 7,
  "fields_total": 7
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 86 · 0.82 s

{"service": "StreamNest Premium", "plan": "Annual", "billing_cycle": "annual", "renewal_date": "2026-12-02", "renewal_price": 119.99, "card_last4": "3321", "cancel_notice_hours": 24}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest Premium"
    },
    "plan": {
      "score": 0,
      "expected": "Premium",
      "got": "Annual"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 6,
  "fields_total": 7
}

MiniCPM5 2B

✗ fail · score 86 · 0.87 s

{"service":"StreamNest Premium","plan":"Annual","billing_cycle":"annual","renewal_date":"2026-12-02","renewal_price":119.99,"card_last4":"3321","cancel_notice_hours":24}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest Premium"
    },
    "plan": {
      "score": 0,
      "expected": "Premium",
      "got": "Annual"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 6,
  "fields_total": 7
}

K2-Horizon 0.9B

✗ fail · score 86 · 0.50 s

{
  "service": "StreamNest Premium",
  "plan": "Annual",
  "billing_cycle": "annual",
  "renewal_date": "2026-12-02",
  "renewal_price": 119.99,
  "card_last4": "3321",
  "cancel_notice_hours": 24
}

Scoring detail

{
  "fields": {
    "service": {
      "score": 1,
      "expected": "StreamNest",
      "got": "StreamNest Premium"
    },
    "plan": {
      "score": 0,
      "expected": "Premium",
      "got": "Annual"
    },
    "billing_cycle": {
      "score": 1,
      "expected": "annual",
      "got": "annual"
    },
    "renewal_date": {
      "score": 1,
      "expected": "2026-12-02",
      "got": "2026-12-02"
    },
    "renewal_price": {
      "score": 1,
      "expected": 119.99,
      "got": 119.99
    },
    "card_last4": {
      "score": 1,
      "expected": "3321",
      "got": "3321"
    },
    "cancel_notice_hours": {
      "score": 1,
      "expected": 24,
      "got": 24
    }
  },
  "fields_correct": 6,
  "fields_total": 7
}

← summarize-extract-024All Summarize + extract itemssummarize-extract-026 →