Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Summarize + extract

summarize-extract-024

hard extractionemail

The prompt

user

Extract information from the email below. Reply with only a JSON object with exactly these keys:
- "unit" (string): apartment number
- "current_rent" (number): current monthly rent
- "new_rent" (number): new monthly rent if they renew
- "new_rent_start_date" (string "YYYY-MM-DD"): date the new rent starts
- "term_months" (number): renewal term in months
- "pet_rent" (number): monthly pet rent
- "sign_by" (string "YYYY-MM-DD"): deadline to sign
- "month_to_month_rent" (number): rent if they don't renew
- "parking_fee" (number): monthly parking fee

Rules: include every key listed. Use null for any field that is not stated in the text (do not guess). Dates are YYYY-MM-DD. Times are HH:MM in 24-hour format. Amounts and counts are plain JSON numbers (no currency symbols, no commas). Phone numbers are in international format with + and the country code, for example +1 312 555 0100; leave out any national trunk prefix 0, so (0)20 or 020 becomes 20 after +44.

Email:
"""
From: Corinne Waverly, Property Manager <leasing@birchwoodcommons.com>
Date: Thursday, September 10, 2026 2:18 PM
To: Apt 3C - Jamal & Tessa Whitford
Subject: Lease renewal offer - Apt 3C

Hello Jamal and Tessa,

Your current lease ends soon, and we'd love to have you stay. Here is your renewal offer:

- Your current rent of $1,640/month will increase to $1,712/month.
- The new rate starts on the first day of the month after next.
- Renewal term: 12 months.
- Pet rent for Biscuit stays at $35/month.
- Your security deposit of $1,640 stays on file; no additional deposit is needed.

To accept, please sign the renewal through the resident portal by September 30. If we don't hear from you, your lease will convert to month-to-month at $1,890/month.

Best,
Corinne
"""

What we expected

{
  "kind": "extract",
  "answer": {
    "unit": "3C",
    "current_rent": 1640,
    "new_rent": 1712,
    "new_rent_start_date": "2026-11-01",
    "term_months": 12,
    "pet_rent": 35,
    "sign_by": "2026-09-30",
    "month_to_month_rent": 1890,
    "parking_fee": null
  },
  "fields": {
    "unit": {
      "type": "code",
      "accept": [
        "Apt 3C",
        "Apartment 3C"
      ]
    },
    "current_rent": {
      "type": "number"
    },
    "new_rent": {
      "type": "number"
    },
    "new_rent_start_date": {
      "type": "date"
    },
    "term_months": {
      "type": "number"
    },
    "pet_rent": {
      "type": "number"
    },
    "sign_by": {
      "type": "date"
    },
    "month_to_month_rent": {
      "type": "number"
    },
    "parking_fee": {
      "type": "number"
    }
  }
}

What each model answered

MiMo-V2.6 Distill 9B

✗ fail · score 89 · 3.50 s

{"unit": "3C", "current_rent": 1640, "new_rent": 1712, "new_rent_start_date": "2026-10-01", "term_months": 12, "pet_rent": 35, "sign_by": "2026-09-30", "month_to_month_rent": 1890, "parking_fee": null}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-10-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

Ornith-1.5 9B

✗ fail · score 89 · 2.83 s

{"unit": "3C", "current_rent": 1640, "new_rent": 1712, "new_rent_start_date": null, "term_months": 12, "pet_rent": 35, "sign_by": "2026-09-30", "month_to_month_rent": 1890, "parking_fee": null}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": null
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

Granite 4.2 8B

✗ fail · score 89 · 3.75 s

{
  "unit": "3C",
  "current_rent": 1640,
  "new_rent": 1712,
  "new_rent_start_date": "2026-10-01",
  "term_months": 12,
  "pet_rent": 35,
  "sign_by": "2026-09-30",
  "month_to_month_rent": 1890,
  "parking_fee": null
}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-10-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

Spark-X2.5 4B

✗ fail · score 89 · 2.68 s

{
  "unit": "Apt 3C",
  "current_rent": 1640,
  "new_rent": 1712,
  "new_rent_start_date": "2026-10-01",
  "term_months": 12,
  "pet_rent": 35,
  "sign_by": "2026-09-30",
  "month_to_month_rent": 1890,
  "parking_fee": null
}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "Apt 3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-10-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

Mellum2.1 12B-A2.5B

✗ fail · score 89 · 1.89 s

{
  "unit": "3C",
  "current_rent": 1640,
  "new_rent": 1712,
  "new_rent_start_date": "2026-10-01",
  "term_months": 12,
  "pet_rent": 35,
  "sign_by": "2026-09-30",
  "month_to_month_rent": 1890,
  "parking_fee": null
}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-10-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.19 s

{"unit": "3C", "current_rent": 1640, "new_rent": 1712, "new_rent_start_date": "2026-11-01", "term_months": 12, "pet_rent": 35, "sign_by": "2026-09-30", "month_to_month_rent": 1890, "parking_fee": null}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 1,
      "expected": "2026-11-01",
      "got": "2026-11-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 9,
  "fields_total": 9
}

MiniCPM5 2B

✗ fail · score 89 · 1.24 s

{"unit":"Apt 3C","current_rent":1640,"new_rent":1712,"new_rent_start_date":"2026-10-01","term_months":12,"pet_rent":35,"sign_by":"2026-09-30","month_to_month_rent":1890,"parking_fee":null}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "Apt 3C"
    },
    "current_rent": {
      "score": 1,
      "expected": 1640,
      "got": 1640
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-10-01"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 8,
  "fields_total": 9
}

K2-Horizon 0.9B

✗ fail · score 78 · 0.67 s

{
  "unit": "Apt 3C",
  "current_rent": 1630,
  "new_rent": 1712,
  "new_rent_start_date": "2026-09-11",
  "term_months": 12,
  "pet_rent": 35,
  "sign_by": "2026-09-30",
  "month_to_month_rent": 1890,
  "parking_fee": null
}

Scoring detail

{
  "fields": {
    "unit": {
      "score": 1,
      "expected": "3C",
      "got": "Apt 3C"
    },
    "current_rent": {
      "score": 0,
      "expected": 1640,
      "got": 1630
    },
    "new_rent": {
      "score": 1,
      "expected": 1712,
      "got": 1712
    },
    "new_rent_start_date": {
      "score": 0,
      "expected": "2026-11-01",
      "got": "2026-09-11"
    },
    "term_months": {
      "score": 1,
      "expected": 12,
      "got": 12
    },
    "pet_rent": {
      "score": 1,
      "expected": 35,
      "got": 35
    },
    "sign_by": {
      "score": 1,
      "expected": "2026-09-30",
      "got": "2026-09-30"
    },
    "month_to_month_rent": {
      "score": 1,
      "expected": 1890,
      "got": 1890
    },
    "parking_fee": {
      "score": 1,
      "expected": null,
      "got": null
    }
  },
  "fields_correct": 7,
  "fields_total": 9
}

← summarize-extract-023All Summarize + extract itemssummarize-extract-025 →