Summarize + extract

Pull exact JSON out of messy receipts, invoices, invites, shipping emails and signatures, and summarize emails, transcripts and articles.

Results

Model	Accuracy	Pass	Easy	Medium	Hard	Median latency	Tok/s	Job time
MiMo-V2.6 Distill 9B	96.1	75.0%	80.0%	76.9%	68.8%	3.62 s	37.6	2m 55s
Mellum2.1 12B-A2.5B	92.2	56.8%	73.3%	46.2%	50.0%	1.62 s	101.2	1m 18s
Ornith-1.5 9B	89.9	59.1%	66.7%	76.9%	37.5%	3.27 s	46.1	2m 36s
Granite 4.2 8B	89.5	45.5%	60.0%	38.5%	37.5%	3.70 s	43.8	3m 24s
MiniCPM5 2B	88.8	47.7%	60.0%	23.1%	56.2%	1.32 s	98.3	58.5 s
Spark-X2.5 4B	88.0	54.5%	66.7%	53.8%	43.8%	2.35 s	65.1	1m 51s
Ling-3.0-tiny (7.9B-A1.3B)	87.1	47.7%	53.3%	46.2%	43.8%	1.07 s	116.3	48.4 s
K2-Horizon 0.9B	74.9	20.5%	26.7%	23.1%	12.5%	0.77 s	210.2	46.5 s

How it is scored

Extraction: per-field exact match after normalization (numbers +-0.005, phones digits-only, emails case-insensitive, null must be null, lists order-insensitive); score = fraction of fields correct, pass = all correct, invalid JSON = 0. Summaries: deterministic partial score = key-fact coverage x (0.7 + 0.15 word-limit + 0.15 no-invented-numbers), plus a judge (rubric.md) for fluency and faithfulness.

Temperature 0 · max 1024 output tokens.

System prompt every model got

You are a careful assistant that extracts information from documents and summarizes text. Follow the requested output format exactly. Use only information stated in the text.

Every item

Final score per item (0-100), which for this job blends the scripted checks with the judge panel's mark, as in the published results. Click any row to see the prompt, the expected answer and every model's reply. Columns sort.

Item	Level	Prompt	Passed by	MiMo-V2.6 Distill 9B	Ornith-1.5 9B	Granite 4.2 8B	Spark-X2.5 4B	Mellum2.1 12B-A2.5B	Ling-3.0-tiny (7.9B-A1.3B)	MiniCPM5 2B	K2-Horizon 0.9B
summarize-extract-001	easy	Extract information from the receipt below. Reply with only a JSON object with exactly these keys: - "merchant…	8/8	100	100	100	100	100	100	100	100
summarize-extract-002	medium	Extract information from the receipt (OCR text from a photo) below. Reply with only a JSON object with exactly…	6/8	100	100	100	100	100	100	88 ✗	63 ✗
summarize-extract-003	hard	Extract information from the receipt (OCR text from a photo) below. Reply with only a JSON object with exactly…	4/8	86 ✗	100	100	100	89 ✗	93 ✗	100	63 ✗
summarize-extract-004	hard	Extract information from the restaurant receipt below. Reply with only a JSON object with exactly these keys: …	8/8	100	100	100	100	100	100	100	100
summarize-extract-005	easy	Extract information from the fuel receipt below. Reply with only a JSON object with exactly these keys: - "sta…	7/8	100	100	100	100	100	100	100	83 ✗
summarize-extract-006	medium	Extract information from the receipt below. Reply with only a JSON object with exactly these keys: - "merchant…	6/8	100	100	100	88 ✗	100	100	100	88 ✗
summarize-extract-007	easy	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "vendor" (s…	8/8	100	100	100	100	100	100	100	100
summarize-extract-008	hard	Extract information from the billing email below. Reply with only a JSON object with exactly these keys: - "pr…	6/8	100	100	100	90 ✗	100	100	100	80 ✗
summarize-extract-009	hard	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "invoice_nu…	5/8	100	88 ✗	88 ✗	100	100	100	100	75 ✗
summarize-extract-010	medium	Extract information from the utility bill (text version) below. Reply with only a JSON object with exactly the…	7/8	100	88 ✗	100	100	100	100	100	100
summarize-extract-011	easy	Extract information from the party invitation below. Reply with only a JSON object with exactly these keys: - …	6/8	100	100	100	89 ✗	100	100	100	89 ✗
summarize-extract-012	medium	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "title" (st…	5/8	100	100	100	86 ✗	100	100	86 ✗	57 ✗
summarize-extract-013	hard	Extract information from the email thread (newest message first) below. Reply with only a JSON object with exa…	2/8	100	0 ✗	33 ✗	67 ✗	100	67 ✗	83 ✗	50 ✗
summarize-extract-014	medium	Extract information from the wedding invitation below. Reply with only a JSON object with exactly these keys: …	4/8	100	100	88 ✗	100	100	88 ✗	88 ✗	88 ✗
summarize-extract-015	easy	Extract information from the shipping notification below. Reply with only a JSON object with exactly these key…	8/8	100	100	100	100	100	100	100	100
summarize-extract-016	medium	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "order_numb…	7/8	100	100	86 ✗	100	100	100	100	100
summarize-extract-017	hard	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "order_numb…	2/8	88 ✗	88 ✗	88 ✗	88 ✗	88 ✗	100	100	88 ✗
summarize-extract-018	easy	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "name" (str…	6/8	100	100	100	83 ✗	100	100	100	83 ✗
summarize-extract-019	medium	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "name" (str…	4/8	100	100	70 ✗	100	90 ✗	100	80 ✗	60 ✗
summarize-extract-020	hard	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "name" (str…	5/8	100	100	100	0 ✗	100	0 ✗	100	0 ✗
summarize-extract-021	hard	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "name" (str…	6/8	100	100	100	100	100	100	82 ✗	55 ✗
summarize-extract-022	easy	Extract information from the appointment reminder below. Reply with only a JSON object with exactly these keys…	6/8	100	100	100	100	100	100	89 ✗	89 ✗
summarize-extract-023	hard	Extract information from the travel itinerary below. Reply with only a JSON object with exactly these keys: - …	6/8	100	100	100	100	100	80 ✗	100	80 ✗
summarize-extract-024	hard	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "unit" (str…	1/8	89 ✗	89 ✗	89 ✗	89 ✗	89 ✗	100	89 ✗	78 ✗
summarize-extract-025	easy	Extract information from the email below. Reply with only a JSON object with exactly these keys: - "service" (…	5/8	100	100	100	100	100	86 ✗	86 ✗	86 ✗
summarize-extract-026	hard	Extract information from the meeting notes below. Reply with only a JSON object with exactly these keys: - "me…	3/8	100	75 ✗	88 ✗	100	100	88 ✗	63 ✗	88 ✗
summarize-extract-027	easy	Summarize the email below in at most 40 words. Reply with only the summary. """ From: Dana Whitfield To: Front…	1/8	83 ✗	83 ✗	85 ✗	83 ✗	71 ✗	83 ✗	95	80 ✗
summarize-extract-028	easy	Summarize the school notice below in at most 50 words. Reply with only the summary. """ Dear Grade 4 Families,…	1/8	100	70 ✗	83 ✗	85 ✗	91 ✗	77 ✗	85 ✗	0 ✗
summarize-extract-029	easy	Summarize the neighborhood association notice below in at most 50 words. Reply with only the summary. """ Subj…	5/8	100	100	94 ✗	98	98	83 ✗	85 ✗	93
summarize-extract-030	medium	Summarize the meeting transcript below in at most 80 words. Focus on decisions, owners and deadlines. Reply wi…	2/8	100	100	83 ✗	82 ✗	91 ✗	82 ✗	64 ✗	53 ✗
summarize-extract-031	medium	Summarize the news article below in at most 80 words. Reply with only the summary. """ Eastgate Library to clo…	2/8	82 ✗	95	83 ✗	80 ✗	87 ✗	87 ✗	86 ✗	93
summarize-extract-032	medium	Summarize the family email thread (newest message first) below in at most 70 words. Say what was finally decid…	4/8	98	100	98	90	78 ✗	65 ✗	80 ✗	57 ✗
summarize-extract-033	hard	Summarize the meeting transcript below in at most 120 words. Cover what was decided (including anything that c…	1/8	83 ✗	83 ✗	70 ✗	88	74 ✗	73 ✗	80 ✗	76 ✗
summarize-extract-034	hard	Summarize the news article below in at most 100 words. Reply with only the summary. """ Council passes $412 mi…	0/8	85 ✗	73 ✗	83 ✗	78 ✗	74 ✗	71 ✗	77 ✗	54 ✗
summarize-extract-035	easy	Summarize the customer email below in at most 60 words. Reply with only the summary. """ Subject: Damaged sofa…	4/8	100	86 ✗	83 ✗	98	80 ✗	95	93	75 ✗
summarize-extract-036	hard	Summarize the email thread (newest message first) below in at most 70 words. Give the final plan only: the mov…	1/8	100	73 ✗	73 ✗	78 ✗	85 ✗	73 ✗	73 ✗	68 ✗
summarize-extract-037	easy	Summarize the news brief below in at most 50 words. Reply with only the summary. """ Kettlebrook recalls 61,00…	5/8	93	100	98	95	95	68 ✗	83 ✗	94 ✗
summarize-extract-038	medium	Summarize the podcast transcript below in at most 90 words. Focus on the guest's advice for beginners. Reply w…	0/8	87 ✗	74 ✗	84 ✗	84 ✗	76 ✗	76 ✗	63 ✗	58 ✗
summarize-extract-039	easy	Summarize the pet discharge instructions below in at most 70 words. Write it as a summary the owner can follow…	3/8	94 ✗	88 ✗	85 ✗	83	98	86 ✗	85	85 ✗
summarize-extract-040	hard	Summarize the building notice below in at most 80 words. Make sure each resident can tell when their own water…	4/8	98	87 ✗	85 ✗	76 ✗	85 ✗	95	98	93
summarize-extract-041	easy	Summarize the event recap email below in at most 50 words. Reply with only the summary. """ Subject: Thank you…	0/8	85 ✗	83 ✗	83 ✗	70 ✗	80 ✗	85 ✗	89 ✗	80 ✗
summarize-extract-042	medium	Summarize the science news article below in at most 90 words. Reply with only the summary. """ Smart thermosta…	3/8	98	98	85 ✗	88	83 ✗	76 ✗	74 ✗	65 ✗
summarize-extract-043	hard	Summarize the voicemail transcript below in at most 50 words. Give the corrected facts only. Reply with only t…	2/8	95	89 ✗	80 ✗	83 ✗	78 ✗	85 ✗	95	62 ✗
summarize-extract-044	medium	Summarize the incident report email below in at most 80 words. Reply with only the summary. """ From: Owen Fai…	0/8	89 ✗	49 ✗	80 ✗	58 ✗	78 ✗	73 ✗	71 ✗	69 ✗

✗ marks a fail (below the job's pass bar). A dash means no valid result.

← All jobs