Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-032

hard answerablesingle_doc

The prompt

Document excerpts provided from: afterschool_program, home_maintenance_log, sitter_notes, soccer_league

If we had pulled Maya out of soccer on September 10, how much of her registration fee would we have gotten back?

What we expected

{
  "answerable": true,
  "answer": "50% of the $135 fee, i.e. $67.50 (withdrawals Sept 1–15 get a 50% refund).",
  "key_facts": [
    [
      "67.50",
      "67.5"
    ]
  ],
  "citations": [
    "soccer_league"
  ],
  "contra": [],
  "evidence": [
    "Withdrawal from September 1 through September 15: 50% refund",
    "Registration fee was $135"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.33 s

Based on [doc:soccer_league], a withdrawal from September 1 through September 15 receives a 50% refund. September 10 falls in that range, so you would have received $67.50 of the $135 registration fee.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 4.57 s

I can't find that in your documents.

The registration fee ($135) and refund policy are in [doc:soccer_league], which states that a withdrawal from September 1 through September 15 yields a 50% refund. Applying that to September 10 gives $67.50 back. However, the soccer league documents don't confirm Maya's actual registration fee or that she was registered, so the exact amount she'd get back can't be determined from the excerpts.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✗ fail · score 30 · 4.43 s

Based on the refund policy in [1] [doc:soccer_league]: withdrawal from September 1 through September 15 receives a 50% refund. September 10 falls within this period, so you would receive 50% of the $135 registration fee back.

[doc:soccer_league]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 1.88 s

[doc:soccer_league]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 3.49 s

A withdrawal on September 10 falls between September 1 and September 15, so the MYSA refund policy gives a 50% refund of the $135 registration fee, i.e., $67.50 [doc:soccer_league].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.67 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

MiniCPM5 2B

✗ fail · score 30 · 1.49 s

The registration fee was $135 [doc:soccer_league]. Withdrawal after September 15 results in no refund [doc:soccer_league]. Since September 10 is after September 15, there would be no refund.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "soccer_league"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.47 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-031All Ask my documents itemsrag-033 →