Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-009
hard answerablesingle_doc
The prompt
Document excerpts provided from: auto_insurance, home_maintenance_log, medical_visit_daniel, renters_insurance
What is the comprehensive deductible on the Crestline?
What we expected
{
"answerable": true,
"answer": "$500 (the Tessera's comprehensive deductible is $250, but the Crestline's is $500).",
"key_facts": [
[
"$500",
"500 dollars"
]
],
"citations": [
"auto_insurance"
],
"contra": [],
"evidence": [
"Comprehensive (other than collision): $500 deductible"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.36 s
The comprehensive deductible on the 2022 Halden Crestline is $500 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 2.62 s
The comprehensive deductible on the Crestline is $500. [doc:auto_insurance]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 3.80 s
The comprehensive deductible on the Crestline is $500. This is stated in document [1] where it says "Vehicle 2 — 2022 Halden Crestline" has "Comprehensive (other than collision): $500 deductible". [1]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✗ fail · score 30 · 1.67 s
[doc:auto_insurance]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.98 s
[doc:auto_insurance] Vehicle 2 — 2022 Halden Crestline - Collision: $500 deductible - Comprehensive (other than collision): $500 deductible - Full Glass endorsement: $0 deductible for windshield and window repair or replacement The comprehensive deductible for the Crestline is $500. [doc:auto_insurance]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.87 s
The comprehensive deductible for the Crestline is $500, as stated in [doc:auto_insurance]. I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.17 s
The comprehensive deductible for the Crestline is $500, as stated in [doc:auto_insurance]. The glass coverage for the Tessera has a $250 deductible under [doc:auto_insurance]. I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 30 · 0.70 s
[1]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}