Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-035
hard unanswerable
The prompt
Document excerpts provided from: medical_visit_daniel, medical_visit_theo, school_calendar, sitter_notes
What dose of ibuprofen did Dr. Feld recommend for Daniel?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"10 ml",
"10ml",
"100 mg"
],
"trap": "Only Theo's pediatric visit gives an ibuprofen dose (10 mL of 100 mg/5 mL); Daniel's visit has none.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.50 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 3.95 s
I can't find that in your documents. The documents about Daniel ([doc:medical_visit_daniel], [doc:medical_visit_daniel]) are his medical visit and don't mention ibuprofen. The ibuprofen recommendation is in [doc:medical_visit_theo], but that's for Theo, not Daniel.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": [
"medical_visit_daniel",
"medical_visit_theo"
]
}Granite 4.2 8B
✓ pass · score 100 · 3.17 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 2.15 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.34 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.79 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.10 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}K2-Horizon 0.9B
✓ pass · score 100 · 0.55 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}