Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-039
easy unanswerable
The prompt
Document excerpts provided from: auto_insurance, renters_insurance
What is the policy number of our umbrella insurance policy?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"hrm-4471-90215",
"sra-2209-3381-07",
"4471-90215",
"2209-3381"
],
"trap": "Only the renters (HRM-4471-90215) and auto (SRA-2209-3381-07) policy numbers exist.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.68 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 2.99 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Granite 4.2 8B
✓ pass · score 100 · 3.37 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 2.18 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.48 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.86 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.10 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}K2-Horizon 0.9B
✓ pass · score 100 · 0.46 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}