Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-023
hard unanswerable
The prompt
Document excerpts provided from: auto_insurance, home_maintenance_log, renters_insurance
How much did the brake job on the Crestline cost?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"289",
"438.40",
"438.4"
],
"trap": "The Brookside invoice is for the Tessera's front brakes ($289.00 line, $438.40 total); there is no Crestline brake record.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 2.85 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 2.26 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Granite 4.2 8B
✓ pass · score 100 · 2.81 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 1.75 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.14 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.68 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.27 s
The brake job cost is not specified in the provided documents. [doc:home_maintenance_log] mentions a brake job on the Tessera but does not provide a cost. [doc:home_maintenance_log] also mentions the Crestline's oil change but no cost. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": [
"home_maintenance_log"
]
}K2-Horizon 0.9B
✓ pass · score 100 · 0.71 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}