Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-004
medium unanswerable
The prompt
Document excerpts provided from: lease, medical_visit_theo, warranty_dishwasher, warranty_laptop
How long is the warranty on our refrigerator?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"2 years",
"two years",
"2-year",
"two-year",
"1 year",
"one year",
"1-year",
"one-year",
"10 years",
"ten years",
"10-year",
"ten-year"
],
"trap": "The corpus has the dishwasher (2-year), washer (1-year) and thermostat (2-year) warranties; the lease only lists the refrigerator as an owner appliance.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 2.68 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 2.21 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Granite 4.2 8B
✓ pass · score 100 · 2.66 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 1.69 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.11 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.66 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 0.84 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}K2-Horizon 0.9B
✓ pass · score 100 · 0.64 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}