Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-017
hard unanswerable
The prompt
Document excerpts provided from: pet_vaccination, sitter_notes, travel_itinerary, warranty_dishwasher
What is Juniper's microchip number?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"985112004417336"
],
"trap": "Only Biscuit's microchip number is on the vet record.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.93 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 3.28 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Granite 4.2 8B
✓ pass · score 100 · 4.34 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 2.47 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 2.30 s
[doc:pet_vaccination] Juniper's microchip number is not listed in the excerpt. [doc:sitter_notes] No microchip information for Juniper. [doc:travel_itinerary] No microchip information for Juniper. [doc:warranty_dishwasher] No microchip information. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": [
"pet_vaccination",
"sitter_notes",
"travel_itinerary",
"warranty_dishwasher"
]
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.99 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.13 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}K2-Horizon 0.9B
✗ fail · score 0 · 1.01 s
985112004417336
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"985112004417336"
],
"cited": []
}