Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-030
easy unanswerable
The prompt
Document excerpts provided from: internet_contract, manual_thermostat, sitter_notes, travel_itinerary
What is the Wi-Fi password at the Silver Pines Lodge?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"maple-lantern-42",
"maple lantern"
],
"trap": "The sitter notes contain the home Wi-Fi password, not the lodge's.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.32 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Ornith-1.5 9B
✓ pass · score 100 · 2.80 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Granite 4.2 8B
✓ pass · score 100 · 3.23 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Spark-X2.5 4B
✓ pass · score 100 · 2.05 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✗ fail · score 0 · 2.50 s
[doc:travel_itinerary] does not mention Wi‑Fi password. [doc:internet_contract] does not mention Wi‑Fi password. [doc:manual_thermostat] does not mention Wi‑Fi password. [doc:sitter_notes] mentions Wi‑Fi password as "maple-lantern-42" but that is for the sitter's house, not Silver Pines Lodge. No excerpt provides the Wi‑Fi password for Silver Pines Lodge. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"maple-lantern-42"
],
"cited": [
"travel_itinerary",
"internet_contract",
"manual_thermostat",
"sitter_notes"
]
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.81 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.22 s
The Wi-Fi password at the Silver Pines Lodge is not provided in the documents. [doc:sitter_notes]
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": [
"sitter_notes"
]
}K2-Horizon 0.9B
✓ pass · score 100 · 0.54 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}