Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-033
easy answerablesingle_doc
The prompt
Document excerpts provided from: sitter_notes, travel_itinerary, warranty_dishwasher
Where in the house is Maya's EpiPen kept?
What we expected
{
"answerable": true,
"answer": "In the red pouch in the kitchen drawer to the left of the stove (a second one is in the front pocket of her purple backpack).",
"key_facts": [
[
"red pouch"
],
[
"left of the stove",
"kitchen drawer",
"drawer"
]
],
"citations": [
"sitter_notes"
],
"contra": [],
"evidence": [
"RED POUCH in the kitchen drawer to the left of the stove"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.01 s
Maya's EpiPen is kept in two places: one in the RED POUCH in the kitchen drawer to the left of the stove, and one in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✗ fail · score 0 · 2.58 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}Granite 4.2 8B
✓ pass · score 100 · 3.75 s
Maya's EpiPen is in the RED POUCH in the kitchen drawer to the left of the stove, and a second one is in the front pocket of her purple backpack. [doc:sitter_notes]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✗ fail · score 30 · 1.75 s
[doc:sitter_notes]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 2.84 s
Maya's EpiPen is kept in the RED POUCH in the kitchen drawer to the left of the stove [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 70 · 1.18 s
[doc:1] states Maya's EpiPen is in the RED POUCH in the kitchen drawer to the left of the stove, with a second one in the front pocket of her purple backpack. [doc:3] also lists Maya's EpiPen x2 as something to pack.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"1",
"3"
],
"citation_recall": 0,
"extra_citations": [
"1",
"3"
],
"citation_score": 0
}MiniCPM5 2B
✓ pass · score 100 · 1.53 s
According to [doc:sitter_notes], Maya's EpiPen is kept in the RED POUCH in the kitchen drawer to the left of the stove, and a second one is in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 70 · 1.10 s
The EpiPen is stored in the RED POUCH in the kitchen drawer, to the left of the stove.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}