Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-038
medium answerablemulti_doc
The prompt
Document excerpts provided from: afterschool_program, school_calendar, sitter_notes, travel_itinerary
What does the after-school program require us to have on file for Maya's peanut allergy, and where do we keep her EpiPen at home?
What we expected
{
"answerable": true,
"answer": "Kids Club (nut-aware, not nut-free) needs an Allergy Action Plan signed by her doctor and two epinephrine auto-injectors on file; at home her EpiPen is in the red pouch in the kitchen drawer left of the stove (a second is in her backpack).",
"key_facts": [
[
"allergy action plan",
"action plan"
],
[
"two epinephrine",
"two auto-injectors",
"2 epinephrine",
"2 auto-injectors",
"two epipens",
"2 epipens",
"two epi",
"two auto injectors",
"2 auto injectors",
"two injectors"
],
[
"red pouch"
]
],
"citations": [
"afterschool_program",
"sitter_notes"
],
"contra": [],
"evidence": [
"**Allergy Action Plan signed by their doctor**",
"**two epinephrine auto-injectors**",
"RED POUCH"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 5.22 s
The after-school program requires an Allergy Action Plan signed by her doctor and two epinephrine auto-injectors on file before her first day [doc:afterschool_program]. At home, one EpiPen is in the red pouch in the kitchen drawer to the left of the stove, and a second is in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 4.27 s
The after-school program requires an **Allergy Action Plan signed by a doctor** and **two epinephrine auto-injectors** on file before Maya's first day [doc:afterschool_program]. At home, her EpiPen is kept in the **RED POUCH in the kitchen drawer to the left of the stove** [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✗ fail · score 77 · 4.40 s
The after-school program requires two epinephrine auto-injectors on file for Maya’s peanut allergy [doc:afterschool_program]. At home, one EpiPen is in the RED POUCH in the kitchen drawer to the left of the stove, and a second is in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
true,
true
],
"contra_hit": [],
"coverage": 0.6667,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.93 s
For Maya's peanut allergy, the after-school program requires an **Allergy Action Plan signed by her doctor** and **two epinephrine auto-injectors** to be on file [doc:afterschool_program]. At home, her EpiPen is kept in the **RED POUCH in the kitchen drawer to the left of the stove** [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 5.12 s
The afterschool program requires an Allergy Action Plan signed by the doctor and two epinephrine auto-injectors on file [doc:afterschool_program]; at home the EpiPen is kept in the RED POUCH in the kitchen drawer to the left of the stove and in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.29 s
[doc:afterschool_program] requires an Allergy Action Plan signed by Maya's doctor and two epinephrine auto-injectors on file. At home, the EpiPen is kept in the red pouch in the kitchen drawer to the left of the stove and in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.65 s
The after-school program requires an Allergy Action Plan signed by their doctor and two epinephrine auto-injectors on file for children with a life-threatening allergy [doc:afterschool_program]. Maya's EpiPens are kept in her RED POUCH in the kitchen drawer to the left of the stove and in the front pocket of her purple backpack [doc:sitter_notes].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"afterschool_program",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 15 · 0.47 s
[doc:afterschool_program]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"afterschool_program"
],
"citation_recall": 0.5,
"extra_citations": [],
"citation_score": 0.5
}