Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-015
medium answerablesingle_doc
The prompt
Document excerpts provided from: afterschool_program, manual_washer, travel_itinerary
How do I turn on the child lock on our washing machine?
What we expected
{
"answerable": true,
"answer": "Press and hold the TEMP and SPIN buttons together for 3 seconds (the lock icon lights).",
"key_facts": [
[
"temp"
],
[
"spin"
],
[
"3 seconds",
"three seconds",
"3 sec",
"3-second",
"three-second"
]
],
"citations": [
"manual_washer"
],
"contra": [],
"evidence": [
"press and hold the TEMP and SPIN buttons together for 3 seconds"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.57 s
Press and hold the TEMP and SPIN buttons together for 3 seconds; the lock icon will light [doc:manual_washer].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 2.92 s
To turn on the child lock, press and hold the TEMP and SPIN buttons together for 3 seconds. The lock icon will light. [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 4.51 s
To turn on the child lock on your washing machine, press and hold the TEMP and SPIN buttons together for 3 seconds. The lock icon will light, indicating the child lock is on. [doc:manual_washer] and [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.32 s
To turn the child lock on, press and hold the TEMP and SPIN buttons together for 3 seconds. The lock icon will light [doc:manual_washer].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.42 s
[doc:manual_washer] To turn Child Lock on or off, press and hold the TEMP and SPIN buttons together for 3 seconds. The lock icon will light.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.06 s
To turn the child lock on, press and hold the TEMP and SPIN buttons together for 3 seconds. The lock icon will light. The Power button still works while Child Lock is on. [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.37 s
To turn on the child lock, press and hold the TEMP and SPIN buttons together for 3 seconds; the lock icon will light [doc:manual_washer]. The Power button still works while Child Lock is on [doc:manual_washer]. I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true,
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 30 · 0.49 s
[doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}