Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-014
easy answerablesingle_doc
The prompt
Document excerpts provided from: home_maintenance_log, manual_thermostat, manual_washer
What does error code E21 on our washer mean?
What we expected
{
"answerable": true,
"answer": "Drain fault: the water didn't drain within 8 minutes. Check the drain hose for kinks and clean the drain pump filter.",
"key_facts": [
[
"drain"
],
[
"filter",
"hose"
]
],
"citations": [
"manual_washer"
],
"contra": [],
"evidence": [
"E21 — Drain fault"
],
"gold_in_context": true,
"citations_ok": [
"home_maintenance_log"
]
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.97 s
E21 is a drain fault: water did not drain within 8 minutes. Check that the drain hose is not kinked or pushed too far into the standpipe, and clean the drain pump filter [doc:manual_washer].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 3.23 s
Error code E21 is a drain fault: water did not drain within 8 minutes. Check that the drain hose is not kinked or pushed too far into the standpipe, and clean the drain pump filter. [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 5.48 s
[doc:manual_washer] states that E21 is a drain fault: "Water did not drain within 8 minutes. Check that the drain hose is not kinked or pushed too far into the standpipe, and clean the drain pump filter (Section 7)." [doc:manual_washer] also notes in Section 8 that cleaning the drain pump filter every 3 months and when E21 appears is recommended. [doc:home_maintenance_log] confirms that when E21 appeared, cleaning the drain filter fixed the issue.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer",
"home_maintenance_log"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.57 s
Error code E21 on the washer means a **drain fault**, where water did not drain within 8 minutes [doc:manual_washer]. The manual advises checking that the drain hose is not kinked or pushed too far into the standpipe, and cleaning the drain pump filter [doc:manual_washer].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.89 s
[doc:manual_washer] [doc:manual_washer] [doc:home_maintenance_log] [doc:home_maintenance_log] E21 — Drain fault. Water did not drain within 8 minutes. Check that the drain hose is not kinked or pushed too far into the standpipe, and clean the drain pump filter (Section 7).
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer",
"home_maintenance_log"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 65 · 0.80 s
E21 means a drain fault, where water did not drain within 8 minutes. [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
false
],
"contra_hit": [],
"coverage": 0.5,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.44 s
Error code E21 means the drain pump filter needs cleaning because water did not drain within 8 minutes [doc:manual_washer]. The manual says to check that the drain hose is not kinked and to clean the filter [doc:manual_washer]. Your maintenance log shows you fixed this by draining water and finding a hair tie and two quarters [doc:home_maintenance_log].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer",
"home_maintenance_log"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✓ pass · score 100 · 0.57 s
Error code E21 on the washer means the drain pump filter needs cleaning, as stated in the manual: [doc:manual_washer]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"manual_washer"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}