Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-011
easy answerablesingle_doc
The prompt
Document excerpts provided from: manual_washer, warranty_dishwasher, warranty_laptop
How long does the dishwasher's full parts-and-labor warranty last?
What we expected
{
"answerable": true,
"answer": "Two years from the original purchase date (April 18, 2026), i.e. through April 18, 2028.",
"key_facts": [
[
"2 years",
"two years",
"2-year",
"two-year",
"24 months",
"april 18, 2028",
"apr 18, 2028"
]
],
"citations": [
"warranty_dishwasher"
],
"contra": [],
"evidence": [
"For two (2) years from the original date of purchase"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.96 s
The dishwasher's full parts-and-labor warranty lasts two years from the original date of purchase (April 18, 2026), covering defects in materials or workmanship plus labor and in-home service [doc:warranty_dishwasher].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 2.79 s
The dishwasher's full parts-and-labor warranty lasts two (2) years from the original date of purchase. [doc:warranty_dishwasher]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 4.99 s
The full parts-and-labor warranty for the dishwasher lasts two years from the original date of purchase. This is stated in [1] and confirmed in [2], which describe the "Full Two-Year Warranty" covering repair or replacement of any defective part and paying for all labor during this period. [doc:warranty_dishwasher] and [doc:warranty_dishwasher] contain this information.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.12 s
The dishwasher's full parts-and-labor warranty lasts for two (2) years from the original date of purchase [doc:warranty_dishwasher].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 2.02 s
[doc:warranty_dishwasher] says "Full Two-Year Warranty: For two (2) years from the original date of purchase, Kestrel will repair or replace, free of charge, any part that fails because of a defect in materials or workmanship. During this period Kestrel also pays for all labor and in-home service to repair the defect." So the full parts-and-labor warranty lasts two years.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.89 s
The dishwasher's full parts-and-labor warranty lasts two years from the original date of purchase. [doc:warranty_dishwasher]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.39 s
The dishwasher has a Full Two-Year Warranty from the original date of purchase [doc:warranty_dishwasher]. There is also a Limited Fifth-Year Warranty for specific components and a Limited Ten-Year Tub Warranty [doc:warranty_dishwasher]. The answer is covered under the two-year full warranty.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 70 · 0.67 s
The dishwasher’s full parts‑and‑labor warranty covers two (2) years from the original purchase date.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}