Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-012
hard answerablesingle_doc
The prompt
Document excerpts provided from: hoa_rules, manual_washer, warranty_dishwasher
If the dishwasher's electronic control board fails in its fourth year, will Kestrel pay for the labor?
What we expected
{
"answerable": true,
"answer": "No. In years 3–5 Kestrel only supplies the replacement control board (parts only); you pay labor and service calls.",
"key_facts": [
[
"you pay for labor",
"you pay labor",
"you pay for the labor",
"pay for labor yourself",
"labor is not covered",
"labor isn't covered",
"labor not covered",
"not cover labor",
"not cover the labor",
"doesn't cover labor",
"does not cover labor",
"won't pay for labor",
"will not pay for labor",
"won't pay for the labor",
"will not pay for the labor",
"not pay for labor",
"not pay for the labor",
"no labor",
"parts only",
"part only",
"only the part",
"only parts",
"only the replacement",
"excluding labor",
"labor excluded",
"labor is your",
"labor would be",
"labor costs are yours",
"you'd pay for labor",
"you would pay for labor",
"you'll pay for labor",
"you will pay for labor",
"you would have to pay",
"you'd have to pay",
"you will have to pay",
"you'll have to pay",
"you have to pay for labor",
"you must pay for labor",
"responsible for labor",
"responsible for the labor",
"responsible for labor costs",
"labor is on you",
"labor's on you",
"your responsibility",
"labor is not included",
"labor isn't included",
"labor not included",
"not include labor",
"doesn't include labor",
"does not include labor",
"labor would be your",
"labor costs would be",
"labor is extra",
"labor is at your",
"at your own cost",
"at your expense",
"you cover the labor",
"you cover labor",
"only the board",
"only the control board",
"only supplies the",
"only provide the",
"only provides the"
]
],
"citations": [
"warranty_dishwasher"
],
"contra": [
"yes,",
"yes.",
"yes -",
"yes!",
"yes;"
],
"evidence": [
"the electronic control board",
"You pay for labor, service calls, and shipping during this period"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.85 s
No. Under the Limited Fifth-Year Warranty, Kestrel provides replacement parts free of charge for the electronic control board during years 3 through 5, but you pay for labor, service calls, and shipping during that period [doc:warranty_dishwasher].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✗ fail · score 30 · 3.81 s
Yes. According to the Limited Fifth-Year Warranty, Kestrel provides replacement parts free of charge for the electronic control board during years 3 through 5, but you pay for labor, service calls, and shipping during this period [doc:warranty_dishwasher]. Since the failure occurs in the fourth year (within years 3–5), the board would be replaced free of charge, but labor is not covered.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [
"yes."
],
"coverage": 0,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 10.3 s
Based on the excerpts provided: - In [1] (doc:warranty_dishwasher), the Limited Fifth-Year Warranty (years 3–5) states: "You pay for labor, service calls, and shipping during this period" for replacement parts (including the electronic control board). - The dishwasher was purchased on April 18, 2026 (see [2] doc:warranty_dishwasher), so year 1 is 2026–2027, year 2 is 2027–2028, year 3 is 2028–2029, year 4 is 2029–2030. Thus, year 4 falls within the fifth-year warranty period (years 3–5). - Since the electronic control board is covered under the Limited Fifth-Year Warranty (years 3–5), and during years 3–5 "you pay for labor, service calls, and shipping," Kestrel will not pay for labor if the electronic control board fails in year 4. Therefore, Kestrel will not pay for the labor. [doc:warranty_dishwasher] and [doc:warranty_dishwasher] (the relevant warranty details in [1] and the purchase date in [2] confirm the timeframe).
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✗ fail · score 0 · 1.77 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 2.63 s
No, Kestrel will not pay for labor; during the fifth-year warranty (years 3‑5) the warranty covers replacement parts only and you pay for labor, service calls, and shipping [doc:warranty_dishwasher].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.24 s
No, Kestrel will not pay for the labor. The Limited Fifth-Year Warranty [doc:warranty_dishwasher] covers the electronic control board free of charge for parts only in years 3 through 5, but labor is not covered. The Ten-Year Tub Warranty [doc:warranty_dishwasher] does not apply to the electronic control board.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✗ fail · score 30 · 1.28 s
Yes, Kestrel will provide replacement parts for the electronic control board from years 3 through 5, but you will pay for labor, service calls, and shipping during this period [1]. Since the failure occurs in the fourth year, you would pay for the labor [1].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [
"yes,"
],
"coverage": 0,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 30 · 1.61 s
Yes. The limited fifth‑year warranty covers the electronic control board for years 3‑5, so Kestrel will pay for the labor in the fourth year. [doc:warranty_dishwasher]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [
"yes."
],
"coverage": 0,
"cited": [
"warranty_dishwasher"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}