Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-027
hard answerablesingle_doc
The prompt
Document excerpts provided from: car_service, home_maintenance_log, warranty_dishwasher, warranty_laptop
How long is the warranty on the brake work Brookside did on the Tessera?
What we expected
{
"answerable": true,
"answer": "12 months or 12,000 miles, whichever comes first (parts and labor).",
"key_facts": [
[
"12 months",
"12-month",
"twelve months",
"one year",
"1 year",
"1-year",
"one-year"
],
[
"12,000",
"12000",
"12k"
]
],
"citations": [
"car_service"
],
"contra": [
"brake work is covered under the two-year",
"covered under the two-year limited warranty",
"brake work has a two-year",
"brake work is covered by the dishwasher",
"warranty for the kestrel dishwasher",
"duration for the brake work isn't",
"duration for the brake work is not",
"duration of the brake warranty isn't",
"duration of the brake warranty is not"
],
"evidence": [
"12 months or 12,000 miles"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.31 s
The Brookside warranty on the Tessera brake work is 12 months or 12,000 miles, whichever comes first [doc:car_service].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✗ fail · score 0 · 2.97 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}Granite 4.2 8B
✓ pass · score 100 · 5.61 s
The brake work on the Tessera was performed by Brookside Auto Care on 09/29/2026, and the warranty on the repair is 12 months or 12,000 miles, whichever comes first, as stated in the warranty section of the invoice [doc:car_service]. The brake pads and rotor resurfacing were covered under this warranty [doc:car_service]. The warranty period applies to the brake repairs performed by Brookside Auto Care.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.76 s
The warranty on the brake work performed by Brookside Auto Care on the Tessera is **12 months or 12,000 miles, whichever comes first** [doc:car_service].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 96 · 2.63 s
[doc:car_service] shows the brake work was done on 9/29/2026 and [doc:warranty_laptop] is unrelated, but [doc:car_service] also states: "Brookside parts & labor warranty on today's repairs: 12 months or 12,000 miles, whichever comes first." That is the warranty on the brake work. So answer: 12 months or 12,000 miles. Cite [doc:car_service].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"warranty_laptop"
],
"citation_recall": 1,
"extra_citations": [
"warranty_laptop"
],
"citation_score": 0.85
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.57 s
The brake work was covered by the Brookside parts and labor warranty, which is 12 months or 12,000 miles, whichever comes first [doc:car_service]. The Tessera's brake repair was part of the $438.40 service on Sep 29, 2026 [doc:car_service].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✗ fail · score 0 · 1.73 s
The warranty on the brake work at Brookside is covered under the two-year limited warranty for the Kestrel dishwasher, but the Tessera service is covered under a separate 12-month or 12,000-mile warranty from the shop. The specific duration for the brake work isn't stated in the documents. [doc:warranty_dishwasher] [doc:car_service]
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true,
true
],
"contra_hit": [
"covered under the two-year limited warranty",
"warranty for the kestrel dishwasher",
"duration for the brake work isn't"
],
"coverage": 0,
"cited": [
"warranty_dishwasher",
"car_service"
],
"citation_recall": 1,
"extra_citations": [
"warranty_dishwasher"
],
"citation_score": 0.85
}K2-Horizon 0.9B
✗ fail · score 30 · 0.57 s
[doc:car_service]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"car_service"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}