Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-008
hard answerablemulti_doc
The prompt
Document excerpts provided from: credit_card_benefits, warranty_laptop
Counting the Northfield Visa extended warranty benefit, until what date is the Novabook laptop covered for hardware defects?
What we expected
{
"answerable": true,
"answer": "Until March 3, 2028: Lumera's 1-year warranty from the March 3, 2026 purchase, plus 1 extra year because it was paid in full with the Northfield Visa Signature.",
"key_facts": [
[
"march 3, 2028",
"mar 3, 2028",
"mar. 3, 2028",
"3/3/2028",
"03/03/2028",
"2028-03-03",
"3 march 2028",
"march 3rd, 2028",
"3 mar 2028",
"mar 3 2028",
"3/3/28",
"03/03/28",
"march 2028"
]
],
"citations": [
"warranty_laptop",
"credit_card_benefits"
],
"contra": [],
"evidence": [
"Date: Tue, Mar 3, 2026",
"ONE (1) YEAR from the date of original retail purchase",
"by **one (1) additional year**"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 0 · 5.69 s
The Novabook has a 1-year Lumera hardware warranty [doc:warranty_laptop], and the Northfield Visa extended warranty adds one additional year for eligible 3-year-or-less warranties [doc:credit_card_benefits]. The purchase was paid entirely with the Visa card [doc:warranty_laptop], so the extended warranty applies. However, the excerpts do not state the original purchase date, so the exact coverage end date cannot be determined. The warranty period runs one year from the date of original retail purchase [doc:warranty_laptop].
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✗ fail · score 15 · 4.02 s
The Lumera Novabook 14 Air has a 1-year manufacturer warranty (from March 3, 2026), which the Northfield Visa extended warranty benefit extends by one additional year to 3 years or less [doc:credit_card_benefits]. Therefore, the laptop is covered for hardware defects until **March 3, 2027**.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"credit_card_benefits"
],
"citation_recall": 0.5,
"extra_citations": [],
"citation_score": 0.5
}Granite 4.2 8B
✗ fail · score 30 · 7.52 s
The Novabook laptop has a one-year manufacturer warranty from the date of purchase. The Northfield Bank extended warranty benefit adds one additional year to the manufacturer’s warranty for eligible items purchased in full with the Visa card. Therefore, with the extended warranty applied, the laptop would be covered for hardware defects for one year after the manufacturer’s warranty period ends, i.e., for a total of two years from the purchase date. [doc:warranty_laptop] shows the manufacturer warranty lasts ONE (1) YEAR from the date of original retail purchase. [doc:credit_card_benefits] shows the extended warranty protection extends the original manufacturer's written U.S. repair warranty by one (1) additional year on eligible warranties of three (3) years or less. Since the manufacturer warranty is one year, the extended benefit adds one year, making total coverage two years from the purchase date. Thus, the laptop is covered for hardware defects for two years from the purchase date.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✗ fail · score 0 · 1.80 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}Mellum2.1 12B-A2.5B
✗ fail · score 30 · 4.14 s
The Novabook laptop's original warranty runs one year from purchase [doc:warranty_laptop], and the Northfield Visa extended warranty adds an additional year [doc:credit_card_benefits], covering hardware defects until March 3, 2027.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.52 s
The Novabook laptop is covered for hardware defects for 3 years from the date of original retail purchase. [doc:warranty_laptop] states the manufacturer's warranty is 1 year, and [doc:credit_card_benefits] extends it by 1 additional year for items with a 3-year or less manufacturer warranty. The purchase was made on March 3, 2026, so the coverage ends on March 3, 2028.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✗ fail · score 30 · 1.54 s
The Lumera Limited Hardware Warranty covers the Novabook hardware for ONE YEAR from original purchase [doc:warranty_laptop]. The Northfield Visa extended warranty adds one additional year to eligible warranties of three years or less [doc:credit_card_benefits]. Therefore, the laptop is covered for hardware defects until the original warranty year plus one additional year from the extended benefit.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 30 · 0.66 s
The warranty period for the Novabook laptop is one (1) YEAR from the date of original retail purchase, as stated in [doc:warranty_laptop] and [doc:credit_card_benefits].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"warranty_laptop",
"credit_card_benefits"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}