Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-040

hard answerablemulti_doc

The prompt

Document excerpts provided from: car_service, home_maintenance_log, lease, manual_thermostat

If we stick with the thermostat's default filter reminder, when is our next HVAC filter change due?

What we expected

{
  "answerable": true,
  "answer": "November 1, 2026: the filter was replaced August 3, 2026, and the default reminder is every 90 days.",
  "key_facts": [
    [
      "november 1",
      "nov 1",
      "nov. 1",
      "11/1",
      "11/01",
      "1 november",
      "1 nov",
      "2026-11-01",
      "nov 1st",
      "november 1st"
    ]
  ],
  "citations": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "contra": [],
  "evidence": [
    "default filter reminder is every 90 days",
    "**Aug 3, 2026** — Replaced the HVAC return filter"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.53 s

The default filter reminder is every 90 days [doc:manual_thermostat], and you reset the count on Aug 3, 2026 after replacing the filter [doc:home_maintenance_log]. So the next HVAC filter change is due around **Nov 1, 2026** (90 days later).

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✗ fail · score 0 · 4.02 s

I can't find that in your documents.

The thermostat manual [doc:manual_thermostat] states the default filter reminder is every 90 days, and your home log [doc:home_maintenance_log] shows the last filter replacement was Aug 3, 2026. However, the log doesn't include a current date, so I can't calculate when the next change is due.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 4.82 s

Based on the default filter reminder of every 90 days mentioned in [1] [doc:manual_thermostat], and the filter was replaced on Aug 3, 2026 (as noted in [2] [doc:home_maintenance_log]), the next HVAC filter change would be due approximately 90 days later, around Nov 1, 2026. The default reminder setting was not changed, as stated in [2] [doc:home_maintenance_log].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 2.68 s

The default filter reminder is every 90 days [doc:manual_thermostat]. Based on the maintenance log, the filter was replaced on Aug 3, 2026 [doc:home_maintenance_log], so the next change is due on **Nov 3, 2026**.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 5.70 s

The next HVAC filter change is due 90 days after the last replacement on Aug 3, 2026, around Nov 1, 2026 [doc:manual_thermostat][doc:home_maintenance_log].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.41 s

The default filter reminder is every 90 days [doc:manual_thermostat]. The last filter change was on Aug 3, 2026 [doc:home_maintenance_log]. Adding 90 days from Aug 3, 2026 gives Nov 1, 2026.

Next HVAC filter change due: Nov 1, 2026.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✗ fail · score 30 · 1.65 s

The thermostat's default filter reminder is every 90 days [doc:manual_thermostat]. The home maintenance log shows the last filter change was on Aug 3, 2026 [doc:home_maintenance_log]. Therefore, the next filter change is due in 90 days, on Sep 1, 2026 [doc:manual_thermostat].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "manual_thermostat",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.47 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-039All Ask my documents items