Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-021

hard answerablemulti_doc

The prompt

Document excerpts provided from: medical_visit_theo, pet_vaccination, travel_itinerary

Biscuit is boarding at Happy Tails during the Tahoe trip. Which of his vaccines will need a booster before he goes, and when is it due?

What we expected

{
  "answerable": true,
  "answer": "Bordetella: it is due December 10, 2026, before the December 21 drop-off, and Happy Tails requires it to be current through the stay.",
  "key_facts": [
    [
      "bordetella",
      "kennel cough"
    ],
    [
      "december 10",
      "dec 10",
      "dec. 10",
      "12/10",
      "12/10",
      "10 december",
      "10 dec",
      "2026-12-10"
    ]
  ],
  "citations": [
    "pet_vaccination",
    "travel_itinerary"
  ],
  "contra": [],
  "evidence": [
    "Bordetella (intranasal)     12/10/2025   12/10/2026",
    "Drop-off **Mon Dec 21 by 6:00 p.m.**",
    "rabies, DHPP, and Bordetella vaccines are current through the last night of the stay"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 5.41 s

Biscuit's Bordetella booster is due 12/10/2026, and it must be current through the last night of his boarding stay at Happy Tails [doc:pet_vaccination], [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination",
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 5.28 s

Biscuit's **Bordetella** vaccine will need a booster before boarding. It's due **12/10/2026** (its current Bordetella expires 12/10/2026), and Happy Tails requires proof that the Bordetella vaccine is current through the last night of the stay.

[doc:pet_vaccination][doc:travel_itinerary]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination",
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 85 · 7.59 s

Biscuit's Bordetella vaccine will need a booster before boarding, and it is due by December 10, 2026. This is stated in [2] [doc:pet_vaccination] which notes "Biscuit: Bordetella booster due 12/10/2026 — boarding facilities will not accept an expired Bordetella." Additionally, [3] [doc:pet_vaccination] confirms Bordetella (intranasal) was given on 12/10/2025 with expiration 12/10/2026, supporting the need for a booster before boarding.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 0.5,
  "extra_citations": [],
  "citation_score": 0.5
}

Spark-X2.5 4B

✓ pass · score 100 · 3.36 s

[doc:travel_itinerary] [doc:pet_vaccination]

Biscuit's **Bordetella** vaccine needs a booster. It is due **12/10/2026** [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary",
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 3.77 s

[doc:travel_itinerary] shows Biscuit boarding at Happy Tails Pet Resort from Mon Dec 21 by 6:00 p.m. to Mon Jan 4 at 10:00 a.m. [doc:pet_vaccination] (the first one) lists REMINDERS ON FILE: "Biscuit: Bordetella booster due 12/10/2026 — boarding facilities will not accept an expired Bordetella." Also [doc:pet_vaccination] (the third one) for Biscuit shows "Bordetella (intranasal) — 12/10/2025 — Required for boarding & daycare". So the booster is due 12/10/2026. The answer: Biscuit needs a Bordetella booster, due 12/10/2026. Use citations.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary",
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 85 · 1.62 s

Biscuit's Bordetella vaccine is due on 12/10/2026 [doc:pet_vaccination], and it is required for boarding [doc:pet_vaccination]. The boarding facility will not accept an expired Bordetella [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 0.5,
  "extra_citations": [],
  "citation_score": 0.5
}

MiniCPM5 2B

✓ pass · score 100 · 1.77 s

Biscuit's Bordetella booster is due 12/10/2026 [doc:travel_itinerary][doc:pet_vaccination]. The travel itinerary confirms boarding at Happy Tails during the trip, and the pet vaccination record notes the expired Bordetella vaccine would be rejected by boarding facilities [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary",
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 15 · 0.70 s

[doc:travel_itinerary]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false,
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 0.5,
  "extra_citations": [],
  "citation_score": 0.5
}

← rag-020All Ask my documents itemsrag-022 →