Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-025

medium answerablesingle_doc

The prompt

Document excerpts provided from: medical_visit_theo, pet_vaccination, travel_itinerary

When does Biscuit's rabies vaccine expire?

What we expected

{
  "answerable": true,
  "answer": "June 3, 2028 (3-year rabies given June 3, 2025).",
  "key_facts": [
    [
      "june 3, 2028",
      "jun 3, 2028",
      "jun. 3, 2028",
      "06/03/2028",
      "6/3/2028",
      "june 3 2028",
      "3 june 2028",
      "2028-06-03",
      "june 3rd, 2028",
      "3 jun 2028",
      "jun 3 2028",
      "6/3/28",
      "06/03/28"
    ]
  ],
  "citations": [
    "pet_vaccination"
  ],
  "contra": [],
  "evidence": [
    "06/03/2028"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.65 s

Biscuit's rabies vaccine expires 06/03/2028 [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.86 s

Biscuit's rabies vaccine expires on 06/03/2028 [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 96 · 5.10 s

[doc:pet_vaccination] shows Juniper's records but not Biscuit's rabies vaccine. [doc:travel_itinerary] mentions proof requirements but not expiration. [doc:pet_vaccination] (second instance) shows Biscuit's rabies vaccine given 06/03/2025 with expiration 06/03/2028. Thus, Biscuit's rabies vaccine expires 06/03/2028.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination",
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [
    "travel_itinerary"
  ],
  "citation_score": 0.85
}

Spark-X2.5 4B

✗ fail · score 0 · 2.54 s

[doc:travel_itinerary]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 0,
  "extra_citations": [
    "travel_itinerary"
  ],
  "citation_score": 0
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 3.13 s

Biscuit's rabies vaccine expires 06/03/2028 [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.31 s

[doc:pet_vaccination]
Biscuit's rabies vaccine expires on 06/03/2028.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 1.05 s

According to [doc:pet_vaccination], Biscuit's rabies vaccine expires 06/03/2028 [doc:pet_vaccination].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✓ pass · score 100 · 2.26 s

[doc:pet_vaccination] (tag?) Actually the excerpt: [doc:pet_vaccination] appears twice. The first excerpt is pet_vaccination. The second is travel_itinerary. The third is pet_vaccination. The fourth is pet_vaccination. Need to answer: When does Biscuit's rabies vaccine expire? Use the tag for the relevant doc. The relevant doc is [doc:pet_vaccination]? Actually the excerpt for Biscuit's vaccination record: [doc:pet_vaccination]? Wait, there is [doc:pet_vaccination] and [doc:pet_vaccination] but both same. The first excerpt includes Biscuit's rabies (3-year) given 06/03/2025 expires 06/03/2028. So answer: Rabies vaccine expires 06/03/2028. Provide short answer.
</ifm|think_faster>
[doc:pet_vaccination] (first excerpt) shows Rabies (3-year) given 06/03/2025, expires 06/03/2028. So answer: Rabies vaccine expires 06/03/2028.
</ifm|think_faster>
[doc:pet_vaccination]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "pet_vaccination"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

← rag-024All Ask my documents itemsrag-026 →