Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-020

easy answerablesingle_doc

The prompt

Document excerpts provided from: internet_contract, manual_washer, travel_itinerary

What is the confirmation code for our Cascadia Air flights to Reno?

What we expected

{
  "answerable": true,
  "answer": "QX7RLM.",
  "key_facts": [
    [
      "qx7rlm"
    ]
  ],
  "citations": [
    "travel_itinerary"
  ],
  "contra": [],
  "evidence": [
    "Confirmation code: **QX7RLM**"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.52 s

The confirmation code for our Cascadia Air flights is **QX7RLM** [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 2.91 s

The confirmation code for the Cascadia Air flights to Reno is **QX7RLM** [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 3.77 s

[1] [doc:travel_itinerary] contains the flight details and confirms the Cascadia Air flights with confirmation code **QX7RLM**. The confirmation code for the Cascadia Air flights to Reno is QX7RLM.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 1.97 s

[doc:travel_itinerary]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 30 · 1.24 s

[doc:travel_itinerary]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.92 s

The confirmation code for the Cascadia Air flights to Reno is **QX7RLM**, found in [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 0.92 s

The confirmation code for the Cascadia Air flights is QX7RLM, as listed in [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✓ pass · score 100 · 0.63 s

The confirmation code for the Cascadia Air flights to Reno is **QX7RLM**, as stated in the travel itinerary excerpt [doc:travel_itinerary].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

← rag-019All Ask my documents itemsrag-021 →