Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-024

easy answerablesingle_doc

The prompt

Document excerpts provided from: auto_insurance, manual_thermostat, medical_visit_daniel, travel_itinerary

What was Daniel's LDL cholesterol at his last physical?

What we expected

{
  "answerable": true,
  "answer": "142 mg/dL (borderline high).",
  "key_facts": [
    [
      "142"
    ]
  ],
  "citations": [
    "medical_visit_daniel"
  ],
  "contra": [],
  "evidence": [
    "LDL cholesterol: 142 mg/dL"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.69 s

Daniel's LDL cholesterol was 142 mg/dL at his last physical [doc:medical_visit_daniel].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.34 s

Daniel's LDL cholesterol at his last physical was 142 mg/dL (borderline high).

[doc:medical_visit_daniel]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 4.52 s

[1] states: "LDL cholesterol: 142 mg/dL (borderline high)" [2] repeats the same lab result including "LDL cholesterol: 142 mg/dL (borderline high)". Thus, Daniel's LDL cholesterol at his last physical was 142 mg/dL.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 2.12 s

[doc:medical_visit_daniel]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.55 s

[doc:medical_visit_daniel] LDL cholesterol: 142 mg/dL (borderline high).

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.10 s

[doc:medical_visit_daniel] states Daniel's LDL cholesterol was 142 mg/dL (borderline high).

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 1.22 s

The user's documents show Daniel's LDL cholesterol was 142 mg/dL, which is noted as borderline high in the lab results [doc:medical_visit_daniel].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 30 · 0.58 s

[doc:medical_visit_daniel]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "medical_visit_daniel"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

← rag-023All Ask my documents itemsrag-025 →