Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-022

easy answerablesingle_doc

The prompt

Document excerpts provided from: manual_washer, medical_visit_theo, sitter_notes, warranty_dishwasher

How much amoxicillin is Theo supposed to take, and how often?

What we expected

{
  "answerable": true,
  "answer": "7.5 mL (400 mg/5 mL suspension) by mouth twice a day for 10 days.",
  "key_facts": [
    [
      "7.5 ml",
      "7.5ml",
      "7.5 milliliters",
      "7.5 millilitres"
    ],
    [
      "twice",
      "two times",
      "2 times",
      "every 12 hours",
      "morning and evening",
      "2x",
      "bid",
      "b.i.d",
      "2 times a day"
    ]
  ],
  "citations": [
    "medical_visit_theo"
  ],
  "contra": [],
  "evidence": [
    "give 7.5 mL by mouth twice a day (morning and evening) for 10 days"
  ],
  "gold_in_context": true,
  "citations_ok": [
    "sitter_notes"
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.99 s

Theo is prescribed Amoxicillin oral suspension 400 mg/5 mL: 7.5 mL by mouth twice a day (morning and evening) for 10 days [doc:medical_visit_theo]. The sitter notes confirm the evening dose is 7.5 mL with dinner, using the syringe in the cup [doc:sitter_notes].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo",
    "sitter_notes"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.48 s

Theo is supposed to take 7.5 mL of amoxicillin oral suspension by mouth twice a day (morning and evening) for 10 days. [doc:medical_visit_theo]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 4.90 s

Amoxicillin is supposed to be given as 7.5 mL by mouth twice a day (morning and evening) for 10 days [doc:medical_visit_theo]. The sitter notes confirm the evening dose is 7.5 mL with dinner [doc:sitter_notes].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo",
    "sitter_notes"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 65 · 2.56 s

Theo is supposed to take **Amoxicillin oral suspension 400 mg/5 mL**, and it should be given **twice a day** (morning and evening) [doc:medical_visit_theo].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false,
    true
  ],
  "contra_hit": [],
  "coverage": 0.5,
  "cited": [
    "medical_visit_theo"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 2.78 s

[doc:medical_visit_theo] says: "Amoxicillin oral suspension 400 mg/5 mL — give 7.5 mL by mouth twice a day (morning and evening) for 10 days."

[doc:sitter_notes] says: "He's on an antibiotic for his ear (amoxicillin, the pink stuff in the fridge) through Oct 15. Evening dose is 7.5 mL with dinner — use the syringe in the cup, not a kitchen spoon."

Both confirm 7.5 mL twice a day. So answer: Theo is supposed to take 7.5 mL of amoxicillin twice a day.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo",
    "sitter_notes"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 1.44 s

Theo is supposed to take 7.5 mL of amoxicillin oral suspension by mouth twice a day (morning and evening) for 10 days [doc:medical_visit_theo]. The sitter notes confirm the evening dose is 7.5 mL with dinner and to use the syringe in the cup, not a kitchen spoon [doc:sitter_notes].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo",
    "sitter_notes"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 1.51 s

Theo is supposed to take Amoxicillin oral suspension 400 mg/5 mL, giving 7.5 mL by mouth twice a day for 10 days [doc:medical_visit_theo]. The instructions also specify that the evening dose should be given with dinner using the syringe in the cup, not a kitchen spoon [doc:sitter_notes].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo",
    "sitter_notes"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✓ pass · score 100 · 1.18 s

Theo should take 7.5 mL of amoxicillin twice each day (morning and evening) for a total of 10 days. [doc:medical_visit_theo]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true,
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "medical_visit_theo"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

← rag-021All Ask my documents itemsrag-023 →