Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-016

easy answerablesingle_doc

The prompt

Document excerpts provided from: afterschool_program, school_calendar, sitter_notes

What time are the kids let out of school on Wednesdays?

What we expected

{
  "answerable": true,
  "answer": "1:15 p.m. (early release every Wednesday).",
  "key_facts": [
    [
      "1:15"
    ]
  ],
  "citations": [
    "school_calendar"
  ],
  "contra": [],
  "evidence": [
    "Early release every Wednesday: 1:15 p.m."
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.57 s

The kids are let out at 1:15 p.m. on Wednesdays due to early release [doc:school_calendar].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.11 s

On Wednesdays, the kids are let out of school at **1:15 p.m.** due to early release.

[doc:school_calendar]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 96 · 6.68 s

Based on the excerpts provided, the answer to the question "What time are the kids let out of school on Wednesdays?" is: Wednesday early release dismissal is at 1:15 p.m. (see [4] [doc:school_calendar]).

The after-school care hours for Wednesdays are from 1:15 p.m. until 6:00 p.m. (see [1] [doc:afterschool_program]).

Thus, kids are let out of school on Wednesdays at 1:15 p.m. (early release time).

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar",
    "afterschool_program"
  ],
  "citation_recall": 1,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.85
}

Spark-X2.5 4B

✓ pass · score 100 · 2.26 s

The kids are let out of school on Wednesdays at **1:15 p.m.** [doc:school_calendar]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✗ fail · score 70 · 2.54 s

[doc:afterschool_program] says: "After-school care: dismissal until **6:00 p.m.** - Wednesdays (early release): from 1:15 p.m. until 6:00 p.m." So the let-out time on Wednesdays is 6:00 p.m. (early release starts at 1:15 p.m. but dismissal is still until 6:00 p.m.). The question: "What time are the kids let out of school on Wednesdays?" Answer: 6:00 p.m. Cite [doc:afterschool_program].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "afterschool_program"
  ],
  "citation_recall": 0,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 70 · 1.58 s

On Wednesdays, the after-school care dismissal is at 1:15 p.m. [doc:1], and the early release day schedule is from 1:15 p.m. until 6:00 p.m. [doc:1]. The dismissal time for the regular school day is 2:45 p.m., but on Wednesdays it is 1:15 p.m. [doc:4].

Answer: 1:15 p.m.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "1",
    "4"
  ],
  "citation_recall": 0,
  "extra_citations": [
    "1",
    "4"
  ],
  "citation_score": 0
}

MiniCPM5 2B

✓ pass · score 96 · 1.34 s

According to [doc:afterschool_program], the after-school care dismissal time on Wednesdays is until 6:00 p.m. The school calendar also mentions early release on Wednesdays at 1:15 p.m. [doc:school_calendar].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "afterschool_program",
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.85
}

K2-Horizon 0.9B

✗ fail · score 0 · 1.01 s

The kids are let out of school on Wednesdays at **6:00 p.m.**

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-015All Ask my documents itemsrag-017 →