Unified Memory

Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-018

medium answerablesingle_doc

The prompt

Document excerpts provided from: afterschool_program, school_calendar

What day does school start back up after winter break?
What we expected
{
  "answerable": true,
  "answer": "Monday, January 4, 2027.",
  "key_facts": [
    [
      "january 4",
      "jan 4",
      "jan. 4",
      "1/4",
      "01/04",
      "4 january",
      "4 jan",
      "2027-01-04"
    ]
  ],
  "citations": [
    "school_calendar"
  ],
  "contra": [],
  "evidence": [
    "Jan 4 (Mon): School resumes"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.44 s

School resumes on Jan 4 (Mon) after winter break [doc:school_calendar].
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 2.96 s

School resumes on Monday, January 4, 2027, after winter break. [doc:school_calendar]
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 4.10 s

After winter break, school resumes on Jan 4 (Mon) in December 2026/January 2027, as stated in [1] [doc:school_calendar].
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 2.07 s

[doc:school_calendar]
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.42 s

[doc:school_calendar] Jan 4 (Mon): School resumes.
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.92 s

School resumes on January 4, 2027 [doc:school_calendar].
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 1.14 s

According to [doc:school_calendar], school resumes on Jan 4 (Mon) after winter break [doc:school_calendar].
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 30 · 0.55 s

[doc:school_calendar]
Scoring detail
{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

← rag-017All Ask my documents itemsrag-019 →