Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-003

medium answerablesingle_doc

The prompt

Document excerpts provided from: hoa_rules, lease, soccer_league

What time do quiet hours start on a Saturday night at Larkspur Commons?

What we expected

{
  "answerable": true,
  "answer": "11:00 p.m. (Friday and Saturday nights, until 8:00 a.m.).",
  "key_facts": [
    [
      "11:00 p.m.",
      "11 p.m.",
      "11 pm",
      "11:00 pm",
      "11pm",
      "11:00pm",
      "23:00",
      "eleven"
    ]
  ],
  "citations": [
    "hoa_rules"
  ],
  "contra": [],
  "evidence": [
    "11:00 p.m. to 8:00 a.m. on Friday and Saturday nights"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.42 s

Quiet hours on Saturday nights at Larkspur Commons start at 11:00 p.m. [doc:hoa_rules]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 2.95 s

Quiet hours at Larkspur Commons start at 11:00 p.m. on Saturday nights. [doc:hoa_rules]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 3.27 s

Quiet hours on Saturday nights start at 11:00 p.m. [doc:hoa_rules]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✓ pass · score 100 · 2.05 s

The quiet hours on a Saturday night start at **11:00 p.m.** [doc:hoa_rules]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 2.03 s

[doc:hoa_rules] Quiet hours are 10:00 p.m. to 7:00 a.m. Sunday night through Thursday night, and 11:00 p.m. to 8:00 a.m. on Friday and Saturday nights. So on Saturday night, quiet hours start at 11:00 p.m.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.67 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

MiniCPM5 2B

✓ pass · score 100 · 1.08 s

Quiet hours on Saturday nights start at 11:00 p.m. [doc:hoa_rules].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 1.17 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-002All Ask my documents itemsrag-004 →