Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-019
medium answerablesingle_doc
The prompt
Document excerpts provided from: afterschool_program, school_calendar, sitter_notes
What number do I call to report Maya absent from school?
What we expected
{
"answerable": true,
"answer": "The attendance line, (541) 555-0251 (call by 9:00 a.m.).",
"key_facts": [
[
"555-0251"
]
],
"citations": [
"school_calendar"
],
"contra": [],
"evidence": [
"Attendance Line at (541) 555-0251"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 3.95 s
You call the Attendance Line at (541) 555-0251, or use the ParentLink app by 9:00 a.m. [doc:school_calendar]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 3.16 s
You call the Attendance Line at (541) 555-0251. [doc:school_calendar]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 5.02 s
[doc:school_calendar] states: "Please call the Attendance Line at (541) 555-0251 or use the ParentLink app by 9:00 a.m. on any day your child will be absent or late." This is the number to call to report Maya's absence.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.41 s
To report Maya absent from school, you should call the Attendance Line at (541) 555-0251 [doc:school_calendar].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.81 s
[doc:school_calendar] says: "Attendance Line: (541) 555-0251". That is the number to report an absence. So answer: You call (541) 555-0251. Cite [doc:school_calendar].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 96 · 1.26 s
The Attendance Line is (541) 555-0251, as stated in [doc:school_calendar]. [doc:sitter_notes] confirms Maya's allergy but does not provide a separate reporting number. I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [
"sitter_notes"
],
"citation_score": 0.85
}MiniCPM5 2B
✓ pass · score 96 · 1.43 s
The Attendance Line number is (541) 555-0251 [doc:school_calendar]. If Maya is absent, you must call this number by 9:00 a.m. [doc:school_calendar]. The after-school program notes also reference the same Attendance Line [doc:afterschool_program].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"school_calendar",
"afterschool_program"
],
"citation_recall": 1,
"extra_citations": [
"afterschool_program"
],
"citation_score": 0.85
}K2-Horizon 0.9B
✗ fail · score 0 · 0.55 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}