Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-026
medium answerablemulti_doc
The prompt
Document excerpts provided from: auto_insurance, car_service, credit_card_benefits
What's the phone number for Brookside Auto Care, and what is our collision deductible on the Tessera?
What we expected
{
"answerable": true,
"answer": "Brookside Auto Care, (541) 555-0340; the Tessera's collision deductible is $1,000.",
"key_facts": [
[
"555-0340"
],
[
"1,000",
"1000",
"one thousand"
]
],
"citations": [
"car_service",
"auto_insurance"
],
"contra": [],
"evidence": [
"Phone (541) 555-0340",
"Collision: $1,000 deductible"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.67 s
Brookside Auto Care's phone number is (541) 555-0340 [doc:car_service], and the Tessera's collision deductible is $1,000 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 3.75 s
Brookside Auto Care's phone number is (541) 555-0340 [doc:car_service]. The collision deductible on the Tessera is $1,000 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 4.23 s
[doc:car_service] provides the phone number for Brookside Auto Care: (541) 555-0340. [1] shows the collision deductible for the Tessera (Vehicle 1) is $1,000.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.97 s
The phone number for Brookside Auto Care is (541) 555-0340 [doc:car_service], and the collision deductible for the 2019 Arden Tessera is $1,000 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 10.3 s
Brookside Auto Care phone number: [doc:car_service]2275 Mill Race Road, Millbrook, OR 97342 Phone (541) 555-0340[doc:car_service] · Open Mon–Fri 7:30a–6p, Sat 8a–2p[doc:car_service]Brookside Auto Care[doc:car_service]2275 Mill Race Road, Millbrook, OR 97342[doc:car_service]Phone (541) 555-0340[doc:car_service]Collision deductible on the Tessera: [doc:auto_insurance]Vehicle 1 — 2019 Arden Tessera[doc:auto_insurance]- Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible[doc:auto_insurance]Collision: $1,000 deductible
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.44 s
The phone number for Brookside Auto Care is (541) 555-0340 [doc:car_service]. The collision deductible on the 2019 Arden Tessera is $1,000 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.24 s
The phone number for Brookside Auto Care is (541) 555-0340 [doc:car_service]. The collision deductible for the Tessera is $1,000 [doc:auto_insurance].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"car_service",
"auto_insurance"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 15 · 1.05 s
[doc:car_service]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"car_service"
],
"citation_recall": 0.5,
"extra_citations": [],
"citation_score": 0.5
}