Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-037
medium unanswerable
The prompt
Document excerpts provided from: afterschool_program, hoa_rules, medical_visit_daniel
What are the pool hours at our gym?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"9:00 p.m.",
"9 p.m.",
"9 pm",
"9pm",
"9:00 pm",
"9:00pm"
],
"trap": "FitHaven lists no pool; the HOA pool (8:00 a.m.–9:00 p.m.) and the lodge pool are different facilities.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 0 · 4.75 s
The pool hours are 8:00 a.m. to 9:00 p.m. daily from Memorial Day weekend through September 30, and the spa is open all year from 8:00 a.m. to 10:00 p.m. [doc:hoa_rules]
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}Ornith-1.5 9B
✗ fail · score 0 · 3.69 s
I can't find that in your documents. The pool hours are listed in [doc:hoa_rules] (Memorial Day weekend through September 30, 8:00 a.m. to 9:00 p.m.), but no document mentions pool hours at a gym.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}Granite 4.2 8B
✗ fail · score 0 · 4.17 s
The pool hours are from 8:00 a.m. to 9:00 p.m. daily from Memorial Day weekend through September 30. This information is found in [1] [doc:hoa_rules].
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}Spark-X2.5 4B
✓ pass · score 100 · 1.81 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✗ fail · score 0 · 1.50 s
[doc:hoa_rules] The pool is open from Memorial Day weekend through September 30, 8:00 a.m. to 9:00 p.m. daily.
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 0 · 1.01 s
The pool is open from Memorial Day weekend through September 30, 8:00 a.m. to 9:00 p.m. daily [doc:hoa_rules].
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}MiniCPM5 2B
✗ fail · score 0 · 1.78 s
The pool is open from Memorial Day weekend through September 30, 8:00 a.m. to 9:00 p.m. daily [doc:hoa_rules]. The spa is open all year, 8:00 a.m. to 10:00 p.m. [doc:hoa_rules]. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"9:00 p.m."
],
"cited": [
"hoa_rules"
]
}K2-Horizon 0.9B
✗ fail · score 0 · 0.98 s
The pool is open from Memorial Day weekend through September 30, 8:00 a.m. to 9:00 p.m. daily.
Scoring detail
{
"answerable": false,
"refusal": false,
"forbidden_hit": [
"9:00 p.m."
],
"cited": []
}