Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-034
medium answerablemulti_doc
The prompt
Document excerpts provided from: gym_membership, hoa_rules, lease
My sister is coming to visit. How long can her car stay in guest parking, and how many consecutive days can a guest stay in our unit without written consent from management?
What we expected
{
"answerable": true,
"answer": "Her car can stay in guest parking for up to 72 consecutive hours (with a guest pass); she can stay up to 14 consecutive days (and no more than 30 days in 12 months) without written consent.",
"key_facts": [
[
"72 hours",
"72 consecutive hours",
"72-hour",
"72 hrs",
"three days",
"3 days",
"72-hours"
],
[
"14 consecutive days",
"14 days",
"two weeks",
"14-day",
"fourteen days",
"2 weeks"
]
],
"citations": [
"hoa_rules",
"lease"
],
"contra": [],
"evidence": [
"no more than 72 consecutive hours",
"more than **14 consecutive days**"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.74 s
Her car can stay in guest parking for no more than 72 consecutive hours, and she must display a temporary guest pass [doc:hoa_rules]. A guest may stay at the Premises for no more than 14 consecutive days, or more than 30 days in any 12-month period, without prior written consent from Management [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 3.26 s
Your sister's car can stay in guest parking for no more than **72 consecutive hours** [doc:hoa_rules]. A guest may stay in the unit for no more than **14 consecutive days** without prior written consent from Management [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 3.75 s
The guest car may remain in guest parking for no more than 72 consecutive hours [doc:hoa_rules]. A guest may stay in the unit for no more than 14 consecutive days without written consent [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 3.39 s
Based on the documents: * **Guest car parking:** According to [doc:hoa_rules], a guest vehicle may remain in guest parking for no more than **72 consecutive hours**. * **Guest stay duration:** According to [doc:lease], no guest may stay at the Premises for more than **14 consecutive days**, or more than **30 days in any 12-month period**, without the prior written consent of Management. [doc:hoa_rules] [doc:lease]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 4.30 s
A guest car may stay in guest parking for up to 72 consecutive hours [doc:hoa_rules]; a guest may stay in the unit for up to 14 consecutive days without written consent [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 1.14 s
The car can stay in guest parking for no more than 72 consecutive hours [doc:hoa_rules]. A guest may stay for no more than 14 consecutive days or 30 days in any 12-month period without prior written consent from Management [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✓ pass · score 100 · 1.60 s
According to [doc:hoa_rules], a guest vehicle may remain in guest parking for no more than 72 consecutive hours. According to [doc:lease], no guest may stay at the Premises for more than 14 consecutive days without prior written consent of Management.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"hoa_rules",
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 70 · 1.45 s
The lease states that a guest may not stay in the unit for more than **14 consecutive days** without Management’s prior written consent. The parking policy limits a guest vehicle in guest parking to **no more than 72 consecutive hours** (and requires a temporary guest pass).
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}