Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-028
medium answerablesingle_doc
The prompt
Document excerpts provided from: gym_membership, manual_thermostat
How do I cancel our FitHaven gym membership?
What we expected
{
"answerable": true,
"answer": "Give 30 days' written notice, in person at the front desk (cancellation form) or by certified mail; email and phone requests are not accepted.",
"key_facts": [
[
"30 days",
"30-day",
"thirty days",
"30 day"
],
[
"certified mail",
"in person",
"front desk"
]
],
"citations": [
"gym_membership"
],
"contra": [],
"evidence": [
"you must give 30 days' written notice, delivered in person at the front desk"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✓ pass · score 100 · 4.54 s
To cancel your FitHaven membership, give 30 days' written notice delivered in person at the front desk using a cancellation form, or sent by certified mail to the club address [doc:gym_membership]. Email or phone cancellation requests are not accepted, and dues will be drafted for any billing date within the 30-day notice period [doc:gym_membership].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 3.31 s
To cancel your FitHaven gym membership, you must give 30 days' written notice, delivered in person at the front desk (using a cancellation form) or sent by certified mail to the club address. Cancellation requests by email or phone are not accepted. [doc:gym_membership]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✓ pass · score 100 · 4.79 s
To cancel your FitHaven gym membership, you must give 30 days' written notice delivered in person at the front desk (using the cancellation form) or by certified mail to the club address. Email or phone cancellation requests are not accepted. Dues will be drafted for any billing date that falls within the 30‑day notice period, and the annual enhancement fee is not refundable once billed. [doc:gym_membership]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✓ pass · score 100 · 2.72 s
To cancel your FitHaven gym membership, you must give **30 days' written notice**. This notice must be delivered **in person at the front desk** (using a cancellation form) or **sent by certified mail** to the club address. Cancellation requests by email or phone are not accepted [doc:gym_membership].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 1.99 s
[doc:gym_membership] [doc:gym_membership] [doc:gym_membership] [doc:gym_membership] [doc:gym_membership] To cancel your FitHaven gym membership, you must give 30 days' written notice delivered in person at the front desk (cancellation form) or sent by certified mail to the club address. Email or phone cancellation requests are not accepted. [doc:gym_membership]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 70 · 1.09 s
To cancel, you must give 30 days' written notice, delivered in person at the front desk (cancellation form) or sent by certified mail to the club address. [doc:lease] I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"lease"
],
"citation_recall": 0,
"extra_citations": [
"lease"
],
"citation_score": 0
}MiniCPM5 2B
✓ pass · score 100 · 1.28 s
To cancel your FitHaven gym membership, you must give 30 days' written notice, delivered in person at the front desk or sent by certified mail to the club address. Cancellation requests by email or phone are not accepted [doc:gym_membership].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true,
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"gym_membership"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}K2-Horizon 0.9B
✗ fail · score 0 · 0.45 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false,
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}