Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-010
hard unanswerable
The prompt
Document excerpts provided from: afterschool_program, gym_membership
What's the late fee if we pay the Kids Club after-school tuition late?
What we expected
{
"answerable": false,
"answer": "I can't find that in your documents.",
"key_facts": [],
"citations": [],
"forbidden": [
"$1 per minute",
"1 per minute",
"$1 a minute",
"1 dollar per minute",
"one dollar per minute",
"$1/minute",
"$1/min",
"$1 per min"
],
"trap": "The handbook lists a late PICKUP fee ($1 per minute per child) but no late tuition-payment fee.",
"gold_in_context": null
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 0 · 4.88 s
The after-school tuition is billed on the 25th and drafted automatically on the 1st, but the excerpts don't specify a late fee for tuition payments. The only late fee mentioned is for late pickups: $1 per minute per child after 6:00 p.m. [doc:afterschool_program]
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"$1 per minute",
"1 per minute"
],
"cited": [
"afterschool_program"
]
}Ornith-1.5 9B
✗ fail · score 0 · 3.84 s
I can't find that in your documents. The documents describe a late *pickup* fee ($1 per minute per child after 6:00 p.m.) [doc:afterschool_program], but they do not mention any late fee for paying the after-school tuition late.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"$1 per minute",
"1 per minute"
],
"cited": [
"afterschool_program"
]
}Granite 4.2 8B
✗ fail · score 0 · 5.59 s
The late fee for after-school tuition is not mentioned in the provided excerpts. However, the late fee for late pickup is $1 per minute per child, as stated in [2] [doc:afterschool_program]. Since the question asks about paying Kids Club after-school tuition late, and no specific late fee for late tuition payment is provided in the excerpts, the answer cannot be given. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"$1 per minute",
"1 per minute"
],
"cited": [
"afterschool_program"
]
}Spark-X2.5 4B
✓ pass · score 100 · 2.19 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}Mellum2.1 12B-A2.5B
✗ fail · score 0 · 2.51 s
[doc:afterschool_program] [doc:pickup] [doc:afterschool_program] [doc:afterschool_program] [doc:gym_membership] [doc:gym_membership] The late fee is $1 per minute per child after 6:00 p.m. as stated in [doc:pickup]. However, the question asks about late fee for paying the Kids Club after-school tuition late, not late pickup. The tuition late fee isn't mentioned in the excerpts. So I cannot find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [
"$1 per minute",
"1 per minute"
],
"cited": [
"afterschool_program",
"pickup",
"gym_membership"
]
}Ling-3.0-tiny (7.9B-A1.3B)
✓ pass · score 100 · 0.88 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}MiniCPM5 2B
✓ pass · score 100 · 1.21 s
The late fee for Kids Club after-school tuition is not specified in the provided documents. I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}K2-Horizon 0.9B
✓ pass · score 100 · 0.95 s
I can't find that in your documents.
Scoring detail
{
"answerable": false,
"refusal": true,
"forbidden_hit": [],
"cited": []
}