Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents
rag-002
hard answerablesingle_doc
The prompt
Document excerpts provided from: hoa_rules, lease, sitter_notes, travel_itinerary
Including the pet rent for Biscuit and Juniper, how much do we pay the landlord each month?
What we expected
{
"answerable": true,
"answer": "$2,215 a month: $2,145 rent plus $35 pet rent for each of the 2 pets ($70).",
"key_facts": [
[
"2,215",
"2215"
]
],
"citations": [
"lease"
],
"contra": [],
"evidence": [
"$2,145.00",
"$35.00 per pet per month"
],
"gold_in_context": true
}What each model answered
MiMo-V2.6 Distill 9B
✗ fail · score 30 · 4.66 s
$2,585.00 per month [doc:lease]. That is $2,145.00 rent plus $35.00 pet rent for Biscuit and $35.00 pet rent for Juniper.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ornith-1.5 9B
✓ pass · score 100 · 4.70 s
Based on the documents, the monthly rent is **$2,145.00** [doc:lease], and the pet rent is **$35.00 per pet per month** [doc:lease]. Including pet rent for both Biscuit and Juniper, you pay the landlord **$2,145.00 + $70.00 = $2,215.00** each month.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Granite 4.2 8B
✗ fail · score 30 · 4.83 s
Including the pet rent for Biscuit and Juniper, we pay the landlord $2,145.00 in rent plus $70.00 in pet rent each month ($35.00 per pet × 2 pets) [doc:lease] [doc:lease]. The pet rent is due with rent as stated in the lease [doc:lease] and the pet rent amount is specified in the lease [doc:lease].
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Spark-X2.5 4B
✗ fail · score 30 · 2.05 s
[doc:lease]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Mellum2.1 12B-A2.5B
✓ pass · score 100 · 5.89 s
[doc:lease]Monthly rent is **$2,145.00**... [doc:lease]Tenant will also pay **pet rent of $35.00 per pet per month**... Total monthly payment including pet rent for Biscuit and Juniper is **$2,215.00**.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
true
],
"contra_hit": [],
"coverage": 1,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}Ling-3.0-tiny (7.9B-A1.3B)
✗ fail · score 30 · 1.45 s
The monthly pet rent for Biscuit and Juniper is $70.00. [doc:lease] states the pet rent is $35.00 per pet per month, and [doc:lease] confirms Biscuit and Juniper are the approved pets. The total monthly pet rent is $35.00 × 2 = $70.00.
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"lease"
],
"citation_recall": 1,
"extra_citations": [],
"citation_score": 1
}MiniCPM5 2B
✗ fail · score 26 · 1.44 s
Monthly rent is $2,145.00, and there is a pet rent of $35.00 per pet per month for Biscuit and Juniper. The total monthly payment would be the rent plus the pet rent for the two pets. [doc:lease] [doc:sitter_notes]
Scoring detail
{
"answerable": true,
"refusal": false,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [
"lease",
"sitter_notes"
],
"citation_recall": 1,
"extra_citations": [
"sitter_notes"
],
"citation_score": 0.85
}K2-Horizon 0.9B
✗ fail · score 0 · 0.47 s
I can't find that in your documents.
Scoring detail
{
"answerable": true,
"refusal": true,
"facts_matched": [
false
],
"contra_hit": [],
"coverage": 0,
"cited": [],
"citation_recall": 0,
"extra_citations": [],
"citation_score": 0
}