Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-036

hard answerablemulti_doc

The prompt

Document excerpts provided from: gym_membership, internet_contract, lease

Once the Clearwave internet promo rate ends, what will our FitHaven monthly dues plus internet cost combined?

What we expected

{
  "answerable": true,
  "answer": "$174 a month: $89 FitHaven dues plus Clearwave's $85 standard rate (before internet taxes and the $1.49 fee).",
  "key_facts": [
    [
      "174",
      "175.49"
    ]
  ],
  "citations": [
    "gym_membership",
    "internet_contract"
  ],
  "contra": [],
  "evidence": [
    "TOTAL MONTHLY DUES ....................................... $89.00",
    "currently $85.00/month"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.40 s

The Clearwave promo rate ends August 4, 2027, after which the standard rate is $85.00/month [doc:internet_contract]. FitHaven monthly dues are $89.00 [doc:gym_membership]. Combined, that is $174.00 per month.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "internet_contract",
    "gym_membership"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✗ fail · score 0 · 3.69 s

I can't find that in your documents.

The excerpts contain the FitHaven monthly dues ($89.00, from [doc:gym_membership]) and the Clearwave internet standard rate ($85.00, from [doc:internet_contract]), but they don't include a combined total.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "gym_membership",
    "internet_contract"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 7.40 s

[doc:gym_membership] shows the total monthly dues for FitHaven are $89.00 (Core Monthly $49.00 + Household Add-On $25.00 + Kids Club $15.00) [1][2][3]. [doc:internet_contract] shows the standard rate after the promo ends is $85.00/month [5]. Adding these gives $89.00 + $85.00 = $174.00 per month.

[doc:gym_membership] shows the total monthly dues for FitHaven are $89.00 (Core Monthly $49.00 + Household Add-On $25.00 + Kids Club $15.00) [1][2][3]. [doc:internet_contract] shows the standard rate after the promo ends is $85.00/month [5]. Adding these gives $89.00 + $85.00 = $174.00 per month.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "gym_membership",
    "internet_contract"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 0 · 1.79 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

Mellum2.1 12B-A2.5B

✗ fail · score 0 · 1.14 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 81 · 1.47 s

The FitHaven monthly dues are $89.00 [doc:lease]. The Clearwave promo rate ends August 4, 2027, after which the standard rate is $85.00/month [doc:internet_contract]. Combined cost: $89.00 + $85.00 = $174.00.

Answer: $174.00

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "lease",
    "internet_contract"
  ],
  "citation_recall": 0.5,
  "extra_citations": [
    "lease"
  ],
  "citation_score": 0.35
}

MiniCPM5 2B

✗ fail · score 30 · 2.00 s

The Clearwave promo rate ends on August 5, 2027, when the standard rate of $85.00/month applies [doc:internet_contract]. The FitHaven Core Monthly dues are $49.00, and the annual enhancement fee of $39.00 is billed every March 15 [doc:gym_membership]. The combined cost would be $49.00 + $39.00 + $85.00 = $173.00 per month, though the exact billing timing depends on the billing cycle.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "internet_contract",
    "gym_membership"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.50 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-035All Ask my documents itemsrag-037 →