Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-007

medium answerablesingle_doc

The prompt

Document excerpts provided from: credit_card_benefits, renters_insurance

If someone stole one of Priya's bracelets (not a scheduled item), what's the most our renters policy pays for jewelry theft?

What we expected

{
  "answerable": true,
  "answer": "$1,500 (special limit for jewelry, watches, furs and precious stones — loss by theft).",
  "key_facts": [
    [
      "1,500",
      "1500"
    ]
  ],
  "citations": [
    "renters_insurance"
  ],
  "contra": [],
  "evidence": [
    "Jewelry, watches, furs, precious stones — loss by theft: $1,500"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 4.20 s

The renters policy pays up to $1,500 for jewelry theft under the Coverage C special limits [doc:renters_insurance]. The scheduled personal property endorsement applies only to scheduled items, and Priya's bracelet is not a scheduled item [doc:renters_insurance].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.98 s

I can't find that in your documents.

The excerpts don't include a specific scheduled amount for the bracelets, so I can't determine what the renters policy would pay for their theft. The only scheduled item in [doc:renters_insurance] is the engagement ring ($6,800), and the jewelry theft special limit is $1,500 [doc:renters_insurance].

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 5.14 s

[2] states that for jewelry theft under Coverage C special limits of liability, the limit is $1,500 — loss by theft: $1,500. [4] confirms Coverage C — Personal Property: $40,000 (replacement cost), but the special limits in [2] apply to jewelry theft. Since the bracelet is not scheduled (the scheduled item is the engagement ring), the most the renters policy pays for jewelry theft is $1,500. [doc:renters_insurance] and [doc:renters_insurance] both support this.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✗ fail · score 30 · 1.84 s

[doc:renters_insurance]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 2.64 s

The most our renters policy pays for jewelry theft is $1,500 per [doc:renters_insurance].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.74 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

MiniCPM5 2B

✗ fail · score 70 · 1.58 s

The excerpts show that the renters policy has a scheduled personal property endorsement covering specific items, but jewelry theft for non-scheduled items is not explicitly defined. The only relevant limit mentioned is the general property coverage limit of $1,500 for jewelry theft under Coverage C, but this applies to all property in that category and does not increase the Coverage C limit. I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.46 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-006All Ask my documents itemsrag-008 →