Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-006

easy answerablesingle_doc

The prompt

Document excerpts provided from: auto_insurance, credit_card_benefits, lease, renters_insurance

What's the deductible on our renters insurance?

What we expected

{
  "answerable": true,
  "answer": "$500 (applies to all Section I property perils).",
  "key_facts": [
    [
      "$500",
      "500 dollars"
    ]
  ],
  "citations": [
    "renters_insurance"
  ],
  "contra": [],
  "evidence": [
    "Deductible (applies to all Section I perils): $500"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.57 s

The renters insurance deductible is $500, which applies to all Section I perils [doc:renters_insurance].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.07 s

The deductible on your renters insurance is $500, which applies to all Section I perils. [doc:renters_insurance]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 3.63 s

The renters insurance document [1] shows a deductible of $500 (applies to all Section I perils). [doc:renters_insurance]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✓ pass · score 100 · 2.23 s

The deductible on your renters insurance is **$500** [doc:renters_insurance].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 1.45 s

[doc:renters_insurance] shows the deductible for the renters insurance is $500.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 0.76 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

MiniCPM5 2B

✓ pass · score 100 · 1.36 s

The deductible for renters insurance is $500, as stated in [doc:renters_insurance]. The document also mentions that the tenant must carry renters insurance with at least $100,000 in personal liability coverage, but the deductible specifically is $500.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "renters_insurance"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 70 · 0.90 s

The renters insurance policy lists a deductible of $500 for all Section I perils.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-005All Ask my documents itemsrag-007 →