Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-005

medium answerablesingle_doc

The prompt

Document excerpts provided from: hoa_rules, home_maintenance_log, sitter_notes

Are we allowed to keep a charcoal grill on our balcony?

What we expected

{
  "answerable": true,
  "answer": "No. Charcoal, gas and other open-flame grills are prohibited on balconies; electric grills are permitted.",
  "key_facts": [
    [
      "prohibited",
      "not allowed",
      "not permitted",
      "banned",
      "can't keep",
      "cannot keep",
      "can not keep",
      "can't have",
      "cannot have",
      "can't use",
      "cannot use",
      "aren't allowed",
      "isn't allowed",
      "are not allowed",
      "is not allowed",
      "no charcoal",
      "not ok",
      "forbidden",
      "no,",
      "no.",
      "no -",
      "no —",
      "not on the balcony",
      "not on balconies"
    ]
  ],
  "citations": [
    "hoa_rules"
  ],
  "contra": [
    "yes,",
    "yes.",
    "yes -",
    "yes!",
    "yes;",
    "charcoal grills are allowed",
    "charcoal grills are permitted"
  ],
  "evidence": [
    "Charcoal grills, gas grills, and any open-flame device are prohibited on balconies",
    "Electric grills are permitted"
  ],
  "gold_in_context": true,
  "citations_ok": [
    "home_maintenance_log"
  ]
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 3.18 s

No — [doc:hoa_rules] prohibits charcoal grills on balconies, and also within 10 feet of any building.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✓ pass · score 100 · 3.26 s

No, you are not allowed to keep a charcoal grill on your balcony. Per [doc:hoa_rules], "Charcoal grills, gas grills, and any open-flame device are prohibited on balconies and within 10 feet of any building, per fire code."

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Granite 4.2 8B

✓ pass · score 100 · 4.26 s

No, a charcoal grill is not allowed on a balcony. This is stated in [1] [doc:hoa_rules] which says: "Charcoal grills, gas grills, and any open-flame device are prohibited on balconies and within 10 feet of any building, per fire code. Electric grills are permitted."

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Spark-X2.5 4B

✓ pass · score 100 · 1.95 s

No, you are not allowed to keep a charcoal grill on your balcony. [doc:hoa_rules]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Mellum2.1 12B-A2.5B

✓ pass · score 100 · 3.09 s

No, charcoal grills are prohibited on balconies per the HOA rules [doc:hoa_rules].

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ling-3.0-tiny (7.9B-A1.3B)

✓ pass · score 100 · 0.99 s

No. [doc:hoa_rules] states that charcoal grills are prohibited on balconies and within 10 feet of any building. [doc:hoa_rules] also notes that electric grills are permitted.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

MiniCPM5 2B

✓ pass · score 100 · 1.55 s

No, you are not allowed to keep a charcoal grill on a balcony. [doc:hoa_rules] states that charcoal grills, gas grills, and any open-flame device are prohibited on balconies and within 10 feet of any building, per fire code. [doc:home_maintenance_log] also notes that the HOA does not allow a gas grill on balconies.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "hoa_rules",
    "home_maintenance_log"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.46 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-004All Ask my documents itemsrag-006 →