Benchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Ask my documents

rag-013

hard answerablemulti_doc

The prompt

Document excerpts provided from: afterschool_program, school_calendar, travel_itinerary

Will Maya and Theo miss any school days at Millbrook Elementary because of our December Lake Tahoe trip?

What we expected

{
  "answerable": true,
  "answer": "No. The trip (Dec 22, 2026 – Jan 3, 2027) falls inside winter break (Dec 21 – Jan 1), and school resumes Monday, Jan 4, the day after the return flight.",
  "key_facts": [
    [
      "no,",
      "no.",
      "no -",
      "no!",
      "no;",
      "won't miss",
      "will not miss",
      "wouldn't miss",
      "would not miss",
      "won't have to miss",
      "will not have to miss",
      "won't need to miss",
      "don't miss",
      "do not miss",
      "not miss any",
      "no school days",
      "no days of school",
      "no missed",
      "none",
      "zero school days",
      "not missing any",
      "won't be missing",
      "will not be missing",
      "no school is missed",
      "no school will be missed",
      "doesn't overlap",
      "does not overlap",
      "entirely within winter break",
      "entirely during winter break",
      "falls within winter break",
      "falls during winter break",
      "during winter break"
    ]
  ],
  "citations": [
    "travel_itinerary",
    "school_calendar"
  ],
  "contra": [
    "yes,",
    "yes.",
    "yes -",
    "yes!",
    "yes;",
    "will miss school",
    "they will miss",
    "would miss school",
    "they'll miss",
    "will be missing"
  ],
  "evidence": [
    "Dec 21 (Mon) – Jan 1 (Fri): WINTER BREAK",
    "Jan 4 (Mon): School resumes",
    "Sun Jan 3, 2027"
  ],
  "gold_in_context": true
}

What each model answered

MiMo-V2.6 Distill 9B

✓ pass · score 100 · 5.53 s

No — the trip falls entirely within winter break. [doc:travel_itinerary] shows the trip runs Dec 22, 2026 – Jan 3, 2027, and [doc:school_calendar] shows Winter Break runs Dec 21 – Jan 1 with school resuming Jan 4, so neither Maya nor Theo misses a school day.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "travel_itinerary",
    "school_calendar"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

Ornith-1.5 9B

✗ fail · score 11 · 4.50 s

Yes. Theo is enrolled in the 5-day (Mon–Fri) plan and Maya in the 3-day (Mon/Wed/Fri) plan, and the entire Winter Break (Dec 21 – Jan 1) is a NO SCHOOL period, so both will miss school days. [doc:afterschool_program][doc:school_calendar]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    false
  ],
  "contra_hit": [
    "yes.",
    "will miss school"
  ],
  "coverage": 0,
  "cited": [
    "afterschool_program",
    "school_calendar"
  ],
  "citation_recall": 0.5,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.35
}

Granite 4.2 8B

✗ fail · score 11 · 27.2 s

Yes, Maya and Theo will miss school days during the trip because Winter Break runs Dec 17–Jan 1 with NO SCHOOL.

Millbrook Elementary has NO SCHOOL on Dec 17–21 (Dec 17–20 school events/break, Dec 21–21 Winter Break start), and the trip is Dec 22–Jan 3. However, the school calendar shows NO SCHOOL for Dec 21–Jan 1 (Winter Break), so school days missed include Dec 21–Jan 1 except any school days that might occur during the trip overlap.

From [4] [doc:school_calendar]: Dec 21 (Mon) – Jan 1 (Fri): WINTER BREAK — NO SCHOOL. Also, Dec 17 (Thu): Winter Music Program (not school), and Dec 18–20 are school days (no NO SCHOOL listed except Nov events). Since Winter Break starts Dec 21, school resumes Jan 4, the trip runs Dec 22–Jan 3, so school days missed are Dec 21–Jan 1 (Winter Break), but the trip overlaps Dec 22–Jan 3, so school days missed during the trip are Dec 22–Jan 1 (Winter Break days).

From [3] [doc:afterschool_program]: Maya Hartwell (Grade 4) — 3-day plan (Mon/Wed/Fri); Theo Hartwell (Grade 1) — 5-day plan. But school is NO SCHOOL during Winter Break per [4].

Thus, Maya and Theo will miss all school days during Winter Break (Dec 21–Jan 1), which includes the trip dates Dec 22–Jan 3 (Jan 2–3 are school days? No, Jan 2–3 are still Winter Break until Jan 1; Jan 2–3 are Winter Break? [4] says Dec 21 – Jan 1: WINTER BREAK — NO SCHOOL, and Jan 4: School resumes. So Jan 2–3 are still NO SCHOOL. Therefore, all school days during the trip period are NO SCHOOL, so they miss no regular school days; they miss only Winter Break days, which are not school days.

Thus, Maya and Theo will not miss any school days because school is closed during the trip dates (Winter Break).

Wait: The question: "Will Maya and Theo miss any school days at Millbrook Elementary because of our December Lake Tahoe trip?" Since Winter Break = NO SCHOOL, they do not miss any school days (they are not in school). However, need to confirm if any school days fall within trip dates that are not NO SCHOOL. [4] says Dec 21 – Jan 1: NO SCHOOL (Winter Break), and Jan 4: School resumes. So Dec 22–Jan 3 includes Jan 2–3 which are still NO SCHOOL (since Jan 1 is last NO SCHOOL day, Jan 2–3 are also NO SCHOOL as part of Winter Break). Thus, no school days occur during the trip, so they miss no school days.

But need to cite: [4] shows Winter Break NO SCHOOL Dec 21–Jan 1, and [3] shows their enrollment plans, but the key is school calendar. So answer: No, they will not miss any school days because school is on Winter Break (NO SCHOOL) during the trip dates.

But phrase: "Will Maya and Theo miss any school days... because of our trip?" Since school is closed during the trip, they don't miss any school days (they aren't in school). So answer: No.

But need short answer with citations.

Answer: No, Maya and Theo will not miss any school days because school is on Winter Break (NO SCHOOL) from Dec 21 to Jan 1,

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [
    "yes,",
    "will miss school"
  ],
  "coverage": 0,
  "cited": [
    "school_calendar",
    "afterschool_program"
  ],
  "citation_recall": 0.5,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.35
}

Spark-X2.5 4B

✗ fail · score 0 · 2.40 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

Mellum2.1 12B-A2.5B

✓ pass · score 96 · 4.62 s

[doc:afterschool_program] shows Maya is on a 3-day plan (Mon/Wed/Fri) and Theo on a 5-day plan. [doc:school_calendar] lists school days for December: Dec 17, 21-20, Jan 1, etc. The trip is Dec 22–Jan 3. School is in session Dec 21–Jan 1 (winter break starts Dec 21). Actually winter break Dec 21 – Jan 1: NO SCHOOL. So Maya and Theo would not attend school during the trip because it coincides with winter break. However need to confirm if any school days fall within Dec 22–Jan 3. According to school calendar, Dec 21–Jan 1 is winter break NO SCHOOL. Jan 4 resumes. So no school days during trip. Need to cite documents: [doc:afterschool_program] for attendance plans, [doc:school_calendar] for break dates. Also maybe [doc:travel_itinerary] for dates but school calendar has dates. Let's cite both.

Answer: Maya and Theo will not miss any school days because the trip coincides with the winter break period from Dec 21 to Jan 1, when school is closed. [doc:afterschool_program][doc:school_calendar]

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [],
  "coverage": 1,
  "cited": [
    "afterschool_program",
    "school_calendar",
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.85
}

Ling-3.0-tiny (7.9B-A1.3B)

✗ fail · score 0 · 2.19 s

Maya is on a 3-day plan (Mon/Wed/Fri) and Theo is on a 5-day plan [doc:afterschool_program]. The December trip is Dec 22–Jan 3 [doc:travel_itinerary], which includes Dec 23–24 (Mon/Tue), Dec 26 (Thu), and Dec 31 (Mon). Maya misses Mon Dec 23, Tue Dec 24, Thu Dec 26, and Mon Dec 31 — 4 days. Theo misses Mon Dec 23, Tue Dec 24, Thu Dec 26, and Mon Dec 31 — 4 days.

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [
    "afterschool_program",
    "travel_itinerary"
  ],
  "citation_recall": 0.5,
  "extra_citations": [
    "afterschool_program"
  ],
  "citation_score": 0.35
}

MiniCPM5 2B

✗ fail · score 30 · 1.82 s

Yes, Maya and Theo will miss school. The school calendar shows no school from December 21, 2026 to January 1, 2027 [doc:school_calendar], and the travel itinerary confirms the trip dates are December 22, 2026 to January 3, 2027 [doc:travel_itinerary]. Since school is suspended during winter break, they will miss those days.

Scoring detail

{
  "answerable": true,
  "refusal": false,
  "facts_matched": [
    true
  ],
  "contra_hit": [
    "yes,",
    "will miss school",
    "they will miss"
  ],
  "coverage": 0,
  "cited": [
    "school_calendar",
    "travel_itinerary"
  ],
  "citation_recall": 1,
  "extra_citations": [],
  "citation_score": 1
}

K2-Horizon 0.9B

✗ fail · score 0 · 0.61 s

I can't find that in your documents.

Scoring detail

{
  "answerable": true,
  "refusal": true,
  "facts_matched": [
    false
  ],
  "contra_hit": [],
  "coverage": 0,
  "cited": [],
  "citation_recall": 0,
  "extra_citations": [],
  "citation_score": 0
}

← rag-012All Ask my documents itemsrag-014 →