Ask my documents

Answer questions about a folder of 22 household documents, with citations, and refuse when the answer isn't there.

Results

Model	Accuracy	Pass	Easy	Medium	Hard	Median latency	Tok/s	Job time
MiMo-V2.6 Distill 9B	90.8	90.0%	100.0%	92.3%	80.0%	4.11 s	40.2	2m 48s
Granite 4.2 8B	86.3	82.5%	100.0%	84.6%	66.7%	4.52 s	39.3	3m 37s
Mellum2.1 12B-A2.5B	83.8	80.0%	66.7%	92.3%	80.0%	2.53 s	97.3	2m 03s
MiniCPM5 2B	81.7	75.0%	100.0%	84.6%	46.7%	1.38 s	103.6	56.1 s
Ornith-1.5 9B	78.9	77.5%	91.7%	92.3%	53.3%	3.33 s	47.5	2m 20s
Ling-3.0-tiny (7.9B-A1.3B)	76.6	70.0%	66.7%	69.2%	73.3%	1.09 s	111.0	45.1 s
Spark-X2.5 4B	70.9	62.5%	66.7%	76.9%	46.7%	2.19 s	68.3	1m 36s
K2-Horizon 0.9B	41.6	27.5%	50.0%	15.4%	20.0%	0.64 s	200.7	31.7 s

How it is scored

Answerable: 0.7 x key-fact coverage + 0.3 x citation score (gold docs cited, -0.15 per extra doc cited); a refusal scores 0. Unanswerable: 1.0 for a refusal that asserts none of the trap facts, else 0. Pass = score >= 0.8.

Temperature 0 · max 800 output tokens.

System prompt every model got

You answer questions about the user's personal documents. Use only the document excerpts provided in the user's message, never outside knowledge or guesses. Keep the answer short (one to three sentences). Cite every document you used with its tag exactly as shown in the excerpts, for example [doc:lease]. If the excerpts do not contain the answer, reply with exactly this sentence and nothing else: I can't find that in your documents.

Every item

Final score per item (0-100). Click any row to see the prompt, the expected answer and every model's reply. Columns sort.

Item	Level	Prompt	Passed by	MiMo-V2.6 Distill 9B	Ornith-1.5 9B	Granite 4.2 8B	Spark-X2.5 4B	Mellum2.1 12B-A2.5B	Ling-3.0-tiny (7.9B-A1.3B)	MiniCPM5 2B	K2-Horizon 0.9B
rag-001	easy	If our rent is paid after the grace period, what late fee does the lease charge?	6/8	100	100	100	100	30 ✗	100	100	0 ✗
rag-002	hard	Including the pet rent for Biscuit and Juniper, how much do we pay the landlord each month?	2/8	30 ✗	100	30 ✗	30 ✗	100	30 ✗	26 ✗	0 ✗
rag-003	medium	What time do quiet hours start on a Saturday night at Larkspur Commons?	6/8	100	100	100	100	100	0 ✗	100	0 ✗
rag-004	medium	How long is the warranty on our refrigerator?	8/8	100	100	100	100	100	100	100	100
rag-005	medium	Are we allowed to keep a charcoal grill on our balcony?	7/8	100	100	100	100	100	100	100	0 ✗
rag-006	easy	What's the deductible on our renters insurance?	6/8	100	100	100	100	100	0 ✗	100	70 ✗
rag-007	medium	If someone stole one of Priya's bracelets (not a scheduled item), what's the most our renters policy pays for …	4/8	100	100	100	30 ✗	100	0 ✗	70 ✗	0 ✗
rag-008	hard	Counting the Northfield Visa extended warranty benefit, until what date is the Novabook laptop covered for har…	1/8	0 ✗	15 ✗	30 ✗	0 ✗	30 ✗	100	30 ✗	30 ✗
rag-009	hard	What is the comprehensive deductible on the Crestline?	6/8	100	100	100	30 ✗	100	100	100	30 ✗
rag-010	hard	What's the late fee if we pay the Kids Club after-school tuition late?	4/8	0 ✗	0 ✗	0 ✗	100	0 ✗	100	100	100
rag-011	easy	How long does the dishwasher's full parts-and-labor warranty last?	7/8	100	100	100	100	100	100	100	70 ✗
rag-012	hard	If the dishwasher's electronic control board fails in its fourth year, will Kestrel pay for the labor?	4/8	100	30 ✗	100	0 ✗	100	100	30 ✗	30 ✗
rag-013	hard	Will Maya and Theo miss any school days at Millbrook Elementary because of our December Lake Tahoe trip?	2/8	100	11 ✗	11 ✗	0 ✗	96	0 ✗	30 ✗	0 ✗
rag-014	easy	What does error code E21 on our washer mean?	7/8	100	100	100	100	100	65 ✗	100	100
rag-015	medium	How do I turn on the child lock on our washing machine?	7/8	100	100	100	100	100	100	100	30 ✗
rag-016	easy	What time are the kids let out of school on Wednesdays?	5/8	100	100	96	100	70 ✗	70 ✗	96	0 ✗
rag-017	hard	What is Juniper's microchip number?	7/8	100	100	100	100	100	100	100	0 ✗
rag-018	medium	What day does school start back up after winter break?	6/8	100	100	100	30 ✗	100	100	100	30 ✗
rag-019	medium	What number do I call to report Maya absent from school?	7/8	100	100	100	100	100	96	96	0 ✗
rag-020	easy	What is the confirmation code for our Cascadia Air flights to Reno?	6/8	100	100	100	30 ✗	30 ✗	100	100	100
rag-021	hard	Biscuit is boarding at Happy Tails during the Tahoe trip. Which of his vaccines will need a booster before he …	7/8	100	100	85	100	100	85	100	15 ✗
rag-022	easy	How much amoxicillin is Theo supposed to take, and how often?	7/8	100	100	100	65 ✗	100	100	100	100
rag-023	hard	How much did the brake job on the Crestline cost?	8/8	100	100	100	100	100	100	100	100
rag-024	easy	What was Daniel's LDL cholesterol at his last physical?	6/8	100	100	100	30 ✗	100	100	100	30 ✗
rag-025	medium	When does Biscuit's rabies vaccine expire?	7/8	100	100	96	0 ✗	100	100	100	100
rag-026	medium	What's the phone number for Brookside Auto Care, and what is our collision deductible on the Tessera?	7/8	100	100	100	100	100	100	100	15 ✗
rag-027	hard	How long is the warranty on the brake work Brookside did on the Tessera?	5/8	100	0 ✗	100	100	96	100	0 ✗	30 ✗
rag-028	medium	How do I cancel our FitHaven gym membership?	6/8	100	100	100	100	100	70 ✗	100	0 ✗
rag-029	easy	How much is the current electric bill and when is it due?	8/8	100	100	100	100	100	100	100	100
rag-030	easy	What is the Wi-Fi password at the Silver Pines Lodge?	7/8	100	100	100	100	0 ✗	100	100	100
rag-031	hard	With the sibling discount, what is our total monthly Kids Club after-school tuition for Theo and Maya?	6/8	100	100	100	100	100	0 ✗	100	30 ✗
rag-032	hard	If we had pulled Maya out of soccer on September 10, how much of her registration fee would we have gotten bac…	3/8	100	100	30 ✗	30 ✗	100	0 ✗	30 ✗	0 ✗
rag-033	easy	Where in the house is Maya's EpiPen kept?	4/8	100	0 ✗	100	30 ✗	100	70 ✗	100	70 ✗
rag-034	medium	My sister is coming to visit. How long can her car stay in guest parking, and how many consecutive days can a …	7/8	100	100	100	100	100	100	100	70 ✗
rag-035	hard	What dose of ibuprofen did Dr. Feld recommend for Daniel?	8/8	100	100	100	100	100	100	100	100
rag-036	hard	Once the Clearwave internet promo rate ends, what will our FitHaven monthly dues plus internet cost combined?	3/8	100	0 ✗	100	0 ✗	0 ✗	81	30 ✗	0 ✗
rag-037	medium	What are the pool hours at our gym?	1/8	0 ✗	0 ✗	0 ✗	100	0 ✗	0 ✗	0 ✗	0 ✗
rag-038	medium	What does the after-school program require us to have on file for Maya's peanut allergy, and where do we keep …	6/8	100	100	77 ✗	100	100	100	100	15 ✗
rag-039	easy	What is the policy number of our umbrella insurance policy?	8/8	100	100	100	100	100	100	100	100
rag-040	hard	If we stick with the thermostat's default filter reminder, when is our next HVAC filter change due?	4/8	100	0 ✗	100	30 ✗	100	100	30 ✗	0 ✗

✗ marks a fail (below the job's pass bar). A dash means no valid result.

← All jobs