Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Read images
vision-039
hard price_tagreasoning
The prompt
1 image attached (not shown).
user
Two shelf tags, A and B, are shown. Which oats are cheaper per ounce? Reply with only the letter A or B.
What we expected
{
"answer": "B",
"accept": [
"Morning Mill",
"Tag B",
"Option B"
],
"type": "choice",
"options": [
[
"B",
"Tag B",
"Option B",
"Morning Mill"
],
[
"A",
"Tag A",
"Option A",
"Brightfield"
]
]
}What each model answered
LFM2.5-VL 3B
✗ fail · score 0 · 4.52 s
A
Scoring detail
{
"type": "choice",
"normalized": "a",
"options_named": [
1
]
}MiMo-V2.6 Distill 9B
✗ fail · score 0 · 2.13 s
A
Scoring detail
{
"type": "choice",
"normalized": "a",
"options_named": [
1
]
}MiniCPM-V 4.7 1B
✗ fail · score 0 · 1.00 s
A
Scoring detail
{
"type": "choice",
"normalized": "a",
"options_named": [
1
]
}