Unified MemoryBenchmarks / Which Small AI Model Should You Run on a Mac? We Ran 312 Tests / Read images
vision-027
hard chartreasoning
The prompt
1 image attached (not shown).
user
In which month did the Online shop first have more orders than the Downtown store? Reply with only the month as written on the chart.
What we expected
{
"answer": "Jul",
"accept": [
"July"
],
"type": "choice",
"options": [
[
"Jan",
"January"
],
[
"Feb",
"February"
],
[
"Mar",
"March"
],
[
"Apr",
"April"
],
[
"May"
],
[
"Jun",
"June"
],
[
"Jul",
"July"
],
[
"Aug",
"August"
],
[
"Sep",
"Sept",
"September"
],
[
"Oct",
"October"
],
[
"Nov",
"November"
],
[
"Dec",
"December"
]
]
}What each model answered
LFM2.5-VL 3B
✓ pass · score 100 · 1.29 s
Jul
Scoring detail
{
"type": "choice",
"normalized": "jul"
}MiMo-V2.6 Distill 9B
✓ pass · score 100 · 2.11 s
July
Scoring detail
{
"type": "choice",
"normalized": "july"
}MiniCPM-V 4.7 1B
✓ pass · score 100 · 0.95 s
Jul
Scoring detail
{
"type": "choice",
"normalized": "jul"
}