FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition

TL;DR

FIKA-Bench introduces a new benchmark for evaluating systems on fine-grained knowledge acquisition, emphasizing external evidence search and verification, revealing current models' significant limitations.

Contribution

The paper presents FIKA-Bench, a high-quality, leakage-aware benchmark for fine-grained knowledge acquisition, and evaluates state-of-the-art models, highlighting their deficiencies in evidence retrieval and visual judgment.

Findings

Best model achieves only 25.1% accuracy.

Models struggle with entity retrieval and visual judgment.

Equipping models with tools alone is insufficient.

Abstract

Fine-grained recognition in everyday life is often not a closed-book classification problem: when encountering unfamiliar objects, humans actively search, compare visual details, and verify evidence before deciding. Existing benchmarks primarily evaluate visually recognition, leaving this active external knowledge acquisition ability underexplored. We study fine-grained knowledge acquisition, where a system must seek, verify, and use external evidence to answer open-ended fine-grained recognition questions. We introduce FIKA-Bench, a leakage-aware and evidence-grounded collection of 311 public-source and real-life instances. To ensure high quality, every example is filtered against frontier closed-book models to remove memorized cases and audited to eliminate image-answer leakage, retaining only samples supported by verified evidence. Our evaluation of latest Large Multimodal Models (LMMs) and agents reveals that the task remains a formidable challenge: the best system reaches only 25.1% accuracy, with no model exceeding 30%. Crucially, we find that merely equipping models with tools is insufficient to bridge this gap; agent failures are predominantly driven by wrong entity retrieval and poor visual judgement. These results show that reliable knowledge acquisition needs better agent designs that focus on fine-grained recognition.