FOCUS: Internal MLLM Representations for Efficient Fine-Grained Visual Question Answering

TL;DR

FOCUS is a training-free visual cropping method that uses internal MLLM representations to efficiently identify relevant image regions, significantly improving fine-grained VQA accuracy and efficiency without task-specific fine-tuning.

Contribution

It introduces a novel, training-free visual cropping approach leveraging MLLM internal features to enhance fine-grained VQA performance and efficiency.

Findings

Outperforms existing visual cropping methods in accuracy and efficiency.

Achieves comparable results to the best baseline with 3-6.5x less compute.

Effective across multiple datasets and MLLM types.

Abstract

While Multimodal Large Language Models (MLLMs) offer strong perception and reasoning capabilities for image-text input, Visual Question Answering (VQA) focusing on small image details still remains a challenge. Although visual cropping techniques seem promising, recent approaches have several limitations: the need for task-specific fine-tuning, low efficiency due to uninformed exhaustive search, or incompatibility with efficient attention implementations. We address these shortcomings by proposing a training-free visual cropping method, dubbed FOCUS, that leverages MLLM-internal representations to guide the search for the most relevant image region. This is accomplished in four steps: first, we identify the target object(s) in the VQA prompt; second, we compute an object relevance map using the key-value (KV) cache; third, we propose and rank relevant image regions based on the map; and finally, we perform the fine-grained VQA task using the top-ranked region. As a result of this informed search strategy, FOCUS achieves strong performance across four fine-grained VQA datasets and three types of MLLMs. It outperforms three popular visual cropping methods in both accuracy and efficiency, and matches the best-performing baseline, ZoomEye, while requiring 3 - 6.5 x less compute.