Visual Funnel: Resolving Contextual Blindness in Multimodal Large Language Models

TL;DR

Visual Funnel is a training-free method that enhances multimodal large language models by preserving hierarchical visual context, effectively addressing the 'Contextual Blindness' problem caused by structural disconnects.

Contribution

We introduce Visual Funnel, a novel two-step approach that dynamically constructs hierarchical visual context to improve fine-grained perception in multimodal models.

Findings

Visual Funnel outperforms naive single-crop baselines.

Adding unstructured crops offers limited benefits, highlighting the importance of hierarchical structure.

The method effectively resolves 'Contextual Blindness' in multimodal models.

Abstract

Multimodal Large Language Models (MLLMs) demonstrate impressive reasoning capabilities, but often fail to perceive fine-grained visual details, limiting their applicability in precision-demanding tasks. While methods that crop salient regions of an image offer a partial solution, we identify a critical limitation they introduce: "Contextual Blindness". This failure occurs due to structural disconnect between high-fidelity details (from the crop) and the broader global context (from the original image), even when all necessary visual information is present. We argue that this limitation stems not from a lack of information 'Quantity', but from a lack of 'Structural Diversity' in the model's input. To resolve this, we propose Visual Funnel, a training-free, two-step approach. Visual Funnel first performs Contextual Anchoring to identify the region of interest in a single forward pass. It then constructs an Entropy-Scaled Portfolio that preserves the hierarchical context - ranging from focal detail to broader surroundings - by dynamically determining crop sizes based on attention entropy and refining crop centers. Through extensive experiments, we demonstrate that Visual Funnel significantly outperforms naive single-crop and unstructured multi-crop baselines. Our results further validate that simply adding more unstructured crops provides limited or even detrimental benefits, confirming that the hierarchical structure of our portfolio is key to resolving Contextual Blindness.