Downscaling Intelligence: Exploring Perception and Reasoning Bottlenecks in Small Multimodal Models

TL;DR

This paper analyzes how reducing large language model capacity impacts multimodal perception and reasoning, revealing perceptual bottlenecks and proposing a new visual extraction tuning method to improve efficiency and performance.

Contribution

It identifies perceptual bottlenecks caused by LLM downscaling and introduces Extract+Think, a novel approach combining visual extraction tuning with step-by-step reasoning.

Findings

LLM downscaling disproportionately affects visual capabilities.

Visual perception drops sharply with model size, often more than reasoning.

Extract+Think improves efficiency and performance in small multimodal models.

Abstract

Scaling up multimodal models has enabled remarkable advances in visual understanding and reasoning, but practical demands call for smaller, efficient systems. In this work, we conduct a principled analysis of downscaling intelligence in multimodal models, examining how reduced large language model (LLM) capacity affects multimodal capabilities. Our initial findings reveal an interesting trend: LLM downscaling disproportionately affects visual capabilities, rather than abilities inherited from the LLM. We then examine whether this drop mainly reflects the expected decline in visual reasoning or a more fundamental loss of perceptual abilities. Isolating the effect of LLM downscaling on perception, we find performance still drops sharply, often matching or exceeding the impact on reasoning. To address this bottleneck, we introduce visual extraction tuning, which explicitly trains the model to extract instruction-relevant visual details consistently across tasks. With these extracted visual details, we then apply step-by-step reasoning to generate answers. Together, these components form our Extract+Think approach, setting a new standard for efficiency and performance in this space.