AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition

TL;DR

AdaptVision introduces an adaptive visual token acquisition method for vision-language models, inspired by human active vision, which selectively gathers visual information to improve efficiency and accuracy in visual question answering tasks.

Contribution

The paper presents AdaptVision, a novel framework that enables VLMs to dynamically determine the necessary visual tokens using reinforcement learning and a decoupled turn policy optimization approach.

Findings

Achieves higher accuracy with fewer visual tokens compared to existing methods.

Demonstrates significant efficiency improvements across multiple VQA benchmarks.

Outperforms state-of-the-art efficient VLM approaches in experiments.

Abstract

Vision-Language Models (VLMs) have achieved remarkable success in visual question answering tasks, but their reliance on large numbers of visual tokens introduces significant computational overhead. While existing efficient VLM approaches reduce visual tokens through fixed-ratio compression, they operate passively and lack the ability to adapt to varying task requirements. This motivates a fundamental question: Can VLMs autonomously determine the minimum number of visual tokens required for each sample? Inspired by human active vision mechanisms, we introduce AdaptVision, an efficient VLM paradigm that enables adaptive visual token acquisition through a coarse-to-fine approach. Our model initially processes compressed visual tokens from low-resolution images and selectively acquires additional visual information by invoking a bounding box tool to crop key regions when necessary. We train AdaptVision using a reinforcement learning framework that carefully balances accuracy and efficiency. Central to our approach is Decoupled Turn Policy Optimization (DTPO), which decouples the learning objective into two components: (1) tool learning, which optimizes correct tool utilization, and (2) accuracy improvement, which refines the generated responses to improve answer correctness. Based on this formulation, we further decouple advantage estimation by computing separate advantages for tokens associated with each objective. This formulation enables more effective optimization for AdaptVision compared to vanilla GRPO. Comprehensive experiments across multiple VQA benchmarks demonstrate that AdaptVision achieves superior performance while consuming substantially fewer visual tokens than state-of-the-art efficient VLM methods.