ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model

TL;DR

ViPER introduces a self-bootstrapping framework that enhances fine-grained visual perception in vision-language models through iterative self-critique and prediction, leading to improved performance across multiple benchmarks.

Contribution

The paper presents a novel two-stage task and a self-evolution framework, ViPER, that significantly improves visual perception in VLMs without sacrificing general capabilities.

Findings

Qwen-Viper achieves up to 6.0% improvement on perception benchmarks

ViPER demonstrates consistent performance gains across diverse tasks

The framework enables autonomous self-improvement of perceptual abilities

Abstract

The limited capacity for fine-grained visual perception presents a critical bottleneck for Vision-Language Models (VLMs) in real-world applications. Addressing this is challenging due to the scarcity of high-quality data and the limitations of existing methods: supervised fine-tuning (SFT) often compromises general capabilities, while reinforcement fine-tuning (RFT) prioritizes textual reasoning over visual perception. To bridge this gap, we propose a novel two-stage task that structures visual perception learning as a coarse-to-fine progressive process. Based on this task formulation, we develop ViPER, a self-bootstrapping framework specifically designed to enable iterative evolution through self-critiquing and self-prediction. By synergistically integrating image-level and instance-level reconstruction with a two-stage reinforcement learning strategy, ViPER establishes a closed-loop training paradigm, where internally synthesized data directly fuel the enhancement of perceptual ability. Applied to the Qwen2.5-VL family, ViPER produces the Qwen-Viper series. With an average gain of 1.7% on seven comprehensive benchmarks spanning various tasks and up to 6.0% on fine-grained perception, Qwen-Viper consistently demonstrates superior performance across different vision-language scenarios while maintaining generalizability. Beyond enabling self-improvement in perceptual capabilities, ViPER provides concrete evidence for the reciprocal relationship between generation and understanding, a breakthrough to developing more autonomous and capable VLMs.