Lang2Act: Fine-Grained Visual Reasoning through Self-Emergent Linguistic Toolchains

TL;DR

Lang2Act introduces a self-emergent linguistic toolchain approach with RL training to improve visual perception and reasoning in vision-language models, surpassing fixed external tool reliance.

Contribution

It presents a novel RL-based framework for VLMs to develop and utilize self-emergent linguistic tools, enhancing visual reasoning capabilities.

Findings

Achieves over 4% performance improvement in visual perception tasks.

Demonstrates effective self-exploration of high-quality actions for linguistic toolbox construction.

Enhances VLMs' visual reasoning without relying on fixed external tools.

Abstract

Visual Retrieval-Augmented Generation (VRAG) enhances Vision-Language Models (VLMs) by incorporating external visual documents to address a given query. Existing VRAG frameworks usually depend on rigid, pre-defined external tools to extend the perceptual capabilities of VLMs, typically by explicitly separating visual perception from subsequent reasoning processes. However, this decoupled design can lead to unnecessary loss of visual information, particularly when image-based operations such as cropping are applied. In this paper, we propose Lang2Act, which enables fine-grained visual perception and reasoning through self-emergent linguistic toolchains. Rather than invoking fixed external engines, Lang2Act collects self-emergent actions as linguistic tools and leverages them to enhance the visual perception capabilities of VLMs. To support this mechanism, we design a two-stage Reinforcement Learning (RL)-based training framework. Specifically, the first stage optimizes VLMs to self-explore high-quality actions for constructing a reusable linguistic toolbox, and the second stage further optimizes VLMs to exploit these linguistic tools for downstream reasoning effectively. Experimental results demonstrate the effectiveness of Lang2Act in substantially enhancing the visual perception capabilities of VLMs, achieving performance improvements of over 4%. All code and data are available at https://github.com/NEUIR/Lang2Act.