WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning

TL;DR

This paper introduces WeThink, a large multimodal dataset and a reinforcement learning approach to improve general-purpose vision-language reasoning in large models, demonstrating significant performance gains across diverse tasks.

Contribution

The paper presents a novel scalable QA synthesis pipeline, a large annotated multimodal dataset, and a hybrid RL training method to advance general-purpose vision-language reasoning.

Findings

Enhanced performance on 14 MLLM benchmarks

Automated data pipeline increases data diversity and model accuracy

Effective hybrid reward mechanism improves RL training efficiency

Abstract

Building on the success of text-based reasoning models like DeepSeek-R1, extending these capabilities to multimodal reasoning holds great promise. While recent works have attempted to adapt DeepSeek-R1-style reinforcement learning (RL) training paradigms to multimodal large language models (MLLM), focusing on domain-specific tasks like math and visual perception, a critical question remains: How can we achieve the general-purpose visual-language reasoning through RL? To address this challenge, we make three key efforts: (1) A novel Scalable Multimodal QA Synthesis pipeline that autonomously generates context-aware, reasoning-centric question-answer (QA) pairs directly from the given images. (2) The open-source WeThink dataset containing over 120K multimodal QA pairs with annotated reasoning paths, curated from 18 diverse dataset sources and covering various question domains. (3) A comprehensive exploration of RL on our dataset, incorporating a hybrid reward mechanism that combines rule-based verification with model-based assessment to optimize RL training efficiency across various task domains. Across 14 diverse MLLM benchmarks, we demonstrate that our WeThink dataset significantly enhances performance, from mathematical reasoning to diverse general multimodal tasks. Moreover, we show that our automated data pipeline can continuously increase data diversity to further improve model performance.