SAM-R1: Leveraging SAM for Reward Feedback in Multimodal Segmentation via Reinforcement Learning

TL;DR

SAM-R1 introduces a reinforcement learning framework that leverages the Segment Anything Model to enable multimodal large models to perform fine-grained image segmentation reasoning with minimal training data, reducing reliance on costly annotations.

Contribution

It is the first to incorporate fine-grained segmentation into multimodal reasoning training using RL and SAM as a reward provider, improving segmentation reasoning with limited data.

Findings

Achieves strong performance with only 3k training samples

Effectively aligns segmentation and reasoning tasks

Demonstrates reinforcement learning enhances multimodal reasoning

Abstract

Leveraging multimodal large models for image segmentation has become a prominent research direction. However, existing approaches typically rely heavily on manually annotated datasets that include explicit reasoning processes, which are costly and time-consuming to produce. Recent advances suggest that reinforcement learning (RL) can endow large models with reasoning capabilities without requiring such reasoning-annotated data. In this paper, we propose SAM-R1, a novel framework that enables multimodal large models to perform fine-grained reasoning in image understanding tasks. Our approach is the first to incorporate fine-grained segmentation settings during the training of multimodal reasoning models. By integrating task-specific, fine-grained rewards with a tailored optimization objective, we further enhance the model's reasoning and segmentation alignment. We also leverage the Segment Anything Model (SAM) as a strong and flexible reward provider to guide the learning process. With only 3k training samples, SAM-R1 achieves strong performance across multiple benchmarks, demonstrating the effectiveness of reinforcement learning in equipping multimodal models with segmentation-oriented reasoning capabilities.