FineRS: Fine-grained Reasoning and Segmentation of Small Objects with Reinforcement Learning

TL;DR

FineRS is a novel reinforcement learning framework that enhances large language models' ability to reason about and segment extremely small objects in high-resolution images through a coarse-to-fine approach.

Contribution

The paper introduces a two-stage reinforcement learning method with a new dataset for small object segmentation and reasoning in high-resolution scenes, improving over existing MLLM approaches.

Findings

Outperforms state-of-the-art MLLM-based methods on segmentation tasks

Effective coarse-to-fine reasoning and refinement pipeline

Introduces a new dataset, FineRS-4k, for small object analysis

Abstract

Multi-modal Large Language Models (MLLMs) have shown remarkable capabilities across a wide range of vision-language tasks. However, due to the restricted input resolutions, MLLMs face significant challenges in precisely understanding and localizing visual details in high-resolution images -- particularly when dealing with extra-small objects embedded in cluttered contexts. To address this issue, we propose \textsc{FineRS}, a two-stage MLLM-based reinforcement learning framework for jointly reasoning and segmenting extremely small objects within high-resolution scenes. \textsc{FineRS} adopts a coarse-to-fine pipeline comprising Global Semantic Exploration (GSE) and Localized Perceptual Refinement (LPR). Specifically, GSE performs instruction-guided reasoning to generate a textural response and a coarse target region, while LPR refines this region to produce an accurate bounding box and segmentation mask. To couple the two stages, we introduce a locate-informed retrospective reward, where LPR's outputs are used to optimize GSE for more robust coarse region exploration. % Additionally, we present \textsc{FineRS}-4k, a new dataset for evaluating MLLMs on attribute-level reasoning and pixel-level segmentation on subtle, small-scale targets in complex high-resolution scenes. Experimental results on \textsc{FineRS}-4k and public datasets demonstrate that our method consistently outperforms state-of-the-art MLLM-based approaches on both instruction-guided segmentation and visual reasoning tasks.