Discriminative Perception via Anchored Description for Reasoning Segmentation

TL;DR

This paper introduces DPAD, a method that enhances reasoning segmentation by generating descriptive captions to discriminate the target object from its context, improving focus, interpretability, and performance in complex scenes.

Contribution

The paper proposes a novel discriminative perception approach that combines captioning with reinforcement learning to improve reasoning chains and segmentation accuracy.

Findings

cIoU on ReasonSeg increased by 3.09%.

Reasoning chain length decreased by approximately 42%.

Significant performance gains demonstrated on benchmark datasets.

Abstract

Reasoning segmentation increasingly employs reinforcement learning to generate explanatory reasoning chains that guide Multimodal Large Language Models. While these geometric rewards are primarily confined to guiding the final localization, they are incapable of discriminating whether the reasoning process remains anchored on the referred region or strays into irrelevant context. Lacking this discriminative guidance, the model's reasoning often devolves into unfocused and verbose chains that ultimately fail to disambiguate and perceive the target in complex scenes. This suggests a need to complement the RL objective with Discriminative Perception, an ability to actively distinguish a target from its context. To realize this, we propose DPAD to compel the model to generate a descriptive caption of the referred object, which is then used to explicitly discriminate by contrasting the caption's semantic relevance to the referred object against the wider context. By optimizing for this discriminative capability, the model is forced to focus on the unique attributes of the target, leading to a more converged and efficient reasoning chain. The descriptive caption also serves as an interpretability rationale that aligns with the segmentation. Experiments on the benchmarks confirm the validity of our approach, delivering substantial performance gains, with the cIoU on ReasonSeg increasing by 3.09% and the reasoning chain length decreasing by approximately 42%. Code is available at https://github.com/mrazhou/DPAD