Pixel-Level Reasoning Segmentation via Multi-turn Conversations

TL;DR

This paper introduces a new pixel-level reasoning segmentation task based on multi-turn conversations, along with a dataset and a framework that outperform existing methods in fine-grained, interactive segmentation.

Contribution

The work presents a novel multi-turn conversational segmentation task, a new dataset PRIST, and the MIRAS framework that integrates pixel-level reasoning with multi-turn dialogue understanding.

Findings

MIRAS outperforms baseline methods in segmentation accuracy.

PRIST dataset contains 24k utterances from 8.3k scenarios.

Framework generates pixel-grounded explanations aligned with user intent.

Abstract

Existing visual perception systems focus on region-level segmentation in single-turn dialogues, relying on complex and explicit query instructions. Such systems cannot reason at the pixel level and comprehend dynamic user intent that changes over interaction. Our work tackles this issue by introducing a novel task, Pixel-level Reasoning Segmentation (Pixel-level RS) based on multi-turn conversations, tracking evolving user intent via multi-turn interactions for fine-grained segmentation. To establish a benchmark for this novel task, we build a Pixel-level ReasonIng Segmentation Dataset Based on Multi-Turn Conversations (PRIST), comprising 24k utterances from 8.3k multi-turn conversational scenarios with segmentation targets. Building on PRIST, we further propose MIRAS, a Multi-turn Interactive ReAsoning Segmentation framework, integrates pixel-level segmentation with robust multi-turn conversation understanding, generating pixel-grounded explanations aligned with user intent. The PRIST dataset and MIRSA framework fill the gap in pixel-level reasoning segmentation. Experimental results on the PRIST dataset demonstrate that our method outperforms current segmentation-specific baselines in terms of segmentation and LLM-based reasoning metrics. The code and data are available at: https://github.com/ccccai239/PixelRIST.