RClicks: Realistic Click Simulation for Benchmarking Interactive Segmentation

TL;DR

This paper introduces RClicks, a benchmark based on a large dataset of real-user clicks, to evaluate interactive segmentation methods more realistically, revealing that current models often underperform in real-world scenarios.

Contribution

The authors collected 475K real-user clicks, developed a clickability model, and established RClicks benchmark to assess interactive segmentation methods under realistic conditions.

Findings

Current models perform worse on real-user data than in baseline benchmarks.

Most interactive segmentation methods lack robustness to diverse click patterns.

RClicks provides a more accurate evaluation of real-world performance.

Abstract

The emergence of Segment Anything (SAM) sparked research interest in the field of interactive segmentation, especially in the context of image editing tasks and speeding up data annotation. Unlike common semantic segmentation, interactive segmentation methods allow users to directly influence their output through prompts (e.g. clicks). However, click patterns in real-world interactive segmentation scenarios remain largely unexplored. Most methods rely on the assumption that users would click in the center of the largest erroneous area. Nevertheless, recent studies show that this is not always the case. Thus, methods may have poor performance in real-world deployment despite high metrics in a baseline benchmark. To accurately simulate real-user clicks, we conducted a large crowdsourcing study of click patterns in an interactive segmentation scenario and collected 475K real-user clicks. Drawing on ideas from saliency tasks, we develop a clickability model that enables sampling clicks, which closely resemble actual user inputs. Using our model and dataset, we propose RClicks benchmark for a comprehensive comparison of existing interactive segmentation methods on realistic clicks. Specifically, we evaluate not only the average quality of methods, but also the robustness w.r.t. click patterns. According to our benchmark, in real-world usage interactive segmentation models may perform worse than it has been reported in the baseline benchmark, and most of the methods are not robust. We believe that RClicks is a significant step towards creating interactive segmentation methods that provide the best user experience in real-world cases.