Does SAM dream of EIG? Characterizing Interactive Segmenter Performance using Expected Information Gain

TL;DR

This paper proposes a new assessment method for interactive segmentation models based on Bayesian Experimental Design, revealing limitations of existing metrics and demonstrating its effectiveness on multiple models and datasets.

Contribution

It introduces a novel evaluation procedure for interactive segmentation that better captures model understanding of prompts, addressing shortcomings of traditional metrics.

Findings

Oracle Dice index can be misleading for measuring understanding.

The proposed method effectively evaluates models across datasets.

Different models show varying performance with the new assessment.

Abstract

We introduce an assessment procedure for interactive segmentation models. Based on concepts from Bayesian Experimental Design, the procedure measures a model's understanding of point prompts and their correspondence with the desired segmentation mask. We show that Oracle Dice index measurements are insensitive or even misleading in measuring this property. We demonstrate the use of the proposed procedure on three interactive segmentation models and subsets of two large image segmentation datasets.