Segmenting Object Affordances: Reproducibility and Sensitivity to Scale

TL;DR

This paper benchmarks affordance segmentation methods in reproducible setups, demonstrating Mask2Former as the top performer and highlighting models' sensitivity to scale variations in object resolution.

Contribution

It provides a reproducible benchmarking framework for affordance segmentation and evaluates a recent architecture, Mask2Former, showing its superior performance.

Findings

Mask2Former outperforms other models on most test sets.

Models lack robustness to scale variations in object resolution.

Benchmarking setup enhances fair comparison of affordance segmentation methods.

Abstract

Visual affordance segmentation identifies image regions of an object an agent can interact with. Existing methods re-use and adapt learning-based architectures for semantic segmentation to the affordance segmentation task and evaluate on small-size datasets. However, experimental setups are often not reproducible, thus leading to unfair and inconsistent comparisons. In this work, we benchmark these methods under a reproducible setup on two single objects scenarios, tabletop without occlusions and hand-held containers, to facilitate future comparisons. We include a version of a recent architecture, Mask2Former, re-trained for affordance segmentation and show that this model is the best-performing on most testing sets of both scenarios. Our analysis shows that models are not robust to scale variations when object resolutions differ from those in the training set.