IAM: Enhancing RGB-D Instance Segmentation with New Benchmarks

TL;DR

This paper introduces three new RGB-D instance segmentation benchmarks, evaluates baseline models on them, and proposes a simple data integration method to improve scene understanding.

Contribution

It provides new instance-level RGB-D segmentation datasets, comprehensive baseline evaluations, and a novel data integration approach for enhanced scene analysis.

Findings

Baseline models show varied strengths and weaknesses on new benchmarks.

The proposed data integration method improves segmentation performance.

Extensive evaluations validate the effectiveness of the new approach.

Abstract

Image segmentation is a vital task for providing human assistance and enhancing autonomy in our daily lives. In particular, RGB-D segmentation-leveraging both visual and depth cues-has attracted increasing attention as it promises richer scene understanding than RGB-only methods. However, most existing efforts have primarily focused on semantic segmentation and thus leave a critical gap. There is a relative scarcity of instance-level RGB-D segmentation datasets, which restricts current methods to broad category distinctions rather than fully capturing the fine-grained details required for recognizing individual objects. To bridge this gap, we introduce three RGB-D instance segmentation benchmarks, distinguished at the instance level. These datasets are versatile, supporting a wide range of applications from indoor navigation to robotic manipulation. In addition, we present an extensive evaluation of various baseline models on these benchmarks. This comprehensive analysis identifies both their strengths and shortcomings, guiding future work toward more robust, generalizable solutions. Finally, we propose a simple yet effective method for RGB-D data integration. Extensive evaluations affirm the effectiveness of our approach, offering a robust framework for advancing toward more nuanced scene understanding.