Impact of Pseudo Depth on Open World Object Segmentation with Minimal User Guidance

TL;DR

This paper explores how pseudo depth maps, generated by pretrained networks, can improve open world object segmentation, especially for unseen classes, with minimal user guidance, demonstrating significant generalization improvements.

Contribution

It introduces a method leveraging pseudo depth maps and minimal user input to enhance open world object segmentation, especially for unseen classes.

Findings

Depth-based segmentation outperforms RGB-only methods on unseen classes.

Using pseudo depth maps improves IoU scores from 61.57 to 69.79 on unseen classes.

The approach generalizes well with minimal user guidance and partial class training.

Abstract

Pseudo depth maps are depth map predicitions which are used as ground truth during training. In this paper we leverage pseudo depth maps in order to segment objects of classes that have never been seen during training. This renders our object segmentation task an open world task. The pseudo depth maps are generated using pretrained networks, which have either been trained with the full intention to generalize to downstream tasks (LeRes and MiDaS), or which have been trained in an unsupervised fashion on video sequences (MonodepthV2). In order to tell our network which object to segment, we provide the network with a single click on the object's surface on the pseudo depth map of the image as input. We test our approach on two different scenarios: One without the RGB image and one where the RGB image is part of the input. Our results demonstrate a considerably better generalization performance from seen to unseen object types when depth is used. On the Semantic Boundaries Dataset we achieve an improvement from $61.57$ to $69.79$ IoU score on unseen classes, when only using half of the training classes during training and performing the segmentation on depth maps only.