Monocular Depth Estimation and Segmentation for Transparent Object with Iterative Semantic and Geometric Fusion

TL;DR

This paper introduces a novel monocular framework that simultaneously performs depth estimation and segmentation of transparent objects using a single image, leveraging semantic-geometric fusion and iterative refinement to achieve state-of-the-art results.

Contribution

It is the first to jointly address depth estimation and segmentation of transparent objects with a single RGB image, utilizing a fusion module and iterative strategy for improved accuracy.

Findings

Outperforms existing methods by 38.8%-46.2% on synthetic and real datasets.

Effectively integrates multi-scale information between tasks.

Achieves clearer and more accurate predictions through iterative refinement.

Abstract

Transparent object perception is indispensable for numerous robotic tasks. However, accurately segmenting and estimating the depth of transparent objects remain challenging due to complex optical properties. Existing methods primarily delve into only one task using extra inputs or specialized sensors, neglecting the valuable interactions among tasks and the subsequent refinement process, leading to suboptimal and blurry predictions. To address these issues, we propose a monocular framework, which is the first to excel in both segmentation and depth estimation of transparent objects, with only a single-image input. Specifically, we devise a novel semantic and geometric fusion module, effectively integrating the multi-scale information between tasks. In addition, drawing inspiration from human perception of objects, we further incorporate an iterative strategy, which progressively refines initial features for clearer results. Experiments on two challenging synthetic and real-world datasets demonstrate that our model surpasses state-of-the-art monocular, stereo, and multi-view methods by a large margin of about 38.8%-46.2% with only a single RGB input. Codes and models are publicly available at https://github.com/L-J-Yuan/MODEST.