Domain Decluttering: Simplifying Images to Mitigate Synthetic-Real Domain Shift and Improve Depth Estimation

TL;DR

This paper introduces ARC, an attention-based method that removes challenging real-world regions to better align synthetic and real images, significantly improving depth estimation accuracy in a realistic mixed-data setting.

Contribution

We propose an attention module that identifies and removes out-of-domain regions in real images, enhancing depth prediction when combining synthetic data with limited real data.

Findings

ARC outperforms state-of-the-art domain adaptation methods

Removing out-of-domain regions improves depth estimation accuracy

Visualized removed regions offer insights into the domain gap

Abstract

Leveraging synthetically rendered data offers great potential to improve monocular depth estimation and other geometric estimation tasks, but closing the synthetic-real domain gap is a non-trivial and important task. While much recent work has focused on unsupervised domain adaptation, we consider a more realistic scenario where a large amount of synthetic training data is supplemented by a small set of real images with ground-truth. In this setting, we find that existing domain translation approaches are difficult to train and offer little advantage over simple baselines that use a mix of real and synthetic data. A key failure mode is that real-world images contain novel objects and clutter not present in synthetic training. This high-level domain shift isn't handled by existing image translation models. Based on these observations, we develop an attention module that learns to identify and remove difficult out-of-domain regions in real images in order to improve depth prediction for a model trained primarily on synthetic data. We carry out extensive experiments to validate our attend-remove-complete approach (ARC) and find that it significantly outperforms state-of-the-art domain adaptation methods for depth prediction. Visualizing the removed regions provides interpretable insights into the synthetic-real domain gap.