Unsupervised Enhancement of Real-World Depth Images Using Tri-Cycle GAN

TL;DR

This paper introduces an improved unsupervised domain translation method using a Tri-Cycle GAN to enhance degraded real-world depth images from low-cost sensors, overcoming limitations of standard Cycle-GAN.

Contribution

The paper proposes a novel Tri-Cycle loss and modifications to Cycle-GAN, enabling effective unsupervised enhancement of depth images without ground-truth data.

Findings

Significant visual improvement in depth image quality.

Quantitative metrics show enhanced accuracy and detail.

Framework extends to asymmetric translation tasks.

Abstract

Low quality depth poses a considerable challenge to computer vision algorithms. In this work we aim to enhance highly degraded, real-world depth images acquired by a low-cost sensor, for which an analytical noise model is unavailable. In the absence of clean ground-truth, we approach the task as an unsupervised domain-translation between the low-quality sensor domain and a high-quality sensor domain, represented using two unpaired training sets. We employ the highly-successful Cycle-GAN to this task, but find it to perform poorly in this case. Identifying the sources of the failure, we introduce several modifications to the framework, including a larger generator architecture, depth-specific losses that take into account missing pixels, and a novel Tri-Cycle loss which promotes information-preservation while addressing the asymmetry between the domains. We show that the resulting framework dramatically improves over the original Cycle-GAN both visually and quantitatively, extending its applicability to more challenging and asymmetric translation tasks.