Deep RGB-D Canonical Correlation Analysis For Sparse Depth Completion

TL;DR

This paper introduces CFCNet, a deep learning model that leverages correlated features between RGB images and sparse depth data to improve depth completion, extending canonical correlation analysis to 2D for better feature fusion.

Contribution

The paper presents a novel end-to-end deep network that incorporates 2D canonical correlation analysis for enhanced sparse depth completion from RGB-D data.

Findings

CFCNet outperforms state-of-the-art methods on indoor and outdoor datasets.

The 2D deep canonical correlation loss improves feature correlation and depth reconstruction.

The model demonstrates flexibility across different sparse depth patterns.

Abstract

In this paper, we propose our Correlation For Completion Network (CFCNet), an end-to-end deep learning model that uses the correlation between two data sources to perform sparse depth completion. CFCNet learns to capture, to the largest extent, the semantically correlated features between RGB and depth information. Through pairs of image pixels and the visible measurements in a sparse depth map, CFCNet facilitates feature-level mutual transformation of different data sources. Such a transformation enables CFCNet to predict features and reconstruct data of missing depth measurements according to their corresponding, transformed RGB features. We extend canonical correlation analysis to a 2D domain and formulate it as one of our training objectives (i.e. 2d deep canonical correlation, or "2D2CCA loss"). Extensive experiments validate the ability and flexibility of our CFCNet compared to the state-of-the-art methods on both indoor and outdoor scenes with different real-life sparse patterns. Codes are available at: https://github.com/choyingw/CFCNet.