Deep Cost Ray Fusion for Sparse Depth Video Completion

TL;DR

This paper introduces RayFusion, a learning-based framework that fuses sequential cost volumes from multiple viewpoints using attention mechanisms, significantly improving sparse depth video completion.

Contribution

The novel RayFusion framework effectively leverages attention for cost volume fusion, outperforming existing methods with fewer parameters.

Findings

Outperforms state-of-the-art on KITTI, VOID, and ScanNetV2 datasets.

Uses fewer network parameters while maintaining high performance.

Demonstrates robustness across diverse indoor and outdoor scenes.

Abstract

In this paper, we present a learning-based framework for sparse depth video completion. Given a sparse depth map and a color image at a certain viewpoint, our approach makes a cost volume that is constructed on depth hypothesis planes. To effectively fuse sequential cost volumes of the multiple viewpoints for improved depth completion, we introduce a learning-based cost volume fusion framework, namely RayFusion, that effectively leverages the attention mechanism for each pair of overlapped rays in adjacent cost volumes. As a result of leveraging feature statistics accumulated over time, our proposed framework consistently outperforms or rivals state-of-the-art approaches on diverse indoor and outdoor datasets, including the KITTI Depth Completion benchmark, VOID Depth Completion benchmark, and ScanNetV2 dataset, using much fewer network parameters.