Consistent Direct Time-of-Flight Video Depth Super-Resolution

TL;DR

This paper introduces a multi-frame fusion approach for super-resolving low-resolution direct time-of-flight (dToF) depth data using high-resolution RGB guidance, along with a new synthetic dataset for evaluation.

Contribution

It presents the first multi-frame fusion scheme for dToF depth super-resolution and introduces DyDToF, a large-scale synthetic dataset for dynamic indoor environments.

Findings

Multi-frame fusion reduces spatial ambiguity in dToF super-resolution.

Incorporating dToF-specific features improves depth reconstruction.

The proposed method outperforms single-frame approaches on the DyDToF dataset.

Abstract

Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, limited by manufacturing capabilities in a compact module, the dToF data has a low spatial resolution (e.g., $\sim 20\times30$ for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches, which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate spatial ambiguity. To evaluate our models on complex dynamic indoor environments and to provide a large-scale dToF sensor dataset, we introduce DyDToF, the first synthetic RGB-dToF video dataset that features dynamic objects and a realistic dToF simulator following the physical imaging process. We believe the methods and dataset are beneficial to a broad community as dToF depth sensing is becoming mainstream on mobile devices. Our code and data are publicly available: https://github.com/facebookresearch/DVSR/