Consistent Direct Time-of-Flight Video Depth Super-Resolution
Zhanghao Sun, Wei Ye, Jinhui Xiong, Gyeongmin Choe, Jialiang Wang,, Shuochen Su, Rakesh Ranjan

TL;DR
This paper introduces a multi-frame fusion approach for super-resolving low-resolution direct time-of-flight (dToF) depth data using high-resolution RGB guidance, along with a new synthetic dataset for evaluation.
Contribution
It presents the first multi-frame fusion scheme for dToF depth super-resolution and introduces DyDToF, a large-scale synthetic dataset for dynamic indoor environments.
Findings
Multi-frame fusion reduces spatial ambiguity in dToF super-resolution.
Incorporating dToF-specific features improves depth reconstruction.
The proposed method outperforms single-frame approaches on the DyDToF dataset.
Abstract
Direct time-of-flight (dToF) sensors are promising for next-generation on-device 3D sensing. However, limited by manufacturing capabilities in a compact module, the dToF data has a low spatial resolution (e.g., for iPhone dToF), and it requires a super-resolution step before being passed to downstream tasks. In this paper, we solve this super-resolution problem by fusing the low-resolution dToF data with the corresponding high-resolution RGB guidance. Unlike the conventional RGB-guided depth enhancement approaches, which perform the fusion in a per-frame manner, we propose the first multi-frame fusion scheme to mitigate the spatial ambiguity resulting from the low-resolution dToF imaging. In addition, dToF sensors provide unique depth histogram information for each local patch, and we incorporate this dToF-specific feature in our network design to further alleviate…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Optical Sensing Technologies · Image Processing Techniques and Applications · Advanced Vision and Imaging
