ToFormer: Towards Large-scale Scenario Depth Completion for Lightweight ToF Camera

TL;DR

This paper introduces LASER-ToF, a large-scale dataset and a novel sensor-aware depth completion network that significantly improves depth estimation for ToF cameras in large environments, enabling better robotic mapping and planning.

Contribution

The paper presents the first large-scale ToF depth completion dataset and a new network architecture with 3D-2D fusion and SLAM integration for improved depth prediction.

Findings

Achieved 8.6% lower MAE than previous methods.

Demonstrated real-time deployment on a quadrotor at 10Hz.

Enabled reliable large-scale mapping and planning in challenging environments.

Abstract

Time-of-Flight (ToF) cameras possess compact design and high measurement precision to be applied to various robot tasks. However, their limited sensing range restricts deployment in large-scale scenarios. Depth completion has emerged as a potential solution to expand the sensing range of ToF cameras, but existing research lacks dedicated datasets and struggles to generalize to ToF measurements. In this paper, we propose a full-stack framework that enables depth completion in large-scale scenarios for short-range ToF cameras. First, we construct a multi-sensor platform with a reconstruction-based pipeline to collect real-world ToF samples with dense large-scale ground truth, yielding the first LArge-ScalE scenaRio ToF depth completion dataset (LASER-ToF). Second, we propose a sensor-aware depth completion network that incorporates a novel 3D branch with a 3D-2D Joint Propagation Pooling (JPP) module and Multimodal Cross-Covariance Attention (MXCA), enabling effective modeling of long-range relationships and efficient 3D-2D fusion under non-uniform ToF depth sparsity. Moreover, our network can utilize the sparse point cloud from visual SLAM as a supplement to ToF depth to further improve prediction accuracy. Experiments show that our method achieves an 8.6% lower mean absolute error than the second-best method, while maintaining lightweight design to support onboard deployment. Finally, to verify the system's applicability on real robots, we deploy proposed method on a quadrotor at a 10Hz runtime, enabling reliable large-scale mapping and long-range planning in challenging environments for short-range ToF cameras.