Self-Supervised Enhancement for Depth from a Lightweight ToF Sensor with Monocular Images

TL;DR

This paper introduces SelfToF, a self-supervised framework that enhances low-resolution ToF depth maps using monocular images, incorporating a scale-recovery module and robustness to varying sparsity levels, validated on NYU and ScanNet datasets.

Contribution

The paper proposes a novel self-supervised learning framework, SelfToF, for depth enhancement from lightweight ToF sensors using monocular images, including a scale-recovery module and sparsity robustness.

Findings

SelfToF significantly improves depth map quality from ToF sensors.

SelfToF* maintains performance across different ToF sparsity levels.

Extensive experiments demonstrate the method's efficiency and effectiveness.

Abstract

Depth map enhancement using paired high-resolution RGB images offers a cost-effective solution for improving low-resolution depth data from lightweight ToF sensors. Nevertheless, naively adopting a depth estimation pipeline to fuse the two modalities requires groundtruth depth maps for supervision. To address this, we propose a self-supervised learning framework, SelfToF, which generates detailed and scale-aware depth maps. Starting from an image-based self-supervised depth estimation pipeline, we add low-resolution depth as inputs, design a new depth consistency loss, propose a scale-recovery module, and finally obtain a large performance boost. Furthermore, since the ToF signal sparsity varies in real-world applications, we upgrade SelfToF to SelfToF* with submanifold convolution and guided feature fusion. Consequently, SelfToF* maintain robust performance across varying sparsity levels in ToF data. Overall, our proposed method is both efficient and effective, as verified by extensive experiments on the NYU and ScanNet datasets. The code is available at \href{https://github.com/denyingmxd/selftof}{https://github.com/denyingmxd/selftof}.