MIPI 2022 Challenge on RGB+ToF Depth Completion: Dataset and Report

TL;DR

This paper introduces the first MIPI challenge focusing on RGB+ToF depth completion, providing a new dataset and evaluating various algorithms for real-time indoor depth sensing.

Contribution

It presents a new dataset, TetrasRGBD, and benchmarks for RGB+ToF depth completion, fostering research in mobile imaging and sensor fusion.

Findings

Multiple algorithms evaluated on the dataset

Real-time performance achieved on desktop GPUs

Objective and subjective metrics used for assessment

Abstract

Developing and integrating advanced image sensors with novel algorithms in camera systems is prevalent with the increasing demand for computational photography and imaging on mobile platforms. However, the lack of high-quality data for research and the rare opportunity for in-depth exchange of views from industry and academia constrain the development of mobile intelligent photography and imaging (MIPI). To bridge the gap, we introduce the first MIPI challenge including five tracks focusing on novel image sensors and imaging algorithms. In this paper, RGB+ToF Depth Completion, one of the five tracks, working on the fusion of RGB sensor and ToF sensor (with spot illumination) is introduced. The participants were provided with a new dataset called TetrasRGBD, which contains 18k pairs of high-quality synthetic RGB+Depth training data and 2.3k pairs of testing data from mixed sources. All the data are collected in an indoor scenario. We require that the running time of all methods should be real-time on desktop GPUs. The final results are evaluated using objective metrics and Mean Opinion Score (MOS) subjectively. A detailed description of all models developed in this challenge is provided in this paper. More details of this challenge and the link to the dataset can be found at https://github.com/mipi-challenge/MIPI2022.