Virtually increasing the measurement frequency of LIDAR sensor utilizing a single RGB camera

TL;DR

This paper proposes a method to virtually increase LIDAR measurement frequency using a single RGB camera, enabling more frequent dynamic object monitoring in vehicles.

Contribution

It introduces a novel approach combining camera tracking and LIDAR data projection to generate virtual LIDAR measurements at higher frame rates.

Findings

Achieves state-of-the-art accuracy on public datasets.

Effectively tracks dynamic objects with increased measurement frequency.

Improves real-time monitoring capabilities in intelligent vehicles.

Abstract

The frame rates of most 3D LIDAR sensors used in intelligent vehicles are substantially lower than current cameras installed in the same vehicle. This research suggests using a mono camera to virtually enhance the frame rate of LIDARs, allowing the more frequent monitoring of dynamic objects in the surroundings that move quickly. As a first step, dynamic object candidates are identified and tracked in the camera frames. Following that, the LIDAR measurement points of these items are found by clustering in the frustums of 2D bounding boxes. Projecting these to the camera and tracking them to the next camera frame can be used to create 3D-2D correspondences between different timesteps. These correspondences between the last LIDAR frame and the actual camera frame are used to solve the PnP (Perspective-n-Point) problem. Finally, the estimated transformations are applied to the previously measured points to generate virtual measurements. With the proposed estimation, if the ego movement is known, not just static object position can be determined at timesteps where camera measurement is available, but positions of dynamic objects as well. We achieve state-of-the-art performance on large public datasets in terms of accuracy and similarity to real measurements.