Hallucinating Dense Optical Flow from Sparse Lidar for Autonomous Vehicles

TL;DR

This paper introduces a novel method to estimate dense optical flow from sparse lidar data, providing a reliable alternative to image-based methods especially in adverse conditions, using a new dataset and a multiscale neural network architecture.

Contribution

The paper presents a new neural network architecture and a dataset for estimating dense optical flow from sparse lidar data, enabling effective flow estimation without relying on images.

Findings

Achieved comparable dense optical flow accuracy to image-based methods on Kitti.

Developed a multiscale filter architecture for lidar-based flow estimation.

Created a new dataset with 20K lidar samples and pseudo ground-truth flows.

Abstract

In this paper we propose a novel approach to estimate dense optical flow from sparse lidar data acquired on an autonomous vehicle. This is intended to be used as a drop-in replacement of any image-based optical flow system when images are not reliable due to e.g. adverse weather conditions or at night. In order to infer high resolution 2D flows from discrete range data we devise a three-block architecture of multiscale filters that combines multiple intermediate objectives, both in the lidar and image domain. To train this network we introduce a dataset with approximately 20K lidar samples of the Kitti dataset which we have augmented with a pseudo ground-truth image-based optical flow computed using FlowNet2. We demonstrate the effectiveness of our approach on Kitti, and show that despite using the low-resolution and sparse measurements of the lidar, we can regress dense optical flow maps which are at par with those estimated with image-based methods.