Dataset and Benchmark: Novel Sensors for Autonomous Vehicle Perception

TL;DR

This paper introduces the NSAVP dataset, featuring novel sensors like event and thermal cameras, to advance perception tasks in autonomous vehicles under challenging conditions, supported by benchmarking experiments.

Contribution

The paper presents the first dataset combining stereo thermal, event, and monochrome cameras for autonomous vehicle perception research.

Findings

Benchmarking reveals challenges in place recognition with novel sensors.

The dataset enables exploration of sensor fusion for improved perception.

Novel sensors show potential to address adverse weather and lighting conditions.

Abstract

Conventional cameras employed in autonomous vehicle (AV) systems support many perception tasks, but are challenged by low-light or high dynamic range scenes, adverse weather, and fast motion. Novel sensors, such as event and thermal cameras, offer capabilities with the potential to address these scenarios, but they remain to be fully exploited. This paper introduces the Novel Sensors for Autonomous Vehicle Perception (NSAVP) dataset to facilitate future research on this topic. The dataset was captured with a platform including stereo event, thermal, monochrome, and RGB cameras as well as a high precision navigation system providing ground truth poses. The data was collected by repeatedly driving two ~8 km routes and includes varied lighting conditions and opposing viewpoint perspectives. We provide benchmarking experiments on the task of place recognition to demonstrate challenges and opportunities for novel sensors to enhance critical AV perception tasks. To our knowledge, the NSAVP dataset is the first to include stereo thermal cameras together with stereo event and monochrome cameras. The dataset and supporting software suite is available at: https://umautobots.github.io/nsavp