4Seasons: Benchmarking Visual SLAM and Long-Term Localization for Autonomous Driving in Challenging Conditions

TL;DR

This paper introduces 4Seasons, a comprehensive benchmark dataset for evaluating visual SLAM and long-term localization in autonomous driving under diverse and challenging environmental conditions, addressing a gap in real-world scenario testing.

Contribution

The paper presents a new large-scale, multi-environment benchmark dataset with accurate reference poses for evaluating visual SLAM and localization methods in challenging conditions.

Findings

State-of-the-art methods show varied performance across conditions.

The benchmark reveals strengths and weaknesses of current approaches.

Promising directions for future research are identified.

Abstract

In this paper, we present a novel visual SLAM and long-term localization benchmark for autonomous driving in challenging conditions based on the large-scale 4Seasons dataset. The proposed benchmark provides drastic appearance variations caused by seasonal changes and diverse weather and illumination conditions. While significant progress has been made in advancing visual SLAM on small-scale datasets with similar conditions, there is still a lack of unified benchmarks representative of real-world scenarios for autonomous driving. We introduce a new unified benchmark for jointly evaluating visual odometry, global place recognition, and map-based visual localization performance which is crucial to successfully enable autonomous driving in any condition. The data has been collected for more than one year, resulting in more than 300 km of recordings in nine different environments ranging from a multi-level parking garage to urban (including tunnels) to countryside and highway. We provide globally consistent reference poses with up to centimeter-level accuracy obtained from the fusion of direct stereo-inertial odometry with RTK GNSS. We evaluate the performance of several state-of-the-art visual odometry and visual localization baseline approaches on the benchmark and analyze their properties. The experimental results provide new insights into current approaches and show promising potential for future research. Our benchmark and evaluation protocols will be available at https://go.vision.in.tum.de/4seasons.