Verti-Bench: A General and Scalable Off-Road Mobility Benchmark for Vertically Challenging Terrain

TL;DR

Verti-Bench is a comprehensive, scalable benchmark for evaluating off-road mobility in extremely rugged and vertically challenging terrains, using diverse environments, tasks, and simulation data.

Contribution

It introduces a new standardized benchmark with diverse terrains and tasks, scalable to various vehicle platforms, and includes datasets and tools for reinforcement learning research.

Findings

Benchmarking reveals strengths and weaknesses of current off-road mobility systems.

Provides insights into vehicle-terrain interactions and failure modes.

Establishes a foundation for future off-road mobility research.

Abstract

Recent advancement in off-road autonomy has shown promises in deploying autonomous mobile robots in outdoor off-road environments. Encouraging results have been reported from both simulated and real-world experiments. However, unlike evaluating off-road perception tasks on static datasets, benchmarking off-road mobility still faces significant challenges due to a variety of factors, including variations in vehicle platforms and terrain properties. Furthermore, different vehicle-terrain interactions need to be unfolded during mobility evaluation, which requires the mobility systems to interact with the environments instead of comparing against a pre-collected dataset. In this paper, we present Verti-Bench, a mobility benchmark that focuses on extremely rugged, vertically challenging off-road environments. 100 unique off-road environments and 1000 distinct navigation tasks with millions of off-road terrain properties, including a variety of geometry and semantics, rigid and deformable surfaces, and large natural obstacles, provide standardized and objective evaluation in high-fidelity multi-physics simulation. Verti-Bench is also scalable to various vehicle platforms with different scales and actuation mechanisms. We also provide datasets from expert demonstration, random exploration, failure cases (rolling over and getting stuck), as well as a gym-like interface for reinforcement learning. We use Verti-Bench to benchmark ten off-road mobility systems, present our findings, and identify future off-road mobility research directions.