VISTA 2.0: An Open, Data-driven Simulator for Multimodal Sensing and Policy Learning for Autonomous Vehicles

TL;DR

VISTA 2.0 is an open, data-driven simulation platform for autonomous vehicles that integrates multiple sensors, enabling robust policy learning and sim-to-real transfer using high-fidelity, real-world datasets.

Contribution

It introduces VISTA 2.0, a novel open-source simulator that combines diverse sensors and realistic data to improve autonomous vehicle policy training and transfer.

Findings

Policies trained in VISTA transfer effectively to real vehicles.

VISTA enables simulation of diverse sensor modalities and viewpoints.

Policies trained in VISTA show increased robustness compared to real-only training.

Abstract

Simulation has the potential to transform the development of robust algorithms for mobile agents deployed in safety-critical scenarios. However, the poor photorealism and lack of diverse sensor modalities of existing simulation engines remain key hurdles towards realizing this potential. Here, we present VISTA, an open source, data-driven simulator that integrates multiple types of sensors for autonomous vehicles. Using high fidelity, real-world datasets, VISTA represents and simulates RGB cameras, 3D LiDAR, and event-based cameras, enabling the rapid generation of novel viewpoints in simulation and thereby enriching the data available for policy learning with corner cases that are difficult to capture in the physical world. Using VISTA, we demonstrate the ability to train and test perception-to-control policies across each of the sensor types and showcase the power of this approach via deployment on a full scale autonomous vehicle. The policies learned in VISTA exhibit sim-to-real transfer without modification and greater robustness than those trained exclusively on real-world data.