BETTY Dataset: A Multi-modal Dataset for Full-Stack Autonomy

TL;DR

The BETTY dataset provides a comprehensive, multi-modal collection of autonomous racing vehicle data over four years, supporting advanced research in full-stack autonomous systems including perception, state estimation, and dynamics modeling.

Contribution

It introduces a large-scale, multi-modal dataset with diverse environments and dynamic scenarios, enabling development and evaluation of full-stack autonomous vehicle algorithms.

Findings

Contains over 13 hours and 32TB of data from 6 racing environments.

Includes high-speed dynamic events like crashes and tire traction loss.

Supports training and testing of full autonomy pipelines.

Abstract

We present the BETTY dataset, a large-scale, multi-modal dataset collected on several autonomous racing vehicles, targeting supervised and self-supervised state estimation, dynamics modeling, motion forecasting, perception, and more. Existing large-scale datasets, especially autonomous vehicle datasets, focus primarily on supervised perception, planning, and motion forecasting tasks. Our work enables multi-modal, data-driven methods by including all sensor inputs and the outputs from the software stack, along with semantic metadata and ground truth information. The dataset encompasses 4 years of data, currently comprising over 13 hours and 32TB, collected on autonomous racing vehicle platforms. This data spans 6 diverse racing environments, including high-speed oval courses, for single and multi-agent algorithm evaluation in feature-sparse scenarios, as well as high-speed road courses with high longitudinal and lateral accelerations and tight, GPS-denied environments. It captures highly dynamic states, such as 63 m/s crashes, loss of tire traction, and operation at the limit of stability. By offering a large breadth of cross-modal and dynamic data, the BETTY dataset enables the training and testing of full autonomy stack pipelines, pushing the performance of all algorithms to the limits. The current dataset is available at https://pitt-mit-iac.github.io/betty-dataset/.