Action-Based Representation Learning for Autonomous Driving

TL;DR

This paper introduces an action-based representation learning approach for autonomous driving that improves interpretability and performance by leveraging human driving data to pre-train models, outperforming traditional end-to-end and supervised methods.

Contribution

The paper presents a novel action-based representation learning method that enhances autonomous driving models' interpretability and efficiency using weakly annotated data.

Findings

Pre-trained affordance-based models outperform pure end-to-end models.

The approach requires less annotated data and achieves better results.

It surpasses previous inverse dynamics and heavily supervised methods.

Abstract

Human drivers produce a vast amount of data which could, in principle, be used to improve autonomous driving systems. Unfortunately, seemingly straightforward approaches for creating end-to-end driving models that map sensor data directly into driving actions are problematic in terms of interpretability, and typically have significant difficulty dealing with spurious correlations. Alternatively, we propose to use this kind of action-based driving data for learning representations. Our experiments show that an affordance-based driving model pre-trained with this approach can leverage a relatively small amount of weakly annotated imagery and outperform pure end-to-end driving models, while being more interpretable. Further, we demonstrate how this strategy outperforms previous methods based on learning inverse dynamics models as well as other methods based on heavy human supervision (ImageNet).