Label Efficient Visual Abstractions for Autonomous Driving

TL;DR

This paper investigates how reducing annotation costs in semantic segmentation impacts the performance of behavior cloning agents in autonomous driving, revealing that high driving performance is achievable with significantly less labeled data.

Contribution

It systematically analyzes the trade-offs between annotation effort and driving performance using various segmentation-based visual abstractions, providing practical insights for label-efficient autonomous driving.

Findings

High driving performance with minimal annotation effort

Significant reduction in policy variance using visual abstractions

Trade-offs identified between class types, sample size, and granularity

Abstract

It is well known that semantic segmentation can be used as an effective intermediate representation for learning driving policies. However, the task of street scene semantic segmentation requires expensive annotations. Furthermore, segmentation algorithms are often trained irrespective of the actual driving task, using auxiliary image-space loss functions which are not guaranteed to maximize driving metrics such as safety or distance traveled per intervention. In this work, we seek to quantify the impact of reducing segmentation annotation costs on learned behavior cloning agents. We analyze several segmentation-based intermediate representations. We use these visual abstractions to systematically study the trade-off between annotation efficiency and driving performance, i.e., the types of classes labeled, the number of image samples used to learn the visual abstraction model, and their granularity (e.g., object masks vs. 2D bounding boxes). Our analysis uncovers several practical insights into how segmentation-based visual abstractions can be exploited in a more label efficient manner. Surprisingly, we find that state-of-the-art driving performance can be achieved with orders of magnitude reduction in annotation cost. Beyond label efficiency, we find several additional training benefits when leveraging visual abstractions, such as a significant reduction in the variance of the learned policy when compared to state-of-the-art end-to-end driving models.