ETA: Efficiency through Thinking Ahead, A Dual Approach to Self-Driving with Large Models

TL;DR

This paper introduces ETA, a dual-system approach that leverages large models for efficient, real-time self-driving decisions by shifting intensive computations to previous time steps and integrating features for prompt responses.

Contribution

ETA presents a novel asynchronous system that propagates past features and predicts future states to enable large models to respond quickly in self-driving applications.

Findings

Achieves 8% performance improvement on Bench2Drive CARLA benchmark

Maintains near-real-time inference speed at 50 ms

Advances state-of-the-art driving score with efficient large model use

Abstract

How can we benefit from large models without sacrificing inference speed, a common dilemma in self-driving systems? A prevalent solution is a dual-system architecture, employing a small model for rapid, reactive decisions and a larger model for slower but more informative analyses. Existing dual-system designs often implement parallel architectures where inference is either directly conducted using the large model at each current frame or retrieved from previously stored inference results. However, these works still struggle to enable large models for a timely response to every online frame. Our key insight is to shift intensive computations of the current frame to previous time steps and perform a batch inference of multiple time steps to make large models respond promptly to each time step. To achieve the shifting, we introduce Efficiency through Thinking Ahead (ETA), an asynchronous system designed to: (1) propagate informative features from the past to the current frame using future predictions from the large model, (2) extract current frame features using a small model for real-time responsiveness, and (3) integrate these dual features via an action mask mechanism that emphasizes action-critical image regions. Evaluated on the Bench2Drive CARLA Leaderboard-v2 benchmark, ETA advances state-of-the-art performance by 8% with a driving score of 69.53 while maintaining a near-real-time inference speed at 50 ms.