Energy-Aware Imitation Learning for Steering Prediction Using Events and Frames

TL;DR

This paper presents an energy-aware imitation learning framework for steering prediction in autonomous driving, combining event camera data and frames to improve accuracy under challenging conditions.

Contribution

It introduces an energy-driven cross-modality fusion module and an energy-aware decoder, advancing multi-modal fusion for safer steering predictions.

Findings

Outperforms state-of-the-art methods on DDD20 and DRFuser datasets.

Utilizes event cameras to enhance robustness in challenging lighting and motion conditions.

Demonstrates improved reliability and safety in steering predictions.

Abstract

In autonomous driving, relying solely on frame-based cameras can lead to inaccuracies caused by factors like long exposure times, high-speed motion, and challenging lighting conditions. To address these issues, we introduce a bio-inspired vision sensor known as the event camera. Unlike conventional cameras, event cameras capture sparse, asynchronous events that provide a complementary modality to mitigate these challenges. In this work, we propose an energy-aware imitation learning framework for steering prediction that leverages both events and frames. Specifically, we design an Energy-driven Cross-modality Fusion Module (ECFM) and an energy-aware decoder to produce reliable and safe predictions. Extensive experiments on two public real-world datasets, DDD20 and DRFuser, demonstrate that our method outperforms existing state-of-the-art (SOTA) approaches. The codes and trained models will be released upon acceptance.