Dual-AEB: Synergizing Rule-Based and Multimodal Large Language Models for Effective Emergency Braking

TL;DR

Dual-AEB integrates multimodal large language models with rule-based systems to improve emergency braking in autonomous vehicles, combining comprehensive scene understanding with rapid response capabilities.

Contribution

This work introduces the first integration of multimodal large language models into AEB systems, enhancing open-scenario adaptability and safety.

Findings

Validated effectiveness through extensive experiments

First to incorporate MLLMs in AEB systems

Source code publicly available

Abstract

Automatic Emergency Braking (AEB) systems are a crucial component in ensuring the safety of passengers in autonomous vehicles. Conventional AEB systems primarily rely on closed-set perception modules to recognize traffic conditions and assess collision risks. To enhance the adaptability of AEB systems in open scenarios, we propose Dual-AEB, a system combines an advanced multimodal large language model (MLLM) for comprehensive scene understanding and a conventional rule-based rapid AEB to ensure quick response times. To the best of our knowledge, Dual-AEB is the first method to incorporate MLLMs within AEB systems. Through extensive experimentation, we have validated the effectiveness of our method. The source code will be available at https://github.com/ChipsICU/Dual-AEB.