A Novel MLLM-based Approach for Autonomous Driving in Different Weather Conditions

TL;DR

This paper presents a comprehensive analysis of an autonomous driving agent powered by a Multi-modal Large Language Model (MLLM) using GPT-4o, demonstrating its robustness and adaptability in adverse weather and complex traffic scenarios within the CARLA simulator.

Contribution

The study introduces MLLM-AD-4o, a novel autonomous driving framework leveraging GPT-4o, and evaluates its performance under various environmental and perception configurations, highlighting its robustness in challenging conditions.

Findings

The AD agent maintains high safety and efficiency in harsh weather conditions.

Performance varies with different perception setups, including cameras and LiDAR.

MLLM integration enhances decision-making and perception in autonomous driving.

Abstract

Autonomous driving (AD) technology promises to revolutionize daily transportation by making it safer, more efficient, and more comfortable. Their role in reducing traffic accidents and improving mobility will be vital to the future of intelligent transportation systems. Autonomous driving in harsh environmental conditions presents significant challenges that demand robust and adaptive solutions and require more investigation. In this context, we present in this paper a comprehensive performance analysis of an autonomous driving agent leveraging the capabilities of a Multi-modal Large Language Model (MLLM) using GPT-4o within the LimSim++ framework that offers close loop interaction with the CARLA driving simulator. We call it MLLM-AD-4o. Our study evaluates the agent's decision-making, perception, and control under adverse conditions, including bad weather, poor visibility, and complex traffic scenarios. Our results demonstrate the AD agent's ability to maintain high levels of safety and efficiency, even in challenging environments, underscoring the potential of GPT-4o to enhance autonomous driving systems (ADS) in any environment condition. Moreover, we evaluate the performance of MLLM-AD-4o when different perception entities are used including either front cameras only, front and rear cameras, and when combined with LiDAR. The results of this work provide valuable insights into integrating MLLMs with AD frameworks, paving the way for future advancements in this field.