M2BeamLLM: Multimodal Sensing-empowered mmWave Beam Prediction with Large Language Models

TL;DR

M2BeamLLM is a new neural network framework that uses multimodal sensor data and large language models to improve beam prediction accuracy and robustness in mmWave V2I communication systems.

Contribution

It introduces a novel multimodal sensing and LLM-based approach for beam prediction, outperforming traditional models in accuracy and robustness.

Findings

Significantly higher beam prediction accuracy.

Outperforms traditional deep learning models.

Performance improves with more sensing modalities.

Abstract

This paper introduces a novel neural network framework called M2BeamLLM for beam prediction in millimeter-wave (mmWave) massive multi-input multi-output (mMIMO) communication systems. M2BeamLLM integrates multi-modal sensor data, including images, radar, LiDAR, and GPS, leveraging the powerful reasoning capabilities of large language models (LLMs) such as GPT-2 for beam prediction. By combining sensing data encoding, multimodal alignment and fusion, and supervised fine-tuning (SFT), M2BeamLLM achieves significantly higher beam prediction accuracy and robustness, demonstrably outperforming traditional deep learning (DL) models in both standard and few-shot scenarios. Furthermore, its prediction performance consistently improves with increased diversity in sensing modalities. Our study provides an efficient and intelligent beam prediction solution for vehicle-to-infrastructure (V2I) mmWave communication systems.