Multimodal Deep Learning-Empowered Beam Prediction in Future THz ISAC Systems

TL;DR

This paper introduces a multimodal deep learning framework using a mixture-of-experts approach for proactive beam prediction in THz ISAC systems, enhancing accuracy and robustness in dynamic environments.

Contribution

It proposes a novel adaptive multimodal fusion method with a mixture-of-experts model that dynamically weights modalities based on their reliability.

Findings

Outperforms static fusion and unimodal methods in prediction accuracy.

Demonstrates robustness and adaptability in vehicle-to-infrastructure scenarios.

Enhances beam alignment efficiency in THz communication systems.

Abstract

Integrated sensing and communication (ISAC) systems operating at terahertz (THz) bands are envisioned to enable both ultra-high data-rate communication and precise environmental awareness for next-generation wireless networks. However, the narrow width of THz beams makes them prone to misalignment and necessitates frequent beam prediction in dynamic environments. Multimodal sensing, which integrates complementary modalities such as camera images, positional data, and radar measurements, has recently emerged as a promising solution for proactive beam prediction. Nevertheless, existing multimodal approaches typically employ static fusion architectures that cannot adjust to varying modality reliability and contributions, thereby degrading predictive performance and robustness. To address this challenge, we propose a novel and efficient multimodal mixture-of-experts (MoE) deep learning framework for proactive beam prediction in THz ISAC systems. The proposed multimodal MoE framework employs multiple modality-specific expert networks to extract representative features from individual sensing modalities, and dynamically fuses them using adaptive weights generated by a gating network according to the instantaneous reliability of each modality. Simulation results in realistic vehicle-to-infrastructure (V2I) scenarios demonstrate that the proposed MoE framework outperforms traditional static fusion methods and unimodal baselines in terms of prediction accuracy and adaptability, highlighting its potential in practical THz ISAC systems with ultra-massive multiple-input multiple-output (MIMO).