Multi-Modal Fusion-Based Multi-Task Semantic Communication System

TL;DR

This paper introduces a multi-modal, multi-task semantic communication framework that leverages BERT-based fusion to improve information understanding and reduce communication overhead in complex multi-modal tasks.

Contribution

It proposes a novel multi-modal fusion-based multi-task semantic communication system utilizing BERT for effective semantic information fusion across modalities.

Findings

MFMSC outperforms benchmarks in accuracy and efficiency.

The BERT-based fusion module enhances multi-modal semantic understanding.

Reduced communication overhead compared to traditional models.

Abstract

In recent years, there has been significant progress in semantic communication systems empowered by deep learning techniques. It has greatly improved the efficiency of information transmission. Nevertheless, traditional semantic communication models still face challenges, particularly due to their single-task and single-modal orientation. Many of these models are designed for specific tasks, which may result in limitations when applied to multi-task communication systems. Moreover, these models often overlook the correlations among different modal data in multi-modal tasks. It leads to an incomplete understanding of complex information, causing increased communication overhead and diminished performance. To address these problems, we propose a multi-modal fusion-based multi-task semantic communication (MFMSC) framework. In contrast to traditional semantic communication approaches, MFMSC can effectively handle various tasks across multiple modalities. Furthermore, we design a fusion module based on Bidirectional Encoder Representations from Transformers (BERT) for multi-modal semantic information fusion. By leveraging the powerful semantic understanding capabilities and self-attention mechanism of BERT, we achieve effective fusion of semantic information from different modalities. We compare our model with multiple benchmarks. Simulation results show that MFMSC outperforms these models in terms of both performance and communication overhead.