CommGPT: A Graph and Retrieval-Augmented Multimodal Communication Foundation Model

TL;DR

CommGPT is a specialized multimodal foundation model for communication that integrates knowledge graphs and retrieval mechanisms to enhance understanding and knowledge access, addressing data, modality, and retrieval challenges in communication AI.

Contribution

The paper introduces CommGPT, a novel multimodal foundation model with a graph and retrieval-augmented framework tailored for communication tasks, including new datasets and a multimodal encoder.

Findings

Effective multimodal understanding demonstrated

Enhanced knowledge retrieval capabilities shown

Improved communication task performance

Abstract

Large Language Models (LLMs) possess human-level cognitive and decision-making capabilities, making them a key technology for 6G. However, applying LLMs to the communication domain faces three major challenges: 1) Inadequate communication data; 2) Restricted input modalities; and 3) Difficulty in knowledge retrieval. To overcome these issues, we propose CommGPT, a multimodal foundation model designed specifically for communications. First, we create high-quality pretraining and fine-tuning datasets tailored in communication, enabling the LLM to engage in further pretraining and fine-tuning with communication concepts and knowledge. Then, we design a multimodal encoder to understand and process information from various input modalities. Next, we construct a Graph and Retrieval-Augmented Generation (GRG) framework, efficiently coupling Knowledge Graph (KG) with Retrieval-Augmented Generation (RAG) for multi-scale learning. Finally, we demonstrate the feasibility and effectiveness of the CommGPT through experimental validation.