Addressing Out-of-Distribution Challenges in Image Semantic Communication Systems with Multi-modal Large Language Models

TL;DR

This paper introduces a novel framework using multi-modal large language models to improve semantic communication in wireless networks, especially addressing out-of-distribution challenges through Bayesian optimization and cooperative inference.

Contribution

It proposes a new 'Plan A - Plan B' framework and a Bayesian optimization scheme to enhance MLLM performance in semantic encoding and reconstruction tasks.

Findings

Significant improvement in semantic compression performance.

Effective filtering of irrelevant vocabulary during inference.

Enhanced image reconstruction reliability through cooperative MLLMs.

Abstract

Semantic communication is a promising technology for next-generation wireless networks. However, the out-of-distribution (OOD) problem, where a pre-trained machine learning (ML) model is applied to unseen tasks that are outside the distribution of its training data, may compromise the integrity of semantic compression. This paper explores the use of multi-modal large language models (MLLMs) to address the OOD issue in image semantic communication. We propose a novel "Plan A - Plan B" framework that leverages the broad knowledge and strong generalization ability of an MLLM to assist a conventional ML model when the latter encounters an OOD input in the semantic encoding process. Furthermore, we propose a Bayesian optimization scheme that reshapes the probability distribution of the MLLM's inference process based on the contextual information of the image. The optimization scheme significantly enhances the MLLM's performance in semantic compression by 1) filtering out irrelevant vocabulary in the original MLLM output; and 2) using contextual similarities between prospective answers of the MLLM and the background information as prior knowledge to modify the MLLM's probability distribution during inference. Further, at the receiver side of the communication system, we put forth a "generate-criticize" framework that utilizes the cooperation of multiple MLLMs to enhance the reliability of image reconstruction.