Entropy-and-Channel-Aware Adaptive-Rate Semantic Communication with MLLM-Aided Feature Compensation

TL;DR

This paper introduces an adaptive semantic communication system that dynamically adjusts transmission rates based on channel conditions and content complexity, utilizing MLLMs for feature compensation to optimize resource use and performance.

Contribution

It presents a novel entropy-and-channel-aware adaptive rate control framework that leverages MLLMs for feature compensation, enabling finer-grained and efficient semantic communication over fading channels.

Findings

Achieves adaptive resource allocation based on channel quality.

Maintains high task performance with reduced resource usage.

Uses MLLMs for effective feature map and symbol recovery.

Abstract

Despite the transmission efficiency gains of semantic communication (SemCom) over traditional methods, most existing SemCom schemes still operate at a fixed transmission rate regardless of channel conditions and transmitted content, resulting in wasted resources in favorable channels and degraded performance in harsh channels. To address this issue, we propose a novel SemCom framework that incorporates an entropy-and-channel-aware adaptive rate control mechanism over MIMO Rayleigh fading channels. Specifically, we embed a joint representation of the channel state information (CSI) and the signal-to-noise ratio (SNR) into both the semantic encoder and decoder, thereby realizing channel-aware semantic coding and decoding. Moreover, the proposed method jointly exploits the CSI, the SNR, the feature maps, and their 2D entropy via two policy networks to selectively transmit only a subset of feature maps and, within each selected feature map, only a subset of symbols. Thereby, it achieves finer-grained adaptive rate control than existing methods. At the receiver, leveraging the strong visual understanding capability of multimodal large language models (MLLMs), we deploy the lightweight visual encoder (InternViT-300M) of the pre-trained InternVL3.5 model to compensate for discarded feature maps and symbols, and we fine-tune InternViT using low-rank adaptation (LoRA) for parameter-efficient training. Experimental results show that, with a carefully designed channel-aware loss function, our system automatically allocates more communication resources under poor channels to enhance task performance while reducing resource usage under favorable channels and maintaining high task performance.