Lightweight Task-Oriented Semantic Communication Empowered by Large-Scale AI Models

TL;DR

This paper introduces a fast knowledge distillation approach for task-oriented semantic communication that reduces model complexity and latency, while dynamically adapting to channel conditions using large-scale AI models.

Contribution

It proposes a novel fast distillation method with a pre-stored compression mechanism and a channel adaptive module, improving efficiency and reliability in semantic communication.

Findings

Outperforms baselines in task accuracy

Reduces model size and computation latency

Requires less training data

Abstract

Recent studies have focused on leveraging large-scale artificial intelligence (LAI) models to improve semantic representation and compression capabilities. However, the substantial computational demands of LAI models pose significant challenges for real-time communication scenarios. To address this, this paper proposes utilizing knowledge distillation (KD) techniques to extract and condense knowledge from LAI models, effectively reducing model complexity and computation latency. Nevertheless, the inherent complexity of LAI models leads to prolonged inference times during distillation, while their lack of channel awareness compromises the distillation performance. These limitations make standard KD methods unsuitable for task-oriented semantic communication scenarios. To address these issues, we propose a fast distillation method featuring a pre-stored compression mechanism that eliminates the need for repetitive inference, significantly improving efficiency. Furthermore, a channel adaptive module is incorporated to dynamically adjust the transmitted semantic information based on varying channel conditions, enhancing communication reliability and adaptability. In addition, an information bottleneck-based loss function is derived to guide the fast distillation process. Simulation results verify that the proposed scheme outperform baselines in term of task accuracy, model size, computation latency, and training data requirements.