Large-Scale Model Enabled Semantic Communication Based on Robust Knowledge Distillation

TL;DR

This paper introduces a robust knowledge distillation framework for semantic communication that enables large-scale models to operate efficiently and withstand channel noise, using lightweight architectures and specialized training techniques.

Contribution

It proposes a novel lightweight architecture search and a two-stage knowledge distillation method to improve efficiency and robustness of semantic communication systems.

Findings

Significantly reduces model parameters while maintaining performance.

Enhances robustness against channel noise in semantic communication.

Outperforms existing methods in image classification tasks.

Abstract

Large-scale models (LSMs) can be an effective framework for semantic representation and understanding, thereby providing a suitable tool for designing semantic communication (SC) systems. However, their direct deployment is often hindered by high computational complexity and resource requirements. In this paper, a novel robust knowledge distillation based semantic communication (RKD-SC) framework is proposed to enable efficient and \textcolor{black}{channel-noise-robust} LSM-powered SC. The framework addresses two key challenges: determining optimal compact model architectures and effectively transferring knowledge while maintaining robustness against channel noise. First, a knowledge distillation-based lightweight differentiable architecture search (KDL-DARTS) algorithm is proposed. This algorithm integrates knowledge distillation loss and a complexity penalty into the neural architecture search process to identify high-performance, lightweight semantic encoder architectures. Second, a novel two-stage robust knowledge distillation (RKD) algorithm is developed to transfer semantic capabilities from an LSM (teacher) to a compact encoder (student) and subsequently enhance system robustness. To further improve resilience to channel impairments, a channel-aware transformer (CAT) block is introduced as the channel codec, trained under diverse channel conditions with variable-length outputs. Extensive simulations on image classification tasks demonstrate that the RKD-SC framework significantly reduces model parameters while preserving a high degree of the teacher model's performance and exhibiting superior robustness compared to existing methods.