Lightweight Neural Network with Knowledge Distillation for CSI Feedback

TL;DR

This paper proposes a knowledge distillation approach to develop lightweight neural networks for CSI feedback, significantly reducing complexity while maintaining high performance, thus enabling practical deployment on resource-limited devices.

Contribution

It introduces two novel knowledge distillation methods tailored for CSI feedback autoencoders, improving lightweight models' performance and generalization capabilities.

Findings

Student autoencoder performance improved significantly.

Inference time reduced to about 14% of teacher network.

Generalization across scenarios and environments enhanced.

Abstract

Deep learning has shown promise in enhancing channel state information (CSI) feedback. However, many studies indicate that better feedback performance often accompanies higher computational complexity. Pursuing better performance-complexity tradeoffs is crucial to facilitate practical deployment, especially on computation-limited devices, which may have to use lightweight autoencoder with unfavorable performance. To achieve this goal, this paper introduces knowledge distillation (KD) to achieve better tradeoffs, where knowledge from a complicated teacher autoencoder is transferred to a lightweight student autoencoder for performance improvement. Specifically, two methods are proposed for implementation. Firstly, an autoencoder KD-based method is introduced by training a student autoencoder to mimic the reconstructed CSI of a pretrained teacher autoencoder. Secondly, an encoder KD-based method is proposed to reduce training overhead by performing KD only on the student encoder. Additionally, a variant of encoder KD is introduced to protect user equipment and base station vendor intellectual property. Numerical simulations demonstrate that the proposed methods can significantly improve the student autoencoder's performance, while reducing the number of floating point operations and inference time to 3.05%-5.28% and 13.80%-14.76% of the teacher network, respectively. Furthermore, the variant encoder KD method effectively enhances the student autoencoder's generalization capability across different scenarios, environments, and bandwidths.