Tiny-WiFo: A Lightweight Wireless Foundation Model for Channel Prediction via Multi-Component Adaptive Knowledge Distillation

TL;DR

This paper introduces Tiny-WiFo, a lightweight wireless foundation model optimized for real-time channel prediction on edge devices, using a novel multi-component adaptive knowledge distillation framework.

Contribution

It presents a new knowledge distillation framework with a cross-attention-based feature selection and an autonomous learning strategy, enabling efficient model compression for wireless channel prediction.

Findings

Tiny-WiFo achieves 1.6 ms inference time on edge devices.

Retains over 98% of WiFo's performance and zero-shot capabilities.

Contains only 5.5 million parameters.

Abstract

The massive scale of Wireless Foundation Models (FMs) hinders their real-time deployment on edge devices. This letter moves beyond standard knowledge distillation by introducing a novel Multi-Component Adaptive Knowledge Distillation (MCAKD) framework. Key innovations include a Cross-Attention-Based Knowledge Selection (CA-KS) module that selectively identifies critical features from the teacher model, and an Autonomous Learning-Passive Learning (AL-PL) strategy that balances knowledge transfer with independent learning to achieve high training efficiency at a manageable computational cost. When applied to the WiFo FM, the distilled Tiny-WiFo model, with only 5.5M parameters, achieves a 1.6 ms inference time while retaining over 98% of WiFo's performance and its crucial zero-shot generalization capability, making real-time FM deployment viable.