MemKD: Memory-Discrepancy Knowledge Distillation for Efficient Time Series Classification

TL;DR

MemKD introduces a novel knowledge distillation method that captures memory discrepancies in time series models, enabling the creation of compact yet high-performing models suitable for resource-constrained environments.

Contribution

We propose MemKD, a knowledge distillation framework specifically designed for time series models that accounts for memory retention discrepancies, improving efficiency and performance.

Findings

Reduces model size and memory usage by ~500 times.

Maintains comparable performance to large teacher models.

Outperforms existing KD methods on time series tasks.

Abstract

Deep learning models, particularly recurrent neural networks and their variants, such as long short-term memory, have significantly advanced time series data analysis. These models capture complex, sequential patterns in time series, enabling real-time assessments. However, their high computational complexity and large model sizes pose challenges for deployment in resource-constrained environments, such as wearable devices and edge computing platforms. Knowledge Distillation (KD) offers a solution by transferring knowledge from a large, complex model (teacher) to a smaller, more efficient model (student), thereby retaining high performance while reducing computational demands. Current KD methods, originally designed for computer vision tasks, neglect the unique temporal dependencies and memory retention characteristics of time series models. To this end, we propose a novel KD framework termed Memory-Discrepancy Knowledge Distillation (MemKD). MemKD leverages a specialized loss function to capture memory retention discrepancies between the teacher and student models across subsequences within time series data, ensuring that the student model effectively mimics the teacher model's behaviour. This approach facilitates the development of compact, high-performing recurrent neural networks suitable for real-time, time series analysis tasks. Our extensive experiments demonstrate that MemKD significantly outperforms state-of-the-art KD methods. It reduces parameter size and memory usage by approximately 500 times while maintaining comparable performance to the teacher model.