Distilling Universal and Joint Knowledge for Cross-Domain Model Compression on Time Series Data

TL;DR

This paper introduces UNI-KD, an end-to-end framework for cross-domain time series model compression that transfers universal and joint knowledge via adversarial learning, improving deployment efficiency in resource-limited environments.

Contribution

The paper proposes a novel universal and joint knowledge distillation framework for cross-domain time series model compression, utilizing adversarial learning to transfer domain-invariant features and shared knowledge.

Findings

Outperforms state-of-the-art methods on four datasets

Effectively transfers universal feature knowledge across domains

Enhances model deployment in resource-constrained environments

Abstract

For many real-world time series tasks, the computational complexity of prevalent deep leaning models often hinders the deployment on resource-limited environments (e.g., smartphones). Moreover, due to the inevitable domain shift between model training (source) and deploying (target) stages, compressing those deep models under cross-domain scenarios becomes more challenging. Although some of existing works have already explored cross-domain knowledge distillation for model compression, they are either biased to source data or heavily tangled between source and target data. To this end, we design a novel end-to-end framework called Universal and joint knowledge distillation (UNI-KD) for cross-domain model compression. In particular, we propose to transfer both the universal feature-level knowledge across source and target domains and the joint logit-level knowledge shared by both domains from the teacher to the student model via an adversarial learning scheme. More specifically, a feature-domain discriminator is employed to align teacher's and student's representations for universal knowledge transfer. A data-domain discriminator is utilized to prioritize the domain-shared samples for joint knowledge transfer. Extensive experimental results on four time series datasets demonstrate the superiority of our proposed method over state-of-the-art (SOTA) benchmarks.