Knowledge Distillation-Empowered Digital Twin for Anomaly Detection

TL;DR

This paper introduces KDDT, a novel digital twin framework utilizing language models, LSTM, and knowledge distillation to enhance anomaly detection in train control systems, achieving high accuracy with limited data.

Contribution

The paper presents a new digital twin method combining language models, LSTM, and knowledge distillation for improved anomaly detection in cyber-physical systems.

Findings

KDDT achieved F1 scores of 0.931 and 0.915 on real industry datasets.

Knowledge distillation contributed an average 6.05% improvement in F1 score.

The combined approach outperforms individual components in anomaly detection accuracy.

Abstract

Cyber-physical systems (CPSs), like train control and management systems (TCMS), are becoming ubiquitous in critical infrastructures. As safety-critical systems, ensuring their dependability during operation is crucial. Digital twins (DTs) have been increasingly studied for this purpose owing to their capability of runtime monitoring and warning, prediction and detection of anomalies, etc. However, constructing a DT for anomaly detection in TCMS necessitates sufficient training data and extracting both chronological and context features with high quality. Hence, in this paper, we propose a novel method named KDDT for TCMS anomaly detection. KDDT harnesses a language model (LM) and a long short-term memory (LSTM) network to extract contexts and chronological features, respectively. To enrich data volume, KDDT benefits from out-of-domain data with knowledge distillation (KD). We evaluated KDDT with two datasets from our industry partner Alstom and obtained the F1 scores of 0.931 and 0.915, respectively, demonstrating the effectiveness of KDDT. We also explored individual contributions of the DT model, LM, and KD to the overall performance of KDDT, via a comprehensive empirical study, and observed average F1 score improvements of 12.4%, 3%, and 6.05%, respectively.