Transformer autoencoder with local attention for sparse and irregular time series with application on risk estimation

TL;DR

This paper presents a Transformer Autoencoder with local attention tailored for sparse, irregular time series, demonstrating improved risk estimation in electrical power systems with real-world data.

Contribution

The proposed framework uniquely combines local attention in a Transformer Autoencoder to effectively handle sparse, irregular time series for risk estimation tasks.

Findings

Achieves high recall and precision in risk detection.

Produces highly discriminative latent features.

Outperforms existing state-of-the-art methods.

Abstract

This paper introduces a framework specifically designed for sparse and irregular time series {risk estimation}. It is based on a Transformer Autoencoder with local attention, which leverages the powerful pattern identification capabilities of transformers complemented by traditional data cleaning and normalization methods. It efficiently captures relevant patterns within irregular sequences suffering from sparse data collection, benefiting from the discriminative ability of the local attention mechanism. The proposed framework is applied to a real-world case study, on the risk estimation of non-technical losses in electrical power systems in a wide area in Greece. Non-technical losses in electrical power systems, primarily stemming from electricity theft, pose significant economic and operational challenges. Detecting these anomalies is particularly challenging due to the inherent sparse and irregular nature of real-world data collection practices. Traditional risk estimation methods struggle with effectively capturing long-range dependencies and robustly handling such data characteristics. We demonstrate that our approach effectively yields highly discriminative latent features, which results in more consistent risk estimation compared with existing state-of-the-art and widely used methods. It achieves high recall and precision, meeting the critical objectives of the problem. As such, our solution offers a robust and effective tool for risk detection in irregular time series datasets.