Uncertainty estimation for time series classification: Exploring predictive uncertainty in transformer-based models for variable stars

TL;DR

This paper enhances transformer-based models for classifying variable stars by integrating uncertainty estimation techniques, improving reliability and interpretability in astronomical light curve analysis.

Contribution

It introduces a hybrid uncertainty estimation method, HA-MC Dropout, for transformer models, demonstrating superior performance over existing techniques in variable star classification.

Findings

HA-MC Dropout achieves higher macro F1-scores across datasets.

Uncertainty estimates from HA-MC Dropout outperform other methods.

The approach improves reliability of automated astronomical classifications.

Abstract

Classifying variable stars is key for understanding stellar evolution and galactic dynamics. With the demands of large astronomical surveys, machine learning models, especially attention-based neural networks, have become the state-of-the-art. While achieving high accuracy is crucial, enhancing model interpretability and uncertainty estimation is equally important to ensure that insights are both reliable and comprehensible. We aim to enhance transformer-based models for classifying astronomical light curves by incorporating uncertainty estimation techniques to detect misclassified instances. We tested our methods on labeled datasets from MACHO, OGLE-III, and ATLAS, introducing a framework that significantly improves the reliability of automated classification for the next-generation surveys. We used Astromer, a transformer-based encoder designed for capturing representations of single-band light curves. We enhanced its capabilities by applying three methods for quantifying uncertainty: Monte Carlo Dropout (MC Dropout), Hierarchical Stochastic Attention (HSA), and a novel hybrid method combining both approaches, which we have named Hierarchical Attention with Monte Carlo Dropout (HA-MC Dropout). We compared these methods against a baseline of deep ensembles (DEs). To estimate uncertainty estimation scores for the misclassification task, we selected Sampled Maximum Probability (SMP), Probability Variance (PV), and Bayesian Active Learning by Disagreement (BALD) as uncertainty estimates. In predictive performance tests, HA-MC Dropout outperforms the baseline, achieving macro F1-scores of 79.8+-0.5 on OGLE, 84+-1.3 on ATLAS, and 76.6+-1.8 on MACHO. When comparing the PV score values, the quality of uncertainty estimation by HA-MC Dropout surpasses that of all other methods, with improvements of 2.5+-2.3 for MACHO, 3.3+-2.1 for ATLAS and 8.5+-1.6 for OGLE-III.