How to Train Your Flare Prediction Model: Revisiting Robust Sampling of Rare Events

TL;DR

This paper investigates solar flare prediction using multivariate time series data, emphasizing the importance of proper sampling, handling class imbalance, and understanding temporal coherence to improve model performance.

Contribution

It revisits methods for addressing class imbalance and temporal coherence in solar flare forecasting, providing experimental insights and discussing the impact of data manipulation and evaluation metrics.

Findings

Proper handling of temporal coherence is crucial for realistic performance assessment.

Class imbalance significantly affects model training and evaluation in flare prediction.

Time series forecasting offers advantages over point-in-time methods when challenges are properly addressed.

Abstract

We present a case study of solar flare forecasting by means of metadata feature time series, by treating it as a prominent class-imbalance and temporally coherent problem. Taking full advantage of pre-flare time series in solar active regions is made possible via the Space Weather Analytics for Solar Flares (SWAN-SF) benchmark dataset; a partitioned collection of multivariate time series of active region properties comprising 4075 regions and spanning over 9 years of the Solar Dynamics Observatory (SDO) period of operations. We showcase the general concept of temporal coherence triggered by the demand of continuity in time series forecasting and show that lack of proper understanding of this effect may spuriously enhance models' performance. We further address another well-known challenge in rare event prediction, namely, the class-imbalance issue. The SWAN-SF is an appropriate dataset for this, with a 60:1 imbalance ratio for GOES M- and X-class flares and a 800:1 for X-class flares against flare-quiet instances. We revisit the main remedies for these challenges and present several experiments to illustrate the exact impact that each of these remedies may have on performance. Moreover, we acknowledge that some basic data manipulation tasks such as data normalization and cross validation may also impact the performance -- we discuss these problems as well. In this framework we also review the primary advantages and disadvantages of using true skill statistic and Heidke skill score, as two widely used performance verification metrics for the flare forecasting task. In conclusion, we show and advocate for the benefits of time series vs. point-in-time forecasting, provided that the above challenges are measurably and quantitatively addressed.