Toward Reliable Benchmarking of Solar Flare Forecasting Methods

TL;DR

This paper evaluates the reliability of solar flare forecasting methods by comparing Poisson probability models with other systems, emphasizing the importance of proper verification metrics like TSS for assessing forecast accuracy.

Contribution

It introduces a standardized approach using Poisson probabilities and advocates for the true skill statistic (TSS) as a benchmark for solar flare forecast evaluation.

Findings

Poisson probabilities perform comparably to complex models.

Maximum TSS values vary across flare classes, indicating forecast difficulty.

Using average flare rates has limitations in representing flaring probability distributions.

Abstract

Solar flares occur in complex sunspot groups, but it remains unclear how the probability of producing a flare of a given magnitude relates to the characteristics of the sunspot group. Here, we use Geostationary Operational Environmental Satellite X-ray flares and McIntosh group classifications from solar cycles 21 and 22 to calculate average flare rates for each McIntosh class and use these to determine Poisson probabilities for different flare magnitudes. Forecast verification measures are studied to find optimum thresholds to convert Poisson flare probabilities into yes/no predictions of cycle 23 flares. A case is presented to adopt the true skill statistic (TSS) as a standard for forecast comparison over the commonly used Heidke skill score (HSS). In predicting flares over 24 hr, the maximum values of TSS achieved are 0.44 (C-class), 0.53 (M-class), 0.74 (X-class), 0.54 (>=M1.0), and 0.46 (>=C1.0). The maximum values of HSS are 0.38 (C-class), 0.27 (M-class), 0.14 (X-class), 0.28 (>=M1.0), and 0.41 (>=C1.0). These show that Poisson probabilities perform comparably to some more complex prediction systems, but the overall inaccuracy highlights the problem with using average values to represent flaring rate distributions.