Uncertainties within Weather Regime definitions for the Euro-Atlantic sector in ERA5 and CMIP6

TL;DR

This study examines the uncertainties in defining Weather Regimes for the Euro-Atlantic sector, assessing their stability and suitability in climate models like ERA5 and CMIP6, especially under climate change impacts.

Contribution

It provides a comprehensive validation and sensitivity analysis of diverse Weather Regime definitions and evaluates their stability and applicability in climate impact assessments.

Findings

Regime persistence and occurrence are highly sensitive to methodological changes.

CMIP6 models show significant instability in replicating historical Weather Regimes.

Caution is needed when using Weather Regimes in impact models due to high uncertainty.

Abstract

Certain Weather Regimes (WR) are associated with a higher risk of energy shortages, i.e. Blocking regimes for European winters. However, there are many uncertainties tied to the implementation of WRs and associated risks in the energy sector. Especially the impact of climate change is unknown. We investigate these uncertainties by looking at three methodologically diverse Euro-Atlantic WR definitions. We carry out a thorough validation of these methods and analyse their methodological and spatio-temporal sensitivity using ERA5 data. Furthermore, we look into the suitability of CMIP6 models for WR based impact assessments. Our sensitivity assessment showed that the persistence and occurrence of regimes are sensitive to small changes in the methodology. We show that the training period used has a very significant impact on the persistence and occurrence of the regimes found. For both WR4 and WR7, this results in instability of the regime patterns. All CMIP6 models investigated show instability of the regimes. Meaning that the normalised distance between the CMIP6 model regimes and our baseline regimes exceeds 0.4 or are visually extremely dissimilar. Only the WR4 regimes clustered on historical CMIP6 model data consistently have a normalised distance to our baseline regimes smaller than 0.4 and are visually identifiable. The WR6 definition exceeds the normalised distance threshold for all investigated CMIP6 experiments. Though all CMIP6 model experiments clustered with the WR7 definition have a normalised distance to the baseline regimes below 0.4, visual inspection of the regimes indicates instability. Great caution should be taken when applying WR's in impact models for the energy sector, due to this large instability and uncertainties associated with WR definitions.

Tables

Table 1. Table 1 : The three Weather Regime (WR) definitions (WR4, WR6 & WR7) applied on piControl (piC) and historical (hist) data streams of seven CMIP6 models. Here it is assessed whether or not the expected regimes are found when the clustering algorithm is trained on the indicated data stream. For normalised distances < 0.4 <0.4 we consider the model to perform well enough to produce the expected regimes and the cells are coloured green . When the normalised distance is > 0.4 >0.4 the distances to the respective ERA5 reanalysis regimes are too large to be considered the same regimes, so there the model/data stream combination is coloured red . The blue cells indicate that for that model data stream the dataset was empty. Empty cells for the CESM2 data streams indicate that no successful clustering has been performed yet. For the UKESM1-0-LL and MPI-ESM1-2-LR no distance could be calculated in a reasonable time frame, but the clustering was performed with varying success.

	WR4 piC	WR4 hist	WR6 piC	WR6 hist	WR7 piC	WR7 hist
CMCC-CM2-SR5	0.430	0.361	0.437	0.585	0.286	0.319
CESM2	0.280	0.256
NorESM2-MM	0.214	0.217	0.444	0.540	0.349	0.280
NorESM2-LM	0.421	0.276	0.679	0.468	0.375	0.341
UKESM1-0-LL	0.230	0.372		0.556	0.239
MPI-ESM1-2-LR	0.552	0.303		0.447	0.475	0.368
EC-Earth3-Veg-LR	0.552	0.277	0.850	0.426	0.475	0.344

Table 2. Table S2 : CMIP6 models that are considered to assess the suitability of WR methodology on climate models and their relevant references.

CMIP6 Models
CMCC-CM2-SR5	[78]
CESM2	[85]
NorESM2-MM	[88]
NorESM2-LM	[88]
UKESM1-0-LL	[80]
MPI-ESM1-2-LR	[79]
EC-Earth3-Veg-LR	[97]

Table 3. Table S3 : Settings Weather Regime identification and assignment for 4 Weather Regimes (WR4) in [ 83 ] and settings used for model validation in this project.

	WR4 [83]	WR4 validation
Period	1979–2018	1979–2018
Months	December – February (DJF)	DJF
Region	90∘W-30∘E, 20∘-80∘N	90∘W-30∘E, 20∘-80∘N
Seasonal Cycle Removed	No	No
Times resolution	00:00UTC	00:00UTC
Grid Resolution	0.25∘	1∘
# EOFs	14	14
Low-pass filter	No	No
Assignment	Hard Assignment	Hard Assignment
Data	ERA5	ERA5

Table 4. Table S4 : Settings Weather Regime identification and assignment for 6 Weather Regimes (WR6) in [ 86 ] and settings used for model validation in this project.

	WR6 [86]	WR6 validation
Period	1979–2018	1979–2018
Months	December – March (DJFM)	DJFM
Region	90∘W-30∘E, 20∘-80∘N	90∘W-30∘E, 20∘-80∘N
Seasonal Cycle Removed	No	No
Times resolution	00:00UTC	00:00UTC
Grid Resolution	2.5∘	2.5∘
# EOFs	n.a.	n.a.
Low-pass filter	No	No
Assignment	Hard Assignment	Hard Assignment
Data	ERA5-Interim	ERA5

Table 5. Table S5 : Settings Weather Regime (WR) identification and assignment for WR7 in [ 73 ] and settings used for model validation in this project.

	WR7 [73]	WR7 validation
Period	1979–2015	1979–2015
Months	Full year	Full year
Region	80∘W-40∘E, 30∘-90∘N	80∘W-40∘E, 30∘-90∘N
Seasonal Cycle Removed	Yes, 30 day rolling window	Yes, monthly mean removed
Times resolution	00:00,06:00,12:00,18:00 UTC	00:00,06:00,12:00,18:00 UTC
Grid Resolution	1∘	1∘
# EOFs	7	7
Low-pass filter	10 days	No
Assignment	WR Index	Hard Assignment
Data	ERA5-Interim	ERA5

Table 6. Table S6 : Baseline settings used for the baseline regimes, found with ERA5 reanalysis data, with the goal of comparing regimes found with the ERA5 reanalysis data and regimes found with CMIP6 experiment data. The CMIP6 experiment data uses the same settings, with the exception of the period, there the entire available period is used.

Period	1971–2020
Grid resolution	$1^{\circ }$
Region	90∘W-30∘E, 20∘-80∘N
Time resolution	00:00UTC

Table 7. Table S7 : Names Weather Regimes used in this paper and their equivalent name in Grams(2017)

Name Weather Regime (WR)	Equivalent WR in Grams (2017)
North Atlantic Oscillation + (NAO+)	Zonal regime (ZO)
North Atlantic Oscillation - (NAO-)	Greenland Blocking (GL)
Scandinavian Blocking + (SB+)	Scandinavian Blocking (ScBL)
Scandinavian Blocking - (SB-)	Scandinavian Trough (ScTr)
Atlantic Ridge + (AR+)	Atlantic Ridge (AR)
Atlantic Ridge - (AR-)	Atlantic Trough (AT)
European Blocking (EB+)	European Blocking (EuBL)

Table 8. Table S8 : Occurrence (Occ) and persistence (Per) rates of the validation experiment for the WR4 definition compared to the values from [ 83 ] . Please note that no persistance of the WR4 was provided by the reference paper.

WR	Occ	Occ literature	Per (days)
NAO+	32.1%	33%	6.0
NAO-	19.4%	20%	6.4
SB+	27.6%	28%	4.8
AR+	21.0%	20%	4.1

Table 9. Table S9 : Occurrence (Occ) and persistence (Per) rates of the validation experiment for the WR6 definition compared to the available literature values from [ 86 ] .

WR	Occ	Occ literature	Per (days)	Per literature (days)
NAO+	18.1%	16.9%	4.1	4.0
NAO-	14.1%	15.5%	5.9	6.5
SB+	19.3%	19.6%	4.2	4.0
SB-	13.9%	16.1%	4.1	3.7
AR+	15.6%	15.6%	3.3	3.5
AR-	18.0%	16.3%	4.1	4.7

Table 10. Table S10 : Occurrence (Occ) and persistence (Per) rates of the validation experiment for the WR7 definition.

WR	Occ	Per(days)
NAO+	14.2%	3.1
NAO-	8.6%	3.6
SB+	16.8%	3.1
SB-	16.5%	2.9
AR+	10.8%	3.0
AR-	15.6%	2.7
EB+	17.5%	2.7

Full text

We investigate these uncertainties by looking at three methodologically diverse Euro-Atlantic WR definitions. We carry out a thorough validation of these methods and analyse their methodological and spatio-temporal sensitivity using ERA5 data. Furthermore, we look into the suitability of CMIP6 models for WR based impact assessments.

Our sensitivity assessment showed that the persistence and occurrence of regimes are sensitive to small changes in the methodology. We show that the training period used has a very significant impact on the persistence and occurrence of the regimes found. For both WR4 and WR7, this results in instability of the regime patterns.

All CMIP6 models investigated show instability of the regimes. Meaning that the normalised distance between the CMIP6 model regimes and our baseline regimes exceeds 0.4 or are visually extremely dissimilar. Only the WR4 regimes clustered on historical CMIP6 model data consistently have a normalised distance to our baseline regimes smaller than 0.4 and are visually identifiable. The WR6 definition exceeds the normalised distance threshold for all investigated CMIP6 experiments. Though all CMIP6 model experiments clustered with the WR7 definition have a normalised distance to the baseline regimes below 0.4, visual inspection of the regimes indicates instability.

Great caution should be taken when applying WR’s in impact models for the energy sector, due to this large instability and uncertainties associated with WR definitions.

1 Introduction

Human well-being is at risk due to Earth’s changing climate [100]. To mitigate greenhouse gas emissions, and their adverse effects on our climate, society is shifting towards renewable energy sources [94]. Although these are more sustainable, they are dependent on the ever changing weather and its associated uncertainties, such as the unknowns of climate variability [93, 90, 111].

Energy system planners and operators around the world guide this transition towards renewables. For their work, they need insightful knowledge on the weather-driven threats that might pose a risk to the adequacy of the electricity grid in the future [93, 95]. A critical situation in a power system where load cannot be met by demand may not always be due to an outage of a power plant. It can also be due to structural shortage of energy due to the specific weather conditions at that time. Shortages can be related to the synoptic-scale state of the atmosphere  [83, 77, 113, 107, 114]. For instance, weather driven shortages can be related to dunkelflautes [82, 109, 112]. A dunkelflaute is a period of little sun and little wind. Insightful knowledge could be gained by better understanding the synoptic-scale atmospheric circulation patterns, their natural variability and the impact of climate change on them. One tool for analysing these patterns is the framework of Weather Regimes (WR).

Weather Regimes are classifications of the meteorological variability into quasi-stationary, persistent and recurrent large-scale circulation patterns of the atmosphere [63]. Here we will focus on WRs in the Euro-Atlantic sector. Because the weather in the winter period in Europe is more persistent, European Weather Regimes are often defined for the (extended) winter period [63, 71, 86], although year-round definitions exist [73].

The framework of WR’s is a well-established technique within meteorology. Its roots dating back to the 1940’s, when the German weather service developed a classification system for synoptic circulation patterns over Europe [118]. These were used to understand the synoptic variability of the atmosphere, so they are not necessarily persistent patterns. Then, since the 1990’s several methods to cluster large scale patterns have been used to study circulation patterns and their impacts  [62, 60, 61]. Discussions about the ideal number of regimes have been extensive, as this has to be set a priori [86, 63, 64]. However, the same treatment has not been extended to other settings within the WR classification methods, such as the resolution of the grid, the period of the data and the investigated region.

Weather Regimes are now also used to investigate the impact of meteorology within the energy sector. These circulation patterns over Europe influence the generation of renewables and the energy demand in Europe, through their relation to wind and temperature anomalies. There are indicators of a direct relation between certain Weather Regimes and periods of low potential renewable energy generation with high demand. Especially a Scandinavian blocking pattern has been linked frequently to winter-time energy shortages in central-Europe [83, 77, 107]. Furthermore, we need to understand the impact of climate change that are relevant for future energy generation, especially because society moving towards the use of renewables. WR’s can be used as a tool for this. [69, 95, 98, 105]. While [96] show that the Atlantic jet-stream indicates that there is a trend towards less persistent blocking regimes in Europe under climate change, these results uses hybrid geopotential-jet regimes and this definition is rarely used in energy system impact assessments.

While the framework of WR’s has now taken root within the energy sector, the method often used for this is not challenged. WR definitions are not often compared to each other. The classic definition [63] is used without challenge while for applications, like the energy sector where they are used to steer large public investments, it is useful to understand uncertainties of each method. To properly integrate climate projections into energy system applications of WR’s, understanding the energy-meteorological uncertainty is required [95, 99, 111]. In these applications the uncertainty of the impact model itself is often overlooked. In our case, the propagated uncertainties caused by how the Weather Regimes are defined, when they are implemented into impact models [81].

In this paper we investigate the stability of WR definitions. Specifically to support the usability of a WR framework in energy sector impact assessments. For this we analyse the gaps in the knowledge of this framework by comparing the stability of three different Weather Regime definitions. We use ERA5 reanalysis data to identify Weather Regimes, which can provide us with an understanding of the inherent uncertainty of Weather Regime definitions. Furthermore, the suitability of WR-methods for climate change research is scrutinized by using data from the Coupled Model Intercomparison Project in phase 6 (CMIP6) multi-model ensemble.

This paper is structured as follows. In Section 2, we explain how we validated our usage of the established WR definitions, how the sensitivity of the definitions was tested and how the CMIP6 models were used for the suitability analysis. Then, in Section 3 we present the primary findings of our validation analysis, the sensitivity assessment of the WR definitions on ERA5 data and the application of the WR definitions on CMIP6 model data. In Section 4 we discuss the merit of our work in the larger context. In Section  5 we present our conclusions on what these results mean for energy impact models.

2 Method

As mentioned in the introduction, we need to understand the energy-meteorological uncertainties involved to determine the suitability of WR definitions for energy sector impact assessments[95, 111]. Therefore, we need to understand how WR’s are identified, why certain methodological choices are made within WR definitions, and how stable the regimes are with respect to these methodological choices. To support this assessment the methodology is split into four sections.

First, in Section 2.1 we explain how WR’s are characterized. Second, we introduce three different WR definitions, how we apply these definitions and what their main differences are in Section 2.2. Third, in Section 2.3 we discuss how we tested the sensitivity of the results to changes in methodology and varying parameters. Finally, we explain how the suitability of the WR framework is evaluated as an impact model for the energy sector in Section 2.4.

2.1 Framework of Weather Regimes

The framework of WR’s is based on identifying reoccurring persistent patterns that describe the low-frequency variability in highly variable (sub-)daily atmospheric data and assigning each point in the time series to one of these patterns. With both the identification and the assignment of the WR’s methodological choices have to be made that influence the identified patterns and the regime dynamics. In this research we use three different methods of WR identification to study their accuracy and sensitivity (Section 2.2) applied to the ERA5 reanalysis dataset (SI Section B).

The most common method for finding these reoccurring patterns consists of three steps. First, the Empirical Orthogonal Functions (EOFs) and corresponding time-series (principal components, PCs) of the wintertime $500hPa$ geopotential height (gph) anomaly data are computed. Second, a $k$-means clustering algorithm [63, 66] is applied to the first N PCs to separate the dataset in subsets, clusters, that are similar within, but different between them. Both the number of clusters $k$ and number of PCs have to be set beforehand. The more PCs are included, the more variance of the original dataset is retained. In the clustering procedure, each data point in the time series is assigned to a cluster based on the distance between the cluster centre and the data. This is what we call ’hard’ regime assignment. Occurrence then is calculated as the fraction of the number of days in each regime over the total number of days, whereas persistence is computed by counting the number of successive days in a regime and averaging those over the timeseries.

There are several alternative techniques for the assignment of data to the clusters, which are done after the clustering. For example, one could assign a data point to a certain WR if its WR-index, as described in [67], is largest and has been that for at least an x number of days. Another technique for regime assignment is Bayesian assignment, as described in  [104]. Here, the data is not assigned to one cluster or regime, but the probability that the data point belongs to a certain regime is calculated (for more details see Figure 1 of [104]). In this case, computing the occurrence is done by adding the regime probabilities for each regime over the whole time period and dividing by the length of the time series.

2.2 Weather Regime Definitions

To analyse the effect of the methodological choice, three definitions of Weather Regimes are investigated in this study. In the following we discuss each method, their most notable characteristics, and which WR’s are found when using these definitions. Further details and differences between our implementation and the original works can be found in SI Section D.

The three methodologically different methods we consider are the classic 4-regime definition [63], Falkena’s 6-regime definition [86], and Grams’ 7-regime definition [73]. Although each definition consider a different number of regimes, visually similar regime patterns are found across these definitions. Figure 1 shows the seven regime patterns that are found for the WR7 definition; the North Atlantic Oscillation in positive and negative phase (NAO+/-), the Atlantic Ridge in positive and negative phase (AR+/-), Scandinavian Blocking in positive and negative phase (SB+/-) and European Blocking (EB+). The regimes identified by the WR4 definition are NAO+, NAO-, AR+ and SB+. For WR6 these four regimes are found together with AR- and SB-.

The spatial patterns of the regime patterns do differ slightly between the different WR definitions (see Figure S1 and Figure S2). It is important to note that the names describing the WRs vary in literature, where here we standardised the naming based on the terminology used in [86]. In Table S7, the equivalent names that are used in [73] can be found.

The classic wintertime 4-Regime (WR4) definition is based on [63]. It is the most commonly used WR definition, so literature discussing this definition is extensive [65, 68, 74, e.g.] and the connection between these regimes and the energy sector has been well documented  [83, 77, 87, e.g.]. However, due to the widespread application of the four regimes, not one universal methodology has been established. Many factors between the identification methods used in literature differ, which can have an effect on the outcome (most notably the persistence). Here, the method described in [83] is followed. They use the first 14 PCs of the DJF Z500 data to identify the four WRs using k-means clustering.

For Falkena’s wintertime (DJFM) 6-regime definition (WR6), we use the method described in their initial work on weather regimes; [86]. This definition stands out as no EOFs are used, eliminating the practice of double filtering of the data and finding 6 regimes to be optimal instead of 4. Within the WR6 method the grid-point data is directly clustered into 6 clusters with a $k$-means clustering algorithm. Since this is a new method, its use is less established within energy-meteorology.

For Grams’ 7-regime definition (WR7) we aimed to replicate the method described in [73]. Where the previous two methods focus on the winter months, this method analyses the recurrent patterns throughout the year and is applicable year-round. This last aspect is ideal for energy sector applications and this methodology has been used in various energy sector impact assessments [73, 106]. In this method, first the Z500 anomaly data is obtained by subtracting the monthly mean (eliminating seasonal variations) after which the leading 7 PCs are clustered into 7 clusters. A significant methodological difference between this method and the other WR definitions, is that a 10-day low-pass filter was used in the original work. However, we found this to be too computationally heavy to use within the WR framework assessment presented here. Furthermore, this definition uses a ‘No regime’ designation, which we also have not been able to replicate.

For each of the three WR definitions, details on the implementation used here, differences with the original method and further literature are discussed in SI Section D. We validated the WRs for each definition by visually comparing the $500hPa$ gph anomaly fields of the clusters we found to the gph anomaly fields in the relevant literature for each WR definition. When the occurrence and persistence of the WR’s was given in the original work, we also compare those to our findings.

2.3 Sensitivity assessment of WR framework

The aim of this study is to analyse the gaps in knowledge, and the suitability, of the WR framework for energy sector impact assessments. For that, understanding the sensitivity to parameters inherent to the WR definition and data is vital. To assess this we study the effect of varying two types of parameters, WR definition and data parameters, on the occurrence, average persistence and stability of the regimes. Here, stability refers to the ability to identify the expected regimes. To verify this we use automatic matching of the regimes of each sensitivity experiment to those of the baseline. The process of this is explained in more detail in SI Section E.5. The steps of the data processing chain for the sensitivity analysis are visualised in the blue stream of Figure 2.

WR definition parameters that we have varied can be divided into two categories: First the number of EOFs. For WR4 and WR7 the number of EOFs used in the first step of calculating the regimes was varied between 7, 14 and 30 EOFs. This was done because using more EOFs means more of the variance of the dataset is being considered. So, understanding how the number of EOFs influences the persistence or occurrence of certain regimes, can inform which number would be best practice.

Second, the effect of changing the method of assignment, hard or Bayesian, is investigated. The assignment of each data point to the closest cluster is a straightforward technique, but it is not a realistic representation of the atmospheric circulation. Therefore, we also looked into the effect of using a probabilistic method of assignment.

Understanding how the specific data the regimes are clustered on affects the occurrence and persistence is necessary, because weather is extremely dynamic. If the results are highly dependent on the data, this implies instability within the method. Especially because the data-sources vary across methods. Data parameters that we have varied can be divided into two categories; time dependent parameters and space dependent parameters.

The time dependent parameters we consider are the time period, time resolution and which months are taken into account. We look at the time period to understand the stability of the regime definitions across decades. This is done with two different approaches: The first approach is to both cluster and assign the data on the same time period. Another approach we use is to cluster the data on the data from 1971–2020, the full available period, and then assign the clusters to data in the 30-year climatological periods 1971–2000, 1981–2010 and 1991–2020. We also research the effects of altering the time resolution, 6-hourly or daily data, to assess whether or not more data translates into a more accurate representation of reality. Furthermore, which winter months are used varies between the WR4 and WR6 definitions. So we investigate if there is a notable difference to persistence or occurrence of certain regimes if in addition to December, January and February being considered, March is included. During this process, the anomalies are calculated with the same method; for each day the average for that specific day is subtracted.

Space parameters that are investigated are the grid resolution and the region. Because grid resolution greatly affects computational costs, the effect of changing the grid resolution from $1^{\circ}$ to $2.5^{\circ}$ can vastly reduce these costs. Therefore, understanding if reducing the grid resolution has an influence on the quality of the results is important. Various definitions define the Euro-Atlantic region differently. So, we need to understand how defining the region differently affects the geographic location of low- and high-pressure areas within the regimes and how this affects the persistence and occurrence of the regimes. Therefore we vary the region of the data the regimes are clustered on between the region used in  [83] (90∘W-30∘E, 20∘-80∘N), the region used in  [73] (80∘W-40∘E, 30∘-90∘N), and three regions stated in SI Section F.2, to investigate the influence of different Western, Eastern and Northern bounds.

2.4 Sensitivity of WR framework within climate models

To assess the suitability of the WR framework for applications to impact models in the energy sector, we need to assess the stability of the framework when applied to climate model data that include projections of future scenarios. Therefore, we apply the framework of WR’s to climate model data to evaluate its applicability for energy sector impact assessments. To assess the impact of climate change, the aleatoric, epistemic, scenario, and as [81] described, the impact model uncertainty, needs to be assessed in the same manner. The aleatoric, or year-to-year, uncertainty is addressed by the variation of the time periods used (Section 2.3) and the uncertainty of the impact model by using three methodologically different WR definitions. So, to facilitate climate change impact assessments we need to address the emission scenario and the epistemic (e.g. the climate model) uncertainty.

Epistemic uncertainty is tested by applying each of the three WR definitions a sub-ensemble of the Coupled Model Intercomparison Project phase 6 (CMIP6). This process is visually represented in Figure 2 by the green arrows. Our sub-ensemble consisted of seven climate models chosen due their data availability covering the set of the simulations needed, an overview of their references is shown in Table S2. We refer to each combination of CMIP6 model and either piControl or historical data, as one CMIP6 experiment. To assess suitability of our sub-ensemble of CMIP6 models we applied the framework of WR’s and calculating the distance between the regimes found in a CMIP6 experiment and the regimes found in the ERA5 reanalysis data. For each of the WR definitions the standard settings, also used for the sensitivity tests, are used as provided in SI Section D.1, with a few exceptions. These exceptions are shown in Table S6 and we call these the ’baseline’ settings. The regimes from the reanalysis data that we will use to compare are also found by using the baseline settings, which we then call the baseline regimes. Then the regimes from the CMIP6 experiment are matched to the ERA5 regimes using the process described in SI Section E.5. When evaluating if a WR definition produced the expected regimes when clustered on CMIP6 model data, we consider it successful if the normalised distance between the baseline regimes and the CMIP6 experiment regimes is $<0.4$. This value is chosen because we want the WR definition to perform better than an empty array, which can produce the value of 0.475.

Emission scenario uncertainty can be tested by assessing the differences in occurrence and persistence of the regimes in different Shared Socio-Economic Pathways (SSP) data streams from the CMIP6 models. Establishing a difference in persistence and occurrence of regimes between historical and SSP126, SSP370 and SSP585 data streams could be an indication of a change due to climate change. Which in turn could inform impact models used in the energy sector. So, for our CMIP6 models ensemble the persistence and occurrence of each of the regimes found by all three WR definitions using Bayesian assignment are calculated. However, due to difficulties with reproducing the expected regimes, these results will not be discussed in this paper.

3 Results

Here we discuss our analysis of the weather regime framework. First, we discuss the results of our validation of the methods described in Section 2.2. Then we delve into our findings of the sensitivity of these methods to various assumptions for each of the definitions used. Finally, we discuss the results of the sensitivity of the WR framework within the context of climate change, where we trained the WR models on CMIP6 model data.

3.1 Validation of Weather Regime implementation

We performed a validation of our WR implementation by comparing the results produced by our WR definitions to the results of the corresponding original method. For each WR definition we will discuss how they perform in terms of reproducing the regimes, and where possible, compare the occurrence and persistence rates of the regimes to the corresponding literature values. For this the models were set with parameters as close to the original method as possible. It is not possible to perform statistical analysis on how close the regimes are to the original method, for any of the definitions, because the gph anomaly maps of the original regimes were not available to us.

The regimes found with our WR4 model closely resemble the regimes found by [83]. Figure S1 in the SI shows the four regimes found by using the WR4 model with the parameters shown in SI Table S3. Visually, the patterns are similar. Small differences in geography and intensity of the highs and lows can be identified, but no large discrepancies are present. Furthermore we found that, occurrence rates of weather regimes for the WR4 regimes are at most $1\%$ different from the rates found by  [83]. This can be seen in Table S8 in SI Section E.3. Persistence rates were not available in [83], so these cannot be compared.

The regimes found with our WR6 model do resemble the regimes found by [86]. Figure S2 in the SI shows the four regimes found by using the WR4 model with the parameters shown in SI Table S4. Visually, the patterns of the highs and lows are located over the same regions.

SI Table S9 in Section E.3 shows that our persistence rates for the WR6 definition are at most 0.6 days lower than the persistence rates found in [86]. Furthermore, this Table shows that the difference of occurrence rate of the regimes we calculate and the regimes from  [86] are within $2.2\%$.

Visually comparing the regimes found here to the regimes produced in [73], we see that the patterns of the highs and lows are in similar geographical positions. Attempts to recreate a ‘no regime’ designation failed, or produced vastly different rates of occurrence compared to [73]. So we decided to not include the ‘no regime’.

This lack of a ‘no regimes’ means a comparison of occurrence rates is not possible for the other regimes, as this rate is influenced by the occurrence of the ‘no regime’. Persistence rates were not given in [73]. The persistence and occurrence rates we found, are shown in SI Table S10.

3.2 Sensitivity of WR definitions

Within the energy sector the scale of the large public investments require a thorough understanding of the uncertainties in methods that steer them. Therefore we performed an analysis of the sensitivity of the WR framework to both methodological and data selection choices. As a result of our deep-dive into the original implementations and our tuning of the implementation within validation process we developed a robust methodology of three WR definitions, on which we have performed sensitivity tests. The complete analysis of sensitivity with additional findings and figures can be found in SI Section F.

The effect of altering the variables mentioned in Section 2.3, on the occurrence and persistence of Weather Regimes was tested. First we will discuss the results of the parameters that alter the WR definition itself: the number of EOFs and the method of assignment. Then, the effect of altering the input data by varying data parameters is investigated. There we will focus on the effect changing the period the clustering algorithm is trained on, because this causes the largest instabilities and uncertainties.

Figures that include the impact of the WR definition dependent parameters are included in SI Section F.1 for each WR definition.

Figure S5 shows that changing the number of EOFs does not have a large effect for the WR4 definition. However, for the WR7 definition, Figure S7 indicates that a larger number of EOFs increases the persistence for some regimes. The difference between 14 and 30 EOFs is small, so the largest difference is between 7 and 14 EOFs. This implies that the leading 7 EOFs do not capture all the relevant variance.

Changing the method of assignment from hard assignment to Bayesian assignment increases the persistence for all regimes for each WR definition. This change affects the occurrence rate per regime differently for each WR definition. For WR4 only a small change in occurrence can be identified, as seen in Figure S5. Figure S6 shows that for WR6, an increased occurrence for NAO+ and NAO- and a decreased occurrence for SB+ and AR+ can be identified. WR7 also shows an increased occurrence for NAO+ and NAO-, but a decreased occurrence for SB+, SB- and AR-, which is shown in Figure S7. Note that when calculating the persistence of the regimes while using Bayesian assignment, the WR with the highest probability is assigned to each data point.

Figures that include the impact of the data parameters, are included in SI Section F.2 for each WR definition: Figures S8, S9 and S10 for WR4, WR6 and WR7, respectively. Those figures show that grid resolution does not have a large effect on either persistence or occurrence.

Furthermore, the temporal resolution only affects the persistence, which is to be expected when using hard assignment. More data points means a larger chance to switch to a different regime faster. So, for all WR definitions and all regimes the persistence is decreased when switching from daily to 6-hourly data.

In addition to that, changing the winter months from December–February to include March, has a small, but noticeable effect for some regimes. For the WR4 definition, NAO- has a higher occurrence and longer persistence when March is included and SB+ has a lower occurrence, as can be seen in Figure S8. However, Figure S9 shows that for the WR6 definition, the difference for NAO- is negligible when March is excluded. Here the regimes SB+, SB- and AR- have the most notable shift in occurrence and/or persistence.

Moreover, the extent of the region does affect the persistence and occurrence of each of the regimes for all WR definitions. This is explained further in SI Section F.2 and can be seen in Figures S8, S9 and S10, for WR4, WR6 and WR7, respectively.

The largest effects can be found in altering the period. So, we will now focus on how the models are affected by changing the period of the data the models cluster the regimes on. Figure 3 shows the occurrence and persistence of the WR4 definition trained on ERA5 data, where the clustering algorithm is trained on different time periods. The reference period, 1979–2018, is the period used in both  [83] and  [86]. We call 1971–2020 the baseline period. This period was chosen, because it is a longer, presumably more stable, climatological period. A distinction is made between training the clustering algorithm on a specific period and assigning those clusters to that period, or training the clustering algorithm on the period 1971–2020 and only assigning the clusters on a specific period. There is a large spread for all four regimes in both occurrence and persistence, depending on training and/or assigning the regimes on different time periods. Special attention needs to be given to the result of training the clustering algorithm on data of the period 1991–2020. A visualization of the patterns found here is shown in Figure 4. There we can see that no Atlantic Ridge regime can be identified. Instead, a regime resembling its negative phase is found. Linked to this change, the NAO+ is notably different from previously found patterns.

In addition to the instability in time for the WR4 definition, both the WR6 and WR7 definitions are strongly affected by the period in which the clustering algorithm is trained. Even though all expected WR’s can be identified, for the WR6 definition, a large spread in occurrence and/or persistence is found, depending on the period considered. This can be seen in Figure S11 in SI Section F.2.

Furthermore, the maximum anomalies of the WR7 Weather Regimes shift visibly, based on different time period the regimes were clustered on. Figure S12 shows the patterns of the regimes that are found by clustering the data from the period 1991–2020. There it is can be seen that the found regimes are different from the reference regimes. This is especially the case for the AR- regime, where the maximum gph anomaly has shifted from the Atlantic ocean to lying over Great Britain. Due to a lack of computational power, it was not possible to train the model on the period 1971–2020 and assign the regimes to different time periods for the WR7 definition.

3.3 Sensitivity of WR framework within climate models

To assess the suitability of the WR framework for applications to impact models in the energy sector, we need to assess the stability of the framework when applied to climate model data that include projections of future scenarios. With stability we mean that the WR definitions reliably produce the expected regimes for different types of input data. As described in Section 2.4, we used a sub-ensemble of CMIP6, and standard parameters within the WR framework.

Instability of the WR definitions is found when applying the three definitions on various CMIP6 models. Whether or not the expected Weather Regimes are found depends on which model and data set (piControl or historical) is used. Table 1 shows for which combinations of CMIP6 models, data set and WR definition, the expected Weather Regimes can be identified. For normalised distances $<0.4$ we consider the model to perform well enough to produce the expected regimes and the cells are coloured green. When the normalised distance is $>0.4$ the distances to the respective ERA5 reanalysis baseline regimes are too large to be considered the same regimes, so there the model/data set combination is coloured red. Blue cells show data streams that were empty. Furthermore, white cells indicate that no successful clustering has been performed yet and/or the distance between the ERA5 reanalysis data and the model output has not been calculated.111In the case of MPI-ESM1-2-LR piControl, the data set was empty, so no distance was calculated.

WR4 seems to be relatively stable when applied to the CMIP6 model data. Of the twelve CMIP6 experiments, ten produce the expected regimes. For WR6, nine CMIP6 experiments were executed, where none performed well enough to stay below the threshold of the normalised distance from the ERA5 reanalysis regimes. Of the nine CMIP6 experiments performed with the WR7 definition, all of them have a normalised distance to the ERA5 reanalysis regimes smaller than 0.4.

However, for certain experiments, when visually comparing the regimes to ERA5 reanalysis, not all of the regimes found in the data stream can be confidently matched to an ERA5 regime. For instance, when applying the WR7 definition to the NorESM2-MM Picontrol data, the normalised distance is 0.349. Despite the distance passing the threshold, visual inspection shows substantial differences. The regimes produces by this experiment are shown in SI Figure S3, where there is no clear AR+, but instead a regime resembling SB+ is found. Additionally, the EB+ regime is located further towards the Atlantic Ocean than in Figure 1. Similar behaviour, the shifting of regimes or not finding a regime clearly, is present in three other experiments for the WR7 definition. These experiments are: NorESM2-MM historical, MPI-ESM1-2-LR historical and UKESM1-0-LL historical.

An example of the clustering using the WR6 definition of a data stream of a CMIP6 model, piControl of NorESM2-LM, is shown in figure S4. Here some regimes are the regimes we expect to see, like the NAO- and the regime labeled as NAO+ by the matching algorithm resembles SB+. However, the other regimes have vastly different structures compared to SI Figure S2. For instance, there is no regime with an intense low or high over the central Atlantic Ocean, as is expected of the AR- and AR+ regimes. For most CMIP6 experiments that do not pass the normalised distance threshold, some of the expected patterns can be identified, but others are similar to other regimes that have shifted or they are completely different patterns. For WR6 the regime that is absent most often is the SB- regime. However, the NAO- and NAO+ are reliably found by the WR6 definition across models and data sets. For WR4 AR+ is missing for CMCC-CM2-SR5 piControl, but for NorESM2-LM piControl all regimes have a different pattern to the expected regime.

There is no strong evidence that training the clustering model on either piControl data or historical data has an influence on the stability of the regimes. The performance of the WR4 definition in the historical stream, does suggest that the historical streams produce regimes more reliably. However, this is not the case for the WR6 or WR7 definitions, so more climate models need to be investigated to be able to draw a conclusion.

4 Discussion

In this study various methodological choices were made that affected the outcome of the research. Here we will explain why certain choices were made and how these affected our results. Moreover, we will place our results of the sensitivity tests into the context of the larger climate system and discuss the implications for impact models used in the energy sector. Then we discuss important issues regarding the application of the WR framework on CMIP6 data. Finally, this Section concludes with suggesting further steps to improve and expand this research.

4.1 Methods

Even though the reproduction of the WR definitions was carried out with great attention to the original methods, some differences within our method exist. Especially problematic was the inability to reproduce the ‘No regime’ definition from [73] (WR7). Moreover, it was not possible to produce an adequate low-pass filter for the WR7 definition. While a basic filter was implemented, the number of re-runs our sensitivity assessment of the WR framework required and the computational burden that even this inadequate filter had, forced us to not include this filter due to time available.

Furthermore, as discussed in SI Section D, here the geopotential height (gph) anomalies are calculated differently for WR6 and WR7, and likely also for WR4. [86] subtracts a background state from the $500hPa$ gph data. In [73] the anomalies are calculated with respect to a 90-day running mean as reference climatology. For WR4 it is not stated how the anomalies of the $500$ hPa gph fields were calculated in [83]. In our approach, for each day the climatology of that day is subtracted from the $500hPa$ gph data. Using the climatology as the baseline should account for the slow seasonal variability without a-priori assumptions on its properties, while providing a smooth background signal.

Furthermore, when applying the WR7 definition on the CMIP6 data we chose to alter the considered region. This was done because the original method includes the $90^{\circ}$N as a boundary, which consists of one data point. This would skew the results, as one data point would have a larger relative influence, so we opted to standardise the regions across definitions.

All other discrepancies between the original method and the method carried out here, can be found in SI Section D. The various small, and sometimes unknown, differences in the model definitions are the likely culprit for the found differences for both the occurrence and the persistence rates within our validation experiments.

Many works do not carry out a validation step within their WR framework. In general the incomplete descriptions, the lack of published open data, and the absence of (reproducible) code (see  [83] and  [73]), means that a replication of these their findings is a demanding effort. This makes replication difficult and is a roadblock towards standardisation across implementations. Therefore we encourage transparency regarding the methodology of the WR framework and the use of the FAIR guiding principles [72]. See SI Section A for our data & code references.

4.2 Sensitivity of WR framework

The results of our sensitivity tests imply a large dependency on the time period the WR method clusters on. So, short time periods should be avoided when implementing WR definitions, as they cause large uncertainties. Especially considering the variability of the climate system at various shorter timescales.

For instance, when considering the effects of the multi-year El Niño Southern Oscillation (ENSO), if the considered period is too short the effects of which El Niño cycles are included may unintentionally influence the results. As found by  [92], there is a relation between occurrence rates of regimes and the El Niño Southern Oscillation on interannual time-scales. Furthermore, multidecadal variability has been observed in German wind energy generation [84], and European solar energy generation [89]. Therefore, we recommend using periods of at least 50 years, when training WR definitions for climate system dependent implementations, such as the energy sector.

Large scale atmospheric structures in the Euro-Atlantic sector are strongly related to the Atlantic Meridional Overturning Circulation (AMOC). Variations in the AMOC have been linked to changes in some regime properties [70, 101]. As the probability of a AMOC collapse is not insignificant [108, 110], a possible collapse should be accounted for investigating the effects of climate change on WR’s. [117] have applied the WR6 definition to CESM simulations with or without an AMOC. While they found little change in the overall patterns or frequency of the regimes, they also found changes in the precipitation patterns for a scenario without an AMOC. Furthermore, [116] showed that an AMOC collapse would be associated with significant cooling during the winter months, likely leading to more stress on the energy sector due to the sensitivity of electricity demand to temperature. Future research on the effects of climate change on WR’s should be aware of this.

4.3 Application WR framework on CMIP6 data

The distances calculated in Section 3.3 to validate the regimes found in the CMIP6 data streams should be treated with caution. The reason for this is that for some data streams, across WR definitions, normalised distances above 1 were calculated within the distance matrix. We were unable to determine where this error comes from. Thus the specific value of the distance should not be taken at face value. However, as the same method was used for each experiment and the results are in line with what can be concluded visually we assume the calculated distances are still an indication of the quality of the found regimes.

Furthermore, there are four data streams where the regimes produced by the WR7 definition pass the 0.4 normalised distance threshold, but the regimes are notably different to the corresponding ERA5 reanalysis regimes. This is discussed in further detail in SI Section E.4. When applying the WR7 definition to CMIP6 data streams, great caution should be taken when assessing the performance of the clustering.

Keen readers might have noticed the lack of the persistence and occurrence results of the regimes found in CMIP6 Shared Socio-economic Pathways data streams. While the data was retrieved, and the WRs identified (where possible, see Table 1), the results would be prone to misuse. We opted not to show these as we found that the WR framework is unsuitable for such calculations, due to the instability of the regimes found.

4.4 Outlook

This research could be improved and expanded in the following two manners. First, when carrying out the sensitivity tests, a Bayesian assignment, as described in [104], could be used for all the tests. Here we only investigated the results of implementing the Bayesian assignment itself, but varying other data parameters when using a more stable assignment would alter the occurrence and persistence rates. Assessing how sensitive the definitions are, when a more persistent method of assignment is used, could possibly reduce the uncertainty of the Weather Regime definitions.

Second, because of the instability of WR definitions when trained on CMIP6 model data, it should be investigated whether assigning regimes trained on ERA5 reanalysis data to CMIP6 historical data, produces a reliable signal. If that is the case, these regimes trained on ERA5 reanalysis data could be assigned to SSP data streams of the CMIP6 models. That could give insight into the possible future behaviours of regimes under climate change. This has already been done by [91], however, there only the behaviour of the WR4 definition is investigated. Furthermore, [115] applies the technique of using ERA5 reanalysis data as a reference for the regimes and applies it to CMIP6 model data as well, however, there it is applied on a summertime definition of WR’s.

5 Conclusion

The aim of this research was to assess the stability of a Weather Regime framework for analysis and its suitability for impact research within the energy sector.

We carried out an extensive validation of various Weather Regime definitions, however this was hindered by the limited availability of source material. Many works do not carry out this vital validation step within their WR framework. We found that in general the incomplete definitions, the lack of published open data, and the absence of code, means that a strenuous effort is necessary to be able to reliably reproduce WR definitions.

Our sensitivity assessment showed that the persistence and occurrence of regimes produced by all of the WR definitions we investigated (WR4, WR6 & WR7) are sensitive to small changes in the methodology. Both definition dependent parameters (i.e. the number of EOFs and the method of assignment) and data dependent parameters (i.e. region, time- and spatial resolution, considered months and time period) affect the occurrence and persistence of the regimes. Though, for each WR definition, Weather Regime and parameter combination, the occurrence and persistence vary with changes in these parameters to a different degree.

A key sensitivity we identified, is that the stability of the regimes defined with WR4 and WR7 are highly dependent on the period the clustering algorithm is trained on in ERA5. As we showed, not all the expected regimes can be identified when the clustering is performed on data from the period 1991–2020. Therefore, we recommend using periods of at least 50 years, when training WR definitions for climate system dependent implementations, such as the energy sector.

We found that the WR framework is not usable for climate change research as there is no reliable performance within the CMIP6 sub-ensemble analysed. When applying the WR definitions to the piControl and historical data streams of a sub-ensemble of CMIP6 models, none of the definitions could reliably produce the expected circulation patterns. For this we used the normalised distance between the regimes found within CMIP6 and those from ERA5. The WR4 definition produces the expected regimes for 10/12 of our experiments, the WR6 definition for 0/9 experiments, and the WR7 definition for 9/9 experiments.

Though, the WR4 and WR7 definitions perform well in terms of number of experiments that pass the threshold, these results must be treated with caution. The reason for this is the instability of the WR4 and WR7 definitions found within our sensitivity assessment, especially with regards to the time periods in the ERA5 reanalysis data. The occurrence and persistence rates found within the CMIP6 sub-ensemble are not reliable. Our findings imply that CMIP6 models are not suitable for the identification of Weather Regimes and an assessment of climate change effects on them is thus currently unfeasible. The use of the WR framework presented with CMIP6 model data is thus unreliable as an indicator of the frequency of future energy droughts or a more general assessment of climate change impact for the energy sector.

Understanding the uncertainties in Weather Regime definitions, the associated impacts of specific WRs, and specific effects of climate change on these Weather Regimes, will help negate possible risks in the energy transition. Planning for renewable energy infrastructure requires an adequate impact model. To integrate WR analysis into impact analyses for the energy sector, one definition, is not reliable enough for policy making, because of the large variability present in the definitions. Using several standardised methods, with a robust understanding of the uncertainties within each method can improve the reliability of the outcome of an impact model. For this, more transparency regarding methodology and establishing standard WR definitions is required. However for now, these WR definitions are not suitable to be implemented in impact models for the energy transition.

Acknowledgments

The content of this paper and the views expressed in it are solely the author’s responsibility, and do not necessarily reflect the views of TenneT TSO B.V..

We acknowledge the World Climate Research Programme, which, through its Working Group on Coupled Modelling, coordinated and promoted CMIP6. We thank the climate modelling groups for producing and making available their model output, the Earth System Grid Federation (ESGF) for archiving the data and providing access, and the multiple funding agencies who support CMIP6 and ESGF.

This document has been produced in the context of the Copernicus Climate Change Service (C3S).

The activities leading to these results have been contracted by the European Centre for Medium-Range Weather Forecasts, operator of C3S on behalf of the European Union (Delegation Agreement signed on 11/11/2014 and Contribution Agreement signed on 22/07/2021). All information in this document is provided "as is" and no guarantee or warranty is given that the information is fit for any particular purpose.

The users thereof use the information at their sole risk and liability. For the avoidance of all doubt , the European Commission and the European Centre for Medium - Range Weather Forecasts have no liability in respect of this document, which is merely representing the author’s view.

\newrefcontext

[sorting=nty]