# Evaluation of time series models under non-stationarity with application to the comparison of regional climate models

## TL;DR

This paper introduces a methodology for evaluating time series models under non-stationarity by using moving scores based on changepoint analysis, enabling more accurate model comparison in changing conditions.

## Contribution

It proposes a novel approach to model evaluation that accounts for non-stationarity through moving windows and changepoint analysis, improving fairness and accuracy.

## Key findings

- Methodology effectively captures non-stationary behavior.
- Application to climate models demonstrates practical utility.
- Simulation and case study validate the approach.

## Abstract

Different disciplines pursue the aim to develop models which characterize certain phenomena as accurately as possible. Climatology is a prime example, where the temporal evolution of the climate is modeled. In order to compare and improve different models, methodology for a fair model evaluation is indispensable. As models and forecasts of a phenomenon are usually associated with uncertainty, proper scoring rules, which are tools that account for this kind of uncertainty, are an adequate choice for model evaluation. However, under the presence of non-stationarity, such a model evaluation becomes challenging, as the characteristics of the phenomenon of interest change. We provide methodology for model evaluation in the context of non-stationary time series. Our methodology assumes stationarity of the time series in shorter moving time windows. These moving windows, which are selected based on a changepoint analysis, are used to characterize the uncertainty of the phenomenon/model for the corresponding time instances. This leads to the concept of moving scores allowing for a temporal assessment of the model performance. The merits of the proposed methodology are illustrated in a simulation and a case study.