Evaluation of Local Explanation Methods for Multivariate Time Series Forecasting

TL;DR

This paper introduces new evaluation metrics for local interpretability methods in multivariate time series forecasting, enabling better assessment of explanation fidelity and sensitivity.

Contribution

It proposes two novel metrics for evaluating local explanations in time series forecasting and extends theoretical foundations with experimental validation.

Findings

The metrics effectively compare local explanation models.

They reveal differences in sensitivity among explanation methods.

Experimental results on Rossmann and electricity datasets demonstrate the metrics' utility.

Abstract

Being able to interpret a machine learning model is a crucial task in many applications of machine learning. Specifically, local interpretability is important in determining why a model makes particular predictions. Despite the recent focus on AI interpretability, there has been a lack of research in local interpretability methods for time series forecasting while the few interpretable methods that exist mainly focus on time series classification tasks. In this study, we propose two novel evaluation metrics for time series forecasting: Area Over the Perturbation Curve for Regression and Ablation Percentage Threshold. These two metrics can measure the local fidelity of local explanation models. We extend the theoretical foundation to collect experimental results on two popular datasets, \textit{Rossmann sales} and \textit{electricity}. Both metrics enable a comprehensive comparison of numerous local explanation models and find which metrics are more sensitive. Lastly, we provide heuristical reasoning for this analysis.