Holistic Measures for Evaluating Prediction Models in Smart Grids

TL;DR

This paper introduces a comprehensive set of performance measures tailored for evaluating prediction models in smart grids, emphasizing application-specific aspects like reliability, volatility, and cost to improve model selection.

Contribution

It proposes a suite of holistic, customizable performance metrics for smart grid prediction models, addressing limitations of traditional abstract metrics.

Findings

New measures provide deeper insights into model behavior.

Application-specific evaluation improves model selection.

Empirical analysis demonstrates practical usefulness.

Abstract

The performance of prediction models is often based on "abstract metrics" that estimate the model's ability to limit residual errors between the observed and predicted values. However, meaningful evaluation and selection of prediction models for end-user domains requires holistic and application-sensitive performance measures. Inspired by energy consumption prediction models used in the emerging "big data" domain of Smart Power Grids, we propose a suite of performance measures to rationally compare models along the dimensions of scale independence, reliability, volatility and cost. We include both application independent and dependent measures, the latter parameterized to allow customization by domain experts to fit their scenario. While our measures are generalizable to other domains, we offer an empirical analysis using real energy use data for three Smart Grid applications: planning, customer education and demand response, which are relevant for energy sustainability. Our results underscore the value of the proposed measures to offer a deeper insight into models' behavior and their impact on real applications, which benefit both data mining researchers and practitioners.