Construction of an informative hierarchical prior for a small sample with the help of historical data and application to electricity load forecasting

TL;DR

This paper develops a hierarchical Bayesian prior using historical data to improve estimation and prediction in small datasets, demonstrated through simulations and electricity load forecasting.

Contribution

It introduces a novel hierarchical prior constructed from related long-term data to enhance model performance on small datasets.

Findings

Improved prediction accuracy in simulated models

Significant enhancement in electricity load forecasting

Effective use of historical data for small sample inference

Abstract

We are interested in the estimation and prediction of a parametric model on a short dataset upon which it is expected to overfit and perform badly. To overcome the lack of data (relatively to the dimension of the model) we propose the construction of an informative hierarchical Bayesian prior based upon another longer dataset which is assumed to share some similarities with the original, short dataset. We illustrate the performance of our prior on simulated dataset from three standard models. Then we apply the methodology to a working model for the electricity load forecasting on real datasets, where it leads to a substantial improvement of the quality of the predictions.