Ensemble Prediction via Covariate-dependent Stacking

TL;DR

This paper introduces covariate-dependent stacking (CDST), a flexible ensemble method that adjusts model weights based on covariates, improving prediction accuracy in complex scenarios like spatio-temporal data.

Contribution

The paper develops a novel covariate-dependent stacking approach with theoretical guarantees and demonstrates its superior performance over traditional methods in simulations and real data.

Findings

CDST outperforms traditional model averaging in simulations.

It provides consistent improvements in land price prediction.

Theoretical oracle inequality supports its effectiveness.

Abstract

This study proposes a novel approach to ensemble prediction, called "covariate-dependent stacking" (CDST). Unlike traditional stacking and model averaging methods, CDST allows model weights to vary flexibly as a function of covariates, thereby enhancing predictive performance in complex scenarios. We formulate the covariate-dependent weights through combinations of basis functions and estimate them via cross-validation optimization. To analyze the theoretical properties, we establish an oracle inequality regarding the expected loss to be minimized for estimating model weights. Through comprehensive simulation studies and an application to large-scale land price prediction, we demonstrate that the CDST consistently outperforms conventional model averaging methods, particularly on datasets where base models fail to capture the underlying complexity. Our findings suggest that the CDST is especially valuable for, but not limited to, spatio-temporal prediction problems, offering a powerful tool for researchers and practitioners across a wide spectrum of data analysis fields.