Combining predictions from linear models when training and test inputs differ

TL;DR

This paper introduces XAIC and FAIC, new model selection criteria designed for better prediction when training and test inputs differ, especially under covariate shift, outperforming traditional methods like AIC and BIC.

Contribution

The paper proposes unbiased KL divergence-based criteria, XAIC and FAIC, that adapt model selection for scenarios with differing training and test inputs, improving predictive performance.

Findings

XAIC and FAIC outperform AIC, BIC, and Bayesian model averaging under covariate shift.

Both methods are especially effective for deterministic inputs and when training and test inputs differ substantially.

Experimental results demonstrate improved prediction accuracy with the new criteria.

Abstract

Methods for combining predictions from different models in a supervised learning setting must somehow estimate/predict the quality of a model's predictions at unknown future inputs. Many of these methods (often implicitly) make the assumption that the test inputs are identical to the training inputs, which is seldom reasonable. By failing to take into account that prediction will generally be harder for test inputs that did not occur in the training set, this leads to the selection of too complex models. Based on a novel, unbiased expression for KL divergence, we propose XAIC and its special case FAIC as versions of AIC intended for prediction that use different degrees of knowledge of the test inputs. Both methods substantially differ from and may outperform all the known versions of AIC even when the training and test inputs are iid, and are especially useful for deterministic inputs and under covariate shift. Our experiments on linear models suggest that if the test and training inputs differ substantially, then XAIC and FAIC predictively outperform AIC, BIC and several other methods including Bayesian model averaging.