Understanding complex predictive models with Ghost Variables

TL;DR

This paper introduces a novel method to assess the relevance of variables in complex predictive models, including neural networks, by comparing predictions with ghost variables and analyzing joint effects through eigenvalues.

Contribution

It proposes a new relevance measure for explanatory variables that captures both individual and joint effects in complex models, extending beyond traditional significance measures.

Findings

Relevance measures align with standard significance in simple models.

Method reveals joint effects in neural networks not accessible by traditional methods.

Illustrated with simulated data and a large real dataset.

Abstract

We propose a procedure for assigning a relevance measure to each explanatory variable in a complex predictive model. We assume that we have a training set to fit the model and a test set to check the out of sample performance. First, the individual relevance of each variable is computed by comparing the predictions in the test set, given by the model that includes all the variables with those of another model in which the variable of interest is substituted by its ghost variable, defined as the prediction of this variable by using the rest of explanatory variables. Second, we check the joint effects among the variables by using the eigenvalues of a relevance matrix that is the covariance matrix of the vectors of individual effects. It is shown that in simple models, as linear or additive models, the proposed measures are related to standard measures of significance of the variables and in neural networks models (and in other algorithmic prediction models) the procedure provides information about the joint and individual effects of the variables that is not usually available by other methods. The procedure is illustrated with simulated examples and the analysis of a large real data set.