MDA for random forests: inconsistency, and a practical solution via the Sobol-MDA

TL;DR

This paper analyzes the statistical properties of the mean decrease accuracy (MDA) in random forests, reveals its limitations in dependent covariate settings, and introduces the Sobol-MDA as a more reliable importance measure with practical benefits.

Contribution

The authors rigorously analyze MDA's asymptotic behavior, identify its flaws under dependence, and propose the Sobol-MDA as a novel, consistent importance measure for random forests.

Findings

MDA converges to different quantities depending on implementation.

Original MDA does not effectively detect influential covariates with dependent features.

Sobol-MDA outperforms existing importance measures in variable selection tasks.

Abstract

Variable importance measures are the main tools to analyze the black-box mechanisms of random forests. Although the mean decrease accuracy (MDA) is widely accepted as the most efficient variable importance measure for random forests, little is known about its statistical properties. In fact, the definition of MDA varies across the main random forest software. In this article, our objective is to rigorously analyze the behavior of the main MDA implementations. Consequently, we mathematically formalize the various implemented MDA algorithms, and then establish their limits when the sample size increases. This asymptotic analysis reveals that these MDA versions differ as importance measures, since they converge towards different quantities. More importantly, we break down these limits into three components: the first two terms are related to Sobol indices, which are well-defined measures of a covariate contribution to the response variance, widely used in the sensitivity analysis field, as opposed to the third term, whose value increases with dependence within covariates. Thus, we theoretically demonstrate that the MDA does not target the right quantity to detect influential covariates in a dependent setting, a fact that has already been noticed experimentally. To address this issue, we define a new importance measure for random forests, the Sobol-MDA, which fixes the flaws of the original MDA, and consistently estimates the accuracy decrease of the forest retrained without a given covariate, but with an efficient computational cost. The Sobol-MDA empirically outperforms its competitors on both simulated and real data for variable selection. An open source implementation in R and C++ is available online.