Comparing Model-agnostic Feature Selection Methods through Relative Efficiency

TL;DR

This paper compares feature selection methods based on the Generalized Covariance Measure and LOCO estimation, providing theoretical and empirical insights into their relative efficiency across different models and real-world data.

Contribution

It offers a theoretical comparison and empirical evaluation of GCM and LOCO methods, highlighting their efficiency differences in various model settings.

Findings

GCM methods generally outperform LOCO under certain conditions

Theoretical asymptotic efficiency of GCM and LOCO is quantified

Empirical results confirm theoretical predictions across multiple datasets

Abstract

Feature selection and importance estimation in a model-agnostic setting is an ongoing challenge of significant interest. Wrapper methods are commonly used because they are typically model-agnostic, even though they are computationally intensive. In this paper, we focus on feature selection methods related to the Generalized Covariance Measure (GCM) and Leave-One-Covariate-Out (LOCO) estimation, and provide a comparison based on relative efficiency. In particular, we present a theoretical comparison under three model settings: linear models, non-linear additive models, and single index models that mimic a single-layer neural network. We complement this with extensive simulations and real data examples. Our theoretical results, along with empirical findings, demonstrate that GCM-related methods generally outperform LOCO under suitable regularity conditions. Furthermore, we quantify the asymptotic relative efficiency of these approaches. Our simulations and real data analysis include widely used machine learning methods such as neural networks and gradient boosting trees.