Nonparametric Assessment of Variable Selection and Ranking Algorithms

TL;DR

This paper introduces nonparametric methods to compare variable selection and ranking algorithms, providing measures, estimators, and inference techniques to evaluate their effectiveness in predicting outcomes.

Contribution

It proposes a novel framework for assessing variable selection and ranking algorithms using nonparametric measures and asymptotic inference, applicable in fixed-dimensional settings.

Findings

Proposed measures effectively evaluate variable selection quality.

Asymptotic properties established for estimators in fixed dimensions.

Numerical studies demonstrate the methods' practical utility.

Abstract

Selecting from or ranking a set of candidates variables in terms of their capacity for predicting an outcome of interest is an important task in many scientific fields. A variety of methods for variable selection and ranking have been proposed in the literature. In practice, it can be challenging to know which method is most appropriate for a given dataset. In this article, we propose methods of comparing variable selection and ranking algorithms. We first introduce measures of the quality of variable selection and ranking algorithms. We then define estimators of our proposed measures, and establish asymptotic results for our estimators in the regime where the dimension of the covariates is fixed as the sample size grows. We use our results to conduct large-sample inference for our measures, and we propose a computationally efficient partial bootstrap procedure to potentially improve finite-sample inference. We assess the properties of our proposed methods using numerical studies, and we illustrate our methods with an analysis of data for predicting wine quality from its physicochemical properties.