Sobolev Independence Criterion

TL;DR

The Sobolev Independence Criterion (SIC) is a novel, interpretable dependency measure that facilitates nonlinear feature selection and reliable discovery in high-dimensional data, combining kernel and neural network approaches.

Contribution

Introduces SIC, a gradient-regularized IPM-based dependency measure with kernel and neural network variants for interpretable feature selection.

Findings

SIC effectively identifies relevant features in synthetic data.

SIC demonstrates reliable feature selection in real-world datasets.

SIC combined with FDR control methods yields trustworthy discoveries.

Abstract

We propose the Sobolev Independence Criterion (SIC), an interpretable dependency measure between a high dimensional random variable X and a response variable Y . SIC decomposes to the sum of feature importance scores and hence can be used for nonlinear feature selection. SIC can be seen as a gradient regularized Integral Probability Metric (IPM) between the joint distribution of the two random variables and the product of their marginals. We use sparsity inducing gradient penalties to promote input sparsity of the critic of the IPM. In the kernel version we show that SIC can be cast as a convex optimization problem by introducing auxiliary variables that play an important role in feature selection as they are normalized feature importance scores. We then present a neural version of SIC where the critic is parameterized as a homogeneous neural network, improving its representation power as well as its interpretability. We conduct experiments validating SIC for feature selection in synthetic and real-world experiments. We show that SIC enables reliable and interpretable discoveries, when used in conjunction with the holdout randomization test and knockoffs to control the False Discovery Rate. Code is available at http://github.com/ibm/sic.