KNIFE: Kernel Iterative Feature Extraction

TL;DR

KNIFE introduces an iterative method for feature selection in kernel-based models, improving performance by estimating feature weights alongside model coefficients, and revealing important feature correlations.

Contribution

The paper presents KNIFE, a novel iterative algorithm for feature weighting in kernel methods, with an efficient path-finding extension for feature regularization.

Findings

Effective in kernel regression and SVMs with various kernels

Reveals non-linear feature correlations

Improves model performance with relevant feature selection

Abstract

Selecting important features in non-linear or kernel spaces is a difficult challenge in both classification and regression problems. When many of the features are irrelevant, kernel methods such as the support vector machine and kernel ridge regression can sometimes perform poorly. We propose weighting the features within a kernel with a sparse set of weights that are estimated in conjunction with the original classification or regression problem. The iterative algorithm, KNIFE, alternates between finding the coefficients of the original problem and finding the feature weights through kernel linearization. In addition, a slight modification of KNIFE yields an efficient algorithm for finding feature regularization paths, or the paths of each feature's weight. Simulation results demonstrate the utility of KNIFE for both kernel regression and support vector machines with a variety of kernels. Feature path realizations also reveal important non-linear correlations among features that prove useful in determining a subset of significant variables. Results on vowel recognition data, Parkinson's disease data, and microarray data are also given.