Sparse Multiple Kernel Learning: Support Identification via Mirror Stratifiability

TL;DR

This paper introduces a stable feature selection method in kernel learning using forward-backward splitting, demonstrating finite iteration identification of relevant features under certain conditions.

Contribution

It establishes the finite-time support identification property of the forward-backward splitting algorithm for sparse multiple kernel learning, leveraging stratification concepts.

Findings

Support set is identified after finite iterations

Supports are stable under the proposed method

Theoretical guarantees depend on qualification conditions

Abstract

In statistical machine learning, kernel methods allow to consider infinite dimensional feature spaces with a computational cost that only depends on the number of observations. This is usually done by solving an optimization problem depending on a data fit term and a suitable regularizer. In this paper we consider feature maps which are the concatenation of a fixed, possibly large, set of simpler feature maps. The penalty is a sparsity inducing one, promoting solutions depending only on a small subset of the features. The group lasso problem is a special case of this more general setting. We show that one of the most popular optimization algorithms to solve the regularized objective function, the forward-backward splitting method, allows to perform feature selection in a stable manner. In particular, we prove that the set of relevant features is identified by the algorithm after a finite number of iterations if a suitable qualification condition holds. The main tools used in the proofs are the notions of stratification and mirror stratifiability.