Sparse Multiple Kernel Learning: Alternating Best Response and Semidefinite Relaxations

TL;DR

This paper introduces a novel sparse multiple kernel learning method that explicitly constrains kernel selection, solves the non-convex problem efficiently, and outperforms existing methods in prediction accuracy with certified near-optimal solutions.

Contribution

It formulates SMKL with explicit cardinality constraints, develops an alternating best response algorithm, and provides semidefinite relaxations for solution certification and warm-starting.

Findings

Outperforms state-of-the-art MKL methods in accuracy by 3.34% on average.

Achieves better accuracy by 4.05% with warm starting.

Provides certificates of near-optimality for solutions.

Abstract

We study Sparse Multiple Kernel Learning (SMKL), which is the problem of selecting a sparse convex combination of prespecified kernels for support vector binary classification. Unlike prevailing l1 regularized approaches that approximate a sparsifying penalty, we formulate the problem by imposing an explicit cardinality constraint on the kernel weights and add an l2 penalty for robustness. We solve the resulting non-convex minimax problem via an alternating best response algorithm with two subproblems: the alpha subproblem is a standard kernel SVM dual solved via LIBSVM, while the beta subproblem admits an efficient solution via the Greedy Selector and Simplex Projector algorithm. We reformulate SMKL as a mixed integer semidefinite optimization problem and derive a hierarchy of semidefinite convex relaxations which can be used to certify near-optimality of the solutions returned by our best response algorithm and also to warm start it. On ten UCI benchmarks, our method with random initialization outperforms state-of-the-art MKL approaches in out-of-sample prediction accuracy on average by 3.34 percentage points (relative to the best performing benchmark) while selecting a small number of candidate kernels in comparable runtime. With warm starting, our method outperforms the best performing benchmark's out-of-sample prediction accuracy on average by 4.05 percentage points. Our convex relaxations provide a certificate that in several cases, the solution returned by our best response algorithm is the globally optimal solution.