Fast learning rate of multiple kernel learning: Trade-off between sparsity and smoothness

TL;DR

This paper analyzes the learning rates of multiple kernel learning with $ ext{l}_1$ and elastic-net regularizations, revealing how regularization choices affect convergence depending on the ground truth's smoothness.

Contribution

It provides sharper convergence rate bounds for MKL with $ ext{l}_1$ and elastic-net regularizations in sparse settings, and clarifies the impact of regularization on performance.

Findings

Faster convergence rates for elastic-net when the ground truth is smooth.

Faster convergence rates for $ ext{l}_1$-regularization when the ground truth is not smooth.

Sharper theoretical bounds than previous analyses.

Abstract

We investigate the learning rate of multiple kernel learning (MKL) with $\ell_1$ and elastic-net regularizations. The elastic-net regularization is a composition of an $\ell_1$-regularizer for inducing the sparsity and an $\ell_2$-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large, but the number of nonzero components of the ground truth is relatively small, and show sharper convergence rates than the learning rates have ever shown for both $\ell_1$ and elastic-net regularizations. Our analysis reveals some relations between the choice of a regularization function and the performance. If the ground truth is smooth, we show a faster convergence rate for the elastic-net regularization with less conditions than $\ell_1$-regularization; otherwise, a faster convergence rate for the $\ell_1$-regularization is shown.