Optimality Implies Kernel Sum Classifiers are Statistically Efficient

TL;DR

This paper introduces a new theoretical framework combining optimization and learning theory to analyze the sample complexity of optimal kernel sum classifiers, providing insights into their efficiency and justifying prior assumptions.

Contribution

It presents a novel analysis method for optimal classifiers using learning theory bounds, and introduces a new Rademacher complexity for such classes.

Findings

Optimal classifiers have favorable sample complexity.

The analysis justifies assumptions in multiple kernel learning.

New Rademacher complexity bounds for optimal classifiers.

Abstract

We propose a novel combination of optimization tools with learning theory bounds in order to analyze the sample complexity of optimal kernel sum classifiers. This contrasts the typical learning theoretic results which hold for all (potentially suboptimal) classifiers. Our work also justifies assumptions made in prior work on multiple kernel learning. As a byproduct of our analysis, we also provide a new form of Rademacher complexity for hypothesis classes containing only optimal classifiers.

Figures

Figure 1 shows the difference between the true $\bm{\alpha}_{{}_{\Sigma,m}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma,m}}\bm{\alpha}_{{}_{\Sigma,m}}$ and the two bounds above. We can observe that the true curve decreases roughly at the same rate as our bounds.

Equations

$$\underset{\mathrm{w}}{\min }$$

$$1-y_i{\mathrm{w}}^{⊺}\mathbf{\varphi }({\mathrm{x}}_i)\le 0\forall i\in [n]$$

$$\underset{\mathbf{\alpha }}{\max }$$

$${\alpha }_i\ge 0\forall i\in [n]$$

$$1-y_i{\mathrm{w}}^{⊺}\mathbf{\varphi }({\mathrm{x}}_i)\le 0\forall i\in [n]$$

$$\mathrm{w}=\sum _{i=1}^n{\alpha }_i{y}_i\mathbf{\varphi }({\mathrm{x}}_i)$$

$${\alpha }_i\ge 0\forall i\in [n]$$

$${\alpha }_i(1-y_i{\mathrm{w}}^{⊺}\mathbf{\varphi }({\mathrm{x}}_i))=0$$

$$\underset{\mathrm{w}}{\min }$$

$$1-y_i{\mathrm{w}}^{⊺}\mathbf{\varphi }({\mathrm{x}}_i)\le {\xi }_i\forall i\in [n]$$

$${\xi }_i\ge 0\forall i\in [n]$$

$$\underset{\mathbf{\alpha },\mathbf{\xi }}{\max }$$

$$0\le {\alpha }_i\le C{\xi }_i\forall i\in [n]$$

$$\hat{}{\mathfrak{R}}_{\mathcal{S}}(\mathcal{F})=E_{\mathbf{\sigma }}\left[\underset{h\in \mathcal{F}}{\sup }\left(\frac{1}{n}\sum _{i=1}^n{\sigma }_{i}h({\mathrm{x}}_i)\right)\right]$$

$$\mathcal{F}:=\left\{\mathrm{x}\mapsto \sum _{i=1}^n{\alpha }_{i}k(\mathrm{x},{\mathrm{x}}_i),{\mathrm{x}}_i\in \mathcal{X},{\mathbf{\alpha }}^{⊺}\mathbf{K}\mathbf{\alpha }\le B^2\right\}$$

$$\psi (x):=\{\begin{array}{ll}1& x<0\\ 1-\frac{x}{\gamma }& 0\le x\le \gamma \\ 0& x>\gamma \end{array}$$

$$\underset{(\mathrm{x},y)\sim \mathcal{D}}{\mathrm{Pr}}[yf(x)\le 0]\le \frac{1}{n}\sum _{i=1}^n\psi (y_{i}f({\mathrm{x}}_i))+\frac{2}{\gamma }\hat{}{\mathfrak{R}}_{\mathcal{S}}(\mathcal{F})+\epsilon$$

$$\displaystyle{\mathcal{F}}_{{}_{\Sigma}}\;{\vcentcolon=}\;\bigg{\{}$$

$$\mathbf{\alpha }=\mathrm{DualSVM}({\tilde{\mathbf{K}}}_{{}_{\Sigma }}),$$

$$\displaystyle\hskip 14.22636pt\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}~{}~{}\forall t\in[m]\bigg{\}}$$

$$\displaystyle{\mathcal{F}}_{{}_{{\mathcal{P}}}}\;{\vcentcolon=}\;\bigg{\{}$$

$$\mathcal{P}\subseteq [m],\mathbf{\alpha }=\mathrm{DualSVM}({\tilde{\mathbf{K}}}_{{}_{\mathcal{P}}}),$$

$$\displaystyle\hskip 14.22636pt\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}~{}~{}\forall t\in[m]\bigg{\}}$$

$$0$$

$${\alpha }_i$$

$$\sum _{i=1}^n{\alpha }_i$$

$$\Vert \mathbf{\alpha }{\Vert }_1$$

$${\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_{1+2}{\mathbf{\alpha }}_{1+2}\le \frac{1}{3}({\mathbf{\alpha }}_1^{⊺}{\tilde{\mathbf{K}}}_1{\mathbf{\alpha }}_1+{\mathbf{\alpha }}_2^{⊺}{\tilde{\mathbf{K}}}_2{\mathbf{\alpha }}_2)$$

$${\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_{1+2}{\mathbf{\alpha }}_{1+2}\le \frac{2}{3}\max \{{\mathbf{\alpha }}_1^{⊺}{\tilde{\mathbf{K}}}_1{\mathbf{\alpha }}_1,{\mathbf{\alpha }}_2^{⊺}{\tilde{\mathbf{K}}}_2{\mathbf{\alpha }}_2\}$$

$$\Vert {\mathbf{\alpha }}^{\prime }{\Vert }_1-\frac{1}{2}{{\mathbf{\alpha }}^{\prime }}^{⊺}\tilde{\mathbf{K}}{\mathbf{\alpha }}^{\prime }\le \Vert \mathbf{\alpha }{\Vert }_1-\frac{1}{2}{\mathbf{\alpha }}^{⊺}\tilde{\mathbf{K}}\mathbf{\alpha }$$

$$\Vert {\mathbf{\alpha }}_{1+2}{\Vert }_1$$

$$=\Vert {\mathbf{\alpha }}_{1+2}{\Vert }_1-\frac{1}{2}{\mathbf{\alpha }}_{1+2}^{⊺}({\tilde{\mathbf{K}}}_1+{\tilde{\mathbf{K}}}_2){\mathbf{\alpha }}_{1+2}$$

$$=\Vert {\mathbf{\alpha }}_{1+2}{\Vert }_1-\frac{1}{2}{\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_1{\mathbf{\alpha }}_{1+2}$$

$$-\frac{1}{2}{\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_2{\mathbf{\alpha }}_{1+2}$$

$$\Vert {\mathbf{\alpha }}_{1+2}{\Vert }_1$$

$$=\Vert {\mathbf{\alpha }}_{1+2}{\Vert }_1-\frac{1}{2}{\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_1{\mathbf{\alpha }}_{1+2}$$

$$-\frac{1}{2}{\mathbf{\alpha }}_{1+2}^{⊺}{\tilde{\mathbf{K}}}_2{\mathbf{\alpha }}_{1+2}$$

Full text

1 Introduction

Classification is a fundamental task in machine learning [21, 13, 15]. Kernel methods allow classifiers to learn powerful nonlinear relationships [22, 3]. Optimization tools allow these methods to learn efficiently [24]. Under mild assumptions, kernels guarantee that learned models generalize well [4]. However, the overall quality of these models still depends heavily on the choice of kernel. To compensate for this, prior work considers learning how to linearly combine a set of arbitrary kernels into a good data-dependent kernel [25, 16, 2].

It is known that if the learned linear combination of kernels is well behaved, then the kernel classifier generalizes well [11, 8, 1]. We extend this body of work by proving that if our classifier is optimal, then the linear combination of kernels is well behaved. This optimality assumption is well justified because many common machine learning problems are solved using optimization algorithms. For instance, in this paper we consider binary classification with Kernel Support Vector Machines (SVM), which are computed by solving a quadratic programming problem. Specifically, we bound the sample complexity of kernel classifiers in two regimes. In the first, we are forced to classify using the sum of a set of kernels. In the second, we choose which kernels we include in our summation.

There exists substantial prior work considering learning kernels. From the computational perspective, several theoretically sound and experimentally efficient algorithms are known [9, 8, 18, 23, 14]. Much of this work relies on optimization tools such as quadratic programs [6], sometimes specifically considering Kernel SVM [26]. This motivates our focus on optimal classifiers for multiple kernel learning. The literature on sample complexity for these problems always assumes that the learned combination of kernels is well behaved [10, 12, 7, 26, 23]. That is, the prior work assumes that the weighted sum of kernel matrices $\tilde{\bm{K}}_{{}_{\Sigma}}$ is paired with a vector $\bm{\alpha}_{{}_{\Sigma}}$ such that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}\leq C^{2}$ for some constant $C$. It is unclear how $C$ depends on the structure or number of base kernels. Our work provides bounds that explains this relationship for optimal classifiers. Additionally, Rademacher complexity is typically used to control the generalization error over all possible (not necessarily optimal) estimators [4, 19, 17]. We differ from this approach by bounding the Rademacher complexity for only optimal estimators. We are not aware of any prior work that explores such bounds.

**Contributions. ** Our results start with a core technical theorem, which is then applied to two novel hypothesis classes.

• •

We first show that the optimal solution to the Kernel SVM problem using the sum of $m$ kernels is well behaved. That is, we consider the given kernel matrices $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m}$ and the corresponding Dual Kernel SVM solution vectors $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m}$, as well as the sum of these kernel matrices $\tilde{\bm{K}}_{{}_{\Sigma}}$ and its Dual Kernel SVM solution vector $\bm{\alpha}_{{}_{\Sigma}}$. Using Karush-Kuhn-Tucker (KKT) optimality conditions, we prove that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}\leq 3m^{-0.58}B^{2}$ provided that all base kernels fulfill $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ for some constant $B$. We are not aware of any existing bounds of this kind, and we provide Rademacher complexity analysis to leverage this result. Note that the previous bounds for the Rademacher complexity in multiple kernel learning assumes that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$ is bounded.

We provide Rademacher complexity bounds for two novel hypothesis classes. As opposed to traditional Rademacher bounds, our hypothesis classes only contain optimal classifiers. The traditional analysis when using a single kernel provides an empirical Rademacher complexity bound of $O(\frac{BR}{\sqrt{n}})$, where $n$ is the number of samples and $k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{i})\leq R^{2}$ bounds the radius of the samples in every feature space [4].

• •

**Kernel Sums: **In the first set, Kernel SVM is required to use the sum of all $m$ kernels. We show that the empirical Rademacher complexity is bounded by $O(\frac{BR}{\sqrt{n}}m^{0.208})$.

• •

**Kernel Subsets: **In the second set, Kernel SVM is allowed to use the sum of any subset of the $m$ kernels. The classical analysis in this setting would pay a multiplicative factor of $2^{m-1}$. The approach we use instead only pays with a factor of $\sqrt{\ln(m)}$. We prove that the empirical Rademacher complexity is bounded by $O(\frac{BR\sqrt{\ln(m)}}{\sqrt{n}}m^{0.208})$.

Note that these Rademacher bounds compare naturally to the traditional single kernel bound. If we use a sum of $m$ kernels instead of just one kernel, then we pay a multiplicative factor of $m^{0.208}$. If we use any subset of kernels, we only pay an extra factor of $\sqrt{\ln(m)}$. Thus, in this work, we show that optimization tools such as KKT conditions are useful in the analysis of statistical bounds. These optimization bounds are leveraged by learning theoretic tools such as Rademacher complexity, as seen in the second and third bullet points. Overall, we obtain new bounds with natural assumptions that connect the existing literature on optimization and learning theory in a novel fashion. Additionally, these bounds justify assumptions made in the existing literature.

2 Preliminaries

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ denote a dataset of $n$ i.i.d. samples from some distribution ${\mathcal{D}}$, where $\bm{\mathrm{x}}_{i}\in{\mathcal{X}}$ and $y_{i}\in\{-1,1\}$ for some ${\mathcal{X}}$. Let $\|\cdot\|_{2}$ denote the $\ell_{2}$ vector norm and $\|\cdot\|_{1}$ denote the $\ell_{1}$ vector norm. Let $[n]\;{\vcentcolon=}\;\{1,\ldots,n\}$ for any natural number $n$.

Let $k:{\mathcal{X}}\times{\mathcal{X}}\rightarrow{\mathbb{R}}$ denote a kernel function. In this paper, we assume that all kernels fulfill $\left|{k(\bm{\mathrm{x}},\bm{\mathrm{x}})}\right|<\infty$ for all $\bm{\mathrm{x}}\in{\mathcal{X}}$. We consider being given a set of kernels $k_{1},\ldots,k_{m}$. Let $k_{{}_{\Sigma}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t=1}^{m}k_{t}(\cdot,\cdot)$ denote the sum of the $m$ kernels. The above notation will be useful when learning with kernel sums. Let ${\mathcal{P}}\subseteq[m]$. Then define $k_{{}_{\mathcal{P}}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t\in{\mathcal{P}}}k_{t}(\cdot,\cdot)$ as the sum of kernels as described by ${\mathcal{P}}$. The latter notation will be useful when learning kernel subsets.

Given a dataset ${\mathcal{S}}$ and a kernel $k_{t}$, we can build the corresponding kernel matrix $\bm{K}_{t}\in{\mathbb{R}}^{n\times n}$, where $[\bm{K}_{t}]_{i,j}\;{\vcentcolon=}\;k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{j})$. Further, we can build the labeled kernel matrix $\tilde{\bm{K}}_{t}$, defined elementwise as $[\tilde{\bm{K}}_{t}]_{i,j}\;{\vcentcolon=}\;y_{i}y_{j}k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{j})$. To simplify notation, all our results use labeled kernel matrices instead of standard kernel matrices.

2.1 Separable SVM

We now present optimal kernel classification, first in the separable case.

Definition 1** (Primal Kernel SVM).**

Given a dataset ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ and a feature map $\bm{\phi}:{\mathcal{X}}\mapsto{\mathbb{R}}^{d}$, the Primal Kernel SVM problem is equivalent to the following optimization problem:

[TABLE]

We will mainly look at the corresponding dual problem:

Definition 2** (Dual Kernel SVM).**

Given a dataset ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ and a kernel function $k(\cdot,\cdot)$ with associated labeled kernel matrix $\tilde{\bm{K}}$, the Dual Kernel SVM problem is equivalent to the following optimization problem:

[TABLE]

Since the dual optimization problem is defined entirely in terms of $\tilde{\bm{K}}$, we can denote the optimal $\bm{\alpha}$ as a function of the labeled kernel matrix. We write this as $\bm{\alpha}=\operatorname*{DualSVM}(\tilde{\bm{K}})$.

Recall that Karush-Kuhn-Tucker (KKT) conditions are necessary and sufficient for optimality in convex optimization problems [5]. We can express the KKT conditions of the Primal Kernel SVM as follows:

Primal Feasibility:

[TABLE]

Stationarity:

[TABLE]

Dual Feasibility:

[TABLE]

Complementary Slackness:

[TABLE]

The above KKT conditions will be used with learning theoretic tools in order to provide novel generalization bounds.

2.2 Non-separable SVM

The primal and dual SVMs above assume that the given kernel is able to separate the data perfectly. Since this is not always the case, we also consider non-separable data using $\ell_{2}$ slack variables:

Definition 3** (Primal Kernel SVM with Slack Variables).**

Given $C>0$, a dataset ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$, and a feature map $\bm{\phi}:{\mathcal{X}}\mapsto{\mathbb{R}}^{d}$, the Primal Kernel SVM problem is equivalent to the following optimization problem:

[TABLE]

Definition 4** (Dual Kernel SVM with Slack Variables).**

Given $C>0$, a dataset ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ and a kernel function $k(\cdot,\cdot)$ with associated labeled kernel matrix $\tilde{\bm{K}}$, the Dual Kernel SVM problem is equivalent to the following optimization problem:

[TABLE]

We denote the solution to the Dual SVM with Slack Variables using parameter $C$ as $\bm{\alpha}=\operatorname*{DualSVM}_{C}(\tilde{\bm{K}})$.

2.3 Rademacher Complexity for Kernels

We use Rademacher complexity to bound the sample complexity of kernel methods. The empirical Rademacher complexity of a hypothesis class ${\mathcal{F}}$ with dataset ${\mathcal{S}}$ is defined as

[TABLE]

where $\bm{\sigma}\in\{-1,+1\}^{n}$ is a vector of Rademacher variables. Bartlett and Mendelson introduced the analysis of sample complexity for kernel methods via Rademacher complexity when using one kernel [4]. Bartlett and Mendelson considered the following hypothesis class of representer theorem functions:

[TABLE]

Each element of ${\mathcal{F}}$ is defined in terms of a dataset $\bm{\mathrm{x}}_{1},\ldots,\bm{\mathrm{x}}_{n}$ and an $\bm{\alpha}$ vector. Bartlett and Mendelson showed that the probability of misclassification is bounded by the empirical risk of misclassification with a $\gamma$-Lipschitz loss plus a Rademacher term:

Theorem 1** (Theorem 22 from [4]).**

Fix $n\geq 0$, $\gamma\in(0,1)$, and $\delta\in(0,1)$.

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset of $n$ i.i.d. samples from ${\mathcal{D}}$. That is, let ${\mathcal{S}}\sim{\mathcal{D}}^{n}$. Define the $\gamma$-Lipschitz Loss function

[TABLE]

Let $\varepsilon\;{\vcentcolon=}\;(\frac{8}{\gamma}+1)\sqrt{\frac{\ln(4/\delta)}{2n}}$. Then with probability at least $1-\delta$ over the choice of ${\mathcal{S}}$, for all $f\in{\mathcal{F}}$ we have

[TABLE]

In this paper, our interest is in bounding this $\hat{}{\mathfrak{R}}_{\mathcal{S}}({\mathcal{F}})$ term under reasonable assumptions on ${\mathcal{F}}$. We specifically consider two hypothesis classes defined over a set of $m$ kernels. First, we consider optimal kernel sum classification, where we must use the sum of all $m$ given kernels:

[TABLE]

Second, we consider optimal kernel subset classification, where we are allowed to use the sum of any subset of the $m$ given kernels:

[TABLE]

Note that $y_{i}$ is not present in Bartlett and Mendelson’s hypothesis class in \texorpdfstring\hyperref[eq:bartlett-hypothesis]Equation 5Equation 5, but it is in \texorpdfstring\hyperref[eq:cf-sum]Equation 2.3Equation 2.3 and \texorpdfstring\hyperref[eq:cf-subset]Equation 2.3Equation 2.3. Regardless, ${\mathcal{F}}_{{}_{\Sigma}}$ and ${\mathcal{F}}_{{}_{\mathcal{P}}}$ do not allow for a more general set of $\bm{\alpha}$ vectors. This is because $\alpha_{i}$ is allowed to be both positive and negative in ${\mathcal{F}}$. However, in ${\mathcal{F}}_{{}_{\Sigma}}$ and ${\mathcal{F}}_{{}_{\mathcal{P}}}$, $\bm{\alpha}$ is a dual optimal vector. Dual Feasibility implies $\alpha_{i}\geq 0$. Thus, by explicitly mentioning $y_{i}$ in the definitions of ${\mathcal{F}}_{{}_{\Sigma}}$ and ${\mathcal{F}}_{{}_{\mathcal{P}}}$, we are stating that $\alpha_{i}$ in ${\mathcal{F}}$ equals $\alpha_{i}y_{i}$ in ${\mathcal{F}}_{{}_{\Sigma}}$ and ${\mathcal{F}}_{{}_{\mathcal{P}}}$.

Initial Rademacher complexity bounds for learning with a single kernel assume that $\bm{\alpha}^{\intercal}\tilde{\bm{K}}\bm{\alpha}\leq B^{2}$ [4]. Previous lines of work on multiple kernel learning then assume that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}\leq C^{2}$ for some constant $C$ [11, 8, 23, 26]. We are interested in proving what values of $C$ are reasonable. To achieve this, we assume that $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ for all base kernels and show that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$ is indeed bounded.

In \texorpdfstring\hyperref[sec:svm-bounds]Section 3Section 3, we leverage our assumption that $\bm{\alpha}$ is optimal to build this bound on $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$. In \texorpdfstring\hyperref[sec:rademacher-bounds]Section 4Section 4, we demonstrate how our bound can augment existing techniques for bounding the Rademacher complexities of \texorpdfstring\hyperref[eq:cf-sum]Equation 2.3Equation 2.3 and \texorpdfstring\hyperref[eq:cf-subset]Equation 2.3Equation 2.3. That is, we bound $\hat{}{\mathfrak{R}}_{{}_{\mathcal{S}}}({\mathcal{F}}_{{}_{\Sigma}})$ and $\hat{}{\mathfrak{R}}_{{}_{\mathcal{S}}}({\mathcal{F}}_{{}_{\mathcal{P}}})$.

3 SVM Bounds for Sums of Kernels

In this section, we leverage KKT conditions and SVM optimality to control the value of $\bm{\alpha}^{\intercal}\tilde{\bm{K}}\bm{\alpha}$ as the number of kernels grows. To start, we consider a single kernel $k$:

Lemma 1**.**

Let $\bm{\alpha}=\operatorname*{DualSVM}(\tilde{\bm{K}})$ for some kernel matrix $\tilde{\bm{K}}$. Then $\|\bm{\alpha}\|_{1}=\bm{\alpha}^{\intercal}\tilde{\bm{K}}\bm{\alpha}$.

Proof.

This proof follows from the KKT conditions provided in \texorpdfstring\hyperref[sec:prelim]Section 2Section 2. We start by substituting Stationarity (\texorpdfstring\hyperref[eq:stationarity]Equation 2Equation 2) into Complementary Slackness (\texorpdfstring\hyperref[eq:complementary-slackness]Equation 4Equation 4). For all $i\in[n]$,

[TABLE]

We can then take the sum of both sides over all $i$:

[TABLE]

Note that $\sum_{i=1}^{n}\alpha_{i}=\|\bm{\alpha}\|_{1}$ since Dual Feasibility (\texorpdfstring\hyperref[eq:dual-feasibility]Equation 3Equation 3) tells us that $\alpha_{i}\geq 0$. ∎

\texorpdfstring\hyperref

[lem:one-kernel]Lemma 1Lemma 1 is mathematically meaningful since at the optimal point $\bm{\alpha}$, the Dual SVM takes objective value exactly equal to $\frac{1}{2}\bm{\alpha}^{\intercal}\tilde{\bm{K}}\bm{\alpha}$. This connects the objective value at the optimal point to the term we want to control. With this in mind, we now move on to consider having two kernels $k_{1}$ and $k_{2}$.

Theorem 2**.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},k_{2}$ be kernel functions. Define $k_{1+2}(\cdot,\cdot)\;{\vcentcolon=}\;k_{1}(\cdot,\cdot)+k_{2}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\tilde{\bm{K}}_{2},\tilde{\bm{K}}_{1+2}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\bm{\alpha}_{2}\bm{\alpha}_{1+2}$ be the corresponding Dual SVM solutions. Then we have

[TABLE]

Furthermore,

[TABLE]

Proof.

First recall that if $\bm{\alpha}=\operatorname*{DualSVM}(\tilde{\bm{K}})$, then for all other dual feasible $\bm{\alpha}^{\prime}$,

[TABLE]

Also note that $\tilde{\bm{K}}_{1+2}=\tilde{\bm{K}}_{1}+\tilde{\bm{K}}_{2}$. We start the proof by looking at the Dual SVM objective for $k_{1+2}$, and distributing over the labeled kernel matrices:

[TABLE]

Reorganizing the above equation, we get

[TABLE]

Next, we use \texorpdfstring\hyperref[lem:one-kernel]Lemma 1Lemma 1 to simplify all three expression that remain:

• •

$2\|\bm{\alpha}_{1+2}\|_{1}-\frac{1}{2}\bm{\alpha}_{1+2}^{\intercal}\tilde{\bm{K}}_{1+2}\bm{\alpha}_{1+2}=\frac{3}{2}\|\bm{\alpha}_{1+2}\|_{1}$

• •

$\|\bm{\alpha}_{1}\|_{1}-\frac{1}{2}\bm{\alpha}_{1}^{\intercal}\tilde{\bm{K}}_{1}\bm{\alpha}_{1}=\frac{1}{2}\|\bm{\alpha}_{1}\|_{1}$

• •

$\|\bm{\alpha}_{2}\|_{1}-\frac{1}{2}\bm{\alpha}_{2}^{\intercal}\tilde{\bm{K}}_{2}\bm{\alpha}_{2}=\frac{1}{2}\|\bm{\alpha}_{2}\|_{1}$

Returning to our bound from \texorpdfstring\hyperref[ineq:two-kernels-part]Inequality 11Inequality 11, we have

[TABLE]

Once we rearrange the constants in \texorpdfstring\hyperref[ineq:two-kernels-final]Inequality 12Inequality 12, we complete the proof. ∎

The constant of $\frac{2}{3}$ in \texorpdfstring\hyperref[ineq:two-thirds]Inequality 9Inequality 9 is advantageous. Since this ratio is below 1, we can recursively apply this theorem to get a vanishing fraction. As $m$ increases, we should now expect $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$ to decrease. We formalize this notion in the following theorem, where we consider using the sum of $m$ kernels.

Theorem 3**.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},k_{2},\ldots,k_{m}$ be kernel functions. Define $k_{{}_{\Sigma}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t=1}^{m}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\Sigma}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\Sigma}}$ be the corresponding Dual SVM solutions. Then we have

[TABLE]

Furthermore,

[TABLE]

In the special case that $m$ is a power of $2$, we have

[TABLE]

Proof sketch.

We provide an intuitive proof for $m=8$. The full proof is in \texorpdfstring\hyperref[app:svm-many-kernels]Appendix BAppendix B.

Since $m$ is a power of two, we can label each of the base kernels with length $\ell=\log_{2}(m)=3$ bitstrings:

Then, for each pair of kernels that differ only in the last digit, define a new kernel as their sum. For instance, define $k_{10}(\cdot,\cdot)\;{\vcentcolon=}\;k_{100}(\cdot,\cdot)+k_{101}(\cdot,\cdot)$. Repeat this process all the way to the root node.

By \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3, we know that

[TABLE]

Going down one level, by applying \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3 again, we know that

[TABLE]

Therefore, by similarly applying \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3 to $\bm{\alpha}_{1}^{\intercal}\tilde{\bm{K}}_{1}\bm{\alpha}_{1}$, we can combine these claims:

[TABLE]

We can then continue until all 8 kernels are included:

[TABLE]

Note that the exponent of $\frac{1}{3}$ is the depth of the tree, equivalent to the length $\ell$ of our bitstring labels. In the general case, we have

[TABLE]

This completes the analysis if $m$ is a power of 2. If we do not have an exact power of two number of kernels, then our tree has depth $\ell-1$ for some leaves. Therefore, we place a floor function around $\log_{2}(3)$:

[TABLE]

To achieve the final result, we bound the summation with

[TABLE]

and simplify the resulting expression. ∎

We take a moment to reflect on this result. It has been well established that the generalization error of kernel classifiers depends on $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$ [10, 12, 7, 26, 23]. \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3 shows that this term actually decreases in the number of kernels. In the next section, we show how this theorem translates into generalization error results.

4 Rademacher Bounds

In this section we apply \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3 to bound the Rademacher complexity of learning with sums of kernels. To better parse and understand these bounds, we make two common assumptions:

• •

Each base kernel has a bounded Dual SVM solution:

[TABLE]

• •

Each vector has a bounded $\ell_{2}$ norm in each feature space:

[TABLE]

The classical bound in [4] on the Rademacher complexity of kernel functions looks at a single kernel, and provides the bound

[TABLE]

where the hypothesis class ${\mathcal{F}}$ is defined in \texorpdfstring\hyperref[eq:bartlett-hypothesis]Equation 5Equation 5. Our bounds are on the order of $\frac{BR}{\sqrt{n}}m^{0.208}$. That is, when moving from one kernel to many kernels, we pay sublinearly in the number of kernels.

We first see this with our bound on the Rademacher complexity of the kernel sum hypothesis class $\hat{}{\mathcal{R}}_{\mathcal{S}}({\mathcal{F}}_{{}_{\Sigma}})$ defined in \texorpdfstring\hyperref[eq:cf-sum]Equation 2.3Equation 2.3:

Theorem 4**.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},\ldots,k_{m}$ be kernel functions. Define $k_{{}_{\Sigma}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t=1}^{m}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\Sigma}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\Sigma}}$ be the corresponding Dual SVM solutions. Then,

[TABLE]

Furthermore, if we assume that $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ and $k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{i})\leq R^{2}$ for all $t\in[m]$ and $i\in[n]$, then we have

[TABLE]

Our proof parallels that of Lemma 22 in [4], and a full proof is in \texorpdfstring\hyperref[app:rademacher-kernel-sum]Appendix CAppendix C. The key difference between Bartlett and Mendelson’s proof and ours is the assumption that $\bm{\alpha}$ is optimal, allowing us to apply \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3.

Next, we consider learning which kernels to sum. In this setting, we allow an algorithm to pick any subset of kernels to sum, but require that Kernel SVM is used for prediction. This is described by the hypothesis class ${\mathcal{F}}_{{}_{\mathcal{P}}}$ defined in \texorpdfstring\hyperref[eq:cf-subset]Equation 2.3Equation 2.3. Because the algorithm can pick any arbitrary subset, we are intuitively bounded by the worst risk over all subsets of kernels. Specifically, \texorpdfstring\hyperref[thm:rademacher-kernel-sum]Theorem 4Theorem 4 suggests that the risk of ${\mathcal{F}}_{{}_{\mathcal{P}}}$ is bounded by the risk of a subset with size $m$. That is, the risk of ${\mathcal{F}}_{{}_{\mathcal{P}}}$ is bounded by the risk of using all kernels. Our next theorem makes this intuition precise, because we only pay an asymptotic factor of $\sqrt{\ln(m)}$ more when considering all possible subsets of kernels instead of only one subset of kernels.

Theorem 5**.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},\ldots,k_{m}$ be kernel functions. Consider any ${\mathcal{P}}\subseteq[m]$. Define $k_{{}_{\mathcal{P}}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t\in{\mathcal{P}}}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\mathcal{P}}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\mathcal{P}}}$ be the corresponding Dual SVM solutions. Assume $k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{i})\leq R^{2}$ and $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ for all $t\in[m]$ and $i\in[n]$. Then,

[TABLE]

where $\eta_{0}=\frac{23}{22}$.

If we tried to build this bound with the classical analytical method found in Lemma 22 of [4], we would have to deal with a difficult supremum over the $2^{m}$ distinct choices of kernels. This would inflate the bound by a multiplicative factor of $\sqrt{2^{m}}=2^{m-1}$. However, our proof instead follows that of Theorem 1 in [10]. This more complicated proof method allows us to pay a factor of $\sqrt{ln(m)}$ to separate supremum over the choice of kernels and the expectation over the $\bm{\sigma}$ vector. This separation allows us to invoke \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3. However, this proof technique also prevents us from building a claim as general as \texorpdfstring\hyperref[thm:rademacher-kernel-sum]Theorem 4Theorem 4, instead only providing bounds using $B$ and $R$. The full proof is found in \texorpdfstring\hyperref[app:rademacher-kernel-learn]Appendix DAppendix D.

5 Bounds for Non-separable Data

Recall the Primal and Dual SVMs with $\ell_{2}$ slack variables from \texorpdfstring\hyperref[sec:prelim]Section 2Section 2. Now we show that if $C=\frac{1}{2}$, then all the other bounds hold using non-separable SVM instead of the separable one. We achieve this by mirroring \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2, which is used by all other results in this paper.

Theorem 6**.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},k_{2}$ be kernel functions. Define $k_{1+2}(\cdot,\cdot)\;{\vcentcolon=}\;k_{1}(\cdot,\cdot)+k_{2}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\tilde{\bm{K}}_{2},\tilde{\bm{K}}_{1+2}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\bm{\alpha}_{2},\bm{\alpha}_{1+2}$ be the corresponding Dual SVM solutions with parameter $C=\frac{1}{2}$. Then we have

[TABLE]

Furthermore,

[TABLE]

The proof mirrors the proof of \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3, except for some careful book keeping for the slack vectors $\bm{\xi}_{1}$, $\bm{\xi}_{2}$, and $\bm{\xi}_{1+2}$. Again, it is the KKT conditions that allow us to bound and compare the $\bm{\xi}$ vectors of the three Dual SVM problems. The full proof is in \texorpdfstring\hyperref[app:proof-two-kernels-slack]Appendix AAppendix A.

With \texorpdfstring\hyperref[thm:two-kernels-slack]Theorem 6Theorem 6, we can reproduce all other results without any changes to the original proofs. Further, this sort of bound on $C$ being a constant is common in learning theory literature such as PAC Bayes [20].

6 Experiment

We show some experimental results that verify our core theorem, i.e. \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3. Our experiment uses 8 fixed kernels from several kernels families. We have 5 radial basis kernels, 1 linear kernel, 1 polynomial kernel, and 1 cosine kernel. All our data is generated from a mixture of 4 Gaussians. Two of the Gaussians generate the positive class while the other 2 generate the negative class.

We generate $n=300$ samples in ${\mathbb{R}}^{50}$. For each of the 8 base kernels, we solve the Dual Kernel SVM problem, and empirically verify that $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq 320=B^{2}$.

Then, we arbitrarily permute the kernel matrices. We solve the Dual Kernel SVM problem with the first kernel matrix denoted as $\tilde{\bm{K}}_{{}_{\Sigma,1}}$. Then we solve the SVM with the sum of the first two kernels, denoted as $\tilde{\bm{K}}_{{}_{\Sigma,2}}$, and so on until we sum all 8 kernels. Let $\bm{\alpha}_{{}_{\Sigma,m}}$ denote the dual solution vector corresponding sum of the first $m$ of the 8 kernels. That is,

[TABLE]

After solving each SVM problem, we keep track of the value of $\bm{\alpha}_{{}_{\Sigma,m}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma,m}}\bm{\alpha}_{{}_{\Sigma,m}}$ value. We then plot this value against the two bounds provided by \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3:

[TABLE]

\texorpdfstring\hyperref

[fig:many-kernel-exper]Figure 1. Figure 1 shows the difference between the true $\bm{\alpha}_{{}_{\Sigma,m}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma,m}}\bm{\alpha}_{{}_{\Sigma,m}}$ and the two bounds above. We can observe that the true curve decreases roughly at the same rate as our bounds.

7 Conclusion

Here we discuss possible directions to extend our work. First, in the context of classification with kernel sums, we are not aware of any efficient and theoretically sound algorithms for learning which kernels to sum. Additionally, we believe that optimality conditions such as KKT are necessary to build meaningful lower bounds in this setting.

One could also analyze the sample complexity of kernel products. This idea is experimentally considered by [14]. This problem is notably more difficult since it requires understanding the Hadamard product of kernel matrices.

More generally, there is little existing work that leverages optimality conditions to justify assumptions made in learning problems. In this paper, KKT tells us that we can control the quantity $\bm{\alpha}_{{}_{\Sigma}}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$, justifying the assumptions made in prior work [10, 26, 23]. We believe that this overall idea is general and applies to other convex optimization problems and classes of representer theorem problems, as well as other learning problems.

Appendix A Non-Separable Proof of Two Kernels (\texorpdfstring\hyperref[thm:two-kernels-slack]Theorem 6Theorem 6)

In this section, we prove a theorem that mirrors that of \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2, but with the $\ell_{2}$ slack SVM. First, we state the KKT conditions for the slack SVM. Let $\bm{\mathrm{r}}$ be the dual variables associated with the primal $\bm{\xi}\succeq 0$ constraints. Then, we have 8 conditions:

$1-\xi_{i}-y_{i}\bm{\mathrm{w}}^{\intercal}\bm{\phi}(\bm{\mathrm{x}}_{i})\leq 0~{}~{}\forall i\in[n]$ (Primal Feasibility 1) 2. 2.

$\xi_{i}\geq 0~{}~{}\forall i\in[n]$ (Primal Feasibility 2) 3. 3.

$\bm{\mathrm{w}}=\sum_{i=1}^{n}\alpha_{i}y_{i}\bm{\phi}(\bm{\mathrm{x}}_{i})$ (Stationarity 1) 4. 4.

$\bm{\mathrm{r}}=C\bm{\xi}-\bm{\alpha}$ (Stationarity 2) 5. 5.

$\alpha_{i}\geq 0~{}~{}\forall i\in[n]$ (Dual Feasibility 1) 6. 6.

$r_{i}\geq 0~{}~{}\forall i\in[n]$ (Dual Feasibility 2) 7. 7.

$\alpha_{i}(1-\xi_{i}-y_{i}\bm{\mathrm{w}}^{\intercal}\bm{\phi}(\bm{\mathrm{x}}_{i}))=0~{}~{}\forall i\in[n]$ (Complementary Slackness 1) 8. 8.

$r_{i}\xi_{i}=0~{}~{}\forall i\in[n]$ (Complementary Slackness 2)

We also provide two preliminary lemmas before proving the main theorem.

Lemma 2**.**

Let $\bm{\alpha},\bm{\xi}$ be the optimal solution to the $\ell_{2}$ Slack Dual SVM problem with parameter $C$. Then, $\bm{\xi}=\frac{1}{C}\bm{\alpha}$. This also implies $\bm{\alpha}^{\intercal}\bm{\xi}=C\|\bm{\xi}\|_{2}^{2}$.

Proof.

First we substitute Stationarity 2 into Complementary Slackness 2:

[TABLE]

That is, when $\xi_{i}\neq 0$, we know that $\xi_{i}=\frac{\alpha_{i}}{C}$. This allows us to conclude that $\xi_{i}\leq\frac{\alpha_{i}}{C}$, since both $\alpha_{i}$ and $C$ are nonnegative. The dual problem has constraint $\alpha_{i}\leq C\xi_{i}$, which is equivalent to $\xi_{i}\geq\frac{\alpha_{i}}{C}$. Hence $\xi_{i}$ is both upper and lower bounded by $\frac{\alpha_{i}}{C}$. Therefore, $\xi_{i}=\frac{\alpha_{i}}{C}$. ∎

Lemma 3**.**

Let $\bm{\alpha},\bm{\xi}$ be the optimal solution to the $\ell_{2}$ Slack Dual SVM problem on input $\tilde{\bm{K}}$ with parameter $C$. Then $\|\bm{\alpha}\|_{1}=\bm{\alpha}^{\intercal}\tilde{\bm{K}}\bm{\alpha}+C\|\bm{\xi}\|_{2}^{2}$.

Proof.

First substitute Stationarity 1 into Complementary Slackness 1:

[TABLE]

Then, we sum up over all $i\in[n]$:

[TABLE]

∎

Now we prove the main theorem:

\texorpdfstring\hyperref**[thm:two-kernels-slack]Theorem 6Theorem 6 Restated****.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},k_{2}$ be kernel functions. Define $k_{1+2}(\cdot,\cdot)\;{\vcentcolon=}\;k_{1}(\cdot,\cdot)+k_{2}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\tilde{\bm{K}}_{2},\tilde{\bm{K}}_{1+2}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\bm{\alpha}_{2},\bm{\alpha}_{1+2}$ be the corresponding Dual SVM solutions with parameter $C=\frac{1}{2}$. Then we have

[TABLE]

Furthermore,

[TABLE]

Proof.

We start with the dual objective for $k_{1+2}$:

[TABLE]

By applying \texorpdfstring\hyperref[lem:one-kernel-lemma-slack]Lemma 3Lemma 3 and some algebra, we have three useful equations:

• •

$2\|\bm{\alpha}_{1+2}\|_{1}-\frac{1}{2}\bm{\alpha}_{1+2}^{\intercal}\tilde{\bm{K}}_{1+2}\bm{\alpha}_{1+2}-\|\bm{\xi}_{1+2}\|_{2}^{2}=\frac{3}{2}\bm{\alpha}_{1+2}^{\intercal}\tilde{\bm{K}}_{1+2}\bm{\alpha}_{1+2}+(2C-1)\|\bm{\xi}_{1+2}\|_{2}^{2}$

• •

$\|\bm{\alpha}_{1}\|_{1}-\frac{1}{2}\bm{\alpha}_{1}^{\intercal}\tilde{\bm{K}}_{1}\bm{\alpha}_{1}-\frac{1}{2}\|\bm{\xi}_{1}\|_{2}^{2}=\frac{1}{2}\bm{\alpha}_{1}^{\intercal}\tilde{\bm{K}}_{1}\bm{\alpha}_{1}+\frac{2C-1}{2}\|\bm{\xi}_{1}\|_{2}^{2}$

• •

$\|\bm{\alpha}_{2}\|_{1}-\frac{1}{2}\bm{\alpha}_{2}^{\intercal}\tilde{\bm{K}}_{2}\bm{\alpha}_{2}-\frac{1}{2}\|\bm{\xi}_{2}\|_{2}^{2}=\frac{1}{2}\bm{\alpha}_{2}^{\intercal}\tilde{\bm{K}}_{2}\bm{\alpha}_{2}+\frac{2C-1}{2}\|\bm{\xi}_{2}\|_{2}^{2}$

Applying these equations, we continue our inequality from before,

[TABLE]

In the second to last line, we recall that $C=\frac{1}{2}$, which implies $2C-1=0$. ∎

Appendix B Proof of Many Kernels (\texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3)

\texorpdfstring\hyperref**[thm:svm-many-kernels]Theorem 3Theorem 3 Restated****.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},k_{2},\ldots,k_{m}$ be kernel functions. Define $k_{{}_{\Sigma}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t=1}^{m}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\Sigma}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\Sigma}}$ be the corresponding Dual SVM solutions. Then we have

[TABLE]

Furthermore

[TABLE]

Proof.

Let $\ell\;{\vcentcolon=}\;\left\lceil{\log_{2}(m)}\right\rceil$ be the length of labels we give our base kernels. Now, rename each kernel $k_{t}$ with the length $\ell$ bitstring representation of the number $t$. For instance, if $\ell=4$ then we rename $k_{6}$ to $k_{0110}$. For every length $\ell-1$ bitstring $b_{0}b_{1}\ldots b_{\ell-1}$, define a new kernel

[TABLE]

Repeat this process of labeling with length $\ell-2$ bitstrings and so on until we have defined $k_{0}$ and $k_{1}$. Lastly, we define

[TABLE]

.

Now, recall \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2 (or \texorpdfstring\hyperref[thm:two-kernels-slack]Theorem 6Theorem 6 if we are using the SVM with slack). Let $[b_{\ell}]\;{\vcentcolon=}\;\{b_{0}\ldots b_{\ell}|b\in\{0,1\}\}$ denote the set of all length $\ell$ bitstrings. Also, for every kernel $k_{b_{0}\ldots b_{j}}$, compute the associated kernel matrix $\tilde{\bm{K}}_{b_{0}\ldots b_{j}}$ and dual solution vector $\bm{\alpha}_{b_{0}\ldots b_{j}}$.

Claim 1**.**

Fix $j\in[\ell-1]$. Then

[TABLE]

This claim follows from induction. In the base case, $j=1$, and \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2 tells us that $\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}\leq\frac{1}{3}(\bm{\alpha}_{0}^{\intercal}\tilde{\bm{K}}_{0}\bm{\alpha}_{0}+\bm{\alpha}_{1}^{\intercal}\tilde{\bm{K}}_{1}\bm{\alpha}_{1})$, matching the claim. Now, assume the claim holds for $j-1$. Then,

[TABLE]

This completes the proof of the claim.

Now we need to be careful when moving to the length $\ell$ kernel labels because if $m$ is not a power of two, then only some of the kernels have a length $\ell$ label. Let ${\mathcal{A}}$ be the set of all base kernels that have a length $\ell-1$ label. Let ${\mathcal{B}}$ be the rest of the base kernels, with a length $\ell$ label. By \texorpdfstring\hyperref[clm:kernel-sum-induction]Claim 1Claim 1, we know that

[TABLE]

Where the second inequalty applies \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2 and the last equality uses the fact that all base kernels are in either ${\mathcal{B}}$ or ${\mathcal{A}}$.

Lastly, recall that $\ell=\left\lceil{\log_{2}(m)}\right\rceil$.

[TABLE]

Therefore, overall, we have

[TABLE]

∎

Appendix C Proof of Kernel Sum Rademacher (\texorpdfstring\hyperref[thm:rademacher-kernel-sum]Theorem 4Theorem 4)

\texorpdfstring\hyperref**[thm:rademacher-kernel-sum]Theorem 4Theorem 4 Restated****.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},\ldots,k_{m}$ be kernel functions. Define $k_{{}_{\Sigma}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t=1}^{m}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\Sigma}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\Sigma}}$ be the corresponding Dual SVM solutions. Then,

[TABLE]

Further, if we assume $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ and $k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{i})\leq R^{2}$ for all $t\in[m],i\in[n]$, then

[TABLE]

This proof very closely parallels that of Lemma 22 in [4]. We produce the entire proof here for completeness. First, note that

[TABLE]

Where $\bm{\phi}_{{}_{\Sigma}}$ is the concatenation of the feature spaces associated with each of the $m$ kernels, and $B_{{}_{\Sigma}}^{2}=\bm{\alpha}_{{}_{\Sigma}}^{\intercal}\tilde{\bm{K}}_{{}_{\Sigma}}\bm{\alpha}_{{}_{\Sigma}}$. Then,

[TABLE]

The second inequality is Jensen’s, and the last inequality is \texorpdfstring\hyperref[thm:svm-many-kernels]Theorem 3Theorem 3. This completes the first part of the proof. We can then substitute in $B^{2}$ and $R^{2}$:

[TABLE]

Appendix D Proof of Learning Kernels (\texorpdfstring\hyperref[thm:learn-kernel-bound]Theorem 5Theorem 5)

\texorpdfstring\hyperref**[thm:learn-kernel-bound]Theorem 5Theorem 5 Restated****.**

Let ${\mathcal{S}}=\{(\bm{\mathrm{x}}_{1},y_{1}),\ldots,(\bm{\mathrm{x}}_{n},y_{n})\}$ be a dataset. Let $k_{1},\ldots,k_{m}$ be kernel functions. Consider any ${\mathcal{P}}\subseteq[m]$. Define $k_{{}_{\mathcal{P}}}(\cdot,\cdot)\;{\vcentcolon=}\;\sum_{t\in{\mathcal{P}}}k_{t}(\cdot,\cdot)$. Let $\tilde{\bm{K}}_{1},\ldots,\tilde{\bm{K}}_{m},\tilde{\bm{K}}_{{}_{\mathcal{P}}}$ be their labeled kernel matrices and $\bm{\alpha}_{1},\ldots,\bm{\alpha}_{m},\bm{\alpha}_{{}_{\mathcal{P}}}$ be the corresponding Dual SVM solutions. Assume $k_{t}(\bm{\mathrm{x}}_{i},\bm{\mathrm{x}}_{i})\leq R^{2}$ and $\bm{\alpha}_{t}^{\intercal}\tilde{\bm{K}}_{t}\bm{\alpha}_{t}\leq B^{2}$ for all $t\in[m]$ and $i\in[n]$. Then,

[TABLE]

where $\eta_{0}=\frac{23}{22}$.

This proof closely follows that of Theorem 1 in [10].

Proof.

Let $s\;{\vcentcolon=}\;\left|{{\mathcal{P}}}\right|$. Let $\bm{\mathrm{w}}_{{}_{\mathcal{P}}}$ be the optimal Primal SVM solution using subset of kernels ${\mathcal{P}}$. Note that $\bm{\mathrm{w}}_{{}_{\mathcal{P}}}$ is a concatenation of $s$ labeled and scaled feature vectors. To be precise, let $\bm{\phi}_{t}$ be the feature map for the $t^{th}$ kernel and define $\bm{\mathrm{w}}_{t}\;{\vcentcolon=}\;\sum_{i=1}^{n}\alpha_{i}y_{i}\bm{\phi}_{t}(\bm{\mathrm{x}}_{i})$. Then $\bm{\mathrm{w}}_{{}_{\mathcal{P}}}=\left[\begin{smallmatrix}\bm{\mathrm{w}}_{t_{1}}^{\intercal}&\ldots&\bm{\mathrm{w}}_{t_{s}}^{\intercal}\end{smallmatrix}\right]^{\intercal}$, where $t_{i}$ is the $i^{th}$ smallest element of ${\mathcal{P}}$.

Consider some $q,r>1$ such that $\frac{1}{q}+\frac{1}{r}=1$. Then,

[TABLE]

The third line follows exactly from Lemma 5 in [10]. We bound both terms separately. We only substantially differ from the original proof in bounding the first term. To start, note that $f(x)=x^{\nicefrac{{1}}{{q}}}$ is subadditive for $\nicefrac{{1}}{{q}}<1$:

[TABLE]

The second inequality follows from Jensen’s, and the last inequality is \texorpdfstring\hyperref[thm:two-kernels]Theorem 2Theorem 2.

We start our bound of the second term by applying Jensen’s Inequality:

[TABLE]

We detour to bound the inner expectation. Assume that $r$ is an even integer. That is, $r=2p$ for some integer $p$.

[TABLE]

Where the last line follows from Lemma 1 in [11], where $\eta_{0}=\frac{23}{22}$. Returning to the bound of the second term,

[TABLE]

By differentiating, we find that $p=\ln(m)$ minimizes this expression. We required $p$ to be an integer, so we instead take $p=\left\lceil{\ln(m)}\right\rceil$.

[TABLE]

Combining the first and second terms’ bounds, we return to the bound of the Rademacher complexity itself:

[TABLE]

∎