A Modified Construction for a Support Vector Classifier to Accommodate Class Imbalances

TL;DR

This paper proposes a modified support vector classifier that adjusts margins based on class variance to better handle imbalanced data, improving classification accuracy.

Contribution

It introduces a novel SVM formulation with class-specific margins proportional to class standard deviations, enhancing performance on imbalanced datasets.

Findings

Improved classification accuracy on imbalanced datasets

The modified SVM reduces bias towards the majority class

The approach generalizes standard SVM when class variances are equal

Abstract

Given a training set with binary classification, the Support Vector Machine identifies the hyperplane maximizing the margin between the two classes of training data. This general formulation is useful in that it can be applied without regard to variance differences between the classes. Ignoring these differences is not optimal, however, as the general SVM will give the class with lower variance an unjustifiably wide berth. This increases the chance of misclassification of the other class and results in an overall loss of predictive performance. An alternate construction is proposed in which the margins of the separating hyperplane are different for each class, each proportional to the standard deviation of its class along the direction perpendicular to the hyperplane. The construction agrees with the SVM in the case of equal class variances. This paper will then examine the impact to the dual representation of the modified constraint equations.

Equations

$$\{x:f(x)=x^T\beta +{\beta }_0=0\}$$

$$G(x)=\text{sign}[x^T\beta +{\beta }_0]$$

$$\underset{\beta ,{\beta }_0,\Vert \beta \Vert =1}{\max }M$$

$$\text{subject to }{y}_i(x_i^T\beta +{\beta }_0)\ge Mi=1,...,N$$

$$\underset{\beta ,{\beta }_0}{\min }\Vert \beta \Vert$$

$$\text{subject to }{y}_i(x_i^T\beta +{\beta }_0)\ge 1i=1,...,N$$

$${\zeta }_i=\max (0,1-y_i(x_i^T\beta +{\beta }_0))$$

$$\sum _{i=1}^N{\zeta }_i\le \text{constant}$$

$$\underset{\beta ,{\beta }_0}{\min }\frac{1}{2}\Vert \beta {\Vert }^2+C\sum _{i=1}^N{\zeta }_i$$

$$\text{subject to }{\zeta }_i\ge 0,y_i(x_i^T\beta +{\beta }_0)\ge 1-{\zeta }_i\forall i$$

$$L_P=\frac{1}{2}\Vert \beta {\Vert }^2+C\sum _{i=1}^N{\zeta }_i-\sum _{i=1}^N{\alpha }_i[y_i(x_i^T\beta +{\beta }_0)-(1-{\zeta }_i)]-\sum _{i=1}^N{\mu }_i{\zeta }_i$$

$$\beta$$

$$0$$

$${\alpha }_i$$

$$L_D$$

$$=\sum _{i=1}^N{\alpha }_i-\frac{1}{2}\sum _{i=1}^N\sum _{i^{\prime }=1}^N{\alpha }_i{\alpha }_{i^{\prime }}{y}_i{y}_{i^{\prime }}⟨x_i,x_{i^{\prime }}⟩$$

$${\alpha }_i[y_i(x_i^T\beta +{\beta }_0-(1-{\zeta }_i)]$$

$${\mu }_i{\zeta }_i$$

$$y_i(x_i^T\beta +{\beta }_0)-(1-{\zeta }_i)$$

$${\sigma }_{K,\beta }={\sigma }_{y_j,\beta }$$

$$={\left(\sum _{j:y_j=y_i}{\left[\left(x_j-\overline{x}\right)\cdot \left(\frac{\beta }{\Vert \beta \Vert }\right)\right]}^2\right)}^{\frac{1}{2}}$$

$$M_K=\underset{x_i\in K}{\min }{y}_i\left(\frac{x_i^T\beta +{\beta }_0}{{\sigma }_{y_i,\beta }}\right)$$

$$\frac{M_A}{{\sigma }_{A,\beta }}=\frac{M_B}{{\sigma }_{B,\beta }}$$

$$\underset{\beta ,{\beta }_0}{\min }\Vert \beta \Vert$$

$$\text{subject to }{y}_i\left(\frac{x_i^T\beta +{\beta }_0}{{\sigma }_{y_i,\beta }}\right)\ge 1i=1,...,N$$

$${\zeta }_i=\max \left(0,1-y_i\left(\frac{x_i^T\beta +{\beta }_0}{{\sigma }_{y_i,\beta }}\right)\right)$$

$$\underset{\beta ,{\beta }_0}{\min }\frac{1}{2}\Vert \beta {\Vert }^2+C\sum _{i=1}^N{\zeta }_i$$

$$\text{subject to }{\zeta }_i\ge 0,y_i\left(\frac{x_i^T\beta +{\beta }_0}{{\sigma }_{y_i,\beta }}\right)\ge 1-{\zeta }_i\forall i$$

$$L_P=\frac{1}{2}\Vert \beta {\Vert }^2+C\sum _{i=1}^N{\zeta }_i-\sum _{i=1}^N{\alpha }_i\left[y_i{\sigma }_{y_i,\beta }^{-1}(x_i^T\beta +{\beta }_0)-(1-{\zeta }_i)\right]-\sum _{i=1}^N{\mu }_i{\zeta }_i$$

$$0$$

$${\alpha }_i$$

$$0$$

$$={\nabla }_{\beta }\left(\frac{1}{2}\Vert \beta {\Vert }^2-\sum _{i=1}^N\left({\alpha }_i{y}_i\right)\left({\sigma }_{y_i,\beta }^{-1}\right)\left(x_i^T\beta +{\beta }_0\right)\right)$$

$$=\beta -\sum _{i=1}^N{\alpha }_i{y}_i{x}_i{\sigma }_{y_i,\beta }^{-1}+\sum _{i=1}^N({\alpha }_i{y}_i)\left({\sigma }_{y_i,\beta }^{-2}\right)\left(x_i^T\beta +{\beta }_0\right)\left({\nabla }_{\beta }{\sigma }_{y_i,\beta }\right)$$

$${\nabla }_{\beta }\left(\left(x_j-\overline{x}\right)\cdot \left(\frac{\beta }{\Vert \beta \Vert }\right)\right)=\left(x_j-\overline{x}\right)\cdot \left(\frac{\Vert \beta {\Vert }^2-\beta \circ \beta }{\Vert \beta {\Vert }^3}\right)$$

$$\begin{gathered} 0=\beta -\sum _{i=1}^N{\alpha }_i{y}_i{x}_i{\sigma }_{y_i,\beta }^{-1}+ \\ +\sum _{i=1}^N\left[{\alpha }_i{y}_i{\sigma }_{y_i,\beta }^{-3}\left(x_i^T\beta +{\beta }_0\right)\left(\sum _{j:y_j=y_i}\left[\left(x_j-\overline{x}\right)\cdot \left(\frac{\beta }{\Vert \beta \Vert }\right)\right]\left[\left(x_j-\overline{x}\right)\left(\frac{\overrightarrow{1}\Vert \beta {\Vert }^2-\beta \circ \beta }{\Vert \beta {\Vert }^3}\right)\right]\right)\right] \end{gathered}$$

Full text

1 A Recap: The Classical SVM Construction

For Section 1, we follow the construction given by Hastie, Tibshirani, and Freidman in The Elements of Statistical Learning [3]. We will parallel this approach in Section 2 when constructing the alternate method.

Suppose we have training data consisting of pairs of observations and labels, $(x_{i},y_{i})$, for $i=1,...,N,$ with $x_{i}\in\mathbb{R}^{p}$ and $y_{i}\in\{-1,1\}$. We may define a hyperplane by:

[TABLE]

where $\beta$ is a vector perpendicular to the hyperplane. An associated classification rule is induced by:

[TABLE]

The goal of finding a separating hyperplane which maximizes the margin $M$ for a linearly separable dataset, the minimum perpendicular distance to a datapoint of either class, can be formalized as:

[TABLE]

This can be more conveniently rephrased by removing the requirement $\beta$ be a unit vector, and setting $M=\frac{1}{\|\beta\|}$:

[TABLE]

Now define slack variables $\zeta_{i},i=1,...,N$ by

[TABLE]

This gives us a framework to relax the assumption of linear separability. Noting that misclassifications occur when $\zeta_{i}>1$, we see the slack variables are the proportion of the margin by which various points fall within their respective margins. We may control the amount of slack by imposing the additional condition:

[TABLE]

for some constant. This is computationally equivalent to the following expression:

[TABLE]

where the parameter $C$ replaces the constant in the previous expression. The corresponding Lagrange primal function is given by:

[TABLE]

which is to be minimized with respect to $\beta,\beta_{0}$, and $\zeta_{i}$. Setting the respective derivatives equal to zero, we get the equations:

[TABLE]

and positivity constraints $\alpha_{i},\mu_{i},\zeta_{i}\geq 0\forall i$. By substituting the above three equations into the Lagrangian dual we obtain the Wolfe dual, given by:

[TABLE]

In addition, the Karush-Kuhn-Tucker conditions yield:

[TABLE]

for $i=1,...,N$. These equations collectively uniquely define the solution to the dual problem.

2 A Modified Approach: Accommodating Difference in Class Variance

The original construction of the SVM for linearly separable data has the goal of maximizing the margin $M=\frac{1}{\|\beta\|}$. In the event of a noticeable difference between class variances in the direction of $\beta$ (perpendicular to our separating hyperplane), the SVM ends up positioning the decision boundary closer to the class with larger variance [say, class A] than would be optimal. The new construction accommodates these class imbalances by increasing the margin of the class of greater variance.

It will be useful at this point to define a few terms. For class $K$, element $x_{j}\in K$, and separating hyperplane $\{x:x^{T}\beta+\beta_{0}=0\}$, define $\sigma_{K,\beta}=\sigma_{y_{j},\beta}$ to be the standard deviation of elements of class $K$ in the direction of $\beta$:

[TABLE]

and, for class $K$ and arbitrary hyperplane $\{x:x^{T}\beta+\beta_{0}=0\}$, define the margin of class $K$ to be:

[TABLE]

We will now seek to find the separating hyperplane which maximizes $\min_{K}M_{K}$, the minimum margin over all classes. As an aside, a byproduct of the classic construction of the SVM yields the equality $M_{A}=M_{B}$ when separating classes $A$ and $B$, since the maximum margin is obtained when the separating hyperplane is midway between both classes. Our new construction will yield as a byproduct the equality:

[TABLE]

This shows that in the event our classes have equal variance in the direction of $\beta$, the modified construction coincides with the classical SVM.

3 Examining Implications to Dual Representation

Maximizing $\min_{K}M_{K}$ modifies the optimization problem to the pair of equations:

[TABLE]

Slightly redefining slack variables according to the fraction of the respective margins they span yields:

[TABLE]

and the corresponding modified SVM equations are given by:

[TABLE]

We can now formulate the corresponding Lagrangian (primal) function as:

[TABLE]

which we again minimize with respect to $\beta,\beta_{0}$, and $\zeta_{i}$. Setting derivatives with respect to $\beta_{0}$ and $\zeta_{i}$ equal to zero, we get similar results:

[TABLE]

and a slightly more complex equation when doing the same with respect to $\beta$:

[TABLE]

Expanding $\sigma_{y_{i},\beta}$ to its representation in (21), we may utilize the Hadamard product notation $\circ$ and the fact

[TABLE]

where $\circ$ is the Hadamard product, to obtain:

[TABLE]

where $\overrightarrow{\mathbf{1}}$ is the vector of ones [1, … , 1].

This gives us a working representation of the equivalent dual optimization equations under the new construction, and a forthcoming paper will be examining the solvability of the above in general in light of the other constraint equations, as well as consequent impacts to kernelizability of the method. We will also examine in depth the circumstances in which our alternate construction outperforms a traditional Support Vector Classifier, and attempt to quantify them.