The Ellipsoidal Separation Machine

TL;DR

The paper introduces the Ellipsoidal Separation Machine (ESM), a scalable convex classification method based on ellipsoids, capable of handling classification with rejection by identifying indeterminate points, and competitive with SVM on various datasets.

Contribution

It presents the first fully functional implementation of the SCB approach for classification using ellipsoids, with a scalable nonconvex formulation and rejection capabilities.

Findings

ESM is competitive with SVM on multiple datasets.

ESM can explicitly identify indeterminate points for rejection.

The method scales better than standard SDP approaches.

Abstract

We propose the -- to the best of our knowledge -- first fully functional implementation of the ``Separation by a Convex Body'' (SCB) approach first outlined in Grzybowski et al. [1] for classification, separating two data sets using an ellipsoid. A training problem is defined that is structurally similar to the Support Vector Machine (SVM) one, thus leading to call our method the Ellipsoidal Separation Machine (ESM). Like SVM, the training problem is convex, and can in particular be formulated as a Semidefinite Program (SDP); however, solving it by means of standard SDP approaches does not scale to the size required by practical classification task. As an alternative, a nonconvex formulation is proposed that is amenable to a Block-Gauss-Seidel approach alternating between a much smaller SDP and a simple separable Second-Order Cone Program (SOCP). For the purpose of the classification approach the reduced SDP can even be solved approximately by relaxing it in a Lagrangian way and updating the multipliers by fast subgradient-type approaches. A characteristic of ESM is that it necessarily defines ``indeterminate points'', i.e., those that cannot be reliably classified as belonging to one of the two sets. This makes it particularly suitable for Classification with Rejection (CwR) tasks, whereby the system explicitly indicates that classification of some points as belonging to one of the two sets is too doubtful to be reliable. We show that, in many datasets, ESM is competitive with SVM -- with the kernel chosen among the three standard ones and endowed with CwR capabilities using the margin of the classifier -- and in general behaves differently; thus, ESM provides another arrow in the quiver when designing CwR approaches.