Classification by Separating Hypersurfaces: An Entropic Approach

TL;DR

This paper introduces an entropic method for classification that finds separating hypersurfaces, extending to polynomial boundaries, offering a robust alternative to traditional techniques like SVMs and gradient descent.

Contribution

The paper proposes a novel entropy-based approach for finding separating hypersurfaces, including polynomial surfaces, in classification tasks, enhancing robustness and versatility.

Findings

Efficiently separates linear and non-linear data points.

Extends to polynomial decision boundaries.

Demonstrates effectiveness in diverse classification tasks.

Abstract

We consider the following classification problem: Given a population of individuals characterized by a set of attributes represented as a vector in ${\mathbb R}^N$, the goal is to find a hyperplane in ${\mathbb R}^N$ that separates two sets of points corresponding to two distinct classes. This problem, with a history dating back to the perceptron model, remains central to machine learning. In this paper we propose a novel approach by searching for a vector of parameters in a bounded $N$-dimensional hypercube centered at the origin and a positive vector in ${\mathbb R}^M$, obtained through the minimization of an entropy-based function defined over the space of unknown variables. The method extends to polynomial surfaces, allowing the separation of data points by more complex decision boundaries. This provides a robust alternative to traditional linear or quadratic optimization techniques, such as support vector machines and gradient descent. Numerical experiments demonstrate the efficiency and versatility of the method in handling diverse classification tasks, including linear and non-linear separability.