Multithreshold Entropy Linear Classifier

TL;DR

This paper introduces a novel multithreshold linear classifier based on information theory, capable of separating data with multiple hyperplanes and optimizing balanced quality measures, outperforming SVM in some cases.

Contribution

The paper proposes a new multithreshold entropy linear classifier that maximizes balanced quality measures and broadens the hypothesis class compared to traditional SVMs.

Findings

MELC achieves similar or higher scores than SVM on synthetic and real datasets.

The method is data scale invariant and provides insights into data structure.

It is beneficial for cheminformatics tasks like ligand activity prediction.

Abstract

Linear classifiers separate the data with a hyperplane. In this paper we focus on the novel method of construction of multithreshold linear classifier, which separates the data with multiple parallel hyperplanes. Proposed model is based on the information theory concepts -- namely Renyi's quadratic entropy and Cauchy-Schwarz divergence. We begin with some general properties, including data scale invariance. Then we prove that our method is a multithreshold large margin classifier, which shows the analogy to the SVM, while in the same time works with much broader class of hypotheses. What is also interesting, proposed method is aimed at the maximization of the balanced quality measure (such as Matthew's Correlation Coefficient) as opposed to very common maximization of the accuracy. This feature comes directly from the optimization problem statement and is further confirmed by the experiments on the UCI datasets. It appears, that our Multithreshold Entropy Linear Classifier (MELC) obtaines similar or higher scores than the ones given by SVM on both synthetic and real data. We show how proposed approach can be benefitial for the cheminformatics in the task of ligands activity prediction, where despite better classification results, MELC gives some additional insight into the data structure (classes of underrepresented chemical compunds).