Generalized Maximum Entropy for Supervised Classification

TL;DR

This paper introduces a generalized maximum entropy framework for supervised classification, leading to minimax risk classifiers with performance guarantees, connecting to existing methods and providing convex optimization-based learning techniques.

Contribution

It develops a unified approach for deriving minimax risk classifiers using generalized entropy functions and offers convex optimization methods with theoretical performance bounds.

Findings

MRCs are derived using convex optimization techniques.

The framework generalizes existing classification methods.

Performance bounds for MRCs are established.

Abstract

The maximum entropy principle advocates to evaluate events' probabilities using a distribution that maximizes entropy among those that satisfy certain expectations' constraints. Such principle can be generalized for arbitrary decision problems where it corresponds to minimax approaches. This paper establishes a framework for supervised classification based on the generalized maximum entropy principle that leads to minimax risk classifiers (MRCs). We develop learning techniques that determine MRCs for general entropy functions and provide performance guarantees by means of convex optimization. In addition, we describe the relationship of the presented techniques with existing classification methods, and quantify MRCs performance in comparison with the proposed bounds and conventional methods.