The Minimum Information Principle for Discriminative Learning

TL;DR

This paper introduces the minimum mutual information principle for discriminative learning, proposing a new framework that outperforms traditional maximum entropy models in classification tasks.

Contribution

It generalizes maximum entropy models to a minimum mutual information framework, providing algorithms, theoretical bounds, and demonstrating improved classification performance.

Findings

Minimum mutual information classifiers outperform maximum entropy models.

The approach has a game-theoretic interpretation.

Algorithms effectively solve the proposed optimization problem.

Abstract

Exponential models of distributions are widely used in machine learning for classiffication and modelling. It is well known that they can be interpreted as maximum entropy models under empirical expectation constraints. In this work, we argue that for classiffication tasks, mutual information is a more suitable information theoretic measure to be optimized. We show how the principle of minimum mutual information generalizes that of maximum entropy, and provides a comprehensive framework for building discriminative classiffiers. A game theoretic interpretation of our approach is then given, and several generalization bounds provided. We present iterative algorithms for solving the minimum information problem and its convex dual, and demonstrate their performance on various classiffication tasks. The results show that minimum information classiffiers outperform the corresponding maximum entropy models.