Studying the Interplay between Information Loss and Operation Loss in Representations for Classification

TL;DR

This paper explores the relationship between information loss and operational loss in feature representations for classification, showing that minimal information loss can imply minimal classification error, but the design principles based solely on information sufficiency may be overly conservative.

Contribution

It provides theoretical bounds linking information loss to classification error and introduces weaker informational sufficiency notions that still ensure operational sufficiency in learning.

Findings

Weak information loss bounds operational loss in classification.

Vanishing information loss implies vanishing error in representations.

Weaker informational sufficiency can achieve operational sufficiency.

Abstract

Information-theoretic measures have been widely adopted in the design of features for learning and decision problems. Inspired by this, we look at the relationship between i) a weak form of information loss in the Shannon sense and ii) the operation loss in the minimum probability of error (MPE) sense when considering a family of lossy continuous representations (features) of a continuous observation. We present several results that shed light on this interplay. Our first result offers a lower bound on a weak form of information loss as a function of its respective operation loss when adopting a discrete lossy representation (quantization) instead of the original raw observation. From this, our main result shows that a specific form of vanishing information loss (a weak notion of asymptotic informational sufficiency) implies a vanishing MPE loss (or asymptotic operational sufficiency) when considering a general family of lossy continuous representations. Our theoretical findings support the observation that the selection of feature representations that attempt to capture informational sufficiency is appropriate for learning, but this selection is a rather conservative design principle if the intended goal is achieving MPE in classification. Supporting this last point, and under some structural conditions, we show that it is possible to adopt an alternative notion of informational sufficiency (strictly weaker than pure sufficiency in the mutual information sense) to achieve operational sufficiency in learning.