Lossless Transformations and Excess Risk Bounds in Statistical Inference

TL;DR

This paper investigates the concept of lossless transformations in statistical inference, characterizes their properties, develops tests for their detection, and provides bounds on excess risk applicable across various loss functions and applications.

Contribution

It introduces the notion of lossless and delta-lossless transformations, offers tests for their identification, and derives universal bounds on excess risk for broad classes of loss functions.

Findings

Lossless transformations have zero excess risk for all loss functions.

A partitioning test for losslessness is strongly consistent for i.i.d. data.

Universal bounds on excess risk are established for general loss functions.

Abstract

We study the excess minimum risk in statistical inference, defined as the difference between the minimum expected loss in estimating a random variable from an observed feature vector and the minimum expected loss in estimating the same random variable from a transformation (statistic) of the feature vector. After characterizing lossless transformations, i.e., transformations for which the excess risk is zero for all loss functions, we construct a partitioning test statistic for the hypothesis that a given transformation is lossless and show that for i.i.d. data the test is strongly consistent. More generally, we develop information-theoretic upper bounds on the excess risk that uniformly hold over fairly general classes of loss functions. Based on these bounds, we introduce the notion of a delta-lossless transformation and give sufficient conditions for a given transformation to be universally delta-lossless. Applications to classification, nonparametric regression, portfolio strategies, information bottleneck, and deep learning, are also surveyed.