Information Theoretic Perspective on Representation Learning

TL;DR

This paper introduces an information-theoretic framework to analyze learned representations in regression tasks, defining concepts like representation-rate and capacity, and deriving fundamental limits on information encoding and compression.

Contribution

It presents a novel theoretical framework that quantifies the limits of representation learning using information theory, including capacity and rate-distortion bounds.

Findings

Derived limits on representation reliability based on input-source entropy

Defined and analyzed representation capacity in a perturbed setting

Established achievable bounds and unified the theoretical results

Abstract

An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.