Concentration of Measure Inequalities and Their Communication and Information-Theoretic Applications

TL;DR

This paper surveys the role of information-theoretic techniques such as the entropy method and transportation-cost inequalities in deriving concentration of measure inequalities, highlighting their applications in information theory and related fields.

Contribution

It introduces three main information-theoretic techniques for deriving concentration inequalities and illustrates their applications in communications and coding theory.

Findings

Martingale, entropy, and transportation-cost methods are key tools.

Concentration inequalities are crucial in high-dimensional probability.

Applications include bounds in information theory and coding.

Abstract

During the last two decades, concentration of measure has been a subject of various exciting developments in convex geometry, functional analysis, statistical physics, high-dimensional statistics, probability theory, information theory, communications and coding theory, computer science, and learning theory. One common theme which emerges in these fields is probabilistic stability: complicated, nonlinear functions of a large number of independent or weakly dependent random variables often tend to concentrate sharply around their expected values. Information theory plays a key role in the derivation of concentration inequalities. Indeed, both the entropy method and the approach based on transportation-cost inequalities are two major information-theoretic paths toward proving concentration. This brief survey is based on a recent monograph of the authors in the Foundations and Trends in Communications and Information Theory (online available at http://arxiv.org/pdf/1212.4663v8.pdf), and a tutorial given by the authors at ISIT 2015. It introduces information theorists to three main techniques for deriving concentration inequalities: the martingale method, the entropy method, and the transportation-cost inequalities. Some applications in information theory, communications, and coding theory are used to illustrate the main ideas.