Entropy Rate Bounds via Second-Order Statistics

TL;DR

This paper introduces two new single-letter upper bounds on the entropy rate of discrete stationary processes using second-order statistics, which are computationally efficient and outperform standard bounds.

Contribution

It presents novel entropy rate bounds based solely on second-order statistics, applicable to large alphabet models, and provides practical methods for their numerical computation.

Findings

The bounds outperform standard conditional entropy bounds.

Both bounds are expressed as one-dimensional integrals, with the second involving a minimization.

The bounds are suitable for processes with large alphabets.

Abstract

This work contains two single-letter upper bounds on the entropy rate of a discrete-valued stationary stochastic process, which only depend on second-order statistics, and are primarily suitable for models which consist of relatively large alphabets. The first bound stems from Gaussian maximum-entropy considerations and depends on the power spectral density (PSD) function of the process. While the PSD function cannot always be calculated in a closed-form, we also propose a second bound, which merely relies on some finite collection of auto-covariance values of the process. Both of the bounds consist of a one-dimensional integral, while the second bound also consists of a minimization problem over a bounded region, hence they can be efficiently calculated numerically. Examples are also provided to show that the new bounds outperform the standard conditional entropy bound.