Universal Compression of Power-Law Distributions

TL;DR

This paper demonstrates that Zipf distributions, common in natural language, can be more efficiently compressed and predicted than previously thought, with distinct worst-case and expected redundancies.

Contribution

It establishes the relationship between the expected redundancy of Zipf distributions and unrestricted distributions, revealing their unique compression properties.

Findings

Expected redundancy of Zipf distributions scales as the inverse power of the unrestricted case.

Zipf distributions have significantly lower expected redundancy compared to unrestricted distributions.

Worst-case redundancy for Zipf and unrestricted distributions is approximately the same.

Abstract

English words and the outputs of many other natural processes are well-known to follow a Zipf distribution. Yet this thoroughly-established property has never been shown to help compress or predict these important processes. We show that the expected redundancy of Zipf distributions of order $\alpha>1$ is roughly the $1/\alpha$ power of the expected redundancy of unrestricted distributions. Hence for these orders, Zipf distributions can be better compressed and predicted than was previously known. Unlike the expected case, we show that worst-case redundancy is roughly the same for Zipf and for unrestricted distributions. Hence Zipf distributions have significantly different worst-case and expected redundancies, making them the first natural distribution class shown to have such a difference.