Towards Weak Source Coding

TL;DR

This paper introduces a weak decoding approach to source coding, deriving bounds that suggest potential improvements in reliability when classifying many small, sparse atypical source sequence clusters.

Contribution

It presents a novel weak decoding version of Shannon's source coding theorem and derives bounds indicating possible reliability improvements with many small, sparse classes.

Findings

Potential for improved reliability with high class counts

Bound derived for weak decoding regime

Applicable when atypical sequences are small and sparsely distributed

Abstract

In this paper, the authors provide a weak decoding version of the traditional source coding theorem of Claude Shannon. The central bound that is obtained is \[ \chi>\log_{\epsilon}(2^{-n(H(X)+\epsilon)}) \] where \[ \chi=\frac{\log(k)}{n(H(X)+\epsilon)} \] and $k$ is the number of unsupervised learning classes formed out of the non-typical source sequences. The bound leads to the conclusion that if the number of classes is high enough, the reliability function might possibly be improved. The specific regime in which this improvement might be allowable is the one in which the atypical-sequence clusters are small in size and sparsely placed; similar regimes might also show an improvement.