On Finite Memory Universal Data Compression and Classification of Individual Sequences

TL;DR

This paper explores the limits of finite-memory universal data compression and classification for individual sequences, demonstrating that context tree coding and a specific classifier are essentially optimal for these tasks.

Contribution

It shows that context tree coding nearly achieves the best possible universal compression for finite blocks and introduces an optimal universal classifier with linear storage complexity.

Findings

Context tree coding nearly achieves optimal universal compression.

A universal context classifier with linear storage is essentially optimal.

The results support the theoretical basis of PST in learning and biology.

Abstract

Consider the case where consecutive blocks of N letters of a semi-infinite individual sequence X over a finite-alphabet are being compressed into binary sequences by some one-to-one mapping. No a-priori information about X is available at the encoder, which must therefore adopt a universal data-compression algorithm. It is known that if the universal LZ77 data compression algorithm is successively applied to N-blocks then the best error-free compression for the particular individual sequence X is achieved, as $N$ tends to infinity. The best possible compression that may be achieved by any universal data compression algorithm for finite N-blocks is discussed. It is demonstrated that context tree coding essentially achieves it. Next, consider a device called classifier (or discriminator) that observes an individual training sequence X. The classifier's task is to examine individual test sequences of length N and decide whether the test N-sequence has the same features as those that are captured by the training sequence X, or is sufficiently different, according to some appropriatecriterion. Here again, it is demonstrated that a particular universal context classifier with a storage-space complexity that is linear in N, is essentially optimal. This may contribute a theoretical "individual sequence" justification for the Probabilistic Suffix Tree (PST) approach in learning theory and in computational biology.