Efficient LZ78 factorization of grammar compressed text

TL;DR

This paper introduces an efficient algorithm for computing the LZ78 factorization directly from a grammar-compressed text represented as an SLP, significantly improving performance on compressible data.

Contribution

The authors develop a novel algorithm that computes LZ78 factorization from an SLP in sublinear time relative to the uncompressed size, with improvements based on the text's redundancy.

Findings

Algorithm runs in $O(n ext{L} + m ext{log}N)$ time for certain conditions.

Performance improves with higher redundancy and smaller SLP size.

Approaches linear time for highly compressible texts.

Abstract

We present an efficient algorithm for computing the LZ78 factorization of a text, where the text is represented as a straight line program (SLP), which is a context free grammar in the Chomsky normal form that generates a single string. Given an SLP of size $n$ representing a text $S$ of length $N$, our algorithm computes the LZ78 factorization of $T$ in $O(n\sqrt{N}+m\log N)$ time and $O(n\sqrt{N}+m)$ space, where $m$ is the number of resulting LZ78 factors. We also show how to improve the algorithm so that the $n\sqrt{N}$ term in the time and space complexities becomes either $nL$, where $L$ is the length of the longest LZ78 factor, or $(N - \alpha)$ where $\alpha \geq 0$ is a quantity which depends on the amount of redundancy that the SLP captures with respect to substrings of $S$ of a certain length. Since $m = O(N/\log_\sigma N)$ where $\sigma$ is the alphabet size, the latter is asymptotically at least as fast as a linear time algorithm which runs on the uncompressed string when $\sigma$ is constant, and can be more efficient when the text is compressible, i.e. when $m$ and $n$ are small.