LZD-style Compression Scheme with Truncation and Repetitions

TL;DR

This paper introduces enhanced LZD-based compression algorithms, LZD+ and LZDR, which achieve linear-time online compression and outperform existing methods in factorization efficiency on standard datasets.

Contribution

The paper presents LZD+ and LZDR, novel improvements to LZD that enable linear-time compression and better factorization, addressing previous inefficiencies and weaknesses.

Findings

LZD+ achieves expected linear-time compression.

LZDR introduces repetition-based factorization for improved compression.

Benchmarking shows superior factor reduction over existing LZ methods.

Abstract

Lempel-Ziv-Double (LZD) is a variation of the LZ78 compression scheme that achieves better compression on repetitive datasets. Nevertheless, prior research has identified computational inefficiencies and a weakness in its compressibility for certain datasets. In this paper, we introduce LZD+, an enhancement of LZD, which enables expected linear-time online compression by allowing truncated references. To avoid the compressibility weakness exhibited by a lower bound example, we propose LZDR (LZD-runlength compressed), a further enhancement on top of LZD+, which introduces a repetition-based factorization rule while maintaining linear expected time complexity. The both time bounds can be de-randomized by a lookup data structure like a balanced search tree with a logarithmic dependency on the alphabet size. Additionally, we present three flexible parsing variants of LZDR that yield fewer factors in practice. Comprehensive benchmarking on standard corpora reveals that LZD+, LZDR, and its flexible variants outperform existing LZ-based methods in the number of factors while keeping competitive runtime efficiency. However, we note that the difference in the number of factors becomes marginal for large datasets like those of the Pizza&Chili corpus.