LZ-Compressed String Dictionaries

TL;DR

This paper presents a method for compressing string dictionaries using LZ78, achieving high compression ratios especially with repetitive data, while maintaining competitive query performance.

Contribution

It introduces a novel approach applying LZ78 compression to string dictionaries, outperforming existing methods in compression ratio on large, repetitive datasets.

Findings

Achieves superior compression ratios on large, repetitive dictionaries

Maintains competitive query times

Validated on datasets up to 1.5 GB

Abstract

We show how to compress string dictionaries using the Lempel-Ziv (LZ78) data compression algorithm. Our approach is validated experimentally on dictionaries of up to 1.5 GB of uncompressed text. We achieve compression ratios often outperforming the existing alternatives, especially on dictionaries containing many repeated substrings. Our query times remain competitive.