Llamazip: Leveraging LLaMA for Lossless Text Compression and Training Dataset Detection

TL;DR

Llamazip is a lossless text compression method leveraging LLaMA's predictive power, reducing data size and enabling detection of training data origin, thus enhancing storage efficiency and transparency in language models.

Contribution

Introduces Llamazip, a novel compression algorithm based on LLaMA, and demonstrates its ability to identify training data provenance, addressing data privacy and transparency issues.

Findings

Llamazip achieves high compression ratios with minimal data loss.

The method can reliably detect if a document was part of the training dataset.

Performance is influenced by quantization and context window size.

Abstract

This work introduces Llamazip, a novel lossless text compression algorithm based on the predictive capabilities of the LLaMA3 language model. Llamazip achieves significant data reduction by only storing tokens that the model fails to predict, optimizing storage efficiency without compromising data integrity. Key factors affecting its performance, including quantization and context window size, are analyzed, revealing their impact on compression ratios and computational requirements. Beyond compression, Llamazip demonstrates the potential to identify whether a document was part of the training dataset of a language model. This capability addresses critical concerns about data provenance, intellectual property, and transparency in language model training.