Compressing Tabular Data via Latent Variable Estimation

TL;DR

This paper presents a lossless compression method for tabular data that estimates latent variables, partitions data, and applies sequential coding, outperforming classical methods in achieving optimal compression rates.

Contribution

Introduces a novel latent variable-based compression algorithm for tabular data with theoretical analysis and empirical validation, outperforming classical schemes.

Findings

The proposed method achieves the optimal entropy rate.

Classical schemes like Lempel-Ziv do not reach the optimal rate.

The model satisfies an asymptotic equipartition property.

Abstract

Data used for analytics and machine learning often take the form of tables with categorical entries. We introduce a family of lossless compression algorithms for such data that proceed in four steps: $(i)$ Estimate latent variables associated to rows and columns; $(ii)$ Partition the table in blocks according to the row/column latents; $(iii)$ Apply a sequential (e.g. Lempel-Ziv) coder to each of the blocks; $(iv)$ Append a compressed encoding of the latents. We evaluate it on several benchmark datasets, and study optimal compression in a probabilistic model for that tabular data, whereby latent values are independent and table entries are conditionally independent given the latent values. We prove that the model has a well defined entropy rate and satisfies an asymptotic equipartition property. We also prove that classical compression schemes such as Lempel-Ziv and finite-state encoders do not achieve this rate. On the other hand, the latent estimation strategy outlined above achieves the optimal rate.