Test-Time Steering for Lossless Text Compression via Weighted Product of Experts

TL;DR

This paper introduces a novel test-time method that adaptively combines universal and neural language models using a weighted product of experts to improve lossless text compression without additional training.

Contribution

It proposes a test-time steering framework with weighted product of experts to enhance neural compression models' generalization to unseen data.

Findings

Improves compression rates over individual models

No fine-tuning required for new data distributions

Compatible with any autoregressive language model

Abstract

Lossless compression techniques are crucial in an era of rapidly growing data. Traditional universal compressors like gzip offer low computational overhead, high speed, and broad applicability across data distributions. However, they often lead to worse compression rates than modern neural compressors, which leverage large-scale training data to model data distributions more effectively. Despite their advantages, neural compressors struggle to generalize to unseen data. To address this limitation, we propose a novel framework that performs Test-Time Steering via a Weighted Product of Experts (wPoE). At inference, our method adaptively combines a universal compression model with a pretrained neural language model, ensuring the compression rate is at least as good as that of the best individual model. Extensive experiments demonstrate that our approach improves the performance of text compression without requiring fine-tuning. Furthermore, it seamlessly integrates with any autoregressive language model, providing a practical solution for enhancing text compression across diverse data distributions.