Haiku to Opus in Just 10 bits: LLMs Unlock Massive Compression Gains

TL;DR

This paper explores advanced compression techniques for LLM-generated text, including lossless, lossy, and interactive protocols, achieving significant reductions in data size while maintaining model capabilities.

Contribution

It introduces a novel interactive compression protocol called Question-Asking, which transfers information efficiently through yes/no questions, outperforming previous methods.

Findings

LoRA adapters double lossless compression efficiency.

Lossy compression with prompting achieves 0.03 ratio, doubling prior results.

Question-Asking protocol recovers up to 72% of model capability gap with minimal bits.

Abstract

We study the compression of LLM-generated text across lossless and lossy regimes, characterizing a compression-compute frontier where more compression is possible at the cost of more compute. For lossless compression, domain-adapted LoRA adapters can improve LLM-based arithmetic coding by 2x over compression with the base LLM alone. For lossy compression, prompting a model for a succinct rewrite then applying arithmetic coding can achieve compression ratios of approximately 0.03, a 2x improvement over compressing the original response. We further introduce Question-Asking compression (QA), an interactive lossy protocol inspired by the game 'Twenty Questions'. A small model iteratively refines its response by asking yes/no questions to a stronger model, transferring exactly one bit per answer. On 8 benchmarks spanning math, science, and code, 10 binary questions recover 23% to 72% of the capability gap between a small and large model on standard benchmarks and 7% to 38% on harder benchmarks, achieving compression ratios of 0.0006 to 0.004. This is over 100x smaller than prior LLM-based compression (Deletang et al., 2024), suggesting that interactive protocols can transfer knowledge far more efficiently than transmitting full responses.