Compression Hacking: A Supplementary Perspective on Informatics Properties of Language Models from Geometric Distortion

TL;DR

This paper introduces geometric distortion-based metrics to better understand language model representations, revealing that traditional compression metrics can be misleading due to anisotropic distortions.

Contribution

The authors propose three refined compression metrics incorporating geometric distortion analysis, improving the correlation with language model capabilities.

Findings

Refined metrics achieve Spearman correlation > 0.9 with LM capabilities

Geometric distortion analysis reveals anisotropy in compressed representations

Enhanced metrics outperform original compression and structure-based metrics

Abstract

Recently, the concept of ``compression as intelligence'' has provided a novel informatics metric perspective for language models (LMs), emphasizing that highly structured representations signify the intelligence level of LMs. However, from a geometric standpoint, the word representation space of highly compressed LMs tends to degenerate into a highly anisotropic state, which hinders the LM's ability to comprehend instructions and directly impacts its performance. We found this compression-anisotropy synchronicity is essentially the ``Compression Hacking'' in LM representations, where noise-dominated directions tend to create the illusion of high compression rates by sacrificing spatial uniformity. Based on this, we propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into a self-evaluation pipeline. The refined metrics exhibit strong alignment with the LM's comprehensive capabilities, achieving Spearman correlation coefficients above 0.9, significantly outperforming both the original compression and other internal structure-based metrics. This confirms that compression hacking substantially enhances the informatics interpretation of LMs by incorporating geometric distortion of representations.