LittleBit-2: Maximizing the Spectral Energy Gain in Sub-1-Bit LLMs via Latent Geometry Alignment

TL;DR

LittleBit-2 introduces a geometric alignment framework that significantly improves sub-1-bit LLM compression, achieving state-of-the-art results by addressing latent space misalignment.

Contribution

It proposes Internal Latent Rotation and Joint-ITQ to align latent distributions with binary hypercubes, enhancing extreme model compression performance.

Findings

Achieves new state-of-the-art in sub-1-bit LLM compression.

Matches fidelity of leading 1-bit baselines on Llama models.

No inference overhead introduced by the method.

Abstract

We identify the Spectral Energy Gain in extreme model compression, where low-rank binary approximations outperform tiny-rank floating-point baselines for heavy-tailed spectra. However, prior attempts fail to realize this potential, trailing state-of-the-art 1-bit methods. We attribute this degradation to Latent Geometry Misalignment: standard singular vectors exhibit high coherence (spiky distribution), the worst-case geometry for binary quantization. To realize this gain, we propose LittleBit-2, a framework employing Internal Latent Rotation and Joint Iterative Quantization (Joint-ITQ). This approach acts as a geometric preconditioner, aligning coherent latent distributions with the binary hypercube with zero inference overhead. Empirically, LittleBit-2 establishes a new state-of-the-art in the sub-1-bit regime (1$\sim$0.1 bpp) on Llama-2 and Llama-3, matching the fidelity of leading 1-bit baselines.