Compression is Routing: Reconstruction Error as an Intrinsic Signal for Modular Language Models

TL;DR

This paper introduces a novel approach called 'Compression is Routing' using a Transformer Autoencoder that leverages reconstruction error as an intrinsic signal for modular language models, enabling scalable expert scheduling and handling ultra-long contexts.

Contribution

It proposes a new architecture that uses reconstruction error for expert routing, eliminating the need for explicit gating and improving scalability in modular language models.

Findings

Achieved 64x sequence length compression with high in-domain accuracy

Reconstruction error effectively discriminates between in-domain and out-of-distribution data

Demonstrated potential for scalable expert scheduling without explicit gating mechanisms

Abstract

Current Large Language Models (LLMs) face three major challenges: context length limitations, high inference costs, and catastrophic forgetting during continual learning. While Mixture-of-Experts (MoE) architectures mitigate some of these conflicts, their routing mechanisms typically rely on explicitly trained auxiliary classifiers. This not only increases system complexity but also often lacks interpretability when handling mixed-domain inputs. Building upon the premise that ``Compression is Intelligence,'' this paper proposes a novel architectural philosophy: Compression is Routing. We trained an 87M-parameter end-to-end Transformer Autoencoder, achieving a 64x sequence length compression (compressing 512 tokens into 8 latent vectors). Experimental results demonstrate that this compressor possesses extreme domain discriminative capability: it achieves a reconstruction accuracy of 99.47% on the in-domain (code) validation set; accuracy drops sharply to 47.76% on a semi-out-of-distribution domain (Wiki text); and further plummets to just 0.57% on a fully out-of-distribution domain (random sequences). This extreme and systematic performance discrepancy establishes the validity of reconstruction error as an Intrinsic Distribution Fingerprint. Based on this, we propose that expert modules can be automatically scheduled using reconstruction residuals directly, without the need for explicit gating networks. This mechanism offers excellent scalability. Furthermore, this architecture provides a new perspective on ``VRAM compression'' for handling ultra-long contexts. This report aims to verify the physical validity of this foundational architecture, offering a new research perspective for the next generation of scalable modular neural networks.