Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodality

TL;DR

This paper advocates for rethinking token reduction in generative models as a fundamental principle that enhances multimodal integration, coherence, and training stability, beyond just improving efficiency.

Contribution

It redefines token reduction from an efficiency tool to a core principle that influences model architecture and applications across vision, language, and multimodal systems.

Findings

Token reduction facilitates deeper multimodal integration.

It helps mitigate hallucinations and overthinking.

Enhances coherence and training stability.

Abstract

In Transformer architectures, tokens\textemdash discrete units derived from raw data\textemdash are formed by segmenting inputs into fixed-length chunks. Each token is then mapped to an embedding, enabling parallel attention computations while preserving the input's essential information. Due to the quadratic computational complexity of transformer self-attention mechanisms, token reduction has primarily been used as an efficiency strategy. This is especially true in single vision and language domains, where it helps balance computational costs, memory usage, and inference latency. Despite these advances, this paper argues that token reduction should transcend its traditional efficiency-oriented role in the era of large generative models. Instead, we position it as a fundamental principle in generative modeling, critically influencing both model architecture and broader applications. Specifically, we contend that across vision, language, and multimodal systems, token reduction can: (i) facilitate deeper multimodal integration and alignment, (ii) mitigate "overthinking" and hallucinations, (iii) maintain coherence over long inputs, and (iv) enhance training stability, etc. We reframe token reduction as more than an efficiency measure. By doing so, we outline promising future directions, including algorithm design, reinforcement learning-guided token reduction, token optimization for in-context learning, agentic framework design, and broader ML and scientific domains.