Shifting AI Efficiency From Model-Centric to Data-Centric Compression

TL;DR

This paper advocates shifting AI efficiency focus from model compression to data compression, emphasizing data-centric methods to handle the increasing computational costs of processing long sequences in large models.

Contribution

It introduces a unified framework for data-centric compression, reviews current methods, and highlights its importance as a paradigm change for long-context AI.

Findings

Data-centric compression offers a promising alternative to model-centric methods.

The framework unifies existing efficiency strategies under a common perspective.

Identifies key challenges and future directions for data-centric AI compression.

Abstract

The advancement of large language models (LLMs) and multi-modal LLMs (MLLMs) has historically relied on scaling model parameters. However, as hardware limits constrain further model growth, the primary computational bottleneck has shifted to the quadratic cost of self-attention over increasingly long sequences by ultra-long text contexts, high-resolution images, and extended videos. In this position paper, \textbf{we argue that the focus of research for efficient artificial intelligence (AI) is shifting from model-centric compression to data-centric compression}. We position data-centric compression as the emerging paradigm, which improves AI efficiency by directly compressing the volume of data processed during model training or inference. To formalize this shift, we establish a unified framework for existing efficiency strategies and demonstrate why it constitutes a crucial paradigm change for long-context AI. We then systematically review the landscape of data-centric compression methods, analyzing their benefits across diverse scenarios. Finally, we outline key challenges and promising future research directions. Our work aims to provide a novel perspective on AI efficiency, synthesize existing efforts, and catalyze innovation to address the challenges posed by ever-increasing context lengths.