Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens

TL;DR

This paper introduces Semantic Equitable Clustering (SEC), a fast, single-pass clustering method based on semantic relevance, improving efficiency and effectiveness of vision token grouping in Vision Transformers and multimodal models.

Contribution

The paper presents SEC, a novel, efficient clustering algorithm that balances token groups based on semantic relevance in a single pass, enhancing Vision Transformer performance and multimodal model integration.

Findings

SEC achieves token clustering in a single pass.

SEC improves efficiency and accuracy in vision tasks.

SEC enhances multimodal large language models' performance.

Abstract

The Vision Transformer (ViT) has gained prominence for its superior relational modeling prowess. However, its global attention mechanism's quadratic complexity poses substantial computational burdens. A common remedy spatially groups tokens for self-attention, reducing computational requirements. Nonetheless, this strategy neglects semantic information in tokens, possibly scattering semantically-linked tokens across distinct groups, thus compromising the efficacy of self-attention intended for modeling inter-token dependencies. Motivated by these insights, we introduce a fast and balanced clustering method, named Semantic Equitable Clustering (SEC). SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner. In contrast to traditional clustering methods requiring multiple iterations, our method achieves token clustering in a single pass. Additionally, SEC regulates the number of tokens per cluster, ensuring a balanced distribution for effective parallel processing on current computational platforms without necessitating further optimization. Capitalizing on SEC, we propose a versatile vision backbone, SECViT. Comprehensive experiments in image classification, object detection, instance segmentation, and semantic segmentation validate the effectiveness of SECViT. Moreover, SEC can be conveniently and swiftly applied to multimodal large language models (MLLM), such as LLaVA, to serve as a vision language connector, effectively accelerating the model's efficiency while maintaining unchanged or better performance.