Agglomerative Token Clustering

TL;DR

Agglomerative Token Clustering (ATC) is a new hierarchical token merging method that improves performance across various vision tasks without additional learnable parameters, especially effective with low token retention.

Contribution

ATC introduces a hierarchical clustering-based token merging technique that outperforms existing methods across multiple vision tasks without requiring extra parameters.

Findings

ATC achieves state-of-the-art results in image classification, synthesis, and detection.

ATC performs on par with prior methods even without fine-tuning.

ATC is highly effective at low token retention rates.

Abstract

We present Agglomerative Token Clustering (ATC), a novel token merging method that consistently outperforms previous token merging and pruning methods across image classification, image synthesis, and object detection & segmentation tasks. ATC merges clusters through bottom-up hierarchical clustering, without the introduction of extra learnable parameters. We find that ATC achieves state-of-the-art performance across all tasks, and can even perform on par with prior state-of-the-art when applied off-the-shelf, i.e. without fine-tuning. ATC is particularly effective when applied with low keep rates, where only a small fraction of tokens are kept and retaining task performance is especially difficult.