Hierarchical Semantic Alignment for Image Clustering

TL;DR

This paper introduces a hierarchical semantic alignment method for image clustering that leverages caption and noun semantics to improve clustering accuracy without training, outperforming existing approaches.

Contribution

The proposed CAE method uniquely combines caption and noun semantics with optimal transport to enhance image clustering performance in a training-free manner.

Findings

Achieves 4.2% higher accuracy on ImageNet-1K

Surpasses state-of-the-art training-free methods

Effective across 8 diverse datasets

Abstract

Image clustering is a classic problem in computer vision, which categorizes images into different groups. Recent studies utilize nouns as external semantic knowledge to improve clustering performance. However, these methods often overlook the inherent ambiguity of nouns, which can distort semantic representations and degrade clustering quality. To address this issue, we propose a hierarChical semAntic alignmEnt method for image clustering, dubbed CAE, which improves clustering performance in a training-free manner. In our approach, we incorporate two complementary types of textual semantics: caption-level descriptions, which convey fine-grained attributes of image content, and noun-level concepts, which represent high-level object categories. We first select relevant nouns from WordNet and descriptions from caption datasets to construct a semantic space aligned with image features. Then, we align image features with selected nouns and captions via optimal transport to obtain a more discriminative semantic space. Finally, we combine the enhanced semantic and image features to perform clustering. Extensive experiments across 8 datasets demonstrate the effectiveness of our method, notably surpassing the state-of-the-art training-free approach with a 4.2% improvement in accuracy and a 2.9% improvement in adjusted rand index (ARI) on the ImageNet-1K dataset.