Hierarchical Textual Knowledge for Enhanced Image Clustering

TL;DR

This paper introduces a hierarchical textual knowledge approach using large language models to improve image clustering by incorporating rich semantic information beyond visual features.

Contribution

It proposes a knowledge-enhanced clustering method that constructs hierarchical concept-attribute structured knowledge to guide image clustering, outperforming existing methods.

Findings

KEC improves clustering accuracy across 20 datasets.

KEC without training surpasses zero-shot CLIP on 14 datasets.

Naive textual knowledge use can harm performance, but KEC is robust.

Abstract

Image clustering aims to group images in an unsupervised fashion. Traditional methods focus on knowledge from visual space, making it difficult to distinguish between visually similar but semantically different classes. Recent advances in vision-language models enable the use of textual knowledge to enhance image clustering. However, most existing methods rely on coarse class labels or simple nouns, overlooking the rich conceptual and attribute-level semantics embedded in textual space. In this paper, we propose a knowledge-enhanced clustering (KEC) method that constructs a hierarchical concept-attribute structured knowledge with the help of large language models (LLMs) to guide clustering. Specifically, we first condense redundant textual labels into abstract concepts and then automatically extract discriminative attributes for each single concept and similar concept pairs, via structured prompts to LLMs. This knowledge is instantiated for each input image to achieve the knowledge-enhanced features. The knowledge-enhanced features with original visual features are adapted to various downstream clustering algorithms. We evaluate KEC on 20 diverse datasets, showing consistent improvements across existing methods using additional textual knowledge. KEC without training outperforms zero-shot CLIP on 14 out of 20 datasets. Furthermore, the naive use of textual knowledge may harm clustering performance, while KEC provides both accuracy and robustness.