ConceptHash: Interpretable Fine-Grained Hashing via Concept Discovery

TL;DR

ConceptHash introduces interpretable fine-grained hashing by associating each hash sub-code with human-understandable concepts, leveraging vision transformers and language guidance for improved image retrieval accuracy.

Contribution

It presents a novel method that automatically discovers interpretable concepts for each hash sub-code without human annotations, enhancing interpretability and fine-grained discrimination.

Findings

Outperforms previous methods on four benchmarks

Achieves sub-code interpretability aligned with human concepts

Effectively distinguishes highly similar sub-categories

Abstract

Existing fine-grained hashing methods typically lack code interpretability as they compute hash code bits holistically using both global and local features. To address this limitation, we propose ConceptHash, a novel method that achieves sub-code level interpretability. In ConceptHash, each sub-code corresponds to a human-understandable concept, such as an object part, and these concepts are automatically discovered without human annotations. Specifically, we leverage a Vision Transformer architecture and introduce concept tokens as visual prompts, along with image patch tokens as model inputs. Each concept is then mapped to a specific sub-code at the model output, providing natural sub-code interpretability. To capture subtle visual differences among highly similar sub-categories (e.g., bird species), we incorporate language guidance to ensure that the learned hash codes are distinguishable within fine-grained object classes while maintaining semantic alignment. This approach allows us to develop hash codes that exhibit similarity within families of species while remaining distinct from species in other families. Extensive experiments on four fine-grained image retrieval benchmarks demonstrate that ConceptHash outperforms previous methods by a significant margin, offering unique sub-code interpretability as an additional benefit. Code at: https://github.com/kamwoh/concepthash.