PDiscoFormer: Relaxing Part Discovery Constraints with Vision Transformers

TL;DR

This paper leverages pre-trained transformer-based vision models to relax geometric constraints in part discovery, significantly improving interpretability and accuracy in fine-grained classification tasks.

Contribution

It introduces a novel approach using vision transformers with a total variation prior, outperforming previous methods in unsupervised part discovery.

Findings

Outperforms previous methods on CUB, PartImageNet, Oxford Flowers

Achieves better part discovery metrics and classification accuracy

Shows that transformer models require different geometric priors for unsupervised part discovery

Abstract

Computer vision methods that explicitly detect object parts and reason on them are a step towards inherently interpretable models. Existing approaches that perform part discovery driven by a fine-grained classification task make very restrictive assumptions on the geometric properties of the discovered parts; they should be small and compact. Although this prior is useful in some cases, in this paper we show that pre-trained transformer-based vision models, such as self-supervised DINOv2 ViT, enable the relaxation of these constraints. In particular, we find that a total variation (TV) prior, which allows for multiple connected components of any size, substantially outperforms previous work. We test our approach on three fine-grained classification benchmarks: CUB, PartImageNet and Oxford Flowers, and compare our results to previously published methods as well as a re-implementation of the state-of-the-art method PDiscoNet with a transformer-based backbone. We consistently obtain substantial improvements across the board, both on part discovery metrics and the downstream classification task, showing that the strong inductive biases in self-supervised ViT models require to rethink the geometric priors that can be used for unsupervised part discovery.