Learning to Align Generative Appearance Priors for Fine-grained Image Retrieval

TL;DR

GAPan introduces a generative model using normalizing flows to align image features with appearance priors, enhancing fine-grained image retrieval especially for unseen categories.

Contribution

It reformulates FGIR learning from category prediction to appearance modeling using invertible density models, improving generalization to unseen categories.

Findings

Achieves state-of-the-art results on fine- and coarse-grained benchmarks.

Effectively models intra-category appearance variation.

Improves retrieval performance on unseen categories.

Abstract

Fine-grained image retrieval (FGIR) typically relies on supervision from seen categories to learn discriminative embeddings for retrieving unseen categories. However, such supervision often biases retrieval models toward the semantics of seen categories rather than the underlying appearance characteristics that generalize across categories, thereby limiting retrieval performance on unseen categories. To tackle this, we propose GAPan, a Generative Appearance Prior alignment network that reformulates the learning objective from category prediction toward appearance modeling. Technically, GAPan treats retrieval features with an invertible density model based on normalizing flows. In the forward direction, the flow maps all instance features into a latent density space, where each seen category is modeled by a class-conditional Gaussian prior and optimized via exact likelihood estimation. This formulation preserves richer appearance details by leveraging the invertible property of the flows. In the reverse direction, samples from the high-density regions of these learned priors are mapped back to the feature space to produce appearance-aware anchors that reflect intra-category variation. These anchors supervise a prior-driven alignment objective that aligns retrieval embeddings with category-specific appearance distributions, thereby improving generalization to unseen categories. Evaluations demonstrate that our GAPan achieves state-of-the-art performance on both widely-used fine- and coarse-grained benchmarks.