Do Different Deep Metric Learning Losses Lead to Similar Learned Features?

TL;DR

This study compares features learned by different deep metric learning losses, revealing that despite similar performance, they can focus on different image properties and regions, especially between classification and ranking losses.

Contribution

The paper introduces a two-step analysis method to compare learned features across models trained with various loss functions, highlighting differences and influences on feature focus.

Findings

Different loss functions guide models to learn distinct features.

Classification and ranking losses lead to different focus areas.

Irrelevant properties can significantly influence embeddings.

Abstract

Recent studies have shown that many deep metric learning loss functions perform very similarly under the same experimental conditions. One potential reason for this unexpected result is that all losses let the network focus on similar image regions or properties. In this paper, we investigate this by conducting a two-step analysis to extract and compare the learned visual features of the same model architecture trained with different loss functions: First, we compare the learned features on the pixel level by correlating saliency maps of the same input images. Second, we compare the clustering of embeddings for several image properties, e.g. object color or illumination. To provide independent control over these properties, photo-realistic 3D car renders similar to images in the Cars196 dataset are generated. In our analysis, we compare 14 pretrained models from a recent study and find that, even though all models perform similarly, different loss functions can guide the model to learn different features. We especially find differences between classification and ranking based losses. Our analysis also shows that some seemingly irrelevant properties can have significant influence on the resulting embedding. We encourage researchers from the deep metric learning community to use our methods to get insights into the features learned by their proposed methods.