CorrEmbed: Evaluating Pre-trained Model Image Similarity Efficacy with a Novel Metric

TL;DR

CorrEmbed introduces a novel metric to evaluate pre-trained image embeddings by correlating embedding distances with human-generated tag similarities, revealing insights into model performance for image similarity tasks.

Contribution

The paper proposes CorrEmbed, a new evaluation method that assesses pre-trained models' image embeddings based on their alignment with human tag similarities, addressing a gap in existing metrics.

Findings

Linear relationship between ImageNet accuracy and tag-correlation scores

CorrEmbed identifies models that deviate from expected performance patterns

Provides insights into how models capture high-level image features

Abstract

Detecting visually similar images is a particularly useful attribute to look to when calculating product recommendations. Embedding similarity, which utilizes pre-trained computer vision models to extract high-level image features, has demonstrated remarkable efficacy in identifying images with similar compositions. However, there is a lack of methods for evaluating the embeddings generated by these models, as conventional loss and performance metrics do not adequately capture their performance in image similarity search tasks. In this paper, we evaluate the viability of the image embeddings from numerous pre-trained computer vision models using a novel approach named CorrEmbed. Our approach computes the correlation between distances in image embeddings and distances in human-generated tag vectors. We extensively evaluate numerous pre-trained Torchvision models using this metric, revealing an intuitive relationship of linear scaling between ImageNet1k accuracy scores and tag-correlation scores. Importantly, our method also identifies deviations from this pattern, providing insights into how different models capture high-level image features. By offering a robust performance evaluation of these pre-trained models, CorrEmbed serves as a valuable tool for researchers and practitioners seeking to develop effective, data-driven approaches to similar item recommendations in fashion retail.