Efficient Discovery and Effective Evaluation of Visual Perceptual Similarity: A Benchmark and Beyond

TL;DR

This paper introduces a large-scale fashion visual similarity benchmark with expert annotations, proposes a new efficient labeling method, and discusses evaluation metrics to improve the assessment of visual similarity models.

Contribution

It provides the first extensive fashion VSD dataset, a novel labeling procedure, and insights into evaluation metrics beyond proxy tasks.

Findings

Created a dataset with 110K expert-annotated pairs

Proposed an efficient labeling procedure applicable to other datasets

Analyzed limitations and biases of current evaluation metrics

Abstract

Visual similarities discovery (VSD) is an important task with broad e-commerce applications. Given an image of a certain object, the goal of VSD is to retrieve images of different objects with high perceptual visual similarity. Although being a highly addressed problem, the evaluation of proposed methods for VSD is often based on a proxy of an identification-retrieval task, evaluating the ability of a model to retrieve different images of the same object. We posit that evaluating VSD methods based on identification tasks is limited, and faithful evaluation must rely on expert annotations. In this paper, we introduce the first large-scale fashion visual similarity benchmark dataset, consisting of more than 110K expert-annotated image pairs. Besides this major contribution, we share insight from the challenges we faced while curating this dataset. Based on these insights, we propose a novel and efficient labeling procedure that can be applied to any dataset. Our analysis examines its limitations and inductive biases, and based on these findings, we propose metrics to mitigate those limitations. Though our primary focus lies on visual similarity, the methodologies we present have broader applications for discovering and evaluating perceptual similarity across various domains.