fruit-SALAD: A Style Aligned Artwork Dataset to reveal similarity perception in image embeddings

TL;DR

This paper introduces fruit-SALAD, a balanced and diverse dataset of 10,000 fruit images across styles and categories, to systematically evaluate and compare how different models perceive visual similarity in images.

Contribution

The creation of Style Aligned Artwork Datasets (SALADs), specifically fruit-SALAD, providing a benchmark for analyzing semantic and style similarity perception in image embeddings.

Findings

Models show varied sensitivity to semantic and style features.

The dataset reveals differences in similarity weights across models.

SALAD enables controlled, quantitative comparison of similarity perception.

Abstract

The notion of visual similarity is essential for computer vision, and in applications and studies revolving around vector embeddings of images. However, the scarcity of benchmark datasets poses a significant hurdle in exploring how these models perceive similarity. Here we introduce Style Aligned Artwork Datasets (SALADs), and an example of fruit-SALAD with 10,000 images of fruit depictions. This combined semantic category and style benchmark comprises 100 instances each of 10 easy-to-recognize fruit categories, across 10 easy distinguishable styles. Leveraging a systematic pipeline of generative image synthesis, this visually diverse yet balanced benchmark demonstrates salient differences in semantic category and style similarity weights across various computational models, including machine learning models, feature extraction algorithms, and complexity measures, as well as conceptual models for reference. This meticulously designed dataset offers a controlled and balanced platform for the comparative analysis of similarity perception. The SALAD framework allows the comparison of how these models perform semantic category and style recognition task to go beyond the level of anecdotal knowledge, making it robustly quantifiable and qualitatively interpretable.