Totally Looks Like - How Humans Compare, Compared to Machines
Amir Rosenfeld, Markus D. Solbach, John K. Tsotsos

TL;DR
This paper introduces the Totally-Looks-Like dataset, revealing that current deep learning models poorly replicate human judgments of image similarity, highlighting gaps in machine perception of complex visual criteria.
Contribution
The paper presents a new diverse dataset of human-labeled image pairs and evaluates the performance of state-of-the-art models, exposing significant limitations in current machine perception.
Findings
Machine features perform poorly compared to human judgments.
The dataset captures a wide range of criteria used by humans.
Artificially simplified matching tasks do not improve model performance.
Abstract
Perceptual judgment of image similarity by humans relies on rich internal representations ranging from low-level features to high-level concepts, scene properties and even cultural associations. However, existing methods and datasets attempting to explain perceived similarity use stimuli which arguably do not cover the full breadth of factors that affect human similarity judgments, even those geared toward this goal. We introduce a new dataset dubbed Totally-Looks-Like (TLL) after a popular entertainment website, which contains images paired by humans as being visually similar. The dataset contains 6016 image-pairs from the wild, shedding light upon a rich and diverse set of criteria employed by human beings. We conduct experiments to try to reproduce the pairings via features extracted from state-of-the-art deep convolutional neural networks, as well as additional human experiments to…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications · Image Retrieval and Classification Techniques
