Learning an Adaptation Function to Assess Image Visual Similarities

TL;DR

This paper introduces an adaptation function to improve computer vision systems' ability to assess visual similarity between images, especially when semantic content differs, by approximating primate visual cortex responses.

Contribution

It proposes a novel adaptation framework that enhances similarity assessment by learning from neural features, outperforming traditional feature comparison methods.

Findings

Increased retrieval accuracy by 2.25x on the Totally Looks Like dataset.

Comparison of various pre-trained networks for visual similarity assessment.

Demonstrated the effectiveness of neural feature adaptation in mimicking primate visual cortex.

Abstract

Human perception is routinely assessing the similarity between images, both for decision making and creative thinking. But the underlying cognitive process is not really well understood yet, hence difficult to be mimicked by computer vision systems. State-of-the-art approaches using deep architectures are often based on the comparison of images described as feature vectors learned for image categorization task. As a consequence, such features are powerful to compare semantically related images but not really efficient to compare images visually similar but semantically unrelated. Inspired by previous works on neural features adaptation to psycho-cognitive representations, we focus here on the specific task of learning visual image similarities when analogy matters. We propose to compare different supervised, semi-supervised and self-supervised networks, pre-trained on distinct scales and contents datasets (such as ImageNet-21k, ImageNet-1K or VGGFace2) to conclude which model may be the best to approximate the visual cortex and learn only an adaptation function corresponding to the approximation of the the primate IT cortex through the metric learning framework. Our experiments conducted on the Totally Looks Like image dataset highlight the interest of our method, by increasing the retrieval scores of the best model @1 by 2.25x. This research work was recently accepted for publication at the ICIP 2021 international conference [1]. In this new article, we expand on this previous work by using and comparing new pre-trained feature extractors on other datasets.