Attributable Visual Similarity Learning

TL;DR

This paper introduces an attributable visual similarity learning framework that models image similarity using graphs and semantic hierarchies, enhancing accuracy and interpretability over traditional point-based methods.

Contribution

It presents a novel graph-based similarity representation and a hierarchical inference method that improves explainability and accuracy in visual similarity learning.

Findings

Significant performance improvements on benchmark datasets.

Enhanced interpretability through similarity attribution.

Effective correction of unreliable similarity nodes.

Abstract

This paper proposes an attributable visual similarity learning (AVSL) framework for a more accurate and explainable similarity measure between images. Most existing similarity learning methods exacerbate the unexplainability by mapping each sample to a single point in the embedding space with a distance metric (e.g., Mahalanobis distance, Euclidean distance). Motivated by the human semantic similarity cognition, we propose a generalized similarity learning paradigm to represent the similarity between two images with a graph and then infer the overall similarity accordingly. Furthermore, we establish a bottom-up similarity construction and top-down similarity inference framework to infer the similarity based on semantic hierarchy consistency. We first identify unreliable higher-level similarity nodes and then correct them using the most coherent adjacent lower-level similarity nodes, which simultaneously preserve traces for similarity attribution. Extensive experiments on the CUB-200-2011, Cars196, and Stanford Online Products datasets demonstrate significant improvements over existing deep similarity learning methods and verify the interpretability of our framework. Code is available at https://github.com/zbr17/AVSL.