Not just a matter of semantics: the relationship between visual similarity and semantic similarity

TL;DR

This paper investigates the correlation between semantic similarity from WordNet and visual similarity in images, revealing that semantic information can be informative but also potentially misleading for visual tasks.

Contribution

It provides a thorough analysis of the visual-semantic relationship using multiple similarity measures and verifies key hypotheses about their correlation.

Findings

Semantic similarity carries meaningful information about visual similarity.

Wrong semantic information can significantly harm model performance.

Semantic similarity is not solely based on class differences.

Abstract

Knowledge transfer, zero-shot learning and semantic image retrieval are methods that aim at improving accuracy by utilizing semantic information, e.g. from WordNet. It is assumed that this information can augment or replace missing visual data in the form of labeled training images because semantic similarity correlates with visual similarity. This assumption may seem trivial, but is crucial for the application of such semantic methods. Any violation can cause mispredictions. Thus, it is important to examine the visual-semantic relationship for a certain target problem. In this paper, we use five different semantic and visual similarity measures each to thoroughly analyze the relationship without relying too much on any single definition. We postulate and verify three highly consequential hypotheses on the relationship. Our results show that it indeed exists and that WordNet semantic similarity carries more information about visual similarity than just the knowledge of "different classes look different". They suggest that classification is not the ideal application for semantic methods and that wrong semantic information is much worse than none.