The Missing Link: Finding label relations across datasets

TL;DR

This paper introduces methods to automatically discover and classify visual-semantic label relations across datasets, enhancing understanding and transfer learning in computer vision.

Contribution

It proposes novel language, vision, and combined approaches to identify label relations across datasets, addressing a key challenge in dataset integration.

Findings

Effective discovery of label relations and their types across datasets

Label relations depend on dataset construction, not just class names

Applications include understanding relations, identifying missing aspects, and improving transfer learning

Abstract

Computer vision is driven by the many datasets available for training or evaluating novel methods. However, each dataset has a different set of class labels, visual definition of classes, images following a specific distribution, annotation protocols, etc. In this paper we explore the automatic discovery of visual-semantic relations between labels across datasets. We aim to understand how instances of a certain class in a dataset relate to the instances of another class in another dataset. Are they in an identity, parent/child, overlap relation? Or is there no link between them at all? To find relations between labels across datasets, we propose methods based on language, on vision, and on their combination. We show that we can effectively discover label relations across datasets, as well as their type. We apply our method to four applications: understand label relations, identify missing aspects, increase label specificity, and predict transfer learning gains. We conclude that label relations cannot be established by looking at the names of classes alone, as they depend strongly on how each of the datasets was constructed.