Benchmarking Omni-Vision Representation through the Lens of Visual Realms

TL;DR

This paper introduces OmniBenchmark, a comprehensive dataset covering diverse visual realms, and proposes ReCo, a contrastive learning method that improves omni-vision representations by leveraging semantic relations.

Contribution

The paper presents a new benchmark dataset for evaluating omni-vision models and a novel contrastive learning framework that encodes semantic relations to enhance generalization.

Findings

ReCo outperforms other supervised contrastive methods.

OmniBenchmark covers most visual realms without semantic overlap.

ReCo improves omni-vision representations across architectures.

Abstract

Though impressive performance has been achieved in specific visual realms (e.g. faces, dogs, and places), an omni-vision representation generalizing to many natural visual domains is highly desirable. But, existing benchmarks are biased and inefficient to evaluate the omni-vision representation -- these benchmarks either only include several specific realms, or cover most realms at the expense of subsuming numerous datasets that have extensive realm overlapping. In this paper, we propose Omni-Realm Benchmark (OmniBenchmark). It includes 21 realm-wise datasets with 7,372 concepts and 1,074,346 images. Without semantic overlapping, these datasets cover most visual realms comprehensively and meanwhile efficiently. In addition, we propose a new supervised contrastive learning framework, namely Relational Contrastive learning (ReCo), for a better omni-vision representation. Beyond pulling two instances from the same concept closer -- the typical supervised contrastive learning framework -- ReCo also pulls two instances from the same semantic realm closer, encoding the semantic relation between concepts, and facilitating omni-vision representation learning. We benchmark ReCo and other advances in omni-vision representation studies that are different in architectures (from CNNs to transformers) and in learning paradigms (from supervised learning to self-supervised learning) on OmniBenchmark. We illustrate the superior of ReCo to other supervised contrastive learning methods and reveal multiple practical observations to facilitate future research.