Omnivore: A Single Model for Many Visual Modalities

TL;DR

This paper introduces Omnivore, a transformer-based model capable of classifying images, videos, and 3D data with a single set of parameters, achieving competitive results across multiple visual modalities.

Contribution

The paper presents a unified model that handles diverse visual data types using shared parameters, simplifying training and improving cross-modal recognition capabilities.

Findings

Achieves 86.0% on ImageNet

Obtains 84.1% on Kinetics

Reaches 67.1% on SUN RGB-D

Abstract

Prior work has studied different visual modalities in isolation and developed separate architectures for recognition of images, videos, and 3D data. Instead, in this paper, we propose a single model which excels at classifying images, videos, and single-view 3D data using exactly the same model parameters. Our 'Omnivore' model leverages the flexibility of transformer-based architectures and is trained jointly on classification tasks from different modalities. Omnivore is simple to train, uses off-the-shelf standard datasets, and performs at-par or better than modality-specific models of the same size. A single Omnivore model obtains 86.0% on ImageNet, 84.1% on Kinetics, and 67.1% on SUN RGB-D. After finetuning, our models outperform prior work on a variety of vision tasks and generalize across modalities. Omnivore's shared visual representation naturally enables cross-modal recognition without access to correspondences between modalities. We hope our results motivate researchers to model visual modalities together.