OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation

TL;DR

The paper introduces OPT, a comprehensive pre-training framework that jointly models visual, textual, and audio data to enhance cross-modal understanding and generation capabilities.

Contribution

It presents a novel encoder-decoder architecture with multi-task pre-training on large-scale triplet data for improved multi-modal alignment and translation.

Findings

OPT achieves strong multi-modal representations.

It performs well on various cross-modal tasks.

The multi-task scheme effectively models different data granularities.

Abstract

In this paper, we propose an Omni-perception Pre-Trainer (OPT) for cross-modal understanding and generation, by jointly modeling visual, text and audio resources. OPT is constructed in an encoder-decoder framework, including three single-modal encoders to generate token-based embeddings for each modality, a cross-modal encoder to encode the correlations among the three modalities, and two cross-modal decoders to generate text and image respectively. For the OPT's pre-training, we design a multi-task pretext learning scheme to model multi-modal resources from three different data granularities, \ie, token-, modality-, and sample-level modeling, through which OPT learns to align and translate among different modalities. The pre-training task is carried out on a large amount of image-text-audio triplets from Open Images. Experimental results show that OPT can learn strong image-text-audio multi-modal representations and achieve promising results on a variety of cross-modal understanding and generation tasks.