ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalities

TL;DR

This paper introduces ONE-PEACE, a scalable, extensible model with 4B parameters that aligns and integrates multiple modalities like vision, audio, and language through a novel architecture and pretraining tasks, enabling broad multi-modal applications.

Contribution

The paper presents ONE-PEACE, a highly extensible general representation model capable of handling unlimited modalities with a flexible architecture and modality-agnostic pretraining tasks, without relying on pretrained models.

Findings

Achieves leading results on diverse uni-modal and multi-modal tasks.

Supports seamless extension to new modalities by adding adapters and FFNs.

Demonstrates effective cross-modal alignment and fine-grained intra-modal understanding.

Abstract

In this work, we explore a scalable way for building a general representation model toward unlimited modalities. We release ONE-PEACE, a highly extensible model with 4B parameters that can seamlessly align and integrate representations across vision, audio, and language modalities. The architecture of ONE-PEACE comprises modality adapters, shared self-attention layers, and modality FFNs. This design allows for the easy extension of new modalities by adding adapters and FFNs, while also enabling multi-modal fusion through self-attention layers. To pretrain ONE-PEACE, we develop two modality-agnostic pretraining tasks, cross-modal aligning contrast and intra-modal denoising contrast, which align the semantic space of different modalities and capture fine-grained details within modalities concurrently. With the scaling-friendly architecture and pretraining tasks, ONE-PEACE has the potential to expand to unlimited modalities. Without using any vision or language pretrained model for initialization, ONE-PEACE achieves leading results on a wide range of uni-modal and multi-modal tasks, including image classification (ImageNet), semantic segmentation (ADE20K), audio-text retrieval (AudioCaps, Clotho), audio classification (ESC-50, FSD50K, VGGSound), audio question answering (AVQA), image-text retrieval (MSCOCO, Flickr30K), and visual grounding (RefCOCO/+/g). Code is available at https://github.com/OFA-Sys/ONE-PEACE.