Unison: A Fully Automatic, Task-Universal, and Low-Cost Framework for Unified Understanding and Generation

TL;DR

Unison is a low-cost, fully automatic multimodal framework that unifies understanding and generation tasks across text, image, and video, with automatic task parsing and parameter extraction, achieving high performance with minimal training resources.

Contribution

It introduces a low-cost, fully automatic multimodal framework that covers diverse tasks and automatically parses user intentions, improving over prior manual or resource-intensive methods.

Findings

Achieves high accuracy with only 500k samples and 50 GPU hours

Automatically identifies task types and extracts meta-information

Performs well across understanding and generation tasks

Abstract

Unified understanding and generation is a highly appealing research direction in multimodal learning. There exist two approaches: one trains a transformer via an auto-regressive paradigm, and the other adopts a two-stage scheme connecting pre-trained understanding and generative models for alignment fine-tuning. The former demands massive data and computing resources unaffordable for ordinary researchers. Though the latter requires a lower training cost, existing works often suffer from limited task coverage or poor generation quality. Both approaches lack the ability to parse input meta-information (such as task type, image resolution, video duration, etc.) and require manual parameter configuration that is tedious and non-intelligent. In this paper, we propose Unison which adopts the two-stage scheme while preserving the capabilities of the pre-trained models well. With an extremely low training cost, we cover a variety of multimodal understanding tasks, including text, image, and video understanding, as well as diverse generation tasks, such as text-to-visual content generation, editing, controllable generation, and IP-based reference generation. We also equip our model with the ability to automatically parse user intentions, determine the target task type, and accurately extract the meta-information required for the corresponding task. This enables full automation of various multimodal tasks without human intervention. Experiments demonstrate that, under a low-cost setting of only 500k training samples and 50 GPU hours, our model can accurately and automatically identify tasks and extract relevant parameters, and achieve superior performance across a variety of understanding and generation tasks.