Omni-Video: Democratizing Unified Video Understanding and Generation

TL;DR

Omni-Video introduces a unified framework that leverages multimodal large language models and diffusion decoders to advance video understanding, generation, and editing with high efficiency and broad applicability.

Contribution

The paper presents a novel architecture and training scheme that enable unified video modeling using existing multimodal models and diffusion techniques, addressing current limitations.

Findings

Effective video generation, editing, and understanding demonstrated

Lightweight design enables fast training with limited data

Model generalizes well across multiple video tasks

Abstract

Notable breakthroughs in unified understanding and generation modeling have led to remarkable advancements in image understanding, reasoning, production and editing, yet current foundational models predominantly focus on processing images, creating a gap in the development of unified models for video understanding and generation. This report presents Omni-Video, an efficient and effective unified framework for video understanding, generation, as well as instruction-based editing. Our key insight is to teach existing multimodal large language models (MLLMs) to produce continuous visual clues that are used as the input of diffusion decoders, which produce high-quality videos conditioned on these visual clues. To fully unlock the potential of our system for unified video modeling, we integrate several technical improvements: 1) a lightweight architectural design that respectively attaches a vision head on the top of MLLMs and a adapter before the input of diffusion decoders, the former produce visual tokens for the latter, which adapts these visual tokens to the conditional space of diffusion decoders; and 2) an efficient multi-stage training scheme that facilitates a fast connection between MLLMs and diffusion decoders with limited data and computational resources. We empirically demonstrate that our model exhibits satisfactory generalization abilities across video generation, editing and understanding tasks.