Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing

TL;DR

Tele-Omni introduces a versatile multimodal framework that unifies various video generation and editing tasks using structured instructions and diffusion models, enabling flexible control and high-quality outputs.

Contribution

The paper presents Tele-Omni, the first unified multimodal framework capable of handling diverse video tasks with a single model, integrating instruction parsing and diffusion-based synthesis.

Findings

Achieves competitive performance across multiple video tasks

Supports multimodal inputs including text, images, and reference videos

Maintains high temporal coherence and visual consistency

Abstract

Recent advances in diffusion-based video generation have substantially improved visual fidelity and temporal coherence. However, most existing approaches remain task-specific and rely primarily on textual instructions, limiting their ability to handle multimodal inputs, contextual references, and diverse video generation and editing scenarios within a unified framework. Moreover, many video editing methods depend on carefully engineered pipelines tailored to individual operations, which hinders scalability and composability. In this paper, we propose Tele-Omni, a unified multimodal framework for video generation and editing that follows multimodal instructions, including text, images, and reference videos, within a single model. Tele-Omni leverages pretrained multimodal large language models to parse heterogeneous instructions and infer structured generation or editing intents, while diffusion-based generators perform high-quality video synthesis conditioned on these structured signals. To enable joint training across heterogeneous video tasks, we introduce a task-aware data processing pipeline that unifies multimodal inputs into a structured instruction format while preserving task-specific constraints. Tele-Omni supports a wide range of video-centric tasks, including text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing. By decoupling instruction parsing from video synthesis and combining it with task-aware data design, Tele-Omni achieves flexible multimodal control while maintaining strong temporal coherence and visual consistency. Experimental results demonstrate that Tele-Omni achieves competitive performance across multiple tasks.