Kling-Omni Technical Report

TL;DR

Kling-Omni is a versatile multimodal generative framework that synthesizes high-quality videos from diverse visual language inputs, integrating generation, editing, and reasoning in a unified system.

Contribution

It introduces a holistic end-to-end system for multimodal video creation, combining diverse inputs and tasks into a single framework with comprehensive data and optimized training strategies.

Findings

Demonstrates exceptional in-context video generation capabilities

Enables reasoning-based video editing and instruction following

Supports cinematic-quality, highly intelligent video synthesis

Abstract

We present Kling-Omni, a generalist generative framework designed to synthesize high-fidelity videos directly from multimodal visual language inputs. Adopting an end-to-end perspective, Kling-Omni bridges the functional separation among diverse video generation, editing, and intelligent reasoning tasks, integrating them into a holistic system. Unlike disjointed pipeline approaches, Kling-Omni supports a diverse range of user inputs, including text instructions, reference images, and video contexts, processing them into a unified multimodal representation to deliver cinematic-quality and highly-intelligent video content creation. To support these capabilities, we constructed a comprehensive data system that serves as the foundation for multimodal video creation. The framework is further empowered by efficient large-scale pre-training strategies and infrastructure optimizations for inference. Comprehensive evaluations reveal that Kling-Omni demonstrates exceptional capabilities in in-context generation, reasoning-based editing, and multimodal instruction following. Moving beyond a content creation tool, we believe Kling-Omni is a pivotal advancement toward multimodal world simulators capable of perceiving, reasoning, generating and interacting with the dynamic and complex worlds.