MoCha:End-to-End Video Character Replacement without Structural Guidance

TL;DR

MoCha introduces an end-to-end framework for video character replacement that operates without structural guidance, using minimal input and a novel data pipeline to outperform existing methods in complex scenarios.

Contribution

The paper presents MoCha, a novel approach that replaces video characters with only a single mask, bypassing structural guidance and utilizing a new data construction pipeline.

Findings

Outperforms state-of-the-art methods in complex scenarios

Requires only a single arbitrary frame mask for replacement

Uses a new data pipeline with UE5 and synthesized datasets

Abstract

Controllable video character replacement with a user-provided identity remains a challenging problem due to the lack of paired video data. Prior works have predominantly relied on a reconstruction-based paradigm that requires per-frame segmentation masks and explicit structural guidance (e.g., skeleton, depth). This reliance, however, severely limits their generalizability in complex scenarios involving occlusions, character-object interactions, unusual poses, or challenging illumination, often leading to visual artifacts and temporal inconsistencies. In this paper, we propose MoCha, a pioneering framework that bypasses these limitations by requiring only a single arbitrary frame mask. To effectively adapt the multi-modal input condition and enhance facial identity, we introduce a condition-aware RoPE and employ an RL-based post-training stage. Furthermore, to overcome the scarcity of qualified paired-training data, we propose a comprehensive data construction pipeline. Specifically, we design three specialized datasets: a high-fidelity rendered dataset built with Unreal Engine 5 (UE5), an expression-driven dataset synthesized by current portrait animation techniques, and an augmented dataset derived from existing video-mask pairs. Extensive experiments demonstrate that our method substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research. Please refer to our project page for more details: orange-3dv-team.github.io/MoCha