APLA: Additional Perturbation for Latent Noise with Adversarial Training Enables Consistency

TL;DR

This paper introduces APLA, a novel diffusion-based text-to-video generation method that enhances temporal consistency by extracting and refining inherent input information using an auxiliary transformer network.

Contribution

The paper proposes APLA, a new diffusion model framework with an auxiliary transformer network to improve temporal consistency in video generation from a single input.

Findings

Significant improvement in video consistency both qualitatively and quantitatively.

Effective extraction of inherent input information to refine pixel predictions.

Utilization of a hybrid transformer-convolution architecture for temporal modeling.

Abstract

Diffusion models have exhibited promising progress in video generation. However, they often struggle to retain consistent details within local regions across frames. One underlying cause is that traditional diffusion models approximate Gaussian noise distribution by utilizing predictive noise, without fully accounting for the impact of inherent information within the input itself. Additionally, these models emphasize the distinction between predictions and references, neglecting information intrinsic to the videos. To address this limitation, inspired by the self-attention mechanism, we propose a novel text-to-video (T2V) generation network structure based on diffusion models, dubbed Additional Perturbation for Latent noise with Adversarial training (APLA). Our approach only necessitates a single video as input and builds upon pre-trained stable diffusion networks. Notably, we introduce an additional compact network, known as the Video Generation Transformer (VGT). This auxiliary component is designed to extract perturbations from the inherent information contained within the input, thereby refining inconsistent pixels during temporal predictions. We leverage a hybrid architecture of transformers and convolutions to compensate for temporal intricacies, enhancing consistency between different frames within the video. Experiments demonstrate a noticeable improvement in the consistency of the generated videos both qualitatively and quantitatively.