Low-Cost Test-Time Adaptation for Robust Video Editing

TL;DR

Vid-TTA introduces a lightweight, self-supervised test-time adaptation framework that enhances the temporal consistency and robustness of video editing models without significant computational costs.

Contribution

It proposes a novel motion-aware reconstruction and prompt perturbation strategy combined with meta-learning for dynamic loss balancing, improving video editing quality during inference.

Findings

Significantly improves temporal consistency in video editing.

Reduces overfitting to simple prompts.

Maintains low computational overhead during adaptation.

Abstract

Video editing is a critical component of content creation that transforms raw footage into coherent works aligned with specific visual and narrative objectives. Existing approaches face two major challenges: temporal inconsistencies due to failure in capturing complex motion patterns, and overfitting to simple prompts arising from limitations in UNet backbone architectures. While learning-based methods can enhance editing quality, they typically demand substantial computational resources and are constrained by the scarcity of high-quality annotated data. In this paper, we present Vid-TTA, a lightweight test-time adaptation framework that personalizes optimization for each test video during inference through self-supervised auxiliary tasks. Our approach incorporates a motion-aware frame reconstruction mechanism that identifies and preserves crucial movement regions, alongside a prompt perturbation and reconstruction strategy that strengthens model robustness to diverse textual descriptions. These innovations are orchestrated by a meta-learning driven dynamic loss balancing mechanism that adaptively adjusts the optimization process based on video characteristics. Extensive experiments demonstrate that Vid-TTA significantly improves video temporal consistency and mitigates prompt overfitting while maintaining low computational overhead, offering a plug-and-play performance boost for existing video editing models.