V-LASIK: Consistent Glasses-Removal from Videos Using Synthetic Data

TL;DR

This paper introduces V-LASIK, a novel method for consistent glasses removal in videos using synthetic data and diffusion models, addressing challenges of artifact generation and inconsistency in local video editing.

Contribution

The paper presents a weakly supervised approach leveraging synthetic data and pretrained diffusion models for consistent, identity-preserving local video editing, specifically glasses removal.

Findings

Achieves consistent glasses removal in videos with preserved identity.

Outperforms existing methods in local video editing tasks.

Successfully generalizes to facial sticker removal.

Abstract

Diffusion-based generative models have recently shown remarkable image and video editing capabilities. However, local video editing, particularly removal of small attributes like glasses, remains a challenge. Existing methods either alter the videos excessively, generate unrealistic artifacts, or fail to perform the requested edit consistently throughout the video. In this work, we focus on consistent and identity-preserving removal of glasses in videos, using it as a case study for consistent local attribute removal in videos. Due to the lack of paired data, we adopt a weakly supervised approach and generate synthetic imperfect data, using an adjusted pretrained diffusion model. We show that despite data imperfection, by learning from our generated data and leveraging the prior of pretrained diffusion models, our model is able to perform the desired edit consistently while preserving the original video content. Furthermore, we exemplify the generalization ability of our method to other local video editing tasks by applying it successfully to facial sticker-removal. Our approach demonstrates significant improvement over existing methods, showcasing the potential of leveraging synthetic data and strong video priors for local video editing tasks.