InverseCrafter: Efficient Video ReCapture as a Latent Domain Inverse Problem

TL;DR

InverseCrafter introduces an efficient latent space inpainting approach for 4D video generation, enabling high-quality view synthesis and editing without extensive fine-tuning or computational costs.

Contribution

It reformulates 4D video generation as a latent inpainting problem, significantly reducing computational overhead and avoiding catastrophic forgetting of pre-trained models.

Findings

Achieves comparable view synthesis quality to fine-tuned models

Provides superior measurement consistency in camera control

Excels at general-purpose video inpainting and editing

Abstract

Recent approaches to controllable 4D video generation often rely on fine-tuning pre-trained Video Diffusion Models (VDMs). This dominant paradigm is computationally expensive, requiring large-scale datasets and architectural modifications, and frequently suffers from catastrophic forgetting of the model's original generative priors. Here, we propose InverseCrafter, an efficient inpainting inverse solver that reformulates the 4D generation task as an inpainting problem solved in the latent space. The core of our method is a principled mechanism to encode the pixel space degradation operator into a continuous, multi-channel latent mask, thereby bypassing the costly bottleneck of repeated VAE operations and backpropagation. InverseCrafter not only achieves comparable novel view generation and superior measurement consistency in camera control tasks with near-zero computational overhead, but also excels at general-purpose video inpainting with editing. Code is available at https://github.com/yeobinhong/InverseCrafter.