Beyond the Visible: Disocclusion-Aware Editing via Proxy Dynamic Graphs

TL;DR

This paper introduces a novel, training-free image-to-video generation method that allows explicit user control over disoccluded regions by separating motion specification from appearance synthesis using a Proxy Dynamic Graph and diffusion prior.

Contribution

It proposes a lightweight, user-editable Proxy Dynamic Graph for deterministic motion control combined with a diffusion prior for appearance synthesis, enabling controllable video generation without fine-tuning.

Findings

Outperforms state-of-the-art in controllable articulated object videos

Allows user editing of disoccluded regions in generated videos

Enables predictable motion and appearance control in image-to-video tasks

Abstract

We address image-to-video generation with explicit user control over the final frame's disoccluded regions. Current image-to-video pipelines produce plausible motion but struggle to generate predictable, articulated motions while enforcing user-specified content in newly revealed areas. Our key idea is to separate motion specification from appearance synthesis: we introduce a lightweight, user-editable Proxy Dynamic Graph (PDG) that deterministically yet approximately drives part motion, while a frozen diffusion prior is used to synthesize plausible appearance that follows that motion. In our training-free pipeline, the user loosely annotates and reposes a PDG, from which we compute a dense motion flow to leverage diffusion as a motion-guided shader. We then let the user edit appearance in the disoccluded areas of the image, and exploit the visibility information encoded by the PDG to perform a latent-space composite that reconciles motion with user intent in these areas. This design yields controllable articulation and user control over disocclusions without fine-tuning. We demonstrate clear advantages against state-of-the-art alternatives towards images turned into short videos of articulated objects, furniture, vehicles, and deformables. Our method mixes generative control, in the form of loose pose and structure, with predictable controls, in the form of appearance specification in the final frame in the disoccluded regions, unlocking a new image-to-video workflow. Code will be released on acceptance. Project page: https://anranqi.github.io/beyond-visible.github.io/