CamViG: Camera Aware Image-to-Video Generation with Multimodal Transformers

TL;DR

This paper introduces CamViG, a multimodal transformer-based method that incorporates 3D camera motion as a control signal for generating videos from a single frame, enabling precise camera path control and validation.

Contribution

The paper presents a novel approach to include 3D camera motion in generative video models using multimodal transformers, allowing for controlled video synthesis from a single image.

Findings

Successful control of camera during video generation

Accurate 3D camera path prediction using computer vision methods

Effective generation of videos conditioned on camera movement

Abstract

We extend multimodal transformers to include 3D camera motion as a conditioning signal for the task of video generation. Generative video models are becoming increasingly powerful, thus focusing research efforts on methods of controlling the output of such models. We propose to add virtual 3D camera controls to generative video methods by conditioning generated video on an encoding of three-dimensional camera movement over the course of the generated video. Results demonstrate that we are (1) able to successfully control the camera during video generation, starting from a single frame and a camera signal, and (2) we demonstrate the accuracy of the generated 3D camera paths using traditional computer vision methods.