AKiRa: Augmentation Kit on Rays for optical video generation

TL;DR

AKiRa introduces a novel framework that enhances controllability in text-conditioned video generation by allowing fine-tuned manipulation of camera motion and optical parameters, enabling cinematic effects and improved visual storytelling.

Contribution

This paper presents AKiRa, a new augmentation framework that integrates complex camera models into video generation, offering unprecedented control over optical and motion parameters.

Findings

Outperforms state-of-the-art methods in controllable video generation

Enables realistic cinematic effects like zoom and fisheye

Demonstrates effective combination of camera optics in generated videos

Abstract

Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this paper, we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal length, distortion, aperture) to achieve cinematic effects such as zoom, fisheye effect, and bokeh. Extensive experiments demonstrate AKiRa's effectiveness in combining and composing camera optics while outperforming all state-of-the-art methods. This work sets a new landmark in controlled and optically enhanced video generation, paving the way for future optical video generation methods.