AKiRa: Augmentation Kit on Rays for optical video generation
Xi Wang, Robin Courant, Marc Christie, Vicky Kalogeiton

TL;DR
AKiRa introduces a novel framework that enhances controllability in text-conditioned video generation by allowing fine-tuned manipulation of camera motion and optical parameters, enabling cinematic effects and improved visual storytelling.
Contribution
This paper presents AKiRa, a new augmentation framework that integrates complex camera models into video generation, offering unprecedented control over optical and motion parameters.
Findings
Outperforms state-of-the-art methods in controllable video generation
Enables realistic cinematic effects like zoom and fisheye
Demonstrates effective combination of camera optics in generated videos
Abstract
Recent advances in text-conditioned video diffusion have greatly improved video quality. However, these methods offer limited or sometimes no control to users on camera aspects, including dynamic camera motion, zoom, distorted lens and focus shifts. These motion and optical aspects are crucial for adding controllability and cinematic elements to generation frameworks, ultimately resulting in visual content that draws focus, enhances mood, and guides emotions according to filmmakers' controls. In this paper, we aim to close the gap between controllable video generation and camera optics. To achieve this, we propose AKiRa (Augmentation Kit on Rays), a novel augmentation framework that builds and trains a camera adapter with a complex camera model over an existing video generation backbone. It enables fine-tuned control over camera motion as well as complex optical parameters (focal…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsAdvanced Vision and Imaging · Advanced Optical Imaging Technologies
MethodsDiffusion · Adapter · Focus
