TimeRewind: Rewinding Time with Image-and-Events Video Diffusion

TL;DR

TimeRewind introduces a novel method combining event camera data with diffusion models to generate plausible videos that rewind time from a single image, capturing missed moments with high temporal fidelity.

Contribution

The paper presents a new framework integrating neuromorphic event data with diffusion models to synthesize pre-capture videos from static images, addressing a previously unsolved problem.

Findings

Successfully generates high-quality rewind videos from single images.

Demonstrates the effectiveness of event-guided diffusion in capturing motion.

Opens new research directions in computational photography and generative modeling.

Abstract

This paper addresses the novel challenge of ``rewinding'' time from a single captured image to recover the fleeting moments missed just before the shutter button is pressed. This problem poses a significant challenge in computer vision and computational photography, as it requires predicting plausible pre-capture motion from a single static frame, an inherently ill-posed task due to the high degree of freedom in potential pixel movements. We overcome this challenge by leveraging the emerging technology of neuromorphic event cameras, which capture motion information with high temporal resolution, and integrating this data with advanced image-to-video diffusion models. Our proposed framework introduces an event motion adaptor conditioned on event camera data, guiding the diffusion model to generate videos that are visually coherent and physically grounded in the captured events. Through extensive experimentation, we demonstrate the capability of our approach to synthesize high-quality videos that effectively ``rewind'' time, showcasing the potential of combining event camera technology with generative models. Our work opens new avenues for research at the intersection of computer vision, computational photography, and generative modeling, offering a forward-thinking solution to capturing missed moments and enhancing future consumer cameras and smartphones. Please see the project page at https://timerewind.github.io/ for video results and code release.