TL;DR
TimeLens is a novel method that combines event-based and flow-based techniques to improve video frame interpolation, especially in highly dynamic scenes, achieving significant PSNR improvements over existing methods.
Contribution
We introduce TimeLens, a new approach that leverages both synthesis and flow information from event cameras for superior frame interpolation.
Findings
Up to 5.21 dB PSNR improvement over state-of-the-art methods.
Effective in highly dynamic and low-texture scenarios.
Provides a new large-scale dataset for dynamic scene interpolation.
Abstract
State-of-the-art frame interpolation methods generate intermediate frames by inferring object motions in the image from consecutive key-frames. In the absence of additional information, first-order approximations, i.e. optical flow, must be used, but this choice restricts the types of motions that can be modeled, leading to errors in highly dynamic scenarios. Event cameras are novel sensors that address this limitation by providing auxiliary visual information in the blind-time between frames. They asynchronously measure per-pixel brightness changes and do this with high temporal resolution and low latency. Event-based frame interpolation methods typically adopt a synthesis-based approach, where predicted frame residuals are directly applied to the key-frames. However, while these approaches can capture non-linear motions they suffer from ghosting and perform poorly in low-texture…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
