RAIN: Real-time Animation of Infinite Video Stream
Zhilei Shu, Ruili Feng, Yang Cao, Zheng-Jun Zha

TL;DR
RAIN enables real-time, high-quality, and stable infinite video stream animation on consumer GPUs by efficiently managing long-range attention and denoising, significantly improving continuity and consistency.
Contribution
Introduces RAIN, a novel pipeline that achieves real-time infinite video stream animation with low latency and minimal additional computational burden on consumer GPUs.
Findings
Outperforms existing methods in quality, accuracy, and consistency.
Enables real-time animation of super-long videos.
Maintains high visual quality over extended streams.
Abstract
Live animation has gained immense popularity for enhancing online engagement, yet achieving high-quality, real-time, and stable animation with diffusion models remains challenging, especially on consumer-grade GPUs. Existing methods struggle with generating long, consistent video streams efficiently, often being limited by latency issues and degraded visual quality over extended periods. In this paper, we introduce RAIN, a pipeline solution capable of animating infinite video streams in real-time with low latency using a single RTX 4090 GPU. The core idea of RAIN is to efficiently compute frame-token attention across different noise levels and long time-intervals while simultaneously denoising a significantly larger number of frame-tokens than previous stream-based methods. This design allows RAIN to generate video frames with much shorter latency and faster speed, while maintaining…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsHuman Motion and Animation · Computer Graphics and Visualization Techniques · Advanced Vision and Imaging
MethodsSoftmax · Attention Is All You Need · Diffusion
