Promptus: Can Prompts Streaming Replace Video Streaming with Stable Diffusion
Jiangkai Wu, Liming Liu, Yunpeng Tan, Junlin Hao, Xinggong Zhang

TL;DR
Promptus introduces a novel semantic communication system that replaces traditional video streaming with prompt streaming and uses Stable Diffusion for video generation, significantly reducing bandwidth while maintaining quality.
Contribution
This work presents Promptus, a new system that employs prompt streaming and diffusion models for efficient video transmission, with novel prompt fitting and bitrate control algorithms.
Findings
Achieves over 4x bandwidth reduction compared to H.265
Improves perceptual quality at low bitrates over VAE and H.265
Reduces severely distorted frames by over 89%
Abstract
With the exponential growth of video traffic, traditional video streaming systems are approaching their limits in compression efficiency and communication capacity. To further reduce bitrate while maintaining quality, we propose Promptus, a disruptive semantic communication system that streaming prompts instead of video content, which represents real-world video frames with a series of "prompts" for delivery and employs Stable Diffusion to generate videos at the receiver. To ensure that the generated video is pixel-aligned with the original video, a gradient descent-based prompt fitting framework is proposed. Further, a low-rank decomposition-based bitrate control algorithm is introduced to achieve adaptive bitrate. For inter-frame compression, an interpolation-aware fitting algorithm is proposed. Evaluations across various video genres demonstrate that, compared to H.265, Promptus can…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsImage and Video Quality Assessment · Video Coding and Compression Technologies
MethodsDiffusion
