Promptus: Can Prompts Streaming Replace Video Streaming with Stable Diffusion

TL;DR

Promptus introduces a novel semantic communication system that replaces traditional video streaming with prompt streaming and uses Stable Diffusion for video generation, significantly reducing bandwidth while maintaining quality.

Contribution

This work presents Promptus, a new system that employs prompt streaming and diffusion models for efficient video transmission, with novel prompt fitting and bitrate control algorithms.

Findings

Achieves over 4x bandwidth reduction compared to H.265

Improves perceptual quality at low bitrates over VAE and H.265

Reduces severely distorted frames by over 89%

Abstract

With the exponential growth of video traffic, traditional video streaming systems are approaching their limits in compression efficiency and communication capacity. To further reduce bitrate while maintaining quality, we propose Promptus, a disruptive semantic communication system that streaming prompts instead of video content, which represents real-world video frames with a series of "prompts" for delivery and employs Stable Diffusion to generate videos at the receiver. To ensure that the generated video is pixel-aligned with the original video, a gradient descent-based prompt fitting framework is proposed. Further, a low-rank decomposition-based bitrate control algorithm is introduced to achieve adaptive bitrate. For inter-frame compression, an interpolation-aware fitting algorithm is proposed. Evaluations across various video genres demonstrate that, compared to H.265, Promptus can achieve more than a 4x bandwidth reduction while preserving the same perceptual quality. On the other hand, at extremely low bitrates, Promptus can enhance the perceptual quality by 0.139 and 0.118 (in LPIPS) compared to VAE and H.265, respectively, and decreases the ratio of severely distorted frames by 89.3% and 91.7%. Our work opens up a new paradigm for efficient video communication. Promptus is open-sourced at: https://github.com/JiangkaiWu/Promptus.