StreamWise: Serving Multi-Modal Generation in Real-Time at Scale

TL;DR

StreamWise is a modular, adaptive system designed to serve multi-modal generative models in real-time, optimizing latency, quality, and resource use for applications like podcast video creation.

Contribution

It introduces StreamWise, a novel, resource-aware, modular serving system that dynamically manages multi-modal generative workloads at scale in real-time.

Findings

Achieves sub-second startup delay for streaming

Cost-effective generation of 10-minute videos in 1.4 hours

Balances latency, quality, and resource utilization effectively

Abstract

Advances in multi-modal generative models are enabling new applications, from storytelling to automated media synthesis. Most current workloads generate simple outputs (e.g., image generation from a prompt) in batch mode, often requiring several seconds even for basic results. Serving real-time multi-modal workflows at scale is costly and complex, requiring efficient coordination of diverse models (each with unique resource needs) across language, audio, image, and video, all under strict latency and resource constraints. We tackle these challenges through the lens of real-time podcast video generation, integrating LLMs, text-to-speech, and video-audio generation. To meet tight SLOs, we design an adaptive, modular serving system, StreamWise, that dynamically manages quality (e.g., resolution, sharpness), model/content parallelism, and resource-aware scheduling. We leverage heterogeneous hardware to maximize responsiveness and efficiency. For example, the system can lower video resolution and allocate more resources to early scenes. We quantify the trade-offs between latency, cost, and quality. The cheapest setup generates a 10-minute podcast video on A100 GPUs in 1.4 hours (8.4x slower than the real-time) for less than \$25. StreamWise enables high-quality real-time streaming with a sub-second startup delay under $45.