Fast-Vid2Vid: Spatial-Temporal Compression for Video-to-Video Synthesis

TL;DR

Fast-Vid2Vid introduces a spatial-temporal compression framework for video-to-video synthesis, significantly reducing computational costs and inference latency while maintaining high-quality, real-time video generation.

Contribution

It is the first to focus on compressing the data stream in both spatial and temporal dimensions for Vid2Vid, enabling faster inference with minimal quality loss.

Findings

Achieves around 20 FPS on standard benchmarks.

Saves approximately 8x computational cost on a single V100 GPU.

Maintains high-quality video synthesis with reduced latency.

Abstract

Video-to-Video synthesis (Vid2Vid) has achieved remarkable results in generating a photo-realistic video from a sequence of semantic maps. However, this pipeline suffers from high computational cost and long inference latency, which largely depends on two essential factors: 1) network architecture parameters, 2) sequential data stream. Recently, the parameters of image-based generative models have been significantly compressed via more efficient network architectures. Nevertheless, existing methods mainly focus on slimming network architectures and ignore the size of the sequential data stream. Moreover, due to the lack of temporal coherence, image-based compression is not sufficient for the compression of the video task. In this paper, we present a spatial-temporal compression framework, \textbf{Fast-Vid2Vid}, which focuses on data aspects of generative models. It makes the first attempt at time dimension to reduce computational resources and accelerate inference. Specifically, we compress the input data stream spatially and reduce the temporal redundancy. After the proposed spatial-temporal knowledge distillation, our model can synthesize key-frames using the low-resolution data stream. Finally, Fast-Vid2Vid interpolates intermediate frames by motion compensation with slight latency. On standard benchmarks, Fast-Vid2Vid achieves around real-time performance as 20 FPS and saves around 8x computational cost on a single V100 GPU.