MTC-VAE: Multi-Level Temporal Compression with Content Awareness

TL;DR

This paper introduces MTC-VAE, a technique for multi-level temporal video compression that maintains performance at higher compression rates and integrates with diffusion models, supported by empirical evaluations.

Contribution

The paper proposes a minimal fine-tuning method to enable VAEs to support multi-level temporal compression, enhancing efficiency without increasing model complexity.

Findings

Supports higher compression rates with minimal performance loss

Demonstrates effective integration with diffusion-based generative models

Provides empirical evidence on compression performance across video segments

Abstract

Latent Video Diffusion Models (LVDMs) rely on Variational Autoencoders (VAEs) to compress videos into compact latent representations. For continuous Variational Autoencoders (VAEs), achieving higher compression rates is desirable; yet, the efficiency notably declines when extra sampling layers are added without expanding the dimensions of hidden channels. In this paper, we present a technique to convert fixed compression rate VAEs into models that support multi-level temporal compression, providing a straightforward and minimal fine-tuning approach to counteract performance decline at elevated compression rates.Moreover, we examine how varying compression levels impact model performance over video segments with diverse characteristics, offering empirical evidence on the effectiveness of our proposed approach. We also investigate the integration of our multi-level temporal compression VAE with diffusion-based generative models, DiT, highlighting successful concurrent training and compatibility within these frameworks. This investigation illustrates the potential uses of multi-level temporal compression.