Adaptive 1D Video Diffusion Autoencoder

TL;DR

This paper introduces One-DVA, a transformer-based autoencoder for video that adaptively compresses and reconstructs videos using diffusion models, overcoming fixed-rate limitations and enhancing generative capabilities.

Contribution

The paper presents a novel adaptive 1D video autoencoder with transformer-based encoding and diffusion decoding, enabling variable-length compression and improved generative support.

Findings

Achieves comparable reconstruction quality to 3D-CNN VAEs at similar compression ratios.

Supports adaptive compression for higher ratios.

Regularizes latent distribution for better generative modeling.

Abstract

Recent video generation models largely rely on video autoencoders that compress pixel-space videos into latent representations. However, existing video autoencoders suffer from three major limitations: (1) fixed-rate compression that wastes tokens on simple videos, (2) inflexible CNN architectures that prevent variable-length latent modeling, and (3) deterministic decoders that struggle to recover appropriate details from compressed latents. To address these issues, we propose One-Dimensional Diffusion Video Autoencoder (One-DVA), a transformer-based framework for adaptive 1D encoding and diffusion-based decoding. The encoder employs query-based vision transformers to extract spatiotemporal features and produce latent representations, while a variable-length dropout mechanism dynamically adjusts the latent length. The decoder is a pixel-space diffusion transformer that reconstructs videos with the latents as input conditions. With a two-stage training strategy, One-DVA achieves performance comparable to 3D-CNN VAEs on reconstruction metrics at identical compression ratios. More importantly, it supports adaptive compression and thus can achieve higher compression ratios. To better support downstream latent generation, we further regularize the One-DVA latent distribution for generative modeling and fine-tune its decoder to mitigate artifacts caused by the generation process.