GenRec: Unifying Video Generation and Recognition with Diffusion Models

TL;DR

GenRec is a unified diffusion-based framework that enhances video recognition and generation by learning generalized spatial-temporal representations, demonstrating robustness and competitive performance on multiple benchmarks.

Contribution

It introduces the first unified model trained with random-frame conditioning to jointly learn video generation and recognition capabilities.

Findings

Achieves 75.8% accuracy on SSV2 and 87.2% on K400 for recognition.

Sets new state-of-the-art FVD scores for class-conditioned video generation.

Demonstrates robustness with limited input frames.

Abstract

Video diffusion models are able to generate high-quality videos by learning strong spatial-temporal priors on large-scale datasets. In this paper, we aim to investigate whether such priors derived from a generative process are suitable for video recognition, and eventually joint optimization of generation and recognition. Building upon Stable Video Diffusion, we introduce GenRec, the first unified framework trained with a random-frame conditioning process so as to learn generalized spatial-temporal representations. The resulting framework can naturally supports generation and recognition, and more importantly is robust even when visual inputs contain limited information. Extensive experiments demonstrate the efficacy of GenRec for both recognition and generation. In particular, GenRec achieves competitive recognition performance, offering 75.8% and 87.2% accuracy on SSV2 and K400, respectively. GenRec also performs the best on class-conditioned image-to-video generation, achieving 46.5 and 49.3 FVD scores on SSV2 and EK-100 datasets. Furthermore, GenRec demonstrates extraordinary robustness in scenarios that only limited frames can be observed. Code will be available at https://github.com/wengzejia1/GenRec.