From Generation to Generalization: Emergent Few-Shot Learning in Video Diffusion Models

TL;DR

This paper demonstrates that Video Diffusion Models can be fine-tuned with minimal data to perform a variety of vision tasks, revealing their potential as adaptable visual learners beyond generation.

Contribution

The authors introduce a few-shot fine-tuning framework for VDMs that enables task adaptation without altering the model's core architecture.

Findings

VDMs can be repurposed for diverse vision tasks with minimal supervision

The method achieves strong generalization across low- and high-level vision tasks

VDMs serve as adaptable visual learners, not just generative models

Abstract

Video Diffusion Models (VDMs) have emerged as powerful generative tools, capable of synthesizing high-quality spatiotemporal content. Yet, their potential goes far beyond mere video generation. We argue that the training dynamics of VDMs, driven by the need to model coherent sequences, naturally pushes them to internalize structured representations and an implicit understanding of the visual world. To probe the extent of this internal knowledge, we introduce a few-shot fine-tuning framework that repurposes VDMs for new tasks using only a handful of examples. Our method transforms each task into a visual transition, enabling the training of LoRA weights on short input-output sequences without altering the generative interface of a frozen VDM. Despite minimal supervision, the model exhibits strong generalization across diverse tasks, from low-level vision (for example, segmentation and pose estimation) to high-level reasoning (for example, on ARC-AGI). These results reframe VDMs as more than generative engines. They are adaptable visual learners with the potential to serve as the backbone for future foundation models in vision.