AnyView: Synthesizing Any Novel View in Dynamic Scenes

TL;DR

AnyView is a diffusion-based framework that synthesizes novel dynamic views in videos from arbitrary camera angles, leveraging diverse datasets to maintain consistency and realism in highly dynamic scenes.

Contribution

It introduces a generalist spatiotemporal implicit representation trained on multiple data sources for zero-shot dynamic view synthesis without geometric assumptions.

Findings

AnyView achieves competitive results on standard benchmarks.

It maintains high-quality, consistent videos from any viewpoint in extreme dynamic scenarios.

Most baselines fail under extreme conditions, while AnyView remains robust.

Abstract

Modern generative video models excel at producing convincing, high-quality outputs, but struggle to maintain multi-view and spatiotemporal consistency in highly dynamic real-world environments. In this work, we introduce \textbf{AnyView}, a diffusion-based video generation framework for \emph{dynamic view synthesis} with minimal inductive biases or geometric assumptions. We leverage multiple data sources with various levels of supervision, including monocular (2D), multi-view static (3D) and multi-view dynamic (4D) datasets, to train a generalist spatiotemporal implicit representation capable of producing zero-shot novel videos from arbitrary camera locations and trajectories. We evaluate AnyView on standard benchmarks, showing competitive results with the current state of the art, and propose \textbf{AnyViewBench}, a challenging new benchmark tailored towards \emph{extreme} dynamic view synthesis in diverse real-world scenarios. In this more dramatic setting, we find that most baselines drastically degrade in performance, as they require significant overlap between viewpoints, while AnyView maintains the ability to produce realistic, plausible, and spatiotemporally consistent videos when prompted from \emph{any} viewpoint. Results, data, code, and models can be viewed at: https://tri-ml.github.io/AnyView/