NUWA-Infinity: Autoregressive over Autoregressive Generation for Infinite Visual Synthesis

TL;DR

NUWA-Infinity introduces an autoregressive over autoregressive model for infinite visual synthesis, enabling high-resolution, arbitrarily-sized images and long videos with efficient dependency modeling and flexible generation order.

Contribution

It proposes a novel hierarchical autoregressive generation mechanism with a Nearby Context Pool and Arbitrary Direction Controller for scalable, high-quality visual synthesis.

Findings

Supports high-resolution, arbitrary-sized image generation

Enables long-duration video synthesis

Outperforms previous models in resolution and flexibility

Abstract

In this paper, we present NUWA-Infinity, a generative model for infinite visual synthesis, which is defined as the task of generating arbitrarily-sized high-resolution images or long-duration videos. An autoregressive over autoregressive generation mechanism is proposed to deal with this variable-size generation task, where a global patch-level autoregressive model considers the dependencies between patches, and a local token-level autoregressive model considers dependencies between visual tokens within each patch. A Nearby Context Pool (NCP) is introduced to cache-related patches already generated as the context for the current patch being generated, which can significantly save computation costs without sacrificing patch-level dependency modeling. An Arbitrary Direction Controller (ADC) is used to decide suitable generation orders for different visual synthesis tasks and learn order-aware positional embeddings. Compared to DALL-E, Imagen and Parti, NUWA-Infinity can generate high-resolution images with arbitrary sizes and support long-duration video generation additionally. Compared to NUWA, which also covers images and videos, NUWA-Infinity has superior visual synthesis capabilities in terms of resolution and variable-size generation. The GitHub link is https://github.com/microsoft/NUWA. The homepage link is https://nuwa-infinity.microsoft.com.