Spanning Tree Autoregressive Visual Generation

TL;DR

The paper introduces Spanning Tree Autoregressive (STAR) modeling for visual generation, which uses spanning tree traversal orders to incorporate prior image knowledge, enhancing flexibility and performance in image editing tasks.

Contribution

STAR employs spanning tree traversal orders to improve autoregressive image generation, balancing prior knowledge incorporation with flexible sequence ordering at inference.

Findings

Maintains sampling performance with prior knowledge incorporation.

Allows flexible sequence orders for image editing.

Efficient traversal order construction via breadth-first search.

Abstract

We present Spanning Tree Autoregressive (STAR) modeling, which can incorporate prior knowledge of images, such as center bias and locality, to maintain sampling performance while also providing sufficiently flexible sequence orders to accommodate image editing at inference. Approaches that expose randomly permuted sequence orders to conventional autoregressive (AR) models in visual generation for bidirectional context either suffer from a decline in performance or compromise the flexibility in sequence order choice at inference. Instead, STAR utilizes traversal orders of uniform spanning trees sampled in a lattice defined by the positions of image patches. Traversal orders are obtained through breadth-first search, allowing us to efficiently construct a spanning tree whose traversal order ensures that the connected partial observation of the image appears as a prefix in the sequence through rejection sampling. Through the tailored yet structured randomized strategy compared to random permutation, STAR preserves the capability of postfix completion while maintaining sampling performance without any significant changes to the model architecture widely adopted in the language AR modeling.