Visual Autoregressive Modeling for Image Super-Resolution

TL;DR

This paper introduces VARSR, a novel autoregressive model for image super-resolution that effectively balances fidelity and realism, leveraging semantic preservation, spatial encoding, and guidance techniques for high-quality image generation.

Contribution

The paper proposes VARSR, a new autoregressive framework for ISR that integrates semantic tokens, spatial encodings, and guidance to improve image quality and efficiency.

Findings

Outperforms diffusion-based methods in quality and efficiency.

Generates high-fidelity, realistic images with semantic consistency.

Utilizes large-scale data and robust training for superior priors.

Abstract

Image Super-Resolution (ISR) has seen significant progress with the introduction of remarkable generative models. However, challenges such as the trade-off issues between fidelity and realism, as well as computational complexity, have also posed limitations on their application. Building upon the tremendous success of autoregressive models in the language domain, we propose \textbf{VARSR}, a novel visual autoregressive modeling for ISR framework with the form of next-scale prediction. To effectively integrate and preserve semantic information in low-resolution images, we propose using prefix tokens to incorporate the condition. Scale-aligned Rotary Positional Encodings are introduced to capture spatial structures and the diffusion refiner is utilized for modeling quantization residual loss to achieve pixel-level fidelity. Image-based Classifier-free Guidance is proposed to guide the generation of more realistic images. Furthermore, we collect large-scale data and design a training process to obtain robust generative priors. Quantitative and qualitative results show that VARSR is capable of generating high-fidelity and high-realism images with more efficiency than diffusion-based methods. Our codes will be released at https://github.com/qyp2000/VARSR.