Style-Guided Inference of Transformer for High-resolution Image Synthesis

TL;DR

This paper introduces a style-guided method for high-resolution image synthesis using transformers, enabling style control without retraining by re-balancing the prior with style constraints.

Contribution

It proposes a novel style transfer approach that conditions transformer-based image synthesis on style images, improving output style consistency without retraining.

Findings

Generated images match input style in most cases

Method reduces the need for multiple samples to achieve desired style

Style control is achieved without retraining the transformer

Abstract

Transformer is eminently suitable for auto-regressive image synthesis which predicts discrete value from the past values recursively to make up full image. Especially, combined with vector quantised latent representation, the state-of-the-art auto-regressive transformer displays realistic high-resolution images. However, sampling the latent code from discrete probability distribution makes the output unpredictable. Therefore, it requires to generate lots of diverse samples to acquire desired outputs. To alleviate the process of generating lots of samples repetitively, in this article, we propose to take a desired output, a style image, as an additional condition without re-training the transformer. To this end, our method transfers the style to a probability constraint to re-balance the prior, thereby specifying the target distribution instead of the original prior. Thus, generated samples from the re-balanced prior have similar styles to reference style. In practice, we can choose either an image or a category of images as an additional condition. In our qualitative assessment, we show that styles of majority of outputs are similar to the input style.