Generalised Image Outpainting with U-Transformer

TL;DR

This paper introduces U-Transformer, a transformer-based GAN that performs all-side image outpainting with long-range dependency modeling, producing realistic extended images for complex scenes.

Contribution

The paper presents a novel U-shaped transformer generator with multi-view TSP for flexible, all-side image outpainting, outperforming existing methods in visual quality.

Findings

Produces visually appealing outpainted images

Handles complex scenes and structures effectively

Allows arbitrary outpainting sizes during testing

Abstract

In this paper, we develop a novel transformer-based generative adversarial neural network called U-Transformer for generalised image outpainting problem. Different from most present image outpainting methods conducting horizontal extrapolation, our generalised image outpainting could extrapolate visual context all-side around a given image with plausible structure and details even for complicated scenery, building, and art images. Specifically, we design a generator as an encoder-to-decoder structure embedded with the popular Swin Transformer blocks. As such, our novel neural network can better cope with image long-range dependencies which are crucially important for generalised image outpainting. We propose additionally a U-shaped structure and multi-view Temporal Spatial Predictor (TSP) module to reinforce image self-reconstruction as well as unknown-part prediction smoothly and realistically. By adjusting the predicting step in the TSP module in the testing stage, we can generate arbitrary outpainting size given the input sub-image. We experimentally demonstrate that our proposed method could produce visually appealing results for generalized image outpainting against the state-of-the-art image outpainting approaches.