Increasing diversity of omni-directional images generated from single image using cGAN based on MLPMixer

TL;DR

This paper introduces a MLPMixer-based method for generating diverse omni-directional images from a single photo, overcoming CNN limitations by capturing long-range dependencies and reducing memory use.

Contribution

The paper presents a novel MLPMixer-based approach for omni-directional image generation, improving diversity and efficiency over traditional CNN-based methods.

Findings

Achieved higher diversity in generated images.

Reduced memory and computational requirements.

Competitive performance compared to CNN-based models.

Abstract

This paper proposes a novel approach to generating omni-directional images from a single snapshot picture. The previous method has relied on the generative adversarial networks based on convolutional neural networks (CNN). Although this method has successfully generated omni-directional images, CNN has two drawbacks for this task. First, since a convolutional layer only processes a local area, it is difficult to propagate the information of an input snapshot picture embedded in the center of the omni-directional image to the edges of the image. Thus, the omni-directional images created by the CNN-based generator tend to have less diversity at the edges of the generated images, creating similar scene images. Second, the CNN-based model requires large video memory in graphics processing units due to the nature of the deep structure in CNN since shallow-layer networks only receives signals from a limited range of the receptive field. To solve these problems, MLPMixer-based method was proposed in this paper. The MLPMixer has been proposed as an alternative to the self-attention in the transformer, which captures long-range dependencies and contextual information. This enables to propagate information efficiently in the omni-directional image generation task. As a result, competitive performance has been achieved with reduced memory consumption and computational cost, in addition to increasing diversity of the generated omni-directional images.