OPa-Ma: Text Guided Mamba for 360-degree Image Out-painting

TL;DR

This paper introduces OPa-Ma, a novel text-guided framework utilizing a State-Space Model called Mamba for efficient and consistent 360-degree image out-painting from narrow field of view images, addressing existing method limitations.

Contribution

The paper proposes a new text-guided out-painting framework with a State-Space Model and modules for better feature fusion and spatial continuity, achieving state-of-the-art results.

Findings

Outperforms existing methods on 360-degree image datasets

Maintains visual continuity and style consistency

Enhances diversity with textual guidance

Abstract

In this paper, we tackle the recently popular topic of generating 360-degree images given the conventional narrow field of view (NFoV) images that could be taken from a single camera or cellphone. This task aims to predict the reasonable and consistent surroundings from the NFoV images. Existing methods for feature extraction and fusion, often built with transformer-based architectures, incur substantial memory usage and computational expense. They also have limitations in maintaining visual continuity across the entire 360-degree images, which could cause inconsistent texture and style generation. To solve the aforementioned issues, we propose a novel text-guided out-painting framework equipped with a State-Space Model called Mamba to utilize its long-sequence modelling and spatial continuity. Furthermore, incorporating textual information is an effective strategy for guiding image generation, enriching the process with detailed context and increasing diversity. Efficiently extracting textual features and integrating them with image attributes presents a significant challenge for 360-degree image out-painting. To address this, we develop two modules, Visual-textual Consistency Refiner (VCR) and Global-local Mamba Adapter (GMA). VCR enhances contextual richness by fusing the modified text features with the image features, while GMA provides adaptive state-selective conditions by capturing the information flow from global to local representations. Our proposed method achieves state-of-the-art performance with extensive experiments on two broadly used 360-degree image datasets, including indoor and outdoor settings.