SE360: Semantic Edit in 360$^\circ$ Panoramas via Hierarchical Data Construction

TL;DR

SE360 introduces a hierarchical data construction framework for semantic editing in 360° panoramas, enabling realistic, guided object modifications with improved quality and accuracy over existing methods.

Contribution

The paper presents a novel coarse-to-fine data generation pipeline and a Transformer-based diffusion model for semantic editing in 360° panoramas, addressing previous limitations.

Findings

Outperforms existing methods in visual quality

Achieves higher semantic accuracy in edits

Ensures geometrically consistent data generation

Abstract

While instruction-based image editing is emerging, extending it to 360$^\circ$ panoramas introduces additional challenges. Existing methods often produce implausible results in both equirectangular projections (ERP) and perspective views. To address these limitations, we propose SE360, a novel framework for multi-condition guided object editing in 360$^\circ$ panoramas. At its core is a novel coarse-to-fine autonomous data generation pipeline without manual intervention. This pipeline leverages a Vision-Language Model (VLM) and adaptive projection adjustment for hierarchical analysis, ensuring the holistic segmentation of objects and their physical context. The resulting data pairs are both semantically meaningful and geometrically consistent, even when sourced from unlabeled panoramas. Furthermore, we introduce a cost-effective, two-stage data refinement strategy to improve data realism and mitigate model overfitting to erase artifacts. Based on the constructed dataset, we train a Transformer-based diffusion model to allow flexible object editing guided by text, mask, or reference image in 360$^\circ$ panoramas. Our experiments demonstrate that our method outperforms existing methods in both visual quality and semantic accuracy.