Dual-Projection Fusion for Accurate Upright Panorama Generation in Robotic Vision

TL;DR

This paper introduces a dual-stream neural network that jointly estimates camera inclination and reconstructs upright panoramic images, improving accuracy in robotic vision applications with 360-degree cameras.

Contribution

It proposes a novel dual-projection adaptive fusion network with modules for inclination estimation and panorama correction, outperforming existing methods.

Findings

Outperforms existing methods in inclination estimation accuracy.

Achieves superior upright panorama reconstruction on SUN360 and M3D datasets.

Demonstrates the effectiveness of dual-projection fusion and attention mechanisms.

Abstract

Panoramic cameras, capable of capturing a 360-degree field of view, are crucial in robotic vision, particularly in environments with sparse features. However, non-upright panoramas due to unstable robot postures hinder downstream tasks. Traditional IMU-based correction methods suffer from drift and external disturbances, while vision-based approaches offer a promising alternative. This study presents a dual-stream angle-aware generation network that jointly estimates camera inclination angles and reconstructs upright panoramic images. The network comprises a CNN branch that extracts local geometric structures from equirectangular projections and a ViT branch that captures global contextual cues from cubemap projections. These are integrated through a dual-projection adaptive fusion module that aligns spatial features across both domains. To further enhance performance, we introduce a high-frequency enhancement block, circular padding, and channel attention mechanisms to preserve 360{\deg} continuity and improve geometric sensitivity. Experiments on the SUN360 and M3D datasets demonstrate that our method outperforms existing approaches in both inclination estimation and upright panorama generation. Ablation studies further validate the contribution of each module and highlight the synergy between the two tasks. The code and related datasets can be found at: https://github.com/YuhaoShine/DualProjectionFusion.