Unified Spherical Frontend: Learning Rotation-Equivariant Representations of Spherical Images from Any Camera

TL;DR

The paper introduces USF, a flexible, rotation-equivariant spherical image processing framework that works across various camera types without harmonic transforms, improving robustness and generalization in perception tasks.

Contribution

USF provides a modular, distortion-free, lens-agnostic approach for spherical image processing that achieves rotation-equivariance without harmonic transforms, enabling efficient high-resolution analysis.

Findings

USF maintains less than 1% performance drop under random rotations.

USF scales efficiently to high-resolution spherical imagery.

USF enables zero-shot generalization to unseen wide-FoV lenses.

Abstract

Modern perception increasingly relies on fisheye, panoramic, and other wide field-of-view (FoV) cameras, yet most pipelines still apply planar CNNs designed for pinhole imagery on 2D grids, where pixel-space neighborhoods misrepresent physical adjacency and models are sensitive to global rotations. Traditional spherical CNNs partially address this mismatch but require costly spherical harmonic transform that constrains resolution and efficiency. We present Unified Spherical Frontend (USF), a distortion-free lens-agnostic framework that transforms images from any calibrated camera onto the unit sphere via ray-direction correspondences, and performs spherical resampling, convolution, and pooling canonically in the spatial domain. USF is modular: projection, location sampling, value interpolation, and resolution control are fully decoupled. Its configurable distance-only convolution kernels offer rotation-equivariance, mirroring translation-equivariance in planar CNNs while avoiding harmonic transforms entirely. We compare multiple standard planar backbones with their spherical counterparts across classification, detection, and segmentation tasks on synthetic (Spherical MNIST) and real-world (PANDORA, Stanford 2D-3D-S) datasets, and stress-test robustness to extreme lens distortions, varying FoV, and arbitrary rotations. USF scales efficiently to high-resolution spherical imagery and maintains less than 1% performance drop under random test-time rotations without training-time rotational augmentation, and enables zero-shot generalization to any unseen (wide-FoV) lenses with minimal performance degradation.