Learning Spherical Convolution for Fast Features from 360{\deg} Imagery

TL;DR

This paper introduces a novel spherical convolutional neural network that enables efficient and accurate feature extraction directly from 360-degree images in equirectangular projection, leveraging pre-trained perspective CNNs.

Contribution

It presents a method to adapt flat CNN filters for spherical images, reducing computational cost while maintaining high accuracy, and allowing the use of existing pre-trained networks.

Findings

Achieves higher accuracy than alternative methods.

Reduces computational cost by orders of magnitude.

Enables effective use of pre-trained perspective CNNs on 360-degree data.

Abstract

While 360{\deg} cameras offer tremendous new possibilities in vision, graphics, and augmented reality, the spherical images they produce make core feature extraction non-trivial. Convolutional neural networks (CNNs) trained on images from perspective cameras yield "flat" filters, yet 360{\deg} images cannot be projected to a single plane without significant distortion. A naive solution that repeatedly projects the viewing sphere to all tangent planes is accurate, but much too computationally intensive for real problems. We propose to learn a spherical convolutional network that translates a planar CNN to process 360{\deg} imagery directly in its equirectangular projection. Our approach learns to reproduce the flat filter outputs on 360{\deg} data, sensitive to the varying distortion effects across the viewing sphere. The key benefits are 1) efficient feature extraction for 360{\deg} images and video, and 2) the ability to leverage powerful pre-trained networks researchers have carefully honed (together with massive labeled image training sets) for perspective images. We validate our approach compared to several alternative methods in terms of both raw CNN output accuracy as well as applying a state-of-the-art "flat" object detector to 360{\deg} data. Our method yields the most accurate results while saving orders of magnitude in computation versus the existing exact reprojection solution.