Projective Manifold Gradient Layer for Deep Rotation Regression

TL;DR

This paper introduces a novel manifold-aware gradient layer for deep neural networks to improve rotation regression on the SO(3) manifold, achieving state-of-the-art results by directly backpropagating on the manifold.

Contribution

It proposes a regularized projective manifold gradient (RPMG) that enhances gradient backpropagation for rotation regression on SO(3) using Riemannian optimization.

Findings

Achieves state-of-the-art performance in rotation estimation tasks.

The gradient layer can be applied to other smooth manifolds like the unit sphere.

Improves backpropagation efficiency on non-Euclidean manifolds.

Abstract

Regressing rotations on SO(3) manifold using deep neural networks is an important yet unsolved problem. The gap between the Euclidean network output space and the non-Euclidean SO(3) manifold imposes a severe challenge for neural network learning in both forward and backward passes. While several works have proposed different regression-friendly rotation representations, very few works have been devoted to improving the gradient backpropagating in the backward pass. In this paper, we propose a manifold-aware gradient that directly backpropagates into deep network weights. Leveraging Riemannian optimization to construct a novel projective gradient, our proposed regularized projective manifold gradient (RPMG) method helps networks achieve new state-of-the-art performance in a variety of rotation estimation tasks. Our proposed gradient layer can also be applied to other smooth manifolds such as the unit sphere. Our project page is at https://jychen18.github.io/RPMG.