On Non-Linear operators for Geometric Deep Learning

TL;DR

This paper characterizes the form of non-linear operators that commute with manifold symmetries, showing point-wise non-linearities are unique for scalar fields and scalar multiplication for vector fields, impacting geometric deep learning design.

Contribution

It proves the form of non-linear operators commuting with diffeomorphisms for scalar and vector fields, clarifying their role in neural network architectures on manifolds.

Findings

Point-wise non-linearities are the only operators commuting with symmetries for scalar fields.

Scalar multiplication is the only such operator for vector fields.

No universal non-linear operator class exists for vector fields over manifolds.

Abstract

This work studies operators mapping vector and scalar fields defined over a manifold $\mathcal{M}$, and which commute with its group of diffeomorphisms $\text{Diff}(\mathcal{M})$. We prove that in the case of scalar fields $L^p_\omega(\mathcal{M,\mathbb{R}})$, those operators correspond to point-wise non-linearities, recovering and extending known results on $\mathbb{R}^d$. In the context of Neural Networks defined over $\mathcal{M}$, it indicates that point-wise non-linear operators are the only universal family that commutes with any group of symmetries, and justifies their systematic use in combination with dedicated linear operators commuting with specific symmetries. In the case of vector fields $L^p_\omega(\mathcal{M},T\mathcal{M})$, we show that those operators are solely the scalar multiplication. It indicates that $\text{Diff}(\mathcal{M})$ is too rich and that there is no universal class of non-linear operators to motivate the design of Neural Networks over the symmetries of $\mathcal{M}$.