Hierarchical Learning in Euclidean Neural Networks

TL;DR

This paper investigates why Euclidean Neural Networks (e3nn) outperform other models in 3D applications by analyzing the role and hierarchy of non-scalar features, especially in electron density prediction.

Contribution

It reveals the importance of higher order features and their hierarchical organization in e3nn, providing insights for designing more effective equivariant models.

Findings

Higher order features improve model accuracy.

Features organize hierarchically by order, similar to multipole expansion.

The nature of the output influences the effectiveness of non-scalar features.

Abstract

Equivariant machine learning methods have shown wide success at 3D learning applications in recent years. These models explicitly build in the reflection, translation and rotation symmetries of Euclidean space and have facilitated large advances in accuracy and data efficiency for a range of applications in the physical sciences. An outstanding question for equivariant models is why they achieve such larger-than-expected advances in these applications. To probe this question, we examine the role of higher order (non-scalar) features in Euclidean Neural Networks (\texttt{e3nn}). We focus on the previously studied application of \texttt{e3nn} to the problem of electron density prediction, which allows for a variety of non-scalar outputs, and examine whether the nature of the output (scalar $l=0$, vector $l=1$, or higher order $l>1$) is relevant to the effectiveness of non-scalar hidden features in the network. Further, we examine the behavior of non-scalar features throughout training, finding a natural hierarchy of features by $l$, reminiscent of a multipole expansion. We aim for our work to ultimately inform design principles and choices of domain applications for {\tt e3nn} networks.