Subequivariant Reinforcement Learning in 3D Multi-Entity Physical Environments

TL;DR

This paper introduces Subequivariant Hierarchical Neural Networks (SHNN) for efficient multi-entity policy learning in 3D environments, utilizing local views and subequivariant message passing to handle complex symmetries and reduce redundancy.

Contribution

The paper proposes SHNN, a novel neural network architecture that dynamically decouples global space into local graphs and employs subequivariant message passing, advancing multi-entity reinforcement learning.

Findings

SHNN outperforms existing methods on the MEBEN benchmarks.

Task assignment and subequivariance are crucial for performance.

The approach effectively handles Euclidean symmetries in multi-entity systems.

Abstract

Learning policies for multi-entity systems in 3D environments is far more complicated against single-entity scenarios, due to the exponential expansion of the global state space as the number of entities increases. One potential solution of alleviating the exponential complexity is dividing the global space into independent local views that are invariant to transformations including translations and rotations. To this end, this paper proposes Subequivariant Hierarchical Neural Networks (SHNN) to facilitate multi-entity policy learning. In particular, SHNN first dynamically decouples the global space into local entity-level graphs via task assignment. Second, it leverages subequivariant message passing over the local entity-level graphs to devise local reference frames, remarkably compressing the representation redundancy, particularly in gravity-affected environments. Furthermore, to overcome the limitations of existing benchmarks in capturing the subtleties of multi-entity systems under the Euclidean symmetry, we propose the Multi-entity Benchmark (MEBEN), a new suite of environments tailored for exploring a wide range of multi-entity reinforcement learning. Extensive experiments demonstrate significant advancements of SHNN on the proposed benchmarks compared to existing methods. Comprehensive ablations are conducted to verify the indispensability of task assignment and subequivariance.