GCNT: Graph-Based Transformer Policies for Morphology-Agnostic Reinforcement Learning

TL;DR

GCNT introduces a graph-based transformer policy network that enables morphology-agnostic reinforcement learning, allowing robots with diverse structures to learn resilient locomotion behaviors with zero-shot generalization.

Contribution

It proposes a novel GCNT architecture combining GCN and Transformer to extract and utilize morphological information for universal robot control.

Findings

Achieved state-of-the-art performance on 8 benchmark tasks.

Demonstrated zero-shot generalization to unseen robot morphologies.

Generated resilient locomotion across diverse robot configurations.

Abstract

Training a universal controller for robots with different morphologies is a promising research trend, since it can significantly enhance the robustness and resilience of the robotic system. However, diverse morphologies can yield different dimensions of state space and action space, making it difficult to comply with traditional policy networks. Existing methods address this issue by modularizing the robot configuration, while do not adequately extract and utilize the overall morphological information, which has been proven crucial for training a universal controller. To this end, we propose GCNT, a morphology-agnostic policy network based on improved Graph Convolutional Network (GCN) and Transformer. It exploits the fact that GCN and Transformer can handle arbitrary number of modules to achieve compatibility with diverse morphologies. Our key insight is that the GCN is able to efficiently extract morphology information of robots, while Transformer ensures that it is fully utilized by allowing each node of the robot to communicate this information directly. Experimental results show that our method can generate resilient locomotion behaviors for robots with different configurations, including zero-shot generalization to robot morphologies not seen during training. In particular, GCNT achieved the best performance on 8 tasks in the 2 standard benchmarks.