Beyond Fixed Morphologies: Learning Graph Policies with Trust Region Compensation in Variable Action Spaces

TL;DR

This paper analyzes how trust region policy optimization methods behave when the action space varies due to different robot morphologies, combining theoretical insights with empirical tests on a controlled environment.

Contribution

It provides the first theoretical analysis of trust region methods under variable action spaces and evaluates their performance across different morphologies in a benchmark environment.

Findings

Varying action space impacts the optimization landscape significantly.

Trust region methods exhibit different behaviors depending on action dimensionality.

Empirical results show the effectiveness of graph policies in morphological generalization.

Abstract

Trust region-based optimization methods have become foundational reinforcement learning algorithms that offer stability and strong empirical performance in continuous control tasks. Growing interest in scalable and reusable control policies translate also in a demand for morphological generalization, the ability of control policies to cope with different kinematic structures. Graph-based policy architectures provide a natural and effective mechanism to encode such structural differences. However, while these architectures accommodate variable morphologies, the behavior of trust region methods under varying action space dimensionality remains poorly understood. To this end, we conduct a theoretical analysis of trust region-based policy optimization methods, focusing on both Trust Region Policy Optimization (TRPO) and its widely used first-order approximation, Proximal Policy Optimization (PPO). The goal is to demonstrate how varying action space dimensionality influence the optimization landscape, particularly under the constraints imposed by KL-divergence or policy clipping penalties. Complementing the theoretical insights, an empirical evaluation under morphological variation is carried out using the Gymnasium Swimmer environment. This benchmark offers a systematically controlled setting for varying the kinematic structure without altering the underlying task, making it particularly well-suited to study morphological generalization.