MORPH: Design Co-optimization with Reinforcement Learning via a Differentiable Hardware Model Proxy

TL;DR

MORPH presents a reinforcement learning-based co-optimization framework that uses a differentiable hardware proxy to efficiently optimize hardware parameters and control policies simultaneously, ensuring realistic hardware behavior.

Contribution

The paper introduces a differentiable hardware proxy for co-optimization with RL, enabling efficient and realistic hardware and control policy optimization in simulation.

Findings

Effective co-optimization of hardware and control policies demonstrated on simulated tasks.

The differentiable proxy maintains closeness to real hardware while enabling optimization.

Successful application to 2D reaching and 3D manipulation tasks.

Abstract

We introduce MORPH, a method for co-optimization of hardware design parameters and control policies in simulation using reinforcement learning. Like most co-optimization methods, MORPH relies on a model of the hardware being optimized, usually simulated based on the laws of physics. However, such a model is often difficult to integrate into an effective optimization routine. To address this, we introduce a proxy hardware model, which is always differentiable and enables efficient co-optimization alongside a long-horizon control policy using RL. MORPH is designed to ensure that the optimized hardware proxy remains as close as possible to its realistic counterpart, while still enabling task completion. We demonstrate our approach on simulated 2D reaching and 3D multi-fingered manipulation tasks.