C^2:Co-design of Robots via Concurrent Networks Coupling Online and Offline Reinforcement Learning

TL;DR

This paper introduces a concurrent network framework combining online and offline reinforcement learning for robot co-design, significantly improving training efficiency and performance over traditional dual-network methods, validated through simulations and real-world experiments.

Contribution

It proposes a novel concurrent network framework that effectively integrates online and offline RL for robot co-design, enhancing training speed and performance.

Findings

Improved algorithmic performance in simulations.

Effective real-robot application validation.

Addresses limitations of previous dual-network frameworks.

Abstract

With the increasing computing power, using data-driven approaches to co-design a robot's morphology and controller has become a promising way. However, most existing data-driven methods require training the controller for each morphology to calculate fitness, which is time-consuming. In contrast, the dual-network framework utilizes data collected by individual networks under a specific morphology to train a population network that provides a surrogate function for morphology optimization. This approach replaces the traditional evaluation of a diverse set of candidates, thereby speeding up the training. Despite considerable results, the online training of both networks impedes their performance. To address this issue, we propose a concurrent network framework that combines online and offline reinforcement learning (RL) methods. By leveraging the behavior cloning term in a flexible manner, we achieve an effective combination of both networks. We conducted multiple sets of comparative experiments in the simulator and found that the proposed method effectively addresses issues present in the dual-network framework, leading to overall algorithmic performance improvement. Furthermore, we validated the algorithm on a real robot, demonstrating its feasibility in a practical application.