Learning Design and Construction with Varying-Sized Materials via Prioritized Memory Resets

TL;DR

This paper introduces a hierarchical reinforcement learning approach with prioritized memory resetting to enable robots to autonomously design and build bridges using blocks of varying sizes, improving exploration and success rates.

Contribution

It proposes a novel prioritized memory resetting technique for high-level reinforcement learning in complex construction tasks with diverse materials.

Findings

Effective bridge construction in simulation and real robot

High success rate in constructing structures with varying-sized blocks

Enhanced exploration efficiency through PMR

Abstract

Can a robot autonomously learn to design and construct a bridge from varying-sized blocks without a blueprint? It is a challenging task with long horizon and sparse reward -- the robot has to figure out physically stable design schemes and feasible actions to manipulate and transport blocks. Due to diverse block sizes, the state space and action trajectories are vast to explore. In this paper, we propose a hierarchical approach for this problem. It consists of a reinforcement-learning designer to propose high-level building instructions and a motion-planning-based action generator to manipulate blocks at the low level. For high-level learning, we develop a novel technique, prioritized memory resetting (PMR) to improve exploration. PMR adaptively resets the state to those most critical configurations from a replay buffer so that the robot can resume training on partial architectures instead of from scratch. Furthermore, we augment PMR with auxiliary training objectives and fine-tune the designer with the locomotion generator. Our experiments in simulation and on a real deployed robotic system demonstrate that it is able to effectively construct bridges with blocks of varying sizes at a high success rate. Demos can be found at https://sites.google.com/view/bridge-pmr.