Graph-based Reinforcement Learning meets Mixed Integer Programs: An application to 3D robot assembly discovery

TL;DR

This paper presents a hierarchical method combining mixed-integer programming and graph-based reinforcement learning to efficiently assemble 3D structures with robots, demonstrating improved robustness and real-world transfer.

Contribution

It introduces a novel hierarchical framework integrating mixed-integer programming and graph-based RL for robot assembly tasks, enabling better generalization and structural feasibility considerations.

Findings

Outperforms unstructured end-to-end approaches in simulations

Demonstrates successful real-world transfer of the method

Shows robustness and efficiency in complex assembly tasks

Abstract

Robot assembly discovery is a challenging problem that lives at the intersection of resource allocation and motion planning. The goal is to combine a predefined set of objects to form something new while considering task execution with the robot-in-the-loop. In this work, we tackle the problem of building arbitrary, predefined target structures entirely from scratch using a set of Tetris-like building blocks and a robotic manipulator. Our novel hierarchical approach aims at efficiently decomposing the overall task into three feasible levels that benefit mutually from each other. On the high level, we run a classical mixed-integer program for global optimization of block-type selection and the blocks' final poses to recreate the desired shape. Its output is then exploited to efficiently guide the exploration of an underlying reinforcement learning (RL) policy. This RL policy draws its generalization properties from a flexible graph-based representation that is learned through Q-learning and can be refined with search. Moreover, it accounts for the necessary conditions of structural stability and robotic feasibility that cannot be effectively reflected in the previous layer. Lastly, a grasp and motion planner transforms the desired assembly commands into robot joint movements. We demonstrate our proposed method's performance on a set of competitive simulated RAD environments, showcase real-world transfer, and report performance and robustness gains compared to an unstructured end-to-end approach. Videos are available at https://sites.google.com/view/rl-meets-milp .