Mixline: A Hybrid Reinforcement Learning Framework for Long-horizon Bimanual Coffee Stirring Task

TL;DR

This paper introduces Mixline, a hybrid reinforcement learning framework that enables bimanual robots to learn complex, long-horizon tasks like coffee stirring by decomposing tasks and combining learned sub-tasks.

Contribution

The paper presents a novel hybrid RL approach combining online and offline algorithms to address coordination and task decomposition in long-horizon bimanual tasks.

Findings

Successfully learned to grasp, hold, lift, insert, and stir using the framework.

Demonstrated effective task decomposition and composition in a simulated environment.

Potential for extension to other complex bimanual tasks.

Abstract

Bimanual activities like coffee stirring, which require coordination of dual arms, are common in daily life and intractable to learn by robots. Adopting reinforcement learning to learn these tasks is a promising topic since it enables the robot to explore how dual arms coordinate together to accomplish the same task. However, this field has two main challenges: coordination mechanism and long-horizon task decomposition. Therefore, we propose the Mixline method to learn sub-tasks separately via the online algorithm and then compose them together based on the generated data through the offline algorithm. We constructed a learning environment based on the GPU-accelerated Isaac Gym. In our work, the bimanual robot successfully learned to grasp, hold and lift the spoon and cup, insert them together and stir the coffee. The proposed method has the potential to be extended to other long-horizon bimanual tasks.