Learning Multimodal Attention for Manipulating Deformable Objects with Changing States

TL;DR

This paper introduces a multimodal attention-based neural network enabling robots to perceive and manipulate deformable objects with changing states, demonstrated through a cooking task involving eggs.

Contribution

The authors develop a predictive recurrent neural network with an attention mechanism for real-time perception and motion generation in dynamic, deformable object manipulation tasks.

Findings

Robot successfully cooked eggs with unknown ingredients.

Model adapted stirring method based on egg state.

Achieved human-like manipulation skills.

Abstract

To support humans in their daily lives, robots are required to autonomously learn, adapt to objects and environments, and perform the appropriate actions. We tackled on the task of cooking scrambled eggs using real ingredients, in which the robot needs to perceive the states of the egg and adjust stirring movement in real time, while the egg is heated and the state changes continuously. In previous works, handling changing objects was found to be challenging because sensory information includes dynamical, both important or noisy information, and the modality which should be focused on changes every time, making it difficult to realize both perception and motion generation in real time. We propose a predictive recurrent neural network with an attention mechanism that can weigh the sensor input, distinguishing how important and reliable each modality is, that realize quick and efficient perception and motion generation. The model is trained with learning from the demonstration, and allows the robot to acquire human-like skills. We validated the proposed technique using the robot, Dry-AIREC, and with our learning model, it could perform cooking eggs with unknown ingredients. The robot could change the method of stirring and direction depending on the status of the egg, as in the beginning it stirs in the whole pot, then subsequently, after the egg started being heated, it starts flipping and splitting motion targeting specific areas, although we did not explicitly indicate them.