A Multimodal Classifier Generative Adversarial Network for Carry and Place Tasks from Ambiguous Language Instructions

TL;DR

This paper introduces MMC-GAN, a multimodal GAN-based model that improves disambiguation of target areas in carry-and-place tasks for domestic robots by integrating environment and robot state information, reducing interaction time.

Contribution

The paper presents a novel MMC-GAN model that leverages multimodal data for disambiguating ambiguous instructions in robot tasks, outperforming existing methods.

Findings

MMC-GAN significantly improves disambiguation accuracy.

Multimodal approach reduces need for dialogue-based clarification.

Enhanced robot task efficiency in domestic environments.

Abstract

This paper focuses on a multimodal language understanding method for carry-and-place tasks with domestic service robots. We address the case of ambiguous instructions, that is, when the target area is not specified. For instance "put away the milk and cereal" is a natural instruction where there is ambiguity regarding the target area, considering environments in daily life. Conventionally, this instruction can be disambiguated from a dialogue system, but at the cost of time and cumbersome interaction. Instead, we propose a multimodal approach, in which the instructions are disambiguated using the robot's state and environment context. We develop the Multi-Modal Classifier Generative Adversarial Network (MMC-GAN) to predict the likelihood of different target areas considering the robot's physical limitation and the target clutter. Our approach, MMC-GAN, significantly improves accuracy compared with baseline methods that use instructions only or simple deep neural networks.