Grounded Language Understanding for Manipulation Instructions Using GAN-Based Classification

TL;DR

This paper introduces a GAN-based grounded language understanding method for domestic service robots to interpret incomplete manipulation instructions and determine physically feasible actions, improving robot communication and task execution.

Contribution

It presents a novel GAN-based classifier that estimates appropriate objects from instructions and situations, advancing grounded language understanding for robot manipulation.

Findings

The proposed method outperforms baseline methods in accuracy.

A new dataset was developed for evaluation.

Experimental results demonstrate improved understanding of incomplete instructions.

Abstract

The target task of this study is grounded language understanding for domestic service robots (DSRs). In particular, we focus on instruction understanding for short sentences where verbs are missing. This task is of critical importance to build communicative DSRs because manipulation is essential for DSRs. Existing instruction understanding methods usually estimate missing information only from non-grounded knowledge; therefore, whether the predicted action is physically executable or not was unclear. In this paper, we present a grounded instruction understanding method to estimate appropriate objects given an instruction and situation. We extend the Generative Adversarial Nets (GAN) and build a GAN-based classifier using latent representations. To quantitatively evaluate the proposed method, we have developed a data set based on the standard data set used for Visual QA. Experimental results have shown that the proposed method gives the better result than baseline methods.