Moment-based Adversarial Training for Embodied Language Comprehension

TL;DR

This paper introduces Moment-based Adversarial Training (MAT), a novel method that improves embodied language comprehension in robots by enhancing instruction and subgoal understanding, leading to better performance on the ALFRED benchmark.

Contribution

We propose a new adversarial training approach using moment-based perturbations to improve robot understanding of complex instructions in vision-and-language tasks.

Findings

MAT outperforms baseline methods on ALFRED benchmark

Enhanced subgoal inference in robot instruction execution

Improved accuracy across all evaluation metrics

Abstract

In this paper, we focus on a vision-and-language task in which a robot is instructed to execute household tasks. Given an instruction such as "Rinse off a mug and place it in the coffee maker," the robot is required to locate the mug, wash it, and put it in the coffee maker. This is challenging because the robot needs to break down the instruction sentences into subgoals and execute them in the correct order. On the ALFRED benchmark, the performance of state-of-the-art methods is still far lower than that of humans. This is partially because existing methods sometimes fail to infer subgoals that are not explicitly specified in the instruction sentences. We propose Moment-based Adversarial Training (MAT), which uses two types of moments for perturbation updates in adversarial training. We introduce MAT to the embedding spaces of the instruction, subgoals, and state representations to handle their varieties. We validated our method on the ALFRED benchmark, and the results demonstrated that our method outperformed the baseline method for all the metrics on the benchmark.