Improving the Robustness to Variations of Objects and Instructions with a Neuro-Symbolic Approach for Interactive Instruction Following

TL;DR

This paper introduces a neuro-symbolic method for interactive instruction following that enhances robustness to object and instruction variations, significantly improving success rates in unseen environments.

Contribution

The paper presents a novel neuro-symbolic approach using high-level symbolic features to improve robustness in interactive instruction following tasks.

Findings

Outperforms end-to-end neural models by 9-74 points in success rate.

Significant improvements in unseen environment tasks.

Effective mitigation of sensitivity to small input changes.

Abstract

An interactive instruction following task has been proposed as a benchmark for learning to map natural language instructions and first-person vision into sequences of actions to interact with objects in 3D environments. We found that an existing end-to-end neural model for this task tends to fail to interact with objects of unseen attributes and follow various instructions. We assume that this problem is caused by the high sensitivity of neural feature extraction to small changes in vision and language inputs. To mitigate this problem, we propose a neuro-symbolic approach that utilizes high-level symbolic features, which are robust to small changes in raw inputs, as intermediate representations. We verify the effectiveness of our model with the subtask evaluation on the ALFRED benchmark. Our experiments show that our approach significantly outperforms the end-to-end neural model by 9, 46, and 74 points in the success rate on the ToggleObject, PickupObject, and SliceObject subtasks in unseen environments respectively.