To Ask or Not to Ask? Detecting Absence of Information in Vision and Language Navigation

TL;DR

This paper introduces an attention-based module for detecting when a vision-language navigation agent lacks sufficient information, improving decision-making and efficiency in handling vague instructions.

Contribution

It proposes a novel instruction-vagueness estimation module that leverages instruction-to-path alignment to identify uncertain points during navigation.

Findings

Vagueness estimation precision-recall improved by 52%.

Instruction-to-path attention scores outperform cross-modal attention for vagueness detection.

Incorporating the module enhances agent efficiency in VLN tasks.

Abstract

Recent research in Vision Language Navigation (VLN) has overlooked the development of agents' inquisitive abilities, which allow them to ask clarifying questions when instructions are incomplete. This paper addresses how agents can recognize "when" they lack sufficient information, without focusing on "what" is missing, particularly in VLN tasks with vague instructions. Equipping agents with this ability enhances efficiency by reducing potential digressions and seeking timely assistance. The challenge in identifying such uncertain points is balancing between being overly cautious (high recall) and overly confident (high precision). We propose an attention-based instruction-vagueness estimation module that learns associations between instructions and the agent's trajectory. By leveraging instruction-to-path alignment information during training, the module's vagueness estimation performance improves by around 52% in terms of precision-recall balance. In our ablative experiments, we also demonstrate the effectiveness of incorporating this additional instruction-to-path attention network alongside the cross-modal attention networks within the navigator module. Our results show that the attention scores from the instruction-to-path attention network serve as better indicators for estimating vagueness.