Navigating the Nuances: A Fine-grained Evaluation of Vision-Language Navigation

TL;DR

This paper introduces a detailed evaluation framework for Vision-Language Navigation, using a CFG-based approach and LLMs to analyze model performance across instruction categories, revealing key challenges and biases.

Contribution

It proposes a novel CFG-based evaluation framework and a semi-automatic method using LLMs to generate instruction data for fine-grained analysis of VLN models.

Findings

Models show performance gaps across instruction categories.

Numerical comprehension remains a significant challenge.

Directional biases are prevalent in current models.

Abstract

This study presents a novel evaluation framework for the Vision-Language Navigation (VLN) task. It aims to diagnose current models for various instruction categories at a finer-grained level. The framework is structured around the context-free grammar (CFG) of the task. The CFG serves as the basis for the problem decomposition and the core premise of the instruction categories design. We propose a semi-automatic method for CFG construction with the help of Large-Language Models (LLMs). Then, we induct and generate data spanning five principal instruction categories (i.e. direction change, landmark recognition, region recognition, vertical movement, and numerical comprehension). Our analysis of different models reveals notable performance discrepancies and recurrent issues. The stagnation of numerical comprehension, heavy selective biases over directional concepts, and other interesting findings contribute to the development of future language-guided navigation systems.