Vision-Language Navigation with Energy-Based Policy

TL;DR

This paper introduces an energy-based policy for vision-language navigation that models the joint state-action distribution to better imitate expert behavior and reduce errors, improving performance across multiple benchmarks.

Contribution

The paper proposes an energy-based navigation policy that aligns with expert behavior by modeling joint state-action distributions, offering a novel approach to VLN.

Findings

Achieves promising results on R2R, REVERIE, RxR, and R2R-CE datasets.

Effectively models the joint distribution of states and actions.

Improves upon existing VLN models by reducing error accumulation.

Abstract

Vision-language navigation (VLN) requires an agent to execute actions following human instructions. Existing VLN models are optimized through expert demonstrations by supervised behavioural cloning or incorporating manual reward engineering. While straightforward, these efforts overlook the accumulation of errors in the Markov decision process, and struggle to match the distribution of the expert policy. Going beyond this, we propose an Energy-based Navigation Policy (ENP) to model the joint state-action distribution using an energy-based model. At each step, low energy values correspond to the state-action pairs that the expert is most likely to perform, and vice versa. Theoretically, the optimization objective is equivalent to minimizing the forward divergence between the occupancy measure of the expert and ours. Consequently, ENP learns to globally align with the expert policy by maximizing the likelihood of the actions and modeling the dynamics of the navigation states in a collaborative manner. With a variety of VLN architectures, ENP achieves promising performances on R2R, REVERIE, RxR, and R2R-CE, unleashing the power of existing VLN models.