Soft Expert Reward Learning for Vision-and-Language Navigation

TL;DR

This paper introduces a Soft Expert Reward Learning model for Vision-and-Language Navigation that improves generalisation and reduces reward engineering by combining soft expert distillation with a self-perception module, outperforming state-of-the-art methods.

Contribution

The paper proposes a novel Soft Expert Reward Learning framework with two modules, enhancing VLN performance and generalisation without manual reward engineering.

Findings

Outperforms state-of-the-art on most VLN metrics

Improves generalisation to unseen environments

Reduces reliance on manual reward design

Abstract

Vision-and-Language Navigation (VLN) requires an agent to find a specified spot in an unseen environment by following natural language instructions. Dominant methods based on supervised learning clone expert's behaviours and thus perform better on seen environments, while showing restricted performance on unseen ones. Reinforcement Learning (RL) based models show better generalisation ability but have issues as well, requiring large amount of manual reward engineering is one of which. In this paper, we introduce a Soft Expert Reward Learning (SERL) model to overcome the reward engineering designing and generalisation problems of the VLN task. Our proposed method consists of two complementary components: Soft Expert Distillation (SED) module encourages agents to behave like an expert as much as possible, but in a soft fashion; Self Perceiving (SP) module targets at pushing the agent towards the final destination as fast as possible. Empirically, we evaluate our model on the VLN seen, unseen and test splits and the model outperforms the state-of-the-art methods on most of the evaluation metrics.