Weakly-supervised VLM-guided Partial Contrastive Learning for Visual Language Navigation

TL;DR

This paper introduces WPCL, a weakly-supervised learning approach that leverages pre-trained VLMs for improved visual perception in VLN tasks without fine-tuning, resulting in better performance and efficiency.

Contribution

The paper proposes a novel weakly-supervised partial contrastive learning method that integrates pre-trained VLMs into VLN agents without fine-tuning, addressing dynamic viewpoints and computational costs.

Findings

Outperforms baseline methods on multiple VLN benchmarks.

Enhances object identification from dynamic viewpoints.

Maintains computational efficiency without VLM fine-tuning.

Abstract

Visual Language Navigation (VLN) is a fundamental task within the field of Embodied AI, focusing on the ability of agents to navigate complex environments based on natural language instructions. Despite the progress made by existing methods, these methods often present some common challenges. First, they rely on pre-trained backbone models for visual perception, which struggle with the dynamic viewpoints in VLN scenarios. Second, the performance is limited when using pre-trained LLMs or VLMs without fine-tuning, due to the absence of VLN domain knowledge. Third, while fine-tuning LLMs and VLMs can improve results, their computational costs are higher than those without fine-tuning. To address these limitations, we propose Weakly-supervised Partial Contrastive Learning (WPCL), a method that enhances an agent's ability to identify objects from dynamic viewpoints in VLN scenarios by effectively integrating pre-trained VLM knowledge into the perception process, without requiring VLM fine-tuning. Our method enhances the agent's ability to interpret and respond to environmental cues while ensuring computational efficiency. Experimental results have shown that our method outperforms the baseline methods on multiple benchmarks, which validate the effectiveness, robustness and generalizability of our method.