MiniVLN: Efficient Vision-and-Language Navigation by Progressive Knowledge Distillation

TL;DR

This paper presents MiniVLN, a lightweight vision-and-language navigation model created through a two-stage knowledge distillation process, achieving comparable performance to larger models with significantly fewer parameters.

Contribution

The paper introduces a novel two-stage knowledge distillation framework for VLN, effectively reducing model size while maintaining high performance.

Findings

Two-stage distillation outperforms single-stage in narrowing performance gap.

MiniVLN achieves similar results to the teacher model with only 12% of parameters.

The approach demonstrates effective deployment of lightweight models in Embodied AI.

Abstract

In recent years, Embodied Artificial Intelligence (Embodied AI) has advanced rapidly, yet the increasing size of models conflicts with the limited computational capabilities of Embodied AI platforms. To address this challenge, we aim to achieve both high model performance and practical deployability. Specifically, we focus on Vision-and-Language Navigation (VLN), a core task in Embodied AI. This paper introduces a two-stage knowledge distillation framework, producing a student model, MiniVLN, and showcasing the significant potential of distillation techniques in developing lightweight models. The proposed method aims to capture fine-grained knowledge during the pretraining phase and navigation-specific knowledge during the fine-tuning phase. Our findings indicate that the two-stage distillation approach is more effective in narrowing the performance gap between the teacher model and the student model compared to single-stage distillation. On the public R2R and REVERIE benchmarks, MiniVLN achieves performance on par with the teacher model while having only about 12% of the teacher model's parameter count.