VLM-Vac: Enhancing Smart Vacuums through VLM Knowledge Distillation and Language-Guided Experience Replay

TL;DR

This paper introduces VLM-Vac, a framework that combines vision-language models with knowledge distillation and language-guided experience replay to improve smart vacuum cleaners' object detection and adaptability in dynamic environments.

Contribution

The paper presents a novel integration of VLMs with knowledge distillation and language-guided experience replay for enhanced robot vacuum autonomy.

Findings

Smaller models learn effectively from VLMs with fewer queries.

The approach outperforms traditional vision clustering in small object detection.

Energy efficiency is improved through knowledge distillation and experience replay.

Abstract

In this paper, we propose VLM-Vac, a novel framework designed to enhance the autonomy of smart robot vacuum cleaners. Our approach integrates the zero-shot object detection capabilities of a Vision-Language Model (VLM) with a Knowledge Distillation (KD) strategy. By leveraging the VLM, the robot can categorize objects into actionable classes -- either to avoid or to suck -- across diverse backgrounds. However, frequently querying the VLM is computationally expensive and impractical for real-world deployment. To address this issue, we implement a KD process that gradually transfers the essential knowledge of the VLM to a smaller, more efficient model. Our real-world experiments demonstrate that this smaller model progressively learns from the VLM and requires significantly fewer queries over time. Additionally, we tackle the challenge of continual learning in dynamic home environments by exploiting a novel experience replay method based on language-guided sampling. Our results show that this approach is not only energy-efficient but also surpasses conventional vision-based clustering methods, particularly in detecting small objects across diverse backgrounds.