iLearnRobot: An Interactive Learning-Based Multi-Modal Robot with Continuous Improvement

TL;DR

This paper introduces iLearnRobot, a multi-modal robot system that learns interactively from natural dialogues with users, using chain questioning and dual-modality retrieval to adapt and improve performance in diverse environments.

Contribution

The paper presents a novel interactive learning framework for robots using MLLMs, enabling continuous improvement through natural dialogue and multi-modal data integration.

Findings

Demonstrates effective learning from user interactions.

Shows improved robot performance over time.

Validates approach through quantitative and qualitative experiments.

Abstract

It is crucial that robots' performance can be improved after deployment, as they are inherently likely to encounter novel scenarios never seen before. This paper presents an innovative solution: an interactive learning-based robot system powered by a Multi-modal Large Language Model(MLLM). A key feature of our system is its ability to learn from natural dialogues with non-expert users. We also propose chain of question to clarify the exact intent of the question before providing an answer and dual-modality retrieval modules to leverage these interaction events to avoid repeating same mistakes, ensuring a seamless user experience before model updates, which is in contrast to current mainstream MLLM-based robotic systems. Our system marks a novel approach in robotics by integrating interactive learning, paving the way for superior adaptability and performance in diverse environments. We demonstrate the effectiveness and improvement of our method through experiments, both quantitively and qualitatively.