SIME: Enhancing Policy Self-Improvement with Modal-level Exploration

TL;DR

This paper introduces SIME, a method that improves robotic self-learning by promoting diverse interactions through modal-level exploration and selecting valuable data for training, leading to more robust control strategies.

Contribution

The paper proposes a novel modal-level exploration mechanism combined with data selection to enhance robot self-improvement during interaction-based learning.

Findings

Effective self-improvement demonstrated on simulation benchmarks.

Successful real-world robot experiments show increased robustness.

Method reduces training costs while improving control strategy quality.

Abstract

Self-improvement requires robotic systems to initially learn from human-provided data and then gradually enhance their capabilities through interaction with the environment. This is similar to how humans improve their skills through continuous practice. However, achieving effective self-improvement is challenging, primarily because robots tend to repeat their existing abilities during interactions, often failing to generate new, valuable data for learning. In this paper, we identify the key to successful self-improvement: modal-level exploration and data selection. By incorporating a modal-level exploration mechanism during policy execution, the robot can produce more diverse and multi-modal interactions. At the same time, we select the most valuable trials and high-quality segments from these interactions for learning. We successfully demonstrate effective robot self-improvement on both simulation benchmarks and real-world experiments. The capability for self-improvement will enable us to develop more robust and high-success-rate robotic control strategies at a lower cost. Our code and experiment scripts are available at https://ericjin2002.github.io/SIME/