Self-Improving Loops for Visual Robotic Planning

TL;DR

This paper introduces SILVR, a self-improving loop for visual robotic planning that iteratively enhances a video model's performance on robotic tasks through self-collected data, improving generalization and efficiency.

Contribution

The paper presents SILVR, a novel online self-improvement framework for visual robotic planning that updates models using self-produced trajectories without requiring ground-truth rewards or expert data.

Findings

Performance improves over iterations on unseen tasks.

Robustness without ground-truth rewards or expert demonstrations.

Outperforms alternative online experience methods.

Abstract

Video generative models trained on expert demonstrations have been utilized as performant text-conditioned visual planners for solving robotic tasks. However, generalization to unseen tasks remains a challenge. Whereas improved generalization may be facilitated by leveraging learned prior knowledge from additional pre-collected offline data sources, such as web-scale video datasets, in the era of experience we aim to design agents that can continuously improve in an online manner from self-collected behaviors. In this work we thus propose the Self-Improving Loops for Visual Robotic Planning (SILVR), where an in-domain video model iteratively updates itself on self-produced trajectories, and steadily improves its performance for a specified task of interest. We apply SILVR to a diverse suite of MetaWorld tasks, as well as two manipulation tasks on a real robot arm, and find that performance improvements continuously emerge over multiple iterations for novel tasks unseen during initial in-domain video model training. We demonstrate that SILVR is robust in the absence of human-provided ground-truth reward functions or expert-quality demonstrations, and is preferable to alternate approaches that utilize online experience in terms of performance and sample efficiency.