Leveraging Jumpy Models for Planning and Fast Learning in Robotic Domains

TL;DR

This paper introduces jumpy models that learn multi-step dynamics and skill embeddings from unlabeled data, enabling efficient planning and fast reinforcement learning in robotic tasks with long horizons.

Contribution

It proposes a method to learn jumpy models and skill embeddings offline, enhancing planning and learning speed in robotic domains without requiring labeled data.

Findings

Jumpy models enable zero-shot generalization to new tasks.

Planning with learned skills accelerates reinforcement learning.

Jumpy models outperform standard dynamics models in long-horizon tasks.

Abstract

In this paper we study the problem of learning multi-step dynamics prediction models (jumpy models) from unlabeled experience and their utility for fast inference of (high-level) plans in downstream tasks. In particular we propose to learn a jumpy model alongside a skill embedding space offline, from previously collected experience for which no labels or reward annotations are required. We then investigate several options of harnessing those learned components in combination with model-based planning or model-free reinforcement learning (RL) to speed up learning on downstream tasks. We conduct a set of experiments in the RGB-stacking environment, showing that planning with the learned skills and the associated model can enable zero-shot generalization to new tasks, and can further speed up training of policies via reinforcement learning. These experiments demonstrate that jumpy models which incorporate temporal abstraction can facilitate planning in long-horizon tasks in which standard dynamics models fail.