One After Another: Learning Incremental Skills for a Changing World

TL;DR

This paper introduces an incremental skill learning framework that enables agents to adapt to changing environments without forgetting previous skills, outperforming existing methods in static and evolving settings.

Contribution

The authors propose a novel incremental skill discovery approach that sequentially learns skills, balancing adaptation to new environments with retention of prior skills.

Findings

Incremental skills outperform state-of-the-art methods in static environments.

The approach enables fast adaptation to environment changes.

Agents retain previously learned skills effectively.

Abstract

Reward-free, unsupervised discovery of skills is an attractive alternative to the bottleneck of hand-designing rewards in environments where task supervision is scarce or expensive. However, current skill pre-training methods, like many RL techniques, make a fundamental assumption - stationary environments during training. Traditional methods learn all their skills simultaneously, which makes it difficult for them to both quickly adapt to changes in the environment, and to not forget earlier skills after such adaptation. On the other hand, in an evolving or expanding environment, skill learning must be able to adapt fast to new environment situations while not forgetting previously learned skills. These two conditions make it difficult for classic skill discovery to do well in an evolving environment. In this work, we propose a new framework for skill discovery, where skills are learned one after another in an incremental fashion. This framework allows newly learned skills to adapt to new environment or agent dynamics, while the fixed old skills ensure the agent doesn't forget a learned skill. We demonstrate experimentally that in both evolving and static environments, incremental skills significantly outperform current state-of-the-art skill discovery methods on both skill quality and the ability to solve downstream tasks. Videos for learned skills and code are made public on https://notmahi.github.io/disk