Periodic Skill Discovery

TL;DR

The paper introduces Periodic Skill Discovery (PSD), a novel unsupervised framework that learns diverse periodic behaviors in robotic tasks by encoding states into a circular latent space, improving skill diversity and downstream task performance.

Contribution

PSD is the first method to explicitly discover periodic skills by mapping states to a circular latent space, capturing temporal distances and enabling diverse periodic behaviors in complex tasks.

Findings

PSD effectively learns skills with diverse periods in robotic tasks.

Skills learned with PSD perform well on downstream tasks like hurdling.

Integrating PSD with existing methods enhances behavior diversity.

Abstract

Unsupervised skill discovery in reinforcement learning (RL) aims to learn diverse behaviors without relying on external rewards. However, current methods often overlook the periodic nature of learned skills, focusing instead on increasing the mutual dependence between states and skills or maximizing the distance traveled in latent space. Considering that many robotic tasks - particularly those involving locomotion - require periodic behaviors across varying timescales, the ability to discover diverse periodic skills is essential. Motivated by this, we propose Periodic Skill Discovery (PSD), a framework that discovers periodic behaviors in an unsupervised manner. The key idea of PSD is to train an encoder that maps states to a circular latent space, thereby naturally encoding periodicity in the latent representation. By capturing temporal distance, PSD can effectively learn skills with diverse periods in complex robotic tasks, even with pixel-based observations. We further show that these learned skills achieve high performance on downstream tasks such as hurdling. Moreover, integrating PSD with an existing skill discovery method offers more diverse behaviors, thus broadening the agent's repertoire. Our code and demos are available at https://jonghaepark.github.io/psd/