Intrinsically Motivated Self-supervised Learning in Reinforcement Learning

TL;DR

This paper introduces IM-SSR, a method that uses self-supervised loss as an intrinsic reward in reinforcement learning, enhancing exploration, robustness, and sample efficiency in vision-based robotics tasks.

Contribution

It proposes a novel way to utilize self-supervised loss as an intrinsic reward, improving reinforcement learning performance without significant additional costs.

Findings

Enhanced sample efficiency in vision-based RL tasks.

Improved generalization in robotics tasks.

Salient performance gains with sparse rewards.

Abstract

In vision-based reinforcement learning (RL) tasks, it is prevalent to assign auxiliary tasks with a surrogate self-supervised loss so as to obtain more semantic representations and improve sample efficiency. However, abundant information in self-supervised auxiliary tasks has been disregarded, since the representation learning part and the decision-making part are separated. To sufficiently utilize information in auxiliary tasks, we present a simple yet effective idea to employ self-supervised loss as an intrinsic reward, called Intrinsically Motivated Self-Supervised learning in Reinforcement learning (IM-SSR). We formally show that the self-supervised loss can be decomposed as exploration for novel states and robustness improvement from nuisance elimination. IM-SSR can be effortlessly plugged into any reinforcement learning with self-supervised auxiliary objectives with nearly no additional cost. Combined with IM-SSR, the previous underlying algorithms achieve salient improvements on both sample efficiency and generalization in various vision-based robotics tasks from the DeepMind Control Suite, especially when the reward signal is sparse.