Learning Future Representation with Synthetic Observations for Sample-efficient Reinforcement Learning

TL;DR

This paper introduces LFS, a novel self-supervised reinforcement learning method that synthesizes future observations to improve sample efficiency and visual representation learning without relying on rewards or actions.

Contribution

LFS proposes a training-free synthetic observation generation and data selection approach to enhance auxiliary data for RL, enabling better future state understanding and wider application scope.

Findings

LFS achieves state-of-the-art sample efficiency in continuous control tasks.

LFS improves visual pre-training from action-free video demonstrations.

Synthetic observations help the agent anticipate future states effectively.

Abstract

In visual Reinforcement Learning (RL), upstream representation learning largely determines the effect of downstream policy learning. Employing auxiliary tasks allows the agent to enhance visual representation in a targeted manner, thereby improving the sample efficiency and performance of downstream RL. Prior advanced auxiliary tasks all focus on how to extract as much information as possible from limited experience (including observations, actions, and rewards) through their different auxiliary objectives, whereas in this article, we first start from another perspective: auxiliary training data. We try to improve auxiliary representation learning for RL by enriching auxiliary training data, proposing \textbf{L}earning \textbf{F}uture representation with \textbf{S}ynthetic observations \textbf{(LFS)}, a novel self-supervised RL approach. Specifically, we propose a training-free method to synthesize observations that may contain future information, as well as a data selection approach to eliminate unqualified synthetic noise. The remaining synthetic observations and real observations then serve as the auxiliary data to achieve a clustering-based temporal association task for representation learning. LFS allows the agent to access and learn observations that have not yet appeared in advance, so as to quickly understand and exploit them when they occur later. In addition, LFS does not rely on rewards or actions, which means it has a wider scope of application (e.g., learning from video) than recent advanced auxiliary tasks. Extensive experiments demonstrate that our LFS exhibits state-of-the-art RL sample efficiency on challenging continuous control and enables advanced visual pre-training based on action-free video demonstrations.