Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RL

TL;DR

This paper examines how specific SSL objective modifications within the SPR framework affect data-efficient RL, demonstrating that tailored SSL objectives and adjustments significantly improve performance across benchmarks.

Contribution

It provides a detailed analysis of SSL objective modifications in RL, highlighting their impact on performance and guiding better SSL objective selection for data-efficient RL.

Findings

SSL modifications within SPR improve performance

Impact of SSL objectives varies across algorithms

Proper SSL objective selection is crucial for data efficiency

Abstract

In this study, we investigate the effect of SSL objective modifications within the SPR framework, focusing on specific adjustments such as terminal state masking and prioritized replay weighting, which were not explicitly addressed in the original design. While these modifications are specific to RL, they are not universally applicable across all RL algorithms. Therefore, we aim to assess their impact on performance and explore other SSL objectives that do not accommodate these adjustments like Barlow Twins and VICReg. We evaluate six SPR variants on the Atari 100k benchmark, including versions both with and without these modifications. Additionally, we test the performance of these objectives on the DeepMind Control Suite, where such modifications are absent. Our findings reveal that incorporating specific SSL modifications within SPR significantly enhances performance, and this influence extends to subsequent frameworks like SR-SPR and BBF, highlighting the critical importance of SSL objective selection and related adaptations in achieving data efficiency in self-predictive reinforcement learning.