Enhancing Offline Reinforcement Learning with Curriculum Learning-Based Trajectory Valuation

TL;DR

This paper introduces CLTV, a curriculum learning approach that uses transition scoring to improve offline reinforcement learning by effectively prioritizing high-quality trajectories, especially in mixed datasets with domain mismatch.

Contribution

The paper proposes CLTV, a novel trajectory valuation method using transition scores and curriculum learning to address domain mismatch in offline RL with mixed datasets.

Findings

CLTV improves offline RL performance across various algorithms.

The method enhances transferability of policies in MuJoCo environments.

Theoretical analysis supports the effectiveness of CLTV.

Abstract

The success of deep reinforcement learning (DRL) relies on the availability and quality of training data, often requiring extensive interactions with specific environments. In many real-world scenarios, where data collection is costly and risky, offline reinforcement learning (RL) offers a solution by utilizing data collected by domain experts and searching for a batch-constrained optimal policy. This approach is further augmented by incorporating external data sources, expanding the range and diversity of data collection possibilities. However, existing offline RL methods often struggle with challenges posed by non-matching data from these external sources. In this work, we specifically address the problem of source-target domain mismatch in scenarios involving mixed datasets, characterized by a predominance of source data generated from random or suboptimal policies and a limited amount of target data generated from higher-quality policies. To tackle this problem, we introduce Transition Scoring (TS), a novel method that assigns scores to transitions based on their similarity to the target domain, and propose Curriculum Learning-Based Trajectory Valuation (CLTV), which effectively leverages these transition scores to identify and prioritize high-quality trajectories through a curriculum learning approach. Our extensive experiments across various offline RL methods and MuJoCo environments, complemented by rigorous theoretical analysis, demonstrate that CLTV enhances the overall performance and transferability of policies learned by offline RL algorithms.