Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discovery

TL;DR

This paper introduces a flow-based method for offline reinforcement learning that learns lossless primitives, enabling faithful policy representation and significantly improving hierarchical policy performance in offline settings.

Contribution

It proposes a novel flow-based primitive learning approach that preserves the full policy space, addressing limitations of previous primitive extraction methods in offline RL.

Findings

Achieves superior performance on D4RL benchmarks.

Demonstrates the effectiveness of lossless primitives in hierarchical policies.

Shows significant performance gains over existing methods.

Abstract

Offline reinforcement learning (RL) enables the agent to effectively learn from logged data, which significantly extends the applicability of RL algorithms in real-world scenarios where exploration can be expensive or unsafe. Previous works have shown that extracting primitive skills from the recurring and temporally extended structures in the logged data yields better learning. However, these methods suffer greatly when the primitives have limited representation ability to recover the original policy space, especially in offline settings. In this paper, we give a quantitative characterization of the performance of offline hierarchical learning and highlight the importance of learning lossless primitives. To this end, we propose to use a \emph{flow}-based structure as the representation for low-level policies. This allows us to represent the behaviors in the dataset faithfully while keeping the expression ability to recover the whole policy space. We show that such lossless primitives can drastically improve the performance of hierarchical policies. The experimental results and extensive ablation studies on the standard D4RL benchmark show that our method has a good representation ability for policies and achieves superior performance in most tasks.