Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

TL;DR

This paper introduces FINO, a novel offline-to-online reinforcement learning method that uses flow matching with injected noise and entropy-guided sampling to improve exploration and sample efficiency during fine-tuning.

Contribution

FINO is the first approach to integrate flow matching with noise injection and entropy-guided sampling for effective offline-to-online RL.

Findings

FINO outperforms existing methods with limited online interactions.

Injected noise enhances exploration beyond offline data.

Entropy-guided sampling balances exploration and exploitation.

Abstract

Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.