FlowCritic: Bridging Value Estimation with Flow Matching in Reinforcement Learning

TL;DR

FlowCritic introduces a novel generative approach to value estimation in reinforcement learning by applying flow matching, enhancing the modeling of complex value distributions beyond traditional methods.

Contribution

It pioneers the use of flow matching for value distribution modeling in RL, offering a more expressive and sample-based estimation method.

Findings

Demonstrates improved value distribution modeling.

Provides more accurate value estimates in RL tasks.

Enhances policy learning efficiency.

Abstract

Reliable value estimation serves as the cornerstone of reinforcement learning (RL) by evaluating long-term returns and guiding policy improvement, significantly influencing the convergence speed and final performance. Existing works improve the reliability of value function estimation via multi-critic ensembles and distributional RL, yet the former merely combines multi point estimation without capturing distributional information, whereas the latter relies on discretization or quantile regression, limiting the expressiveness of complex value distributions. Inspired by flow matching's success in generative modeling, we propose a generative paradigm for value estimation, named FlowCritic. Departing from conventional regression for deterministic value prediction, FlowCritic leverages flow matching to model value distributions and generate samples for value estimation.