CFlowNets: Continuous Control with Generative Flow Networks

TL;DR

This paper introduces CFlowNets, a novel extension of GFlowNets for continuous control, providing a theoretical framework, a training method, and demonstrating superior exploration capabilities over traditional reinforcement learning methods.

Contribution

The paper develops the first continuous control adaptation of GFlowNets, including theoretical formulation, a training framework, and error bounds, with empirical validation.

Findings

CFlowNets outperform RL methods in exploration tasks

Error in flow approximation decreases rapidly with more samples

Theoretical error bounds are established for flow approximation

Abstract

Generative flow networks (GFlowNets), as an emerging technique, can be used as an alternative to reinforcement learning for exploratory control tasks. GFlowNet aims to generate distribution proportional to the rewards over terminating states, and to sample different candidates in an active learning fashion. GFlowNets need to form a DAG and compute the flow matching loss by traversing the inflows and outflows of each node in the trajectory. No experiments have yet concluded that GFlowNets can be used to handle continuous tasks. In this paper, we propose generative continuous flow networks (CFlowNets) that can be applied to continuous control tasks. First, we present the theoretical formulation of CFlowNets. Then, a training framework for CFlowNets is proposed, including the action selection process, the flow approximation algorithm, and the continuous flow matching loss function. Afterward, we theoretically prove the error bound of the flow approximation. The error decreases rapidly as the number of flow samples increases. Finally, experimental results on continuous control tasks demonstrate the performance advantages of CFlowNets compared to many reinforcement learning methods, especially regarding exploration ability.