Evaluating GFlowNet from partial episodes for stable and flexible policy-based training

TL;DR

This paper introduces an evaluation balance method for GFlowNets that improves the reliability and flexibility of policy-based training by effectively estimating policy divergence from partial episodes, applicable to synthetic and real-world tasks.

Contribution

It presents a novel evaluation balance objective that enhances divergence estimation and supports parameterized backward policies and offline data integration in GFlowNets.

Findings

Evaluation balance improves training stability.

Method supports parameterized backward policies.

Effective on both synthetic and real-world tasks.

Abstract

Generative Flow Networks (GFlowNets) were developed to learn policies for efficiently sampling combinatorial candidates by interpreting their generative processes as trajectories in directed acyclic graphs. In the value-based training workflow, the objective is to enforce the balance over partial episodes between the flows of the learned policy and the estimated flows of the desired policy, implicitly encouraging policy divergence minimization. The policy-based strategy alternates between estimating the policy divergence and updating the policy, but reliable estimation of the divergence under directed acyclic graphs remains a major challenge. This work bridges the two perspectives by showing that flow balance also yields a principled policy evaluator that measures the divergence, and an evaluation balance objective over partial episodes is proposed for learning the evaluator. As demonstrated on both synthetic and real-world tasks, evaluation balance not only strengthens the reliability of policy-based training but also broadens its flexibility by seamlessly supporting parameterized backward policies and enabling the integration of offline data-collection techniques.