Distributional GFlowNets with Quantile Flows

TL;DR

This paper introduces a distributional approach to GFlowNets by using quantile functions, enabling risk-sensitive policies and improving performance on benchmarks, even with deterministic rewards.

Contribution

It proposes a novel distributional GFlowNet framework with quantile matching, enhancing training and enabling risk-sensitive decision-making.

Findings

Significant performance improvements on benchmarks.

Ability to learn risk-sensitive policies.

Effective even with deterministic rewards.

Abstract

Generative Flow Networks (GFlowNets) are a new family of probabilistic samplers where an agent learns a stochastic policy for generating complex combinatorial structure through a series of decision-making steps. Despite being inspired from reinforcement learning, the current GFlowNet framework is relatively limited in its applicability and cannot handle stochasticity in the reward function. In this work, we adopt a distributional paradigm for GFlowNets, turning each flow function into a distribution, thus providing more informative learning signals during training. By parameterizing each edge flow through their quantile functions, our proposed \textit{quantile matching} GFlowNet learning algorithm is able to learn a risk-sensitive policy, an essential component for handling scenarios with risk uncertainty. Moreover, we find that the distributional approach can achieve substantial improvement on existing benchmarks compared to prior methods due to our enhanced training algorithm, even in settings with deterministic rewards.