An Empirical Study of the Effectiveness of Using a Replay Buffer on Mode Discovery in GFlowNets

TL;DR

This paper investigates the impact of using a replay buffer in GFlowNets, showing that it significantly enhances mode discovery and sampling efficiency in discrete domains like molecule synthesis.

Contribution

It is the first empirical study to evaluate the effect of replay buffers on GFlowNets, demonstrating improved mode discovery and training efficiency.

Findings

Replay buffers improve mode discovery speed.

Sampling techniques affect the quality of discovered modes.

Significant gains observed in toy and real-world domains.

Abstract

Reinforcement Learning (RL) algorithms aim to learn an optimal policy by iteratively sampling actions to learn how to maximize the total expected return, $R(x)$. GFlowNets are a special class of algorithms designed to generate diverse candidates, $x$, from a discrete set, by learning a policy that approximates the proportional sampling of $R(x)$. GFlowNets exhibit improved mode discovery compared to conventional RL algorithms, which is very useful for applications such as drug discovery and combinatorial search. However, since GFlowNets are a relatively recent class of algorithms, many techniques which are useful in RL have not yet been associated with them. In this paper, we study the utilization of a replay buffer for GFlowNets. We explore empirically various replay buffer sampling techniques and assess the impact on the speed of mode discovery and the quality of the modes discovered. Our experimental results in the Hypergrid toy domain and a molecule synthesis environment demonstrate significant improvements in mode discovery when training with a replay buffer, compared to training only with trajectories generated on-policy.