MetaGFN: Exploring Distant Modes with Adapted Metadynamics for Continuous GFlowNets

TL;DR

MetaGFN introduces Adapted Metadynamics as an exploration strategy for continuous GFlowNets, significantly improving convergence speed and ability to discover distant reward modes in continuous domains.

Contribution

The paper proposes Adapted Metadynamics for continuous GFlowNets, enabling efficient exploration and faster convergence in continuous reward landscapes.

Findings

MetaGFN accelerates convergence to target distributions.

It discovers more distant reward modes than previous methods.

Effective in various continuous domain experiments.

Abstract

Generative Flow Networks (GFlowNets) are a class of generative models that sample objects in proportion to a specified reward function through a learned policy. They can be trained either on-policy or off-policy, needing a balance between exploration and exploitation for fast convergence to a target distribution. While exploration strategies for discrete GFlowNets have been studied, exploration in the continuous case remains to be investigated, despite the potential for novel exploration algorithms due to the local connectedness of continuous domains. Here, we introduce Adapted Metadynamics, a variant of metadynamics that can be applied to arbitrary black-box reward functions on continuous domains. We use Adapted Metadynamics as an exploration strategy for continuous GFlowNets. We show several continuous domains where the resulting algorithm, MetaGFN, accelerates convergence to the target distribution and discovers more distant reward modes than previous off-policy exploration strategies used for GFlowNets.