Loss-Guided Auxiliary Agents for Overcoming Mode Collapse in GFlowNets

TL;DR

This paper introduces Loss-Guided GFlowNets, an approach where an auxiliary network guides exploration based on the main network's loss, leading to faster discovery of diverse high-reward solutions and overcoming mode collapse.

Contribution

The paper proposes a novel loss-guided exploration method for GFlowNets that directly uses the main network's training loss to improve diversity and efficiency.

Findings

Outperforms baselines in exploration efficiency and diversity

Discovered over 40 times more unique modes in sequence generation

Reduced exploration error by approximately 99%

Abstract

Although Generative Flow Networks (GFlowNets) are designed to capture multiple modes of a reward function, they often suffer from mode collapse in practice, getting trapped in early-discovered modes and requiring prolonged training to find diverse solutions. Existing exploration techniques often rely on heuristic novelty signals. We propose Loss-Guided GFlowNets (LGGFN), a novel approach where an auxiliary GFlowNet's exploration is \textbf{directly driven by the main GFlowNet's training loss}. By prioritizing trajectories where the main model exhibits \textbf{high loss}, LGGFN focuses sampling on poorly understood regions of the state space. This targeted exploration significantly accelerates the discovery of diverse, high-reward samples. Empirically, across \textbf{diverse benchmarks} including grid environments, structured sequence generation, Bayesian structure learning, and biological sequence design, LGGFN consistently \textbf{outperforms} baselines in exploration efficiency and sample diversity. For instance, on a challenging sequence generation task, it discovered over 40 times more unique valid modes while simultaneously reducing the exploration error metric by approximately 99\%.