Looking Backward: Retrospective Backward Synthesis for Goal-Conditioned GFlowNets

TL;DR

This paper introduces RBS, a method that synthesizes backward trajectories to improve training of goal-conditioned GFlowNets, significantly enhancing sample efficiency and diversity in high-dimensional, sparse reward tasks.

Contribution

The paper proposes RBS, a novel backward synthesis technique that enriches training data for goal-conditioned GFlowNets, addressing sparse rewards and limited trajectory coverage issues.

Findings

RBS significantly improves sample efficiency.

RBS outperforms baseline methods on standard benchmarks.

Enhanced trajectory diversity leads to better goal coverage.

Abstract

Generative Flow Networks (GFlowNets), a new family of probabilistic samplers, have demonstrated remarkable capabilities to generate diverse sets of high-reward candidates, in contrast to standard return maximization approaches (e.g., reinforcement learning) which often converge to a single optimal solution. Recent works have focused on developing goal-conditioned GFlowNets, which aim to train a single GFlowNet capable of achieving different outcomes as the task specifies. However, training such models is challenging due to extremely sparse rewards, particularly in high-dimensional problems. Moreover, previous methods suffer from the limited coverage of explored trajectories during training, which presents more pronounced challenges when only offline data is available. In this work, we propose a novel method called \textbf{R}etrospective \textbf{B}ackward \textbf{S}ynthesis (\textbf{RBS}) to address these critical problems. Specifically, RBS synthesizes new backward trajectories in goal-conditioned GFlowNets to enrich training trajectories with enhanced quality and diversity, thereby introducing copious learnable signals for effectively tackling the sparse reward problem. Extensive empirical results show that our method improves sample efficiency by a large margin and outperforms strong baselines on various standard evaluation benchmarks.