Ergodic Generative Flows

TL;DR

This paper introduces Ergodic Generative Flows (EGFs), a new class of generative models leveraging ergodicity to improve training in continuous and imitation learning settings, with tractable loss functions and broad applicability.

Contribution

The work proposes EGFs with globally defined transformations, a new KL-weakFM loss for imitation learning without reward models, and demonstrates their effectiveness on various tasks.

Findings

EGFs enable simpler training with finite transformations.

KL-weakFM loss effectively trains EGFs for imitation learning.

EGFs perform well on toy and real-world datasets.

Abstract

Generative Flow Networks (GFNs) were initially introduced on directed acyclic graphs to sample from an unnormalized distribution density. Recent works have extended the theoretical framework for generative methods allowing more flexibility and enhancing application range. However, many challenges remain in training GFNs in continuous settings and for imitation learning (IL), including intractability of flow-matching loss, limited tests of non-acyclic training, and the need for a separate reward model in imitation learning. The present work proposes a family of generative flows called Ergodic Generative Flows (EGFs) which are used to address the aforementioned issues. First, we leverage ergodicity to build simple generative flows with finitely many globally defined transformations (diffeomorphisms) with universality guarantees and tractable flow-matching loss (FM loss). Second, we introduce a new loss involving cross-entropy coupled to weak flow-matching control, coined KL-weakFM loss. It is designed for IL training without a separate reward model. We evaluate IL-EGFs on toy 2D tasks and real-world datasets from NASA on the sphere, using the KL-weakFM loss. Additionally, we conduct toy 2D reinforcement learning experiments with a target reward, using the FM loss.