Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

TL;DR

FAN is an offline RL algorithm that improves efficiency and performance by simplifying flow policy sampling and distributional critic computations, achieving state-of-the-art results with reduced runtimes.

Contribution

Introducing FAN, a novel offline RL method that reduces computational costs by using single iteration and sample techniques, while maintaining high performance.

Findings

FAN achieves state-of-the-art results on robotic tasks.

FAN significantly reduces training and inference runtimes.

Theoretical analysis confirms convergence and performance bounds.

Abstract

We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that utilizes only a single flow policy iteration and requires only a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.