GenFlowRL: Shaping Rewards with Generative Object-Centric Flow in Visual Reinforcement Learning

TL;DR

GenFlowRL introduces a novel reward shaping method using generated object-centric flow to improve generalization and robustness in visual reinforcement learning for manipulation tasks, outperforming existing approaches.

Contribution

The paper presents GenFlowRL, a new approach that leverages generated object-centric flow for reward shaping, enabling effective policy learning from diverse demonstrations with low-dimensional features.

Findings

Superior performance on 10 manipulation tasks in simulation and real-world.

Effective generalization across diverse and challenging scenarios.

Robust policies learned from cross-embodiment datasets.

Abstract

Recent advances have shown that video generation models can enhance robot learning by deriving effective robot actions through inverse dynamics. However, these methods heavily depend on the quality of generated data and struggle with fine-grained manipulation due to the lack of environment feedback. While video-based reinforcement learning improves policy robustness, it remains constrained by the uncertainty of video generation and the challenges of collecting large-scale robot datasets for training diffusion models. To address these limitations, we propose GenFlowRL, which derives shaped rewards from generated flow trained from diverse cross-embodiment datasets. This enables learning generalizable and robust policies from diverse demonstrations using low-dimensional, object-centric features. Experiments on 10 manipulation tasks, both in simulation and real-world cross-embodiment evaluations, demonstrate that GenFlowRL effectively leverages manipulation features extracted from generated object-centric flow, consistently achieving superior performance across diverse and challenging scenarios. Our Project Page: https://colinyu1.github.io/genflowrl