The Duality of Generative AI and Reinforcement Learning in Robotics: A Review

TL;DR

This review explores how generative AI and reinforcement learning synergize to enhance robotics control policies, highlighting recent advancements, challenges, and future research directions in integrating these technologies.

Contribution

It provides a comprehensive taxonomy of current methods, analyzes the duality between generative AI and RL in robotics, and discusses open challenges and future research insights.

Findings

Generative AI models serve as modular priors for multi-modal input fusion in RL.

RL can effectively train and fine-tune generative models for policy generation.

Identified key challenges include scalability, adaptation, and safety concerns.

Abstract

Recently, generative AI and reinforcement learning (RL) have been redefining what is possible for AI agents that take information flows as input and produce intelligent behavior. As a result, we are seeing similar advancements in embodied AI and robotics for control policy generation. Our review paper examines the integration of generative AI models with RL to advance robotics. Our primary focus is on the duality between generative AI and RL for robotics downstream tasks. Specifically, we investigate: (1) The role of prominent generative AI tools as modular priors for multi-modal input fusion in RL tasks. (2) How RL can train, fine-tune and distill generative models for policy generation, such as VLA models, similarly to RL applications in large language models. We then propose a new taxonomy based on a considerable amount of selected papers. Lastly, we identify open challenges accounting for model scalability, adaptation and grounding, giving recommendations and insights on future research directions. We reflect on which generative AI models best fit the RL tasks and why. On the other side, we reflect on important issues inherent to RL-enhanced generative policies, such as safety concerns and failure modes, and what are the limitations of current methods. A curated collection of relevant research papers is maintained on our GitHub repository, serving as a resource for ongoing research and development in this field: https://github.com/clmoro/Robotics-RL-FMs-Integration.