Behavioral Mode Discovery for Fine-tuning Multimodal Generative Policies

TL;DR

This paper introduces an unsupervised mode discovery framework for fine-tuning multimodal generative policies with reinforcement learning, preserving behavioral diversity and improving task success.

Contribution

It proposes a novel method to uncover latent behavioral modes, using mutual information as an intrinsic reward during RL fine-tuning.

Findings

Outperforms conventional fine-tuning methods in robotic manipulation tasks.

Achieves higher success rates while maintaining multimodal action distributions.

Enhances behavioral diversity during policy refinement.

Abstract

We address the problem of fine-tuning pre-trained generative policies with reinforcement learning (RL) while preserving the multimodality of their action distributions. Existing methods for RL fine-tuning of generative policies (e.g., diffusion policies) improve task performance but often collapse diverse behaviors into a single reward-maximizing mode. To mitigate this issue, we propose an unsupervised mode discovery framework that uncovers latent behavioral modes within generative policies. The discovered modes enable the use of mutual information as an intrinsic reward, regularizing RL fine-tuning to enhance task success while maintaining behavioral diversity. Experiments on robotic manipulation tasks demonstrate that our method consistently outperforms conventional fine-tuning approaches, achieving higher success rates and preserving richer multimodal action distributions.