Prompt Tuning with Diffusion for Few-Shot Pre-trained Policy Generalization

TL;DR

This paper introduces Prompt Diffuser, a diffusion-based method for prompt tuning in offline RL, enabling high-quality prompt generation from noise and improving adaptation to new tasks in meta-RL.

Contribution

It proposes a novel diffusion model approach for prompt tuning, overcoming initialization limitations and enhancing prompt quality for better generalization in offline RL.

Findings

Prompt Diffuser produces superior prompts compared to traditional methods.

The approach improves meta-RL task performance.

Experimental results validate robustness and effectiveness.

Abstract

Offline reinforcement learning (RL) methods harness previous experiences to derive an optimal policy, forming the foundation for pre-trained large-scale models (PLMs). When encountering tasks not seen before, PLMs often utilize several expert trajectories as prompts to expedite their adaptation to new requirements. Though a range of prompt-tuning methods have been proposed to enhance the quality of prompts, these methods often face optimization restrictions due to prompt initialization, which can significantly constrain the exploration domain and potentially lead to suboptimal solutions. To eliminate the reliance on the initial prompt, we shift our perspective towards the generative model, framing the prompt-tuning process as a form of conditional generative modeling, where prompts are generated from random noise. Our innovation, the Prompt Diffuser, leverages a conditional diffusion model to produce prompts of exceptional quality. Central to our framework is the approach to trajectory reconstruction and the meticulous integration of downstream task guidance during the training phase. Further experimental results underscore the potency of the Prompt Diffuser as a robust and effective tool for the prompt-tuning process, demonstrating strong performance in the meta-RL tasks.