Reinforcement Learning for Aligning Large Language Models Agents with Interactive Environments: Quantifying and Mitigating Prompt Overfitting

TL;DR

This paper investigates how reinforcement learning fine-tuning affects large language models' robustness to prompt variations in interactive environments, revealing sensitivity issues and proposing contrastive loss to enhance generalization.

Contribution

It introduces a framework for analyzing prompt sensitivity after RL training and proposes a contrastive loss method to mitigate overfitting to specific prompts.

Findings

LLMs' performance degrades with unseen prompt formulations

Prompt sensitivity is linked to internal representations and salient tokens

Contrastive loss improves robustness and generalization

Abstract

Reinforcement learning (RL) is a promising approach for aligning large language models (LLMs) knowledge with sequential decision-making tasks. However, few studies have thoroughly investigated the impact on LLM agents capabilities of fine-tuning them with RL in a specific environment. In this paper, we propose a novel framework to analyze the sensitivity of LLMs to prompt formulations following RL training in a textual environment. Our findings reveal that the performance of LLMs degrades when faced with prompt formulations different from those used during the RL training phase. Besides, we analyze the source of this sensitivity by examining the model's internal representations and salient tokens. Finally, we propose to use a contrastive loss to mitigate this sensitivity and improve the robustness and generalization capabilities of LLMs.