Risk-Averse Finetuning of Large Language Models

TL;DR

This paper introduces a risk-averse fine-tuning approach for large language models that reduces toxic outputs by optimizing Conditional Value at Risk, leading to safer and more constructive language generation.

Contribution

It presents a novel risk-averse reinforcement learning method using CVaR optimization for fine-tuning LLMs to mitigate harmful content.

Findings

Effective reduction of toxic outputs in LLMs

Improved safety in sentiment modification tasks

Maintains generative performance while reducing risks

Abstract

We consider the challenge of mitigating the generation of negative or toxic content by the Large Language Models (LLMs) in response to certain prompts. We propose integrating risk-averse principles into LLM fine-tuning to minimize the occurrence of harmful outputs, particularly rare but significant events. By optimizing the risk measure of Conditional Value at Risk (CVaR), our methodology trains LLMs to exhibit superior performance in avoiding toxic outputs while maintaining effectiveness in generative tasks. Empirical evaluations on sentiment modification and toxicity mitigation tasks demonstrate the efficacy of risk-averse reinforcement learning with human feedback (RLHF) in promoting a safer and more constructive online discourse environment.