Explainable reinforcement learning from human feedback to improve alignment

TL;DR

This paper introduces a method to enhance reinforcement learning from human feedback for language models by explaining unsatisfactory responses and unlearning problematic training data, leading to improved alignment.

Contribution

It presents a novel post-hoc explanation and unlearning approach to correct causes of unsatisfactory responses in RLHF-trained language models.

Findings

The method effectively explains why certain responses are generated.

Unlearning identified training data improves response quality.

The approach maintains performance on satisfactory responses.

Abstract

A common and effective strategy for humans to improve an unsatisfactory outcome in daily life is to find a cause of this outcome and correct the cause. In this paper, we investigate whether this human improvement strategy can be applied to improving reinforcement learning from human feedback (RLHF) for alignment of language models (LMs). In particular, it is observed in the literature that LMs tuned by RLHF can still output unsatisfactory responses. This paper proposes a method to improve the unsatisfactory responses by correcting their causes. Our method has two parts. The first part proposes a post-hoc explanation method to explain why an unsatisfactory response is generated to a prompt by identifying the training data that lead to this response. We formulate this problem as a constrained combinatorial optimization problem where the objective is to find a set of training data closest to this prompt-response pair in a feature representation space, and the constraint is that the prompt-response pair can be decomposed as a convex combination of this set of training data in the feature space. We propose an efficient iterative data selection algorithm to solve this problem. The second part proposes an unlearning method that improves unsatisfactory responses to some prompts by unlearning the training data that lead to these unsatisfactory responses and, meanwhile, does not significantly degrade satisfactory responses to other prompts. Experimental results demonstrate that our algorithm can improve RLHF.