Language Model Self-improvement by Reinforcement Learning Contemplation

TL;DR

This paper presents SIRLC, an unsupervised reinforcement learning method where LLMs self-assess and improve their performance across NLP tasks without external labels, enhancing accuracy and translation quality.

Contribution

Introduces SIRLC, a novel unsupervised reinforcement learning approach enabling LLMs to self-improve by evaluating and scoring their own generated outputs.

Findings

5.6% increase in reasoning accuracy

BERTScore improved from 0.82 to 0.86 in translation

Effective across various NLP tasks

Abstract

Large Language Models (LLMs) have exhibited remarkable performance across various natural language processing (NLP) tasks. However, fine-tuning these models often necessitates substantial supervision, which can be expensive and time-consuming to obtain. This paper introduces a novel unsupervised method called LanguageModel Self-Improvement by Reinforcement Learning Contemplation (SIRLC) that improves LLMs without reliance on external labels. Our approach is grounded in the observation that it is simpler for language models to assess text quality than to generate text. Building on this insight, SIRLC assigns LLMs dual roles as both student and teacher. As a student, the LLM generates answers to unlabeled questions, while as a teacher, it evaluates the generated text and assigns scores accordingly. The model parameters are updated using reinforcement learning to maximize the evaluation score. We demonstrate that SIRLC can be applied to various NLP tasks, such as reasoning problems, text generation, and machine translation. Our experiments show that SIRLC effectively improves LLM performance without external supervision, resulting in a 5.6% increase in answering accuracy for reasoning tasks and a rise in BERTScore from 0.82 to 0.86 for translation tasks. Furthermore, SIRLC can be applied to models of different sizes, showcasing its broad applicability.