From Tarzan to Tolkien: Controlling the Language Proficiency Level of LLMs for Content Generation

TL;DR

This paper explores methods to control the difficulty level of text generated by large language models, aiming to make content suitable for language learners, and introduces a new model that outperforms existing solutions at lower cost.

Contribution

The paper presents CALM, a novel approach combining finetuning and reinforcement learning to effectively control language proficiency levels in LLM-generated content, surpassing GPT-4 performance.

Findings

Prompt-based strategies show large performance gaps between GPT-4 and open source models.

Finetuning and RL can bridge the performance gap effectively.

CALM outperforms GPT-4 and other methods in controlling language difficulty.

Abstract

We study the problem of controlling the difficulty level of text generated by Large Language Models (LLMs) for contexts where end-users are not fully proficient, such as language learners. Using a novel framework, we evaluate the effectiveness of several key approaches for this task, including few-shot prompting, supervised finetuning, and reinforcement learning (RL), utilising both GPT-4 and open source alternatives like LLama2-7B and Mistral-7B. Our findings reveal a large performance gap between GPT-4 and the open source models when using prompt-based strategies. However, we show how to bridge this gap with a careful combination of finetuning and RL alignment. Our best model, CALM (CEFR-Aligned Language Model), surpasses the performance of GPT-4 and other strategies, at only a fraction of the cost. We further validate the quality of our results through a small-scale human study.