Free-text Rationale Generation under Readability Level Control

TL;DR

This paper explores how large language models generate natural language rationales at different readability levels, revealing that explanations adapt to instructions but do not always match traditional complexity metrics, with human preferences favoring high-school-level explanations.

Contribution

It demonstrates the ability of LLMs to produce explanations at specified readability levels and analyzes their alignment with traditional metrics and human preferences.

Findings

Explanations adapt to readability prompts but differ from traditional complexity scores.

Generated rationales tend to have medium complexity, correlating with quality metrics.

Humans prefer and find high-school-level rationales most satisfactory.

Abstract

Free-text rationales justify model decisions in natural language and thus become likable and accessible among approaches to explanation across many tasks. However, their effectiveness can be hindered by misinterpretation and hallucination. As a perturbation test, we investigate how large language models (LLMs) perform rationale generation under the effects of readability level control, i.e., being prompted for an explanation targeting a specific expertise level, such as sixth grade or college. We find that explanations are adaptable to such instruction, though the observed distinction between readability levels does not fully match the defined complexity scores according to traditional readability metrics. Furthermore, the generated rationales tend to feature medium level complexity, which correlates with the measured quality using automatic metrics. Finally, our human annotators confirm a generally satisfactory impression on rationales at all readability levels, with high-school-level readability being most commonly perceived and favored.