The Sufficiency-Conciseness Trade-off in LLM Self-Explanation from an Information Bottleneck Perspective

TL;DR

This paper investigates the balance between explanation sufficiency and conciseness in large language models, using an information bottleneck approach to optimize explanation length without sacrificing answer accuracy.

Contribution

It introduces an evaluation pipeline based on the information bottleneck principle to measure and optimize explanation conciseness and sufficiency across languages.

Findings

Concise explanations often remain sufficient for correct answers.

Excessive compression reduces explanation effectiveness.

The approach works in both English and Persian datasets.

Abstract

Large Language Models increasingly rely on self-explanations, such as chain of thought reasoning, to improve performance on multi step question answering. While these explanations enhance accuracy, they are often verbose and costly to generate, raising the question of how much explanation is truly necessary. In this paper, we examine the trade-off between sufficiency, defined as the ability of an explanation to justify the correct answer, and conciseness, defined as the reduction in explanation length. Building on the information bottleneck principle, we conceptualize explanations as compressed representations that retain only the information essential for producing correct answers.To operationalize this view, we introduce an evaluation pipeline that constrains explanation length and assesses sufficiency using multiple language models on the ARC Challenge dataset. To broaden the scope, we conduct experiments in both English, using the original dataset, and Persian, as a resource-limited language through translation. Our experiments show that more concise explanations often remain sufficient, preserving accuracy while substantially reducing explanation length, whereas excessive compression leads to performance degradation.