Rethinking the Understanding Ability across LLMs through Mutual Information

TL;DR

This paper introduces an information-theoretic framework based on mutual information to evaluate and enhance the intrinsic linguistic understanding of large language models, revealing differences between model types and benefits of fine-tuning.

Contribution

It formalizes language understanding as mutual information between input and latent representations, derives a computable lower bound for token-level MI, and empirically compares models and fine-tuning effects.

Findings

Encoder-only models retain higher mutual information than decoder-only models.

Decoder-only models show a late-layer 'forgetting' pattern in mutual information.

Fine-tuning to maximize token recoverability improves understanding ability.

Abstract

Recent advances in large language models (LLMs) have revolutionized natural language processing, yet evaluating their intrinsic linguistic understanding remains challenging. Moving beyond specialized evaluation tasks, we propose an information-theoretic framework grounded in mutual information (MI) to achieve this. We formalize the understanding as MI between an input sentence and its latent representation (sentence-level MI), measuring how effectively input information is preserved in latent representation. Given that LLMs learn embeddings for individual tokens, we decompose sentence-level MI into token-level MI between tokens and sentence embeddings, establishing theoretical bounds connecting these measures. Based on this foundation, we theoretically derive a computable lower bound for token-level MI using Fano's inequality, which directly relates to token-level recoverability-the ability to predict original tokens from sentence embedding. We implement this recoverability task to comparatively measure MI across different LLMs, revealing that encoder-only models consistently maintain higher information fidelity than their decoder-only counterparts, with the latter exhibiting a distinctive late-layer "forgetting" pattern where mutual information is first enhanced and then discarded. Moreover, fine-tuning to maximize token-level recoverability consistently improves understanding ability of LLMs on tasks without task-specific supervision, demonstrating that mutual information can serve as a foundation for understanding and improving language model capabilities.