Quantifying Semantic Emergence in Language Models

TL;DR

This paper introduces a new metric called Information Emergence (IE) to quantify how well large language models extract meaningful semantics from input tokens, providing insights into their semantic understanding capabilities.

Contribution

The paper proposes a novel, task- and architecture-agnostic metric, Information Emergence, for measuring semantic extraction in language models, along with a lightweight estimator for mutual information.

Findings

IE reveals patterns consistent with linguistic knowledge

Some IE patterns are unexpected, offering new insights

Experiments validate IE's informativeness in different contexts

Abstract

Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning. Yet, there remains no established metric to quantify this capability. In this work, we introduce a quantitative metric, Information Emergence (IE), designed to measure LLMs' ability to extract semantics from input tokens. We formalize ``semantics'' as the meaningful information abstracted from a sequence of tokens and quantify this by comparing the entropy reduction observed for a sequence of tokens (macro-level) and individual tokens (micro-level). To achieve this, we design a lightweight estimator to compute the mutual information at each transformer layer, which is agnostic to different tasks and language model architectures. We apply IE in both synthetic in-context learning (ICL) scenarios and natural sentence contexts. Experiments demonstrate informativeness and patterns about semantics. While some of these patterns confirm the conventional prior linguistic knowledge, the rest are relatively unexpected, which may provide new insights.