Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs

TL;DR

This paper introduces an axiomatic framework to precisely quantify and distinguish between memorization and in-context reasoning effects in large language models, enabling detailed analysis of their inference patterns.

Contribution

It presents a novel axiomatic system that categorizes and mathematically decomposes memorization and reasoning effects in LLMs, facilitating interpretability.

Findings

Disentanglement of effects enables detailed inference analysis

Categorization of memorization into foundational and chaotic types

Classification of reasoning into enhanced, eliminated, and reversed patterns

Abstract

In this study, we propose an axiomatic system to define and quantify the precise memorization and in-context reasoning effects used by the large language model (LLM) for language generation. These effects are formulated as non-linear interactions between tokens/words encoded by the LLM. Specifically, the axiomatic system enables us to categorize the memorization effects into foundational memorization effects and chaotic memorization effects, and further classify in-context reasoning effects into enhanced inference patterns, eliminated inference patterns, and reversed inference patterns. Besides, the decomposed effects satisfy the sparsity property and the universal matching property, which mathematically guarantee that the LLM's confidence score can be faithfully decomposed into the memorization effects and in-context reasoning effects. Experiments show that the clear disentanglement of memorization effects and in-context reasoning effects enables a straightforward examination of detailed inference patterns encoded by LLMs.