What Makes In-context Learning Effective for Mathematical Reasoning: A Theoretical Analysis

TL;DR

This paper provides a theoretical analysis of in-context learning for mathematical reasoning in large language models, proposing a new demonstration selection method that improves performance across benchmarks.

Contribution

It introduces a theoretical framework linking reasoning performance to semantic similarity and inference stability, and proposes LMS3, a novel demonstration selection method.

Findings

LMS3 outperforms existing methods on multiple benchmarks.

Demonstration relevance and stability are key to reasoning performance.

LMS3 effectively filters unsuitable samples for few-shot learning.

Abstract

Owing to the capability of in-context learning, large language models (LLMs) have shown impressive performance across diverse mathematical reasoning benchmarks. However, we find that few-shot demonstrations can sometimes bring negative performance and their effectiveness on LLMs' reasoning abilities remains unreliable. To this end, in this paper, we aim to theoretically analyze the impact of in-context demonstrations on LLMs' reasoning performance. We prove that the reasoning efficacy (measured by empirical prediction loss) can be bounded by a LLM-oriented semantic similarity and an inference stability of demonstrations, which is general for both one-shot and few-shot scenarios. Based on this finding, we propose a straightforward, generalizable, and low-complexity demonstration selection method named LMS3. It can adaptively facilitate to select the most pertinent samples for different LLMs and includes a novel demonstration rejection mechanism to automatically filter out samples that are unsuitable for few-shot learning. Through experiments on three representative benchmarks, two LLM backbones, and multiple few-shot settings, we verify that our LMS3 has superiority and achieves consistent improvements on all datasets, which existing methods have been unable to accomplish.