Assessing the Limits of In-Context Learning beyond Functions using Partially Ordered Relation

TL;DR

This paper investigates the limitations of in-context learning in large language models when dealing with partially ordered relations, revealing performance constraints as prompt complexity increases, supported by empirical and theoretical analysis.

Contribution

It introduces a framework for evaluating ICL on partially ordered relations and analyzes its performance limits with increasing prompt complexity, both empirically and theoretically.

Findings

ICL performance saturates with increasing prompt complexity.

Effectiveness of ICL remains limited despite sufficient demonstrations.

Theoretical justification of ICL behavior based on implicit optimization.

Abstract

Generating rational and generally accurate responses to tasks, often accompanied by example demonstrations, highlights Large Language Model's (LLM's) remarkable In-Context Learning (ICL) capabilities without requiring updates to the model's parameter space. Despite having an ongoing exploration focused on the inference from a document-level concept, its behavior in learning well-defined functions or relations in context needs a careful investigation. In this article, we present the performance of ICL on partially ordered relation by introducing the notion of inductively increasing complexity in prompts. In most cases, the saturated performance of the chosen metric indicates that while ICL offers some benefits, its effectiveness remains constrained as we increase the complexity in the prompts even in presence of sufficient demonstrative examples. The behavior is evident from our empirical findings and has further been theoretically justified in term of its implicit optimization process. The code is available \href{https://anonymous.4open.science/r/ICLonPartiallyOrderSet}{here}.