In-Context Learning Demonstration Selection via Influence Analysis

TL;DR

This paper introduces InfICL, a demonstration selection method for in-context learning that uses influence functions to identify impactful training samples, improving performance without costly fine-tuning.

Contribution

The paper presents a novel influence-based demonstration selection method, InfICL, which enhances ICL performance efficiently by analyzing training sample influence without fine-tuning.

Findings

InfICL outperforms state-of-the-art baselines in various datasets.

It is cost-effective by only using LLM-generated embeddings.

Demonstrates improved generalization in ICL tasks.

Abstract

Large Language Models (LLMs) have showcased their In-Context Learning (ICL) capabilities, enabling few-shot learning without the need for gradient updates. Despite its advantages, the effectiveness of ICL heavily depends on the choice of demonstrations. Selecting the most effective demonstrations for ICL remains a significant research challenge. To tackle this issue, we propose a demonstration selection method named InfICL, which utilizes influence functions to analyze impacts of training samples. By identifying the most influential training samples as demonstrations, InfICL aims to enhance the ICL generalization performance. To keep InfICL cost-effective, we only use the LLM to generate sample input embeddings, avoiding expensive fine-tuning. Through empirical studies on various real-world datasets, we demonstrate advantages of InfICL compared to state-of-the-art baselines.