Exploring the Role of Diversity in Example Selection for In-Context Learning

TL;DR

This paper investigates how increasing topical diversity in example selection for in-context learning improves task performance, using reranking with maximum marginal relevance to balance relevance and diversity.

Contribution

It introduces a reranking method based on maximum marginal relevance to enhance diversity in example selection for ICL, leading to improved downstream results.

Findings

Diversifying examples improves downstream performance.

Reranking with MMR enhances topical diversity.

Consistent gains across context sizes and similarity functions.

Abstract

In-Context Learning (ICL) has gained prominence due to its ability to perform tasks without requiring extensive training data and its robustness to noisy labels. A typical ICL workflow involves selecting localized examples relevant to a given input using sparse or dense embedding-based similarity functions. However, relying solely on similarity-based selection may introduce topical biases in the retrieved contexts, potentially leading to suboptimal downstream performance. We posit that reranking the retrieved context to enhance topical diversity can improve downstream task performance. To achieve this, we leverage maximum marginal relevance (MMR) which balances topical similarity with inter-example diversity. Our experimental results demonstrate that diversifying the selected examples leads to consistent improvements in downstream performance across various context sizes and similarity functions. The implementation of our approach is made available at https://github.com/janak11111/Diverse-ICL.