Effective In-Context Example Selection through Data Compression

TL;DR

This paper introduces a data compression-based method for selecting in-context examples in large language models, significantly improving their performance by effectively choosing relevant data that retains essential information.

Contribution

It proposes a novel two-stage data compression approach for in-context example selection, enhancing relevance and information retention in large language models.

Findings

Average performance improvement of 5.90% across datasets

Effective selection of relevant examples improves model accuracy

Method applicable to multiple language models

Abstract

In-context learning has been extensively validated in large language models. However, the mechanism and selection strategy for in-context example selection, which is a crucial ingredient in this approach, lacks systematic and in-depth research. In this paper, we propose a data compression approach to the selection of in-context examples. We introduce a two-stage method that can effectively choose relevant examples and retain sufficient information about the training dataset within the in-context examples. Our method shows a significant improvement of an average of 5.90% across five different real-world datasets using four language models.