Bayesian Example Selection Improves In-Context Learning for Speech, Text, and Visual Modalities

TL;DR

This paper introduces ByCS, a Bayesian method for selecting in-context examples that enhances large language models' performance across speech, text, and visual tasks by focusing on inverse inference probabilities.

Contribution

The paper presents a novel Bayesian in-context example selection approach (ByCS) that improves ICL performance across multiple modalities by leveraging inverse inference probabilities.

Findings

ByCS improves ICL accuracy across speech, text, and visual tasks.

The method demonstrates robustness across different models and datasets.

Experimental results show significant performance gains over baseline methods.

Abstract

Large language models (LLMs) can adapt to new tasks through in-context learning (ICL) based on a few examples presented in dialogue history without any model parameter update. Despite such convenience, the performance of ICL heavily depends on the quality of the in-context examples presented, which makes the in-context example selection approach a critical choice. This paper proposes a novel Bayesian in-Context example Selection method (ByCS) for ICL. Extending the inference probability conditioned on in-context examples based on Bayes' theorem, ByCS focuses on the inverse inference conditioned on test input. Following the assumption that accurate inverse inference probability (likelihood) will result in accurate inference probability (posterior), in-context examples are selected based on their inverse inference results. Diverse and extensive cross-tasking and cross-modality experiments are performed with speech, text, and image examples. Experimental results show the efficacy and robustness of our ByCS method on various models, tasks and modalities.