# InSQuAD: In-Context Learning for Efficient Retrieval via Submodular Mutual Information to Enforce Quality and Diversity

## TL;DR

InSQuAD enhances in-context learning by using submodular mutual information to select high-quality, diverse exemplars, improving retrieval and performance across multiple datasets.

## Contribution

The paper introduces a novel SMI-based selection strategy and training paradigm for better quality and diversity in in-context exemplars for ICL models.

## Key findings

- Significant performance improvements on nine benchmark datasets.
- Effective modeling of relevance and diversity in exemplar retrieval.
- Enhanced ICL performance through synthetic paraphrase augmentation.

## Abstract

In this paper, we introduce InSQuAD, designed to enhance the performance of In-Context Learning (ICL) models through Submodular Mutual Information} (SMI) enforcing Quality and Diversity among in-context exemplars. InSQuAD achieves this through two principal strategies: First, we model the ICL task as a targeted selection problem and introduce a unified selection strategy based on SMIs which mines relevant yet diverse in-context examples encapsulating the notions of quality and diversity. Secondly, we address a common pitfall in existing retrieval models which model query relevance, often overlooking diversity, critical for ICL. InSQuAD introduces a combinatorial training paradigm which learns the parameters of an SMI function to enforce both quality and diversity in the retrieval model through a novel likelihood-based loss. To further aid the learning process we augment an existing multi-hop question answering dataset with synthetically generated paraphrases. Adopting the retrieval model trained using this strategy alongside the novel targeted selection formulation for ICL on nine benchmark datasets shows significant improvements validating the efficacy of our approach.