SMITE: Enhancing Fairness in LLMs through Optimal In-Context Example Selection via Dynamic Validation

TL;DR

This paper presents SMITE, a novel method that dynamically selects in-context examples for large language models, improving both fairness and accuracy by using a evolving validation set during inference.

Contribution

It introduces a dynamic validation set and an iterative selection algorithm for optimal in-context examples, a first in LLM fairness and performance enhancement.

Findings

Significant improvements in fairness metrics across tested LLMs.

Enhanced predictive accuracy compared to baseline methods.

First application of dynamic validation in in-context learning for LLMs.

Abstract

Large Language Models (LLMs) are widely used for downstream tasks such as tabular classification, where ensuring fairness in their outputs is critical for inclusivity, equal representation, and responsible AI deployment. This study introduces a novel approach to enhancing LLM performance and fairness through the concept of a dynamic validation set, which evolves alongside the test set, replacing the traditional static validation approach. We also propose an iterative algorithm, SMITE, to select optimal in-context examples, with each example set validated against its corresponding dynamic validation set. The in-context set with the lowest total error is used as the final demonstration set. Our experiments across four different LLMs show that our proposed techniques significantly improve both predictive accuracy and fairness compared to baseline methods. To our knowledge, this is the first study to apply dynamic validation in the context of in-context learning for LLMs.