MarginSel : Max-Margin Demonstration Selection for LLMs

TL;DR

MarginSel is a novel demonstration selection method for LLMs that adaptively chooses hard examples to improve classification performance, achieving significant F1-score gains by inducing max-margin behavior.

Contribution

It introduces a two-step max-margin demonstration selection method that enhances LLM in-context learning by focusing on hard examples, with theoretical and empirical validation.

Findings

Achieves 2-7% absolute F1-score improvement

Induces max-margin behavior in LLMs

Effectively increases margin for hard examples

Abstract

Large Language Models (LLMs) excel at few-shot learning via in-context learning (ICL). However, the effectiveness of ICL is often sensitive to the selection and ordering of demonstration examples. To address this, we present MarginSel: Max-Margin Demonstration Selection for LLMs, a two-step method that selects hard demonstration examples for the ICL prompt, adapting to each test instance. Our approach achieves 2-7% absolute improvement in F1-score across classification tasks, compared to a random selection of examples. We also provide theoretical insights and empirical evidence showing that MarginSel induces max-margin behavior in LLMs by effectively increasing the margin for hard examples, analogous to support vectors, thereby shifting the decision boundary in a beneficial direction.