Enhancing Input-Label Mapping in In-Context Learning with Contrastive Decoding

TL;DR

This paper introduces In-Context Contrastive Decoding (ICCD), a method that improves large language models' ability to utilize input-label mappings in in-context learning, leading to significant performance gains across multiple NLP tasks without extra training.

Contribution

The paper proposes ICCD, a novel contrastive decoding technique that enhances input-label mapping in ICL, demonstrating broad applicability and consistent improvements across various LLM scales.

Findings

Up to +1.8 average performance improvement on 7 NLU tasks

Effective across 6 different LLM scales

Compatible with various demonstration selection methods

Abstract

Large language models (LLMs) excel at a range of tasks through in-context learning (ICL), where only a few task examples guide their predictions. However, prior research highlights that LLMs often overlook input-label mapping information in ICL, relying more on their pre-trained knowledge. To address this issue, we introduce In-Context Contrastive Decoding (ICCD), a novel method that emphasizes input-label mapping by contrasting the output distributions between positive and negative in-context examples. Experiments on 7 natural language understanding (NLU) tasks show that our ICCD method brings consistent and significant improvement (up to +1.8 improvement on average) upon 6 different scales of LLMs without requiring additional training. Our approach is versatile, enhancing performance with various demonstration selection methods, demonstrating its broad applicability and effectiveness. The code and scripts are released at https://github.com/Romainpkq/CD_ICL.