NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning

TL;DR

NoisyICL introduces random noise to model parameters during in-context learning to improve accuracy, calibration, and fairness without extensive fine-tuning, demonstrating effectiveness across multiple models and datasets.

Contribution

It presents a simple noise perturbation method to enhance ICL performance and calibration, offering a cost-effective alternative to fine-tuning.

Findings

Improves prediction accuracy across datasets

Enhances model calibration and confidence faithfulness

Promotes fairer predictions in ICL

Abstract

In-Context Learning (ICL) is suffering from unsatisfactory performance and under-calibration due to high prior bias and unfaithful confidence. Some previous works fine-tuned language models for better ICL performance with enormous datasets and computing costs. In this paper, we propose NoisyICL, simply perturbing the model parameters by random noises to strive for better performance and calibration. Our experiments on two models and 12 downstream datasets show that NoisyICL can help ICL produce more accurate predictions. Our further analysis indicates that NoisyICL enables the model to provide more fair predictions, and also with more faithful confidence. Therefore, we believe that NoisyICL is an effective calibration of ICL. Our experimental code is uploaded to Github.