Calibrate to Discriminate: Improve In-Context Learning with Label-Free Comparative Inference

TL;DR

This paper identifies a miscalibration issue in large language models' in-context learning, introduces new metrics to measure it, and proposes a comparative inference method that enhances calibration and classification accuracy.

Contribution

The paper introduces a novel in-context comparative inference technique and new metrics to better measure and mitigate indiscriminate miscalibration in LLMs.

Findings

Improved calibration and accuracy over baseline methods.

Effective in five diverse datasets.

Addresses a unique miscalibration behavior in LLMs.

Abstract

While in-context learning with large language models (LLMs) has shown impressive performance, we have discovered a unique miscalibration behavior where both correct and incorrect predictions are assigned the same level of confidence. We refer to this phenomenon as indiscriminate miscalibration. We found that traditional calibration metrics, such as Expected Calibrated Errors (ECEs), are unable to capture this behavior effectively. To address this issue, we propose new metrics to measure the severity of indiscriminate miscalibration. Additionally, we develop a novel in-context comparative inference method to alleviate miscalibrations and improve classification performance. Through extensive experiments on five datasets, we demonstrate that our proposed method can achieve more accurate and calibrated predictions compared to regular zero-shot and few-shot prompting.